[RFC PATCH 00/56] Dynamic mitigations

[RFC PATCH 00/56] Dynamic mitigations

[David Kaplan]

Dynamic mitigations enables changing the kernel CPU security mitigations at
runtime without a reboot/kexec.

Previously, mitigation choices had to be made on the kernel cmdline.  With
this feature an administrator can select new mitigation choices by writing
a sysfs file, after which the kernel will re-patch itself based on the new
mitigations.

As the performance cost of CPU mitigations can be significant, selecting
the right set of mitigations is important to achieve the correct balance of
performance/security.

Use
---
As described in the supplied documentation file, new mitigations are
selected by writing cmdline options to a new sysfs file.  Only cmdline
options related to mitigations are recognized via this interface.  All
previous mitigation-related cmdline options are ignored and selections are
done based on the new options.

Examples:
   echo "mitigations=off" > /sys/devices/system/cpu/mitigations
   echo "spectre_v2=retpoline tsa=off" > /sys/devices/system/cpu/mitigations


There are several use cases that will benefit from dynamic mitigations:

Use Cases
---------
1. Runtime Policy

Some workflows rely on booting a generic kernel before customizing the system.
cloud-init is a popular example of this where a VM is started typically with
default settings and then is customized based on a customer-provided
configuration file.

As flows like this rely on configuring the system after boot, they currently
cannot customize the mitigation policy.  With dynamic mitigations, this
configuration information can be augmented to include security policy
information.

For example, a cloud VM which runs only trusted workloads likely does not
need any CPU security mitigations applied.  But as this policy information
is not known at boot time, the kernel will be booted with unnecessary
mitigations enabled.  With dynamic mitigations, these mitigations can be
disabled during boot after policy information is retrieved, improving
performance.

2. Mitigation Changes

Sometimes there are needs to change the mitigation settings in light of new
security findings.  For example, AMD-SB-1036 advised of a security issue
with a spectre v2 mitigation and advised using a different one instead.

With dynamic mitigations, such changes can be made without a reboot/kexec
which minimizes disruption in environments which cannot easily tolerate
such an event.

3. Mitigation Testing

Being able to quickly change between different mitigation settings without
having to restart applications is beneficial when conducting mitigation
development and testing.

Note that some bugs have multiple mitigation options, which may have
varying performance impacts.  Being able to quickly switch between them
makes evaluating such options easier.


Implementation Details
----------------------
Re-patching the kernel is expected to be a very rare operation and is done
under very big hammers.  All tasks are put into the freezer and the
re-patching is then done under the (new) stop_machine_nmi() routine.

To re-patch the kernel, it is first reverted back to its compile-time
state.  The original bytes from alternatives, retpolines, etc. are saved
during boot so they can later be used to restore the original kernel image.
After that, the kernel is patched based on the new feature flags.

This simplifies the re-patch process as restoring the original kernel image
is relatively straightforward.  In other words, instead of having to
re-patch from mitigation A to mitigation B directly, we first restore the
original image and then patch from that to mitigation B, similar to if the
system had booted with mitigation B selected originally.


Performance
-----------
Testing so far has demonstrated that re-patching takes ~50ms on an AMD EPYC
7713 running a typical Ubuntu kernel with around 100 modules loaded.

Guide to Patch Series
---------------------
As this series is rather lengthy, this may help with understanding it:

Patches 3-18 focus on "resetting" mitigations.  Every bug that may set feature
flags, MSRs, static branches, etc. now has matching "reset" functions that will
undo all these changes.  This is used at the beginning of the re-patch flow.

Patches 20-22 move various functions and values out of the .init section.  Most
of the existing mitigation logic was marked as __init and the mitigation
settings as __ro_after_init but now these can be changed at runtime.  The
__ro_after_init marking functioned as a defense-in-depth measure but is arguably
of limited meaningful security value as an attacker who can modify kernel data
can do a lot worse than change some speculation settings.  As re-patching
requires being able to modify these settings, it was simplest to remove them
from that section.

Patches 23-27 involve linker and related modifications to keep alternative
information around at runtime instead of free'ing it after boot.  This does
result in slightly higher runtime memory consumption which is one reason why
this feature is behind a Kconfig option.  On a typical kernel, this was measured
at around 2MB of extra kernel memory usage.

Patches 28-30 focus on the new stop_machine_nmi() which behaves like
stop_machine() but runs the handler in NMI context, thus ensuring that even NMIs
cannot interrupt the handler.  As dynamic mitigations involves re-patching
functions used by NMI entry code, this is required for safety.

Patches 31-40 focus on support for restoring the kernel text at runtime.  This
involves saving the original kernel bytes when patched the first time and adding
support to then restore those later.

Patches 41-44 start building support for updating code, in particular module
code at runtime.

Patches 45-47 focus on support for the Indirect Target Selection mitigation
which is particularly challenging because it requires runtime memory allocations
and permission changes which are not possible in NMI context.  As a result, ITS
memory is pre-allocated before entering NMI context.

Patch 50 adds the complete function for resetting and re-patching the kernel.

Patches 51-53 build the sysfs interface for re-patching and support for parsing
the new options provided.

Patches 54-56 add debugfs interfaces to values which are important for
mitigations.  These are useful for userspace test utilities to be able to force
a CPU to appear to be vulnerable or immune to certain bugs as well as being able
to help verify if the kernel is correctly mitigating various vulnerabilities.

David Kaplan (56):
  Documentation/admin-guide: Add documentation
  x86/Kconfig: Add CONFIG_DYNAMIC_MITIGATIONS
  cpu: Reset global mitigations
  x86/bugs: Reset spectre_v1 mitigations
  x86/bugs: Reset spectre_v2 mitigations
  x86/bugs: Reset retbleed mitigations
  x86/bugs: Reset spectre_v2_user mitigations
  x86/bugs: Reset SSB mitigations
  x86/bugs: Reset L1TF mitigations
  x86/bugs: Reset MDS mitigations
  x86/bugs: Reset MMIO mitigations
  x86/bugs: Reset SRBDS mitigations
  x86/bugs: Reset SRSO mitigations
  x86/bugs: Reset GDS mitigations
  x86/bugs: Reset BHI mitigations
  x86/bugs: Reset ITS mitigation
  x86/bugs: Reset TSA mitigations
  x86/bugs: Reset VMSCAPE mitigations
  x86/bugs: Define bugs_smt_disable()
  x86/bugs: Move bugs.c logic out of .init section
  x86/callthunks: Move logic out of .init
  cpu: Move mitigation logic out of .init
  x86/vmlinux.lds: Move alternative sections
  x86/vmlinux.lds: Move altinstr_aux conditionally
  x86/vmlinux.lds: Define __init_alt_end
  module: Save module ELF info
  x86/mm: Conditionally free alternative sections
  stop_machine: Add stop_machine_nmi()
  x86/apic: Add self-NMI support
  x86/nmi: Add support for stop_machine_nmi()
  x86/alternative: Prepend nops with retpolines
  x86/alternative: Add module param
  x86/alternative: Avoid re-patching init code
  x86/alternative: Save old bytes for alternatives
  x86/alternative: Save old bytes for retpolines
  x86/alternative: Do not recompute len on re-patch
  x86/alternative: Reset alternatives
  x86/callthunks: Reset callthunks
  x86/sync_core: Add sync_core_nmi_safe()
  x86/alternative: Use sync_core_nmi_safe()
  static_call: Add update_all_static_calls()
  module: Make memory writeable for re-patching
  module: Update alternatives
  x86/module: Update alternatives
  x86/alternative: Use boot_cpu_has in ITS code
  x86/alternative: Add ITS re-patching support
  x86/module: Add ITS re-patch support for modules
  x86/bugs: Move code for updating speculation MSRs
  x86/fpu: Qualify warning in os_xsave
  x86/alternative: Add re-patch support
  cpu: Parse string of mitigation options
  x86/bugs: Support parsing mitigation options
  drivers/cpu: Re-patch mitigations through sysfs
  x86/debug: Create debugfs interface to x86_capabilities
  x86/debug: Show return thunk in debugfs
  x86/debug: Show static branch config in debugfs

 .../ABI/testing/sysfs-devices-system-cpu      |   8 +
 .../hw-vuln/dynamic_mitigations.rst           |  75 ++
 Documentation/admin-guide/hw-vuln/index.rst   |   1 +
 arch/x86/Kconfig                              |  12 +
 arch/x86/entry/vdso/vma.c                     |   2 +-
 arch/x86/include/asm/alternative.h            |  51 +-
 arch/x86/include/asm/bugs.h                   |   4 +
 arch/x86/include/asm/module.h                 |  10 +
 arch/x86/include/asm/sync_core.h              |  14 +
 arch/x86/kernel/alternative.c                 | 497 ++++++++++++-
 arch/x86/kernel/apic/ipi.c                    |   7 +
 arch/x86/kernel/callthunks.c                  |  85 ++-
 arch/x86/kernel/cpu/bugs.c                    | 686 +++++++++++++-----
 arch/x86/kernel/cpu/common.c                  |  65 +-
 arch/x86/kernel/cpu/cpu.h                     |   4 -
 arch/x86/kernel/fpu/xstate.h                  |   2 +-
 arch/x86/kernel/module.c                      |  96 ++-
 arch/x86/kernel/nmi.c                         |   4 +
 arch/x86/kernel/static_call.c                 |   3 +-
 arch/x86/kernel/vmlinux.lds.S                 | 110 +--
 arch/x86/mm/init.c                            |  12 +-
 arch/x86/mm/mm_internal.h                     |   2 +
 arch/x86/tools/relocs.c                       |   1 +
 drivers/base/cpu.c                            | 113 +++
 include/linux/cpu.h                           |  10 +
 include/linux/module.h                        |  11 +
 include/linux/static_call.h                   |   2 +
 include/linux/stop_machine.h                  |  32 +
 kernel/cpu.c                                  |  62 +-
 kernel/module/main.c                          |  78 +-
 kernel/static_call_inline.c                   |  22 +
 kernel/stop_machine.c                         |  79 +-
 32 files changed, 1876 insertions(+), 284 deletions(-)
 create mode 100644 Documentation/admin-guide/hw-vuln/dynamic_mitigations.rst


base-commit: a5652f0f2a69fadcfb2f687a11a737a57f15b28e
-- 
2.34.1

[RFC PATCH 01/56] Documentation/admin-guide: Add documentation

[David Kaplan]

Add new documentation for the dynamic mitigation feature.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 .../hw-vuln/dynamic_mitigations.rst           | 75 +++++++++++++++++++
 Documentation/admin-guide/hw-vuln/index.rst   |  1 +
 2 files changed, 76 insertions(+)
 create mode 100644 Documentation/admin-guide/hw-vuln/dynamic_mitigations.rst

diff --git a/Documentation/admin-guide/hw-vuln/dynamic_mitigations.rst b/Documentation/admin-guide/hw-vuln/dynamic_mitigations.rst
new file mode 100644
index 000000000000..9904e6ec9be5
--- /dev/null
+++ b/Documentation/admin-guide/hw-vuln/dynamic_mitigations.rst
@@ -0,0 +1,75 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+Dynamic Mitigations
+-------------------
+
+Dynamic mitigation support enables the re-configuration of CPU vulnerability
+mitigations through sysfs.  The file /sys/devices/system/cpu/mitigations
+contains the current set of mitigation-related options.  The file can be written
+to in order to make the kernel re-select and re-apply mitigations without a
+reboot or kexec.
+
+The data written to the file should be command line options related to
+mitigation controls (e.g., "mitigations=auto spectre_v2=retpoline mds=off").
+When the file is written, all previous selections related to mitigation controls
+are discarded and the new options are evaluated.  Any non-mitigation related
+options are ignored.
+
+Dynamic mitigations are available if the CONFIG_DYNAMIC_MITIGATIONS option is
+selected.
+
+Purpose
+-------
+
+Dynamic mitigations serve two primary purposes:
+
+Move Policy To Userspace
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+Mitigation choices are related to the security policy and posture of the system.
+Most mitigations are only necessary on shared, multi-user systems if untrusted
+code may be run on the system, such as through untrusted userspace or untrusted
+virtual machines.  The kernel may not know how the system will be used on boot,
+and therefore must adopt a strong security posture for safety.
+
+With dynamic mitigations, userspace can re-select mitigations once the needs of
+the system can be determined and more policy information is available.
+
+Mitigation Testing
+^^^^^^^^^^^^^^^^^^
+
+Dynamic mitigation support makes it easy to toggle individual mitigations or
+choose between different mitigation options without the expense of a reboot or
+kexec.  This may be useful when evaluating the performance of various
+mitigation options.  It can also be useful for performing bug fixes without a
+reboot, in case a particular mitigation is undesired or buggy.
+
+Caveats
+-------
+
+There are a few limitations to dynamic mitigation support:
+
+Runtime Limitations
+^^^^^^^^^^^^^^^^^^^
+
+There are a few mitigations that cannot be toggled at runtime due to the way
+they are structured.  Specifically, kernel PTI (page table isolation) cannot be
+toggled because of the complexity of this mitigation.  Additionally, SMT cannot
+be disabled at runtime.  Therefore, if a bug mitigation requires disabling SMT,
+a warning message will be printed.
+
+BPF JIT
+^^^^^^^
+
+There is currently no way to recompile already JIT'd BPF programs.  This can
+present a security problem if moving from a less secure security posture to a
+more secure one.  It is recommended to either unload BPF programs prior to
+re-configuring mitigations, ensure that security settings only become less
+restrictive over time, or disable use of the BPF JIT.
+
+Performance
+-----------
+
+Re-configuring mitigations is done under the biggest of hammers.  All tasks are
+frozen, all cores are stopped, interrupts are masked, etc.  This may affect
+system responsiveness if lots of patching must be done.
diff --git a/Documentation/admin-guide/hw-vuln/index.rst b/Documentation/admin-guide/hw-vuln/index.rst
index 55d747511f83..44418bad5895 100644
--- a/Documentation/admin-guide/hw-vuln/index.rst
+++ b/Documentation/admin-guide/hw-vuln/index.rst
@@ -27,3 +27,4 @@ are configurable at compile, boot or run time.
    old_microcode
    indirect-target-selection
    vmscape
+   dynamic_mitigations
-- 
2.34.1

[RFC PATCH 02/56] x86/Kconfig: Add CONFIG_DYNAMIC_MITIGATIONS

[David Kaplan]

CONFIG_DYNAMIC_MITIGATIONS enables support for runtime re-patching of the
kernel when mitigation selections are changed.  It depends on
CONFIG_LIVEPATCH because it needs modules to preserve all their ELF
information for later re-patching.  It also depends on CONFIG_FREEZER
because re-patching must be done while all tasks are in the freezer to
avoid race conditions.

CONFIG_DEBUG_ENTRY must be false because dynamic mitigations relies on a
non-reentrant NMI handler which does not unmask NMIs early.

CONFIG_DYNAMIC_MITIGATIONS is optional because support for dynamic
mitigations does increase runtime kernel memory usage.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 arch/x86/Kconfig | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index fa3b616af03a..0c8c1e508223 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -2711,6 +2711,18 @@ config MITIGATION_VMSCAPE
 	  Enable mitigation for VMSCAPE attacks. VMSCAPE is a hardware security
 	  vulnerability on Intel and AMD CPUs that may allow a guest to do
 	  Spectre v2 style attacks on userspace hypervisor.
+
+config DYNAMIC_MITIGATIONS
+	bool "Support dynamic reconfiguration of CPU mitigations at runtime"
+	depends on LIVEPATCH && FREEZER && !DEBUG_ENTRY
+	default y
+	help
+	  Allow CPU mitigations to be reconfigured at runtime via
+	  sysfs.  Dynamic mitigation support requires extra kernel memory
+	  to keep around alternative information after kernel boot but
+	  allows CPU mitigation settings to be modified without a
+	  kexec/reboot.
+
 endif
 
 config ARCH_HAS_ADD_PAGES
-- 
2.34.1

[RFC PATCH 03/56] cpu: Reset global mitigations

[David Kaplan]

Reset global mitigations, including attack vectors and then call the
arch-specific function to reset further mitigations.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 include/linux/cpu.h |  3 +++
 kernel/cpu.c        | 18 ++++++++++++++++++
 2 files changed, 21 insertions(+)

diff --git a/include/linux/cpu.h b/include/linux/cpu.h
index 487b3bf2e1ea..3da629a76a49 100644
--- a/include/linux/cpu.h
+++ b/include/linux/cpu.h
@@ -208,6 +208,9 @@ enum smt_mitigations {
 	SMT_MITIGATIONS_ON,
 };
 
+void cpu_reset_mitigations(void);
+void arch_cpu_reset_mitigations(void);
+
 #ifdef CONFIG_CPU_MITIGATIONS
 extern bool cpu_mitigations_off(void);
 extern bool cpu_mitigations_auto_nosmt(void);
diff --git a/kernel/cpu.c b/kernel/cpu.c
index db9f6c539b28..910249f6217a 100644
--- a/kernel/cpu.c
+++ b/kernel/cpu.c
@@ -3171,6 +3171,7 @@ void __init boot_cpu_hotplug_init(void)
 }
 
 #ifdef CONFIG_CPU_MITIGATIONS
+
 /*
  * All except the cross-thread attack vector are mitigated by default.
  * Cross-thread mitigation often requires disabling SMT which is expensive
@@ -3326,3 +3327,20 @@ static int __init mitigations_parse_cmdline(char *arg)
 }
 #endif
 early_param("mitigations", mitigations_parse_cmdline);
+
+#ifdef CONFIG_DYNAMIC_MITIGATIONS
+void __weak arch_cpu_reset_mitigations(void)
+{
+}
+
+void cpu_reset_mitigations(void)
+{
+	smt_mitigations = SMT_MITIGATIONS_AUTO;
+	cpu_mitigations = CPU_MITIGATIONS_AUTO;
+	attack_vectors[CPU_MITIGATE_USER_KERNEL] = true;
+	attack_vectors[CPU_MITIGATE_USER_USER] = true;
+	attack_vectors[CPU_MITIGATE_GUEST_HOST] = IS_ENABLED(CONFIG_KVM);
+	attack_vectors[CPU_MITIGATE_GUEST_GUEST] = IS_ENABLED(CONFIG_KVM);
+	arch_cpu_reset_mitigations();
+}
+#endif
-- 
2.34.1

[RFC PATCH 04/56] x86/bugs: Reset spectre_v1 mitigations

[David Kaplan]

Add function to reset spectre_v1 mitigations back to their boot-time
defaults.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 arch/x86/include/asm/bugs.h |  1 +
 arch/x86/kernel/cpu/bugs.c  | 18 ++++++++++++++++++
 2 files changed, 19 insertions(+)

diff --git a/arch/x86/include/asm/bugs.h b/arch/x86/include/asm/bugs.h
index f25ca2d709d4..e43b9412645e 100644
--- a/arch/x86/include/asm/bugs.h
+++ b/arch/x86/include/asm/bugs.h
@@ -11,5 +11,6 @@ static inline int ppro_with_ram_bug(void) { return 0; }
 #endif
 
 extern void cpu_bugs_smt_update(void);
+void arch_cpu_reset_mitigations(void);
 
 #endif /* _ASM_X86_BUGS_H */
diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
index 6a526ae1fe99..9d5c6a3e50e1 100644
--- a/arch/x86/kernel/cpu/bugs.c
+++ b/arch/x86/kernel/cpu/bugs.c
@@ -46,6 +46,8 @@
  *				   may want to change based on other choices
  *				   made.  This function is optional.
  *   <vuln>_apply_mitigation() -- Enable the selected mitigation.
+ *   <vuln>_reset_mitigation() -- Undo's the apply_mitigation step, this is used
+ *				  with runtime mitigation patching.
  *
  * The compile-time mitigation in all cases should be AUTO.  An explicit
  * command-line option can override AUTO.  If no such option is
@@ -1247,6 +1249,15 @@ static void __init spectre_v1_apply_mitigation(void)
 	pr_info("%s\n", spectre_v1_strings[spectre_v1_mitigation]);
 }
 
+#ifdef CONFIG_DYNAMIC_MITIGATIONS
+static void spectre_v1_reset_mitigation(void)
+{
+	setup_clear_cpu_cap(X86_FEATURE_FENCE_SWAPGS_USER);
+	setup_clear_cpu_cap(X86_FEATURE_FENCE_SWAPGS_KERNEL);
+	spectre_v1_mitigation = SPECTRE_V1_MITIGATION_AUTO;
+}
+#endif
+
 static int __init nospectre_v1_cmdline(char *str)
 {
 	spectre_v1_mitigation = SPECTRE_V1_MITIGATION_NONE;
@@ -3794,3 +3805,10 @@ void __warn_thunk(void)
 {
 	WARN_ONCE(1, "Unpatched return thunk in use. This should not happen!\n");
 }
+
+#ifdef CONFIG_DYNAMIC_MITIGATIONS
+void arch_cpu_reset_mitigations(void)
+{
+	spectre_v1_reset_mitigation();
+}
+#endif
-- 
2.34.1

[RFC PATCH 05/56] x86/bugs: Reset spectre_v2 mitigations

[David Kaplan]

Add function to reset spectre_v2 mitigations back to their boot-time
defaults.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 arch/x86/kernel/cpu/bugs.c | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
index 9d5c6a3e50e1..0430635bb17d 100644
--- a/arch/x86/kernel/cpu/bugs.c
+++ b/arch/x86/kernel/cpu/bugs.c
@@ -2477,6 +2477,24 @@ static void __init spectre_v2_apply_mitigation(void)
 	}
 }
 
+#ifdef CONFIG_DYNAMIC_MITIGATIONS
+static void spectre_v2_reset_mitigation(void)
+{
+	x86_spec_ctrl_base &= ~SPEC_CTRL_IBRS;
+	x86_spec_ctrl_base &= ~SPEC_CTRL_RRSBA_DIS_S;
+	rrsba_disabled = false;
+	setup_clear_cpu_cap(X86_FEATURE_KERNEL_IBRS);
+	setup_clear_cpu_cap(X86_FEATURE_RETPOLINE_LFENCE);
+	setup_clear_cpu_cap(X86_FEATURE_RETPOLINE);
+	setup_clear_cpu_cap(X86_FEATURE_RSB_CTXSW);
+	setup_clear_cpu_cap(X86_FEATURE_USE_IBPB_FW);
+	spectre_v2_enabled = SPECTRE_V2_NONE;
+	nospectre_v2 = false;
+	spectre_v2_cmd = IS_ENABLED(CONFIG_MITIGATION_SPECTRE_V2) ?
+		SPECTRE_V2_CMD_AUTO : SPECTRE_V2_CMD_NONE;
+}
+#endif
+
 static void update_stibp_msr(void * __unused)
 {
 	u64 val = spec_ctrl_current() | (x86_spec_ctrl_base & SPEC_CTRL_STIBP);
@@ -3810,5 +3828,6 @@ void __warn_thunk(void)
 void arch_cpu_reset_mitigations(void)
 {
 	spectre_v1_reset_mitigation();
+	spectre_v2_reset_mitigation();
 }
 #endif
-- 
2.34.1

[RFC PATCH 06/56] x86/bugs: Reset retbleed mitigations

[David Kaplan]

Add function to reset retbleed mitigations back to their boot-time
defaults.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 arch/x86/kernel/cpu/bugs.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
index 0430635bb17d..1f56ccb5f641 100644
--- a/arch/x86/kernel/cpu/bugs.c
+++ b/arch/x86/kernel/cpu/bugs.c
@@ -1542,6 +1542,20 @@ static void __init retbleed_apply_mitigation(void)
 		cpu_smt_disable(false);
 }
 
+#ifdef CONFIG_DYNAMIC_MITIGATIONS
+static void retbleed_reset_mitigation(void)
+{
+	setup_clear_cpu_cap(X86_FEATURE_RETHUNK);
+	setup_clear_cpu_cap(X86_FEATURE_UNRET);
+	setup_clear_cpu_cap(X86_FEATURE_ENTRY_IBPB);
+	setup_clear_cpu_cap(X86_FEATURE_IBPB_ON_VMEXIT);
+	setup_clear_cpu_cap(X86_FEATURE_CALL_DEPTH);
+	x86_return_thunk = __x86_return_thunk;
+	retbleed_mitigation = IS_ENABLED(CONFIG_MITIGATION_RETBLEED) ?
+		RETBLEED_MITIGATION_AUTO : RETBLEED_MITIGATION_NONE;
+}
+#endif
+
 #undef pr_fmt
 #define pr_fmt(fmt)     "ITS: " fmt
 
@@ -3829,5 +3843,6 @@ void arch_cpu_reset_mitigations(void)
 {
 	spectre_v1_reset_mitigation();
 	spectre_v2_reset_mitigation();
+	retbleed_reset_mitigation();
 }
 #endif
-- 
2.34.1

[RFC PATCH 07/56] x86/bugs: Reset spectre_v2_user mitigations

[David Kaplan]

Add function to reset spectre_v2_user mitigations back to their boot-time
defaults.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 arch/x86/kernel/cpu/bugs.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
index 1f56ccb5f641..4ca46f58e384 100644
--- a/arch/x86/kernel/cpu/bugs.c
+++ b/arch/x86/kernel/cpu/bugs.c
@@ -2056,6 +2056,18 @@ static void __init spectre_v2_user_apply_mitigation(void)
 	}
 }
 
+#ifdef CONFIG_DYNAMIC_MITIGATIONS
+static void spectre_v2_user_reset_mitigation(void)
+{
+	static_branch_disable(&switch_vcpu_ibpb);
+	static_branch_disable(&switch_mm_always_ibpb);
+	static_branch_disable(&switch_mm_cond_ibpb);
+	spectre_v2_user_stibp = SPECTRE_V2_USER_NONE;
+	spectre_v2_user_ibpb = SPECTRE_V2_USER_NONE;
+	spectre_v2_user_cmd = SPECTRE_V2_USER_CMD_AUTO;
+}
+#endif
+
 static const char * const spectre_v2_strings[] = {
 	[SPECTRE_V2_NONE]			= "Vulnerable",
 	[SPECTRE_V2_RETPOLINE]			= "Mitigation: Retpolines",
@@ -3844,5 +3856,6 @@ void arch_cpu_reset_mitigations(void)
 	spectre_v1_reset_mitigation();
 	spectre_v2_reset_mitigation();
 	retbleed_reset_mitigation();
+	spectre_v2_user_reset_mitigation();
 }
 #endif
-- 
2.34.1

[RFC PATCH 08/56] x86/bugs: Reset SSB mitigations

[David Kaplan]

Add function to reset SSB mitigations back to their boot-time defaults.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 arch/x86/kernel/cpu/bugs.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
index 4ca46f58e384..cc7b1b67d22d 100644
--- a/arch/x86/kernel/cpu/bugs.c
+++ b/arch/x86/kernel/cpu/bugs.c
@@ -380,6 +380,16 @@ static void x86_amd_ssb_disable(void)
 		wrmsrq(MSR_AMD64_LS_CFG, msrval);
 }
 
+static void x86_amd_ssb_enable(void)
+{
+	u64 msrval = x86_amd_ls_cfg_base;
+
+	if (boot_cpu_has(X86_FEATURE_VIRT_SSBD))
+		wrmsrl(MSR_AMD64_VIRT_SPEC_CTRL, 0);
+	else if (boot_cpu_has(X86_FEATURE_LS_CFG_SSBD))
+		wrmsrl(MSR_AMD64_LS_CFG, msrval);
+}
+
 #undef pr_fmt
 #define pr_fmt(fmt)	"MDS: " fmt
 
@@ -2672,6 +2682,17 @@ static void __init ssb_apply_mitigation(void)
 	}
 }
 
+#ifdef CONFIG_DYNAMIC_MITIGATIONS
+static void ssb_reset_mitigation(void)
+{
+	setup_clear_cpu_cap(X86_FEATURE_SPEC_STORE_BYPASS_DISABLE);
+	x86_spec_ctrl_base &= ~SPEC_CTRL_SSBD;
+	nossb = false;
+	ssb_mode = IS_ENABLED(CONFIG_MITIGATION_SSB) ?
+		SPEC_STORE_BYPASS_AUTO : SPEC_STORE_BYPASS_NONE;
+}
+#endif
+
 #undef pr_fmt
 #define pr_fmt(fmt)     "Speculation prctl: " fmt
 
@@ -2916,6 +2937,8 @@ void x86_spec_ctrl_setup_ap(void)
 
 	if (ssb_mode == SPEC_STORE_BYPASS_DISABLE)
 		x86_amd_ssb_disable();
+	else
+		x86_amd_ssb_enable();
 }
 
 bool itlb_multihit_kvm_mitigation;
@@ -3857,5 +3880,6 @@ void arch_cpu_reset_mitigations(void)
 	spectre_v2_reset_mitigation();
 	retbleed_reset_mitigation();
 	spectre_v2_user_reset_mitigation();
+	ssb_reset_mitigation();
 }
 #endif
-- 
2.34.1

[RFC PATCH 09/56] x86/bugs: Reset L1TF mitigations

[David Kaplan]

Add function to reset L1TF mitigations back to their boot-time defaults.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 arch/x86/kernel/cpu/bugs.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
index cc7b1b67d22d..b61bbeaf82b1 100644
--- a/arch/x86/kernel/cpu/bugs.c
+++ b/arch/x86/kernel/cpu/bugs.c
@@ -3059,6 +3059,15 @@ static void __init l1tf_apply_mitigation(void)
 	setup_force_cpu_cap(X86_FEATURE_L1TF_PTEINV);
 }
 
+#ifdef CONFIG_DYNAMIC_MITIGATIONS
+static void l1tf_reset_mitigation(void)
+{
+	setup_clear_cpu_cap(X86_FEATURE_L1TF_PTEINV);
+	l1tf_mitigation = IS_ENABLED(CONFIG_MITIGATION_L1TF) ?
+		L1TF_MITIGATION_AUTO : L1TF_MITIGATION_OFF;
+}
+#endif
+
 static int __init l1tf_cmdline(char *str)
 {
 	if (!boot_cpu_has_bug(X86_BUG_L1TF))
@@ -3881,5 +3890,6 @@ void arch_cpu_reset_mitigations(void)
 	retbleed_reset_mitigation();
 	spectre_v2_user_reset_mitigation();
 	ssb_reset_mitigation();
+	l1tf_reset_mitigation();
 }
 #endif
-- 
2.34.1

[RFC PATCH 10/56] x86/bugs: Reset MDS mitigations

[David Kaplan]

Add function to reset MDS mitigations back to their boot-time defaults.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 arch/x86/kernel/cpu/bugs.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
index b61bbeaf82b1..5668a8b8821b 100644
--- a/arch/x86/kernel/cpu/bugs.c
+++ b/arch/x86/kernel/cpu/bugs.c
@@ -554,6 +554,16 @@ static void __init mds_apply_mitigation(void)
 	}
 }
 
+#ifdef CONFIG_DYNAMIC_MITIGATIONS
+static void mds_reset_mitigation(void)
+{
+	setup_clear_cpu_cap(X86_FEATURE_CLEAR_CPU_BUF);
+	static_branch_disable(&cpu_buf_idle_clear);
+	mds_mitigation = IS_ENABLED(CONFIG_MITIGATION_MDS) ?
+		MDS_MITIGATION_AUTO : MDS_MITIGATION_OFF;
+}
+#endif
+
 static int __init mds_cmdline(char *str)
 {
 	if (!boot_cpu_has_bug(X86_BUG_MDS))
@@ -3891,5 +3901,6 @@ void arch_cpu_reset_mitigations(void)
 	spectre_v2_user_reset_mitigation();
 	ssb_reset_mitigation();
 	l1tf_reset_mitigation();
+	mds_reset_mitigation();
 }
 #endif
-- 
2.34.1

[RFC PATCH 11/56] x86/bugs: Reset MMIO mitigations

[David Kaplan]

Add function to reset MMIO mitigations back to their boot-time defaults.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 arch/x86/kernel/cpu/bugs.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
index 5668a8b8821b..9139c8187913 100644
--- a/arch/x86/kernel/cpu/bugs.c
+++ b/arch/x86/kernel/cpu/bugs.c
@@ -787,6 +787,16 @@ static void __init mmio_apply_mitigation(void)
 		cpu_smt_disable(false);
 }
 
+#ifdef CONFIG_DYNAMIC_MITIGATIONS
+static void mmio_reset_mitigation(void)
+{
+	static_branch_disable(&cpu_buf_vm_clear);
+	static_branch_disable(&cpu_buf_idle_clear);
+	mmio_mitigation = IS_ENABLED(CONFIG_MITIGATION_MMIO_STALE_DATA) ?
+		MMIO_MITIGATION_AUTO : MMIO_MITIGATION_OFF;
+}
+#endif
+
 static int __init mmio_stale_data_parse_cmdline(char *str)
 {
 	if (!boot_cpu_has_bug(X86_BUG_MMIO_STALE_DATA))
@@ -3902,5 +3912,6 @@ void arch_cpu_reset_mitigations(void)
 	ssb_reset_mitigation();
 	l1tf_reset_mitigation();
 	mds_reset_mitigation();
+	mmio_reset_mitigation();
 }
 #endif
-- 
2.34.1

[RFC PATCH 12/56] x86/bugs: Reset SRBDS mitigations

[David Kaplan]

Add function to reset SRBDS mitigations back to their boot-time defaults.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 arch/x86/kernel/cpu/bugs.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
index 9139c8187913..9a9d5309aa56 100644
--- a/arch/x86/kernel/cpu/bugs.c
+++ b/arch/x86/kernel/cpu/bugs.c
@@ -993,6 +993,17 @@ static void __init srbds_apply_mitigation(void)
 	update_srbds_msr();
 }
 
+#ifdef CONFIG_DYNAMIC_MITIGATIONS
+static void srbds_reset_mitigation(void)
+{
+	/* To cause the MSR bit to be cleared */
+	srbds_mitigation = SRBDS_MITIGATION_FULL;
+	update_srbds_msr();
+	srbds_mitigation = IS_ENABLED(CONFIG_MITIGATION_SRBDS) ?
+		SRBDS_MITIGATION_AUTO : SRBDS_MITIGATION_OFF;
+}
+#endif
+
 static int __init srbds_parse_cmdline(char *str)
 {
 	if (!str)
@@ -3913,5 +3924,6 @@ void arch_cpu_reset_mitigations(void)
 	l1tf_reset_mitigation();
 	mds_reset_mitigation();
 	mmio_reset_mitigation();
+	srbds_reset_mitigation();
 }
 #endif
-- 
2.34.1

[RFC PATCH 13/56] x86/bugs: Reset SRSO mitigations

[David Kaplan]

Add function to reset SRSO mitigations back to their boot-time defaults.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 arch/x86/kernel/cpu/bugs.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
index 9a9d5309aa56..112553058ccc 100644
--- a/arch/x86/kernel/cpu/bugs.c
+++ b/arch/x86/kernel/cpu/bugs.c
@@ -3312,6 +3312,21 @@ static void __init srso_apply_mitigation(void)
 	}
 }
 
+#ifdef CONFIG_DYNAMIC_MITIGATIONS
+static void srso_reset_mitigation(void)
+{
+	setup_clear_cpu_cap(X86_FEATURE_RETHUNK);
+	setup_clear_cpu_cap(X86_FEATURE_UNRET);
+	setup_clear_cpu_cap(X86_FEATURE_SRSO_ALIAS);
+	setup_clear_cpu_cap(X86_FEATURE_SRSO);
+	setup_clear_cpu_cap(X86_FEATURE_ENTRY_IBPB);
+	setup_clear_cpu_cap(X86_FEATURE_IBPB_ON_VMEXIT);
+	x86_pred_cmd = PRED_CMD_IBPB;
+	x86_return_thunk = __x86_return_thunk;
+	srso_mitigation = SRSO_MITIGATION_AUTO;
+}
+#endif
+
 #undef pr_fmt
 #define pr_fmt(fmt)	"VMSCAPE: " fmt
 
@@ -3925,5 +3940,6 @@ void arch_cpu_reset_mitigations(void)
 	mds_reset_mitigation();
 	mmio_reset_mitigation();
 	srbds_reset_mitigation();
+	srso_reset_mitigation();
 }
 #endif
-- 
2.34.1

[RFC PATCH 14/56] x86/bugs: Reset GDS mitigations

[David Kaplan]

Add function to reset GDS mitigations back to their boot-time defaults.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 arch/x86/kernel/cpu/bugs.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
index 112553058ccc..e765ac0b9240 100644
--- a/arch/x86/kernel/cpu/bugs.c
+++ b/arch/x86/kernel/cpu/bugs.c
@@ -1182,6 +1182,18 @@ static void __init gds_apply_mitigation(void)
 	pr_info("%s\n", gds_strings[gds_mitigation]);
 }
 
+#ifdef CONFIG_DYNAMIC_MITIGATIONS
+static void gds_reset_mitigation(void)
+{
+	/* To cause the MSR bit to be cleared. */
+	gds_mitigation = GDS_MITIGATION_OFF;
+	if (x86_arch_cap_msr & ARCH_CAP_GDS_CTRL)
+		update_gds_msr();
+	gds_mitigation =  IS_ENABLED(CONFIG_MITIGATION_GDS) ?
+		GDS_MITIGATION_AUTO : GDS_MITIGATION_OFF;
+}
+#endif
+
 static int __init gds_parse_cmdline(char *str)
 {
 	if (!str)
@@ -3941,5 +3953,6 @@ void arch_cpu_reset_mitigations(void)
 	mmio_reset_mitigation();
 	srbds_reset_mitigation();
 	srso_reset_mitigation();
+	gds_reset_mitigation();
 }
 #endif
-- 
2.34.1

[RFC PATCH 15/56] x86/bugs: Reset BHI mitigations

[David Kaplan]

Add function to reset BHI mitigations back to their boot-time defaults.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 arch/x86/kernel/cpu/bugs.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
index e765ac0b9240..67561e5c2154 100644
--- a/arch/x86/kernel/cpu/bugs.c
+++ b/arch/x86/kernel/cpu/bugs.c
@@ -2360,6 +2360,17 @@ static void __init bhi_apply_mitigation(void)
 	setup_force_cpu_cap(X86_FEATURE_CLEAR_BHB_VMEXIT);
 }
 
+#ifdef CONFIG_DYNAMIC_MITIGATIONS
+static void bhi_reset_mitigation(void)
+{
+	/* RRSBA already cleared in spectre_v2_reset_mitigation() */
+	setup_clear_cpu_cap(X86_FEATURE_CLEAR_BHB_VMEXIT);
+	setup_clear_cpu_cap(X86_FEATURE_CLEAR_BHB_LOOP);
+	bhi_mitigation = IS_ENABLED(CONFIG_MITIGATION_SPECTRE_BHI) ?
+		BHI_MITIGATION_AUTO : BHI_MITIGATION_OFF;
+}
+#endif
+
 static void __init spectre_v2_select_mitigation(void)
 {
 	if ((spectre_v2_cmd == SPECTRE_V2_CMD_RETPOLINE ||
@@ -3954,5 +3965,6 @@ void arch_cpu_reset_mitigations(void)
 	srbds_reset_mitigation();
 	srso_reset_mitigation();
 	gds_reset_mitigation();
+	bhi_reset_mitigation();
 }
 #endif
-- 
2.34.1

[RFC PATCH 16/56] x86/bugs: Reset ITS mitigation

[David Kaplan]

Add function to reset ITS mitigation back to boot-time default.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 arch/x86/kernel/cpu/bugs.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
index 67561e5c2154..bf5de097e1a9 100644
--- a/arch/x86/kernel/cpu/bugs.c
+++ b/arch/x86/kernel/cpu/bugs.c
@@ -1744,6 +1744,17 @@ static void __init its_apply_mitigation(void)
 	}
 }
 
+#ifdef CONFIG_DYNAMIC_MITIGATIONS
+static void its_reset_mitigation(void)
+{
+	setup_clear_cpu_cap(X86_FEATURE_INDIRECT_THUNK_ITS);
+	setup_clear_cpu_cap(X86_FEATURE_RETHUNK);
+	x86_return_thunk = __x86_return_thunk;
+	its_mitigation = IS_ENABLED(CONFIG_MITIGATION_ITS) ?
+		ITS_MITIGATION_AUTO : ITS_MITIGATION_OFF;
+}
+#endif
+
 #undef pr_fmt
 #define pr_fmt(fmt)	"Transient Scheduler Attacks: " fmt
 
@@ -3966,5 +3977,6 @@ void arch_cpu_reset_mitigations(void)
 	srso_reset_mitigation();
 	gds_reset_mitigation();
 	bhi_reset_mitigation();
+	its_reset_mitigation();
 }
 #endif
-- 
2.34.1

[RFC PATCH 17/56] x86/bugs: Reset TSA mitigations

[David Kaplan]

Add function to reset TSA mitigations back to their boot-time defaults.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 arch/x86/kernel/cpu/bugs.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
index bf5de097e1a9..6a3e079a85fc 100644
--- a/arch/x86/kernel/cpu/bugs.c
+++ b/arch/x86/kernel/cpu/bugs.c
@@ -1858,6 +1858,17 @@ static void __init tsa_apply_mitigation(void)
 	}
 }
 
+#ifdef CONFIG_DYNAMIC_MITIGATIONS
+static void tsa_reset_mitigation(void)
+{
+	setup_clear_cpu_cap(X86_FEATURE_CLEAR_CPU_BUF);
+	setup_clear_cpu_cap(X86_FEATURE_CLEAR_CPU_BUF_VM);
+	static_branch_disable(&cpu_buf_idle_clear);
+	tsa_mitigation =
+		IS_ENABLED(CONFIG_MITIGATION_TSA) ? TSA_MITIGATION_AUTO : TSA_MITIGATION_NONE;
+}
+#endif
+
 #undef pr_fmt
 #define pr_fmt(fmt)     "Spectre V2 : " fmt
 
@@ -3978,5 +3989,6 @@ void arch_cpu_reset_mitigations(void)
 	gds_reset_mitigation();
 	bhi_reset_mitigation();
 	its_reset_mitigation();
+	tsa_reset_mitigation();
 }
 #endif
-- 
2.34.1

[RFC PATCH 18/56] x86/bugs: Reset VMSCAPE mitigations

[David Kaplan]

Add function to reset VMSCAPE mitigations back to their boot-time defaults.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 arch/x86/kernel/cpu/bugs.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
index 6a3e079a85fc..fcb1337026f1 100644
--- a/arch/x86/kernel/cpu/bugs.c
+++ b/arch/x86/kernel/cpu/bugs.c
@@ -3446,6 +3446,15 @@ static void __init vmscape_apply_mitigation(void)
 		setup_force_cpu_cap(X86_FEATURE_IBPB_EXIT_TO_USER);
 }
 
+#ifdef CONFIG_DYNAMIC_MITIGATIONS
+static void vmscape_reset_mitigation(void)
+{
+	setup_clear_cpu_cap(X86_FEATURE_IBPB_EXIT_TO_USER);
+	vmscape_mitigation = IS_ENABLED(CONFIG_MITIGATION_VMSCAPE) ?
+		VMSCAPE_MITIGATION_AUTO : VMSCAPE_MITIGATION_NONE;
+}
+#endif
+
 #undef pr_fmt
 #define pr_fmt(fmt) fmt
 
@@ -3990,5 +3999,6 @@ void arch_cpu_reset_mitigations(void)
 	bhi_reset_mitigation();
 	its_reset_mitigation();
 	tsa_reset_mitigation();
+	vmscape_reset_mitigation();
 }
 #endif
-- 
2.34.1

[RFC PATCH 19/56] x86/bugs: Define bugs_smt_disable()

[David Kaplan]

bugs_smt_disable() is a wrapper around cpu_smt_disable() which can only be
called during boot.  Print a warning if any mitigations try to disable SMT
at runtime.

bugs_smt_disable() is marked __ref because cpu_smt_disable() is an __init
function, but it will never be called after init code is free'd.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 arch/x86/kernel/cpu/bugs.c | 20 ++++++++++++++------
 1 file changed, 14 insertions(+), 6 deletions(-)

diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
index fcb1337026f1..6cd7198f7dca 100644
--- a/arch/x86/kernel/cpu/bugs.c
+++ b/arch/x86/kernel/cpu/bugs.c
@@ -455,6 +455,14 @@ static bool __init should_mitigate_vuln(unsigned int bug)
 	}
 }
 
+static void __ref bugs_smt_disable(bool enable)
+{
+	if (system_state == SYSTEM_BOOTING)
+		cpu_smt_disable(enable);
+	else
+		pr_warn("Unable to disable SMT after system boot!\n");
+}
+
 /* Default mitigation for MDS-affected CPUs */
 static enum mds_mitigations mds_mitigation __ro_after_init =
 	IS_ENABLED(CONFIG_MITIGATION_MDS) ? MDS_MITIGATION_AUTO : MDS_MITIGATION_OFF;
@@ -550,7 +558,7 @@ static void __init mds_apply_mitigation(void)
 		setup_force_cpu_cap(X86_FEATURE_CLEAR_CPU_BUF);
 		if (!boot_cpu_has(X86_BUG_MSBDS_ONLY) &&
 		    (mds_nosmt || smt_mitigations == SMT_MITIGATIONS_ON))
-			cpu_smt_disable(false);
+			bugs_smt_disable(false);
 	}
 }
 
@@ -671,7 +679,7 @@ static void __init taa_apply_mitigation(void)
 		setup_force_cpu_cap(X86_FEATURE_CLEAR_CPU_BUF);
 
 		if (taa_nosmt || smt_mitigations == SMT_MITIGATIONS_ON)
-			cpu_smt_disable(false);
+			bugs_smt_disable(false);
 	}
 }
 
@@ -784,7 +792,7 @@ static void __init mmio_apply_mitigation(void)
 		static_branch_enable(&cpu_buf_idle_clear);
 
 	if (mmio_nosmt || smt_mitigations == SMT_MITIGATIONS_ON)
-		cpu_smt_disable(false);
+		bugs_smt_disable(false);
 }
 
 #ifdef CONFIG_DYNAMIC_MITIGATIONS
@@ -1592,7 +1600,7 @@ static void __init retbleed_apply_mitigation(void)
 
 	if (mitigate_smt && !boot_cpu_has(X86_FEATURE_STIBP) &&
 	    (retbleed_nosmt || smt_mitigations == SMT_MITIGATIONS_ON))
-		cpu_smt_disable(false);
+		bugs_smt_disable(false);
 }
 
 #ifdef CONFIG_DYNAMIC_MITIGATIONS
@@ -3109,10 +3117,10 @@ static void __init l1tf_apply_mitigation(void)
 		break;
 	case L1TF_MITIGATION_FLUSH_NOSMT:
 	case L1TF_MITIGATION_FULL:
-		cpu_smt_disable(false);
+		bugs_smt_disable(false);
 		break;
 	case L1TF_MITIGATION_FULL_FORCE:
-		cpu_smt_disable(true);
+		bugs_smt_disable(true);
 		break;
 	}
 
-- 
2.34.1

[RFC PATCH 20/56] x86/bugs: Move bugs.c logic out of .init section

[David Kaplan]

If dynamic mitigations are supported, all the mitigation selection
functions and mitigation choices may change at runtime.  Therefore, none of
the functions may exist in .init and the data must not be read-only.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 arch/x86/kernel/cpu/bugs.c | 312 ++++++++++++++++++-------------------
 1 file changed, 156 insertions(+), 156 deletions(-)

diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
index 6cd7198f7dca..06061bcb08b8 100644
--- a/arch/x86/kernel/cpu/bugs.c
+++ b/arch/x86/kernel/cpu/bugs.c
@@ -55,52 +55,52 @@
  * mitigation option.
  */
 
-static void __init spectre_v1_select_mitigation(void);
-static void __init spectre_v1_apply_mitigation(void);
-static void __init spectre_v2_select_mitigation(void);
-static void __init spectre_v2_update_mitigation(void);
-static void __init spectre_v2_apply_mitigation(void);
-static void __init retbleed_select_mitigation(void);
-static void __init retbleed_update_mitigation(void);
-static void __init retbleed_apply_mitigation(void);
-static void __init spectre_v2_user_select_mitigation(void);
-static void __init spectre_v2_user_update_mitigation(void);
-static void __init spectre_v2_user_apply_mitigation(void);
-static void __init ssb_select_mitigation(void);
-static void __init ssb_apply_mitigation(void);
-static void __init l1tf_select_mitigation(void);
-static void __init l1tf_apply_mitigation(void);
-static void __init mds_select_mitigation(void);
-static void __init mds_update_mitigation(void);
-static void __init mds_apply_mitigation(void);
-static void __init taa_select_mitigation(void);
-static void __init taa_update_mitigation(void);
-static void __init taa_apply_mitigation(void);
-static void __init mmio_select_mitigation(void);
-static void __init mmio_update_mitigation(void);
-static void __init mmio_apply_mitigation(void);
-static void __init rfds_select_mitigation(void);
-static void __init rfds_update_mitigation(void);
-static void __init rfds_apply_mitigation(void);
-static void __init srbds_select_mitigation(void);
-static void __init srbds_apply_mitigation(void);
-static void __init l1d_flush_select_mitigation(void);
-static void __init srso_select_mitigation(void);
-static void __init srso_update_mitigation(void);
-static void __init srso_apply_mitigation(void);
-static void __init gds_select_mitigation(void);
-static void __init gds_apply_mitigation(void);
-static void __init bhi_select_mitigation(void);
-static void __init bhi_update_mitigation(void);
-static void __init bhi_apply_mitigation(void);
-static void __init its_select_mitigation(void);
-static void __init its_update_mitigation(void);
-static void __init its_apply_mitigation(void);
-static void __init tsa_select_mitigation(void);
-static void __init tsa_apply_mitigation(void);
-static void __init vmscape_select_mitigation(void);
-static void __init vmscape_update_mitigation(void);
-static void __init vmscape_apply_mitigation(void);
+static void spectre_v1_select_mitigation(void);
+static void spectre_v1_apply_mitigation(void);
+static void spectre_v2_select_mitigation(void);
+static void spectre_v2_update_mitigation(void);
+static void spectre_v2_apply_mitigation(void);
+static void retbleed_select_mitigation(void);
+static void retbleed_update_mitigation(void);
+static void retbleed_apply_mitigation(void);
+static void spectre_v2_user_select_mitigation(void);
+static void spectre_v2_user_update_mitigation(void);
+static void spectre_v2_user_apply_mitigation(void);
+static void ssb_select_mitigation(void);
+static void ssb_apply_mitigation(void);
+static void l1tf_select_mitigation(void);
+static void l1tf_apply_mitigation(void);
+static void mds_select_mitigation(void);
+static void mds_update_mitigation(void);
+static void mds_apply_mitigation(void);
+static void taa_select_mitigation(void);
+static void taa_update_mitigation(void);
+static void taa_apply_mitigation(void);
+static void mmio_select_mitigation(void);
+static void mmio_update_mitigation(void);
+static void mmio_apply_mitigation(void);
+static void rfds_select_mitigation(void);
+static void rfds_update_mitigation(void);
+static void rfds_apply_mitigation(void);
+static void srbds_select_mitigation(void);
+static void srbds_apply_mitigation(void);
+static void l1d_flush_select_mitigation(void);
+static void srso_select_mitigation(void);
+static void srso_update_mitigation(void);
+static void srso_apply_mitigation(void);
+static void gds_select_mitigation(void);
+static void gds_apply_mitigation(void);
+static void bhi_select_mitigation(void);
+static void bhi_update_mitigation(void);
+static void bhi_apply_mitigation(void);
+static void its_select_mitigation(void);
+static void its_update_mitigation(void);
+static void its_apply_mitigation(void);
+static void tsa_select_mitigation(void);
+static void tsa_apply_mitigation(void);
+static void vmscape_select_mitigation(void);
+static void vmscape_update_mitigation(void);
+static void vmscape_apply_mitigation(void);
 
 /* The base value of the SPEC_CTRL MSR without task-specific bits set */
 u64 x86_spec_ctrl_base;
@@ -118,15 +118,15 @@ EXPORT_PER_CPU_SYMBOL_GPL(x86_spec_ctrl_current);
 DEFINE_PER_CPU(bool, x86_ibpb_exit_to_user);
 EXPORT_PER_CPU_SYMBOL_GPL(x86_ibpb_exit_to_user);
 
-u64 x86_pred_cmd __ro_after_init = PRED_CMD_IBPB;
+u64 x86_pred_cmd = PRED_CMD_IBPB;
 
-static u64 __ro_after_init x86_arch_cap_msr;
+static u64 x86_arch_cap_msr;
 
 static DEFINE_MUTEX(spec_ctrl_mutex);
 
-void (*x86_return_thunk)(void) __ro_after_init = __x86_return_thunk;
+void (*x86_return_thunk)(void) = __x86_return_thunk;
 
-static void __init set_return_thunk(void *thunk)
+static void set_return_thunk(void *thunk)
 {
 	x86_return_thunk = thunk;
 
@@ -169,8 +169,8 @@ EXPORT_SYMBOL_GPL(spec_ctrl_current);
  * AMD specific MSR info for Speculative Store Bypass control.
  * x86_amd_ls_cfg_ssbd_mask is initialized in identify_boot_cpu().
  */
-u64 __ro_after_init x86_amd_ls_cfg_base;
-u64 __ro_after_init x86_amd_ls_cfg_ssbd_mask;
+u64 x86_amd_ls_cfg_base;
+u64 x86_amd_ls_cfg_ssbd_mask;
 
 /* Control conditional STIBP in switch_to() */
 DEFINE_STATIC_KEY_FALSE(switch_to_cond_stibp);
@@ -205,7 +205,7 @@ EXPORT_SYMBOL_GPL(cpu_buf_vm_clear);
 #undef pr_fmt
 #define pr_fmt(fmt)	"mitigations: " fmt
 
-static void __init cpu_print_attack_vectors(void)
+static void cpu_print_attack_vectors(void)
 {
 	pr_info("Enabled attack vectors: ");
 
@@ -235,7 +235,7 @@ static void __init cpu_print_attack_vectors(void)
 	}
 }
 
-void __init cpu_select_mitigations(void)
+void cpu_select_mitigations(void)
 {
 	/*
 	 * Read the SPEC_CTRL MSR to account for reserved bits which may
@@ -399,7 +399,7 @@ static void x86_amd_ssb_enable(void)
  *
  * See Documentation/admin-guide/hw-vuln/attack_vector_controls.rst
  */
-static bool __init should_mitigate_vuln(unsigned int bug)
+static bool should_mitigate_vuln(unsigned int bug)
 {
 	switch (bug) {
 	/*
@@ -464,9 +464,9 @@ static void __ref bugs_smt_disable(bool enable)
 }
 
 /* Default mitigation for MDS-affected CPUs */
-static enum mds_mitigations mds_mitigation __ro_after_init =
+static enum mds_mitigations mds_mitigation =
 	IS_ENABLED(CONFIG_MITIGATION_MDS) ? MDS_MITIGATION_AUTO : MDS_MITIGATION_OFF;
-static bool mds_nosmt __ro_after_init = false;
+static bool mds_nosmt = false;
 
 static const char * const mds_strings[] = {
 	[MDS_MITIGATION_OFF]	= "Vulnerable",
@@ -483,7 +483,7 @@ enum taa_mitigations {
 };
 
 /* Default mitigation for TAA-affected CPUs */
-static enum taa_mitigations taa_mitigation __ro_after_init =
+static enum taa_mitigations taa_mitigation =
 	IS_ENABLED(CONFIG_MITIGATION_TAA) ? TAA_MITIGATION_AUTO : TAA_MITIGATION_OFF;
 
 enum mmio_mitigations {
@@ -494,7 +494,7 @@ enum mmio_mitigations {
 };
 
 /* Default mitigation for Processor MMIO Stale Data vulnerabilities */
-static enum mmio_mitigations mmio_mitigation __ro_after_init =
+static enum mmio_mitigations mmio_mitigation =
 	IS_ENABLED(CONFIG_MITIGATION_MMIO_STALE_DATA) ?	MMIO_MITIGATION_AUTO : MMIO_MITIGATION_OFF;
 
 enum rfds_mitigations {
@@ -505,16 +505,16 @@ enum rfds_mitigations {
 };
 
 /* Default mitigation for Register File Data Sampling */
-static enum rfds_mitigations rfds_mitigation __ro_after_init =
+static enum rfds_mitigations rfds_mitigation =
 	IS_ENABLED(CONFIG_MITIGATION_RFDS) ? RFDS_MITIGATION_AUTO : RFDS_MITIGATION_OFF;
 
 /*
  * Set if any of MDS/TAA/MMIO/RFDS are going to enable VERW clearing
  * through X86_FEATURE_CLEAR_CPU_BUF on kernel and guest entry.
  */
-static bool verw_clear_cpu_buf_mitigation_selected __ro_after_init;
+static bool verw_clear_cpu_buf_mitigation_selected;
 
-static void __init mds_select_mitigation(void)
+static void mds_select_mitigation(void)
 {
 	if (!boot_cpu_has_bug(X86_BUG_MDS)) {
 		mds_mitigation = MDS_MITIGATION_OFF;
@@ -534,7 +534,7 @@ static void __init mds_select_mitigation(void)
 	verw_clear_cpu_buf_mitigation_selected = true;
 }
 
-static void __init mds_update_mitigation(void)
+static void mds_update_mitigation(void)
 {
 	if (!boot_cpu_has_bug(X86_BUG_MDS))
 		return;
@@ -551,7 +551,7 @@ static void __init mds_update_mitigation(void)
 	pr_info("%s\n", mds_strings[mds_mitigation]);
 }
 
-static void __init mds_apply_mitigation(void)
+static void mds_apply_mitigation(void)
 {
 	if (mds_mitigation == MDS_MITIGATION_FULL ||
 	    mds_mitigation == MDS_MITIGATION_VMWERV) {
@@ -572,7 +572,7 @@ static void mds_reset_mitigation(void)
 }
 #endif
 
-static int __init mds_cmdline(char *str)
+static int mds_cmdline(char *str)
 {
 	if (!boot_cpu_has_bug(X86_BUG_MDS))
 		return 0;
@@ -596,7 +596,7 @@ early_param("mds", mds_cmdline);
 #undef pr_fmt
 #define pr_fmt(fmt)	"TAA: " fmt
 
-static bool taa_nosmt __ro_after_init;
+static bool taa_nosmt;
 
 static const char * const taa_strings[] = {
 	[TAA_MITIGATION_OFF]		= "Vulnerable",
@@ -605,12 +605,12 @@ static const char * const taa_strings[] = {
 	[TAA_MITIGATION_TSX_DISABLED]	= "Mitigation: TSX disabled",
 };
 
-static bool __init taa_vulnerable(void)
+static bool taa_vulnerable(void)
 {
 	return boot_cpu_has_bug(X86_BUG_TAA) && boot_cpu_has(X86_FEATURE_RTM);
 }
 
-static void __init taa_select_mitigation(void)
+static void taa_select_mitigation(void)
 {
 	if (!boot_cpu_has_bug(X86_BUG_TAA)) {
 		taa_mitigation = TAA_MITIGATION_OFF;
@@ -635,7 +635,7 @@ static void __init taa_select_mitigation(void)
 		verw_clear_cpu_buf_mitigation_selected = true;
 }
 
-static void __init taa_update_mitigation(void)
+static void taa_update_mitigation(void)
 {
 	if (!taa_vulnerable())
 		return;
@@ -665,7 +665,7 @@ static void __init taa_update_mitigation(void)
 	pr_info("%s\n", taa_strings[taa_mitigation]);
 }
 
-static void __init taa_apply_mitigation(void)
+static void taa_apply_mitigation(void)
 {
 	if (taa_mitigation == TAA_MITIGATION_VERW ||
 	    taa_mitigation == TAA_MITIGATION_UCODE_NEEDED) {
@@ -683,7 +683,7 @@ static void __init taa_apply_mitigation(void)
 	}
 }
 
-static int __init tsx_async_abort_parse_cmdline(char *str)
+static int tsx_async_abort_parse_cmdline(char *str)
 {
 	if (!boot_cpu_has_bug(X86_BUG_TAA))
 		return 0;
@@ -707,7 +707,7 @@ early_param("tsx_async_abort", tsx_async_abort_parse_cmdline);
 #undef pr_fmt
 #define pr_fmt(fmt)	"MMIO Stale Data: " fmt
 
-static bool mmio_nosmt __ro_after_init = false;
+static bool mmio_nosmt = false;
 
 static const char * const mmio_strings[] = {
 	[MMIO_MITIGATION_OFF]		= "Vulnerable",
@@ -715,7 +715,7 @@ static const char * const mmio_strings[] = {
 	[MMIO_MITIGATION_VERW]		= "Mitigation: Clear CPU buffers",
 };
 
-static void __init mmio_select_mitigation(void)
+static void mmio_select_mitigation(void)
 {
 	if (!boot_cpu_has_bug(X86_BUG_MMIO_STALE_DATA)) {
 		mmio_mitigation = MMIO_MITIGATION_OFF;
@@ -741,7 +741,7 @@ static void __init mmio_select_mitigation(void)
 		verw_clear_cpu_buf_mitigation_selected = true;
 }
 
-static void __init mmio_update_mitigation(void)
+static void mmio_update_mitigation(void)
 {
 	if (!boot_cpu_has_bug(X86_BUG_MMIO_STALE_DATA))
 		return;
@@ -767,7 +767,7 @@ static void __init mmio_update_mitigation(void)
 	pr_info("%s\n", mmio_strings[mmio_mitigation]);
 }
 
-static void __init mmio_apply_mitigation(void)
+static void mmio_apply_mitigation(void)
 {
 	if (mmio_mitigation == MMIO_MITIGATION_OFF)
 		return;
@@ -805,7 +805,7 @@ static void mmio_reset_mitigation(void)
 }
 #endif
 
-static int __init mmio_stale_data_parse_cmdline(char *str)
+static int mmio_stale_data_parse_cmdline(char *str)
 {
 	if (!boot_cpu_has_bug(X86_BUG_MMIO_STALE_DATA))
 		return 0;
@@ -835,12 +835,12 @@ static const char * const rfds_strings[] = {
 	[RFDS_MITIGATION_UCODE_NEEDED]		= "Vulnerable: No microcode",
 };
 
-static inline bool __init verw_clears_cpu_reg_file(void)
+static inline bool verw_clears_cpu_reg_file(void)
 {
 	return (x86_arch_cap_msr & ARCH_CAP_RFDS_CLEAR);
 }
 
-static void __init rfds_select_mitigation(void)
+static void rfds_select_mitigation(void)
 {
 	if (!boot_cpu_has_bug(X86_BUG_RFDS)) {
 		rfds_mitigation = RFDS_MITIGATION_OFF;
@@ -861,7 +861,7 @@ static void __init rfds_select_mitigation(void)
 		verw_clear_cpu_buf_mitigation_selected = true;
 }
 
-static void __init rfds_update_mitigation(void)
+static void rfds_update_mitigation(void)
 {
 	if (!boot_cpu_has_bug(X86_BUG_RFDS))
 		return;
@@ -877,13 +877,13 @@ static void __init rfds_update_mitigation(void)
 	pr_info("%s\n", rfds_strings[rfds_mitigation]);
 }
 
-static void __init rfds_apply_mitigation(void)
+static void rfds_apply_mitigation(void)
 {
 	if (rfds_mitigation == RFDS_MITIGATION_VERW)
 		setup_force_cpu_cap(X86_FEATURE_CLEAR_CPU_BUF);
 }
 
-static __init int rfds_parse_cmdline(char *str)
+static int rfds_parse_cmdline(char *str)
 {
 	if (!str)
 		return -EINVAL;
@@ -912,7 +912,7 @@ enum srbds_mitigations {
 	SRBDS_MITIGATION_HYPERVISOR,
 };
 
-static enum srbds_mitigations srbds_mitigation __ro_after_init =
+static enum srbds_mitigations srbds_mitigation =
 	IS_ENABLED(CONFIG_MITIGATION_SRBDS) ? SRBDS_MITIGATION_AUTO : SRBDS_MITIGATION_OFF;
 
 static const char * const srbds_strings[] = {
@@ -962,7 +962,7 @@ void update_srbds_msr(void)
 	wrmsrq(MSR_IA32_MCU_OPT_CTRL, mcu_ctrl);
 }
 
-static void __init srbds_select_mitigation(void)
+static void srbds_select_mitigation(void)
 {
 	if (!boot_cpu_has_bug(X86_BUG_SRBDS)) {
 		srbds_mitigation = SRBDS_MITIGATION_OFF;
@@ -996,7 +996,7 @@ static void __init srbds_select_mitigation(void)
 	pr_info("%s\n", srbds_strings[srbds_mitigation]);
 }
 
-static void __init srbds_apply_mitigation(void)
+static void srbds_apply_mitigation(void)
 {
 	update_srbds_msr();
 }
@@ -1012,7 +1012,7 @@ static void srbds_reset_mitigation(void)
 }
 #endif
 
-static int __init srbds_parse_cmdline(char *str)
+static int srbds_parse_cmdline(char *str)
 {
 	if (!str)
 		return -EINVAL;
@@ -1033,9 +1033,9 @@ enum l1d_flush_mitigations {
 	L1D_FLUSH_ON,
 };
 
-static enum l1d_flush_mitigations l1d_flush_mitigation __initdata = L1D_FLUSH_OFF;
+static enum l1d_flush_mitigations l1d_flush_mitigation = L1D_FLUSH_OFF;
 
-static void __init l1d_flush_select_mitigation(void)
+static void l1d_flush_select_mitigation(void)
 {
 	if (!l1d_flush_mitigation || !boot_cpu_has(X86_FEATURE_FLUSH_L1D))
 		return;
@@ -1044,7 +1044,7 @@ static void __init l1d_flush_select_mitigation(void)
 	pr_info("Conditional flush on switch_mm() enabled\n");
 }
 
-static int __init l1d_flush_parse_cmdline(char *str)
+static int l1d_flush_parse_cmdline(char *str)
 {
 	if (!strcmp(str, "on"))
 		l1d_flush_mitigation = L1D_FLUSH_ON;
@@ -1066,7 +1066,7 @@ enum gds_mitigations {
 	GDS_MITIGATION_HYPERVISOR,
 };
 
-static enum gds_mitigations gds_mitigation __ro_after_init =
+static enum gds_mitigations gds_mitigation =
 	IS_ENABLED(CONFIG_MITIGATION_GDS) ? GDS_MITIGATION_AUTO : GDS_MITIGATION_OFF;
 
 static const char * const gds_strings[] = {
@@ -1123,7 +1123,7 @@ void update_gds_msr(void)
 	WARN_ON_ONCE(mcu_ctrl != mcu_ctrl_after);
 }
 
-static void __init gds_select_mitigation(void)
+static void gds_select_mitigation(void)
 {
 	u64 mcu_ctrl;
 
@@ -1170,7 +1170,7 @@ static void __init gds_select_mitigation(void)
 	}
 }
 
-static void __init gds_apply_mitigation(void)
+static void gds_apply_mitigation(void)
 {
 	if (!boot_cpu_has_bug(X86_BUG_GDS))
 		return;
@@ -1202,7 +1202,7 @@ static void gds_reset_mitigation(void)
 }
 #endif
 
-static int __init gds_parse_cmdline(char *str)
+static int gds_parse_cmdline(char *str)
 {
 	if (!str)
 		return -EINVAL;
@@ -1227,7 +1227,7 @@ enum spectre_v1_mitigation {
 	SPECTRE_V1_MITIGATION_AUTO,
 };
 
-static enum spectre_v1_mitigation spectre_v1_mitigation __ro_after_init =
+static enum spectre_v1_mitigation spectre_v1_mitigation =
 	IS_ENABLED(CONFIG_MITIGATION_SPECTRE_V1) ?
 		SPECTRE_V1_MITIGATION_AUTO : SPECTRE_V1_MITIGATION_NONE;
 
@@ -1257,7 +1257,7 @@ static bool smap_works_speculatively(void)
 	return true;
 }
 
-static void __init spectre_v1_select_mitigation(void)
+static void spectre_v1_select_mitigation(void)
 {
 	if (!boot_cpu_has_bug(X86_BUG_SPECTRE_V1))
 		spectre_v1_mitigation = SPECTRE_V1_MITIGATION_NONE;
@@ -1266,7 +1266,7 @@ static void __init spectre_v1_select_mitigation(void)
 		spectre_v1_mitigation = SPECTRE_V1_MITIGATION_NONE;
 }
 
-static void __init spectre_v1_apply_mitigation(void)
+static void spectre_v1_apply_mitigation(void)
 {
 	if (!boot_cpu_has_bug(X86_BUG_SPECTRE_V1))
 		return;
@@ -1319,14 +1319,14 @@ static void spectre_v1_reset_mitigation(void)
 }
 #endif
 
-static int __init nospectre_v1_cmdline(char *str)
+static int nospectre_v1_cmdline(char *str)
 {
 	spectre_v1_mitigation = SPECTRE_V1_MITIGATION_NONE;
 	return 0;
 }
 early_param("nospectre_v1", nospectre_v1_cmdline);
 
-enum spectre_v2_mitigation spectre_v2_enabled __ro_after_init = SPECTRE_V2_NONE;
+enum spectre_v2_mitigation spectre_v2_enabled = SPECTRE_V2_NONE;
 
 /* Depends on spectre_v2 mitigation selected already */
 static inline bool cdt_possible(enum spectre_v2_mitigation mode)
@@ -1353,7 +1353,7 @@ enum its_mitigation {
 	ITS_MITIGATION_RETPOLINE_STUFF,
 };
 
-static enum its_mitigation its_mitigation __ro_after_init =
+static enum its_mitigation its_mitigation =
 	IS_ENABLED(CONFIG_MITIGATION_ITS) ? ITS_MITIGATION_AUTO : ITS_MITIGATION_OFF;
 
 enum retbleed_mitigation {
@@ -1375,10 +1375,10 @@ static const char * const retbleed_strings[] = {
 	[RETBLEED_MITIGATION_STUFF]	= "Mitigation: Stuffing",
 };
 
-static enum retbleed_mitigation retbleed_mitigation __ro_after_init =
+static enum retbleed_mitigation retbleed_mitigation =
 	IS_ENABLED(CONFIG_MITIGATION_RETBLEED) ? RETBLEED_MITIGATION_AUTO : RETBLEED_MITIGATION_NONE;
 
-static int __ro_after_init retbleed_nosmt = false;
+static int retbleed_nosmt = false;
 
 enum srso_mitigation {
 	SRSO_MITIGATION_NONE,
@@ -1393,9 +1393,9 @@ enum srso_mitigation {
 	SRSO_MITIGATION_BP_SPEC_REDUCE,
 };
 
-static enum srso_mitigation srso_mitigation __ro_after_init = SRSO_MITIGATION_AUTO;
+static enum srso_mitigation srso_mitigation = SRSO_MITIGATION_AUTO;
 
-static int __init retbleed_parse_cmdline(char *str)
+static int retbleed_parse_cmdline(char *str)
 {
 	if (!str)
 		return -EINVAL;
@@ -1435,7 +1435,7 @@ early_param("retbleed", retbleed_parse_cmdline);
 #define RETBLEED_UNTRAIN_MSG "WARNING: BTB untrained return thunk mitigation is only effective on AMD/Hygon!\n"
 #define RETBLEED_INTEL_MSG "WARNING: Spectre v2 mitigation leaves CPU vulnerable to RETBleed attacks, data leaks possible!\n"
 
-static void __init retbleed_select_mitigation(void)
+static void retbleed_select_mitigation(void)
 {
 	if (!boot_cpu_has_bug(X86_BUG_RETBLEED)) {
 		retbleed_mitigation = RETBLEED_MITIGATION_NONE;
@@ -1500,7 +1500,7 @@ static void __init retbleed_select_mitigation(void)
 	}
 }
 
-static void __init retbleed_update_mitigation(void)
+static void retbleed_update_mitigation(void)
 {
 	if (!boot_cpu_has_bug(X86_BUG_RETBLEED))
 		return;
@@ -1544,7 +1544,7 @@ static void __init retbleed_update_mitigation(void)
 	pr_info("%s\n", retbleed_strings[retbleed_mitigation]);
 }
 
-static void __init retbleed_apply_mitigation(void)
+static void retbleed_apply_mitigation(void)
 {
 	bool mitigate_smt = false;
 
@@ -1627,7 +1627,7 @@ static const char * const its_strings[] = {
 	[ITS_MITIGATION_RETPOLINE_STUFF]	= "Mitigation: Retpolines, Stuffing RSB",
 };
 
-static int __init its_parse_cmdline(char *str)
+static int its_parse_cmdline(char *str)
 {
 	if (!str)
 		return -EINVAL;
@@ -1656,7 +1656,7 @@ static int __init its_parse_cmdline(char *str)
 }
 early_param("indirect_target_selection", its_parse_cmdline);
 
-static void __init its_select_mitigation(void)
+static void its_select_mitigation(void)
 {
 	if (!boot_cpu_has_bug(X86_BUG_ITS)) {
 		its_mitigation = ITS_MITIGATION_OFF;
@@ -1697,7 +1697,7 @@ static void __init its_select_mitigation(void)
 		its_mitigation = ITS_MITIGATION_ALIGNED_THUNKS;
 }
 
-static void __init its_update_mitigation(void)
+static void its_update_mitigation(void)
 {
 	if (!boot_cpu_has_bug(X86_BUG_ITS))
 		return;
@@ -1730,7 +1730,7 @@ static void __init its_update_mitigation(void)
 	pr_info("%s\n", its_strings[its_mitigation]);
 }
 
-static void __init its_apply_mitigation(void)
+static void its_apply_mitigation(void)
 {
 	switch (its_mitigation) {
 	case ITS_MITIGATION_OFF:
@@ -1783,10 +1783,10 @@ static const char * const tsa_strings[] = {
 	[TSA_MITIGATION_FULL]		= "Mitigation: Clear CPU buffers",
 };
 
-static enum tsa_mitigations tsa_mitigation __ro_after_init =
+static enum tsa_mitigations tsa_mitigation =
 	IS_ENABLED(CONFIG_MITIGATION_TSA) ? TSA_MITIGATION_AUTO : TSA_MITIGATION_NONE;
 
-static int __init tsa_parse_cmdline(char *str)
+static int tsa_parse_cmdline(char *str)
 {
 	if (!str)
 		return -EINVAL;
@@ -1806,7 +1806,7 @@ static int __init tsa_parse_cmdline(char *str)
 }
 early_param("tsa", tsa_parse_cmdline);
 
-static void __init tsa_select_mitigation(void)
+static void tsa_select_mitigation(void)
 {
 	if (!boot_cpu_has_bug(X86_BUG_TSA)) {
 		tsa_mitigation = TSA_MITIGATION_NONE;
@@ -1848,7 +1848,7 @@ static void __init tsa_select_mitigation(void)
 	pr_info("%s\n", tsa_strings[tsa_mitigation]);
 }
 
-static void __init tsa_apply_mitigation(void)
+static void tsa_apply_mitigation(void)
 {
 	switch (tsa_mitigation) {
 	case TSA_MITIGATION_USER_KERNEL:
@@ -1880,9 +1880,9 @@ static void tsa_reset_mitigation(void)
 #undef pr_fmt
 #define pr_fmt(fmt)     "Spectre V2 : " fmt
 
-static enum spectre_v2_user_mitigation spectre_v2_user_stibp __ro_after_init =
+static enum spectre_v2_user_mitigation spectre_v2_user_stibp =
 	SPECTRE_V2_USER_NONE;
-static enum spectre_v2_user_mitigation spectre_v2_user_ibpb __ro_after_init =
+static enum spectre_v2_user_mitigation spectre_v2_user_ibpb =
 	SPECTRE_V2_USER_NONE;
 
 #ifdef CONFIG_MITIGATION_RETPOLINE
@@ -1954,7 +1954,7 @@ enum spectre_v2_mitigation_cmd {
 	SPECTRE_V2_CMD_IBRS,
 };
 
-static enum spectre_v2_mitigation_cmd spectre_v2_cmd __ro_after_init =
+static enum spectre_v2_mitigation_cmd spectre_v2_cmd =
 	IS_ENABLED(CONFIG_MITIGATION_SPECTRE_V2) ? SPECTRE_V2_CMD_AUTO : SPECTRE_V2_CMD_NONE;
 
 enum spectre_v2_user_mitigation_cmd {
@@ -1967,7 +1967,7 @@ enum spectre_v2_user_mitigation_cmd {
 	SPECTRE_V2_USER_CMD_SECCOMP_IBPB,
 };
 
-static enum spectre_v2_user_mitigation_cmd spectre_v2_user_cmd __ro_after_init =
+static enum spectre_v2_user_mitigation_cmd spectre_v2_user_cmd =
 	IS_ENABLED(CONFIG_MITIGATION_SPECTRE_V2) ? SPECTRE_V2_USER_CMD_AUTO : SPECTRE_V2_USER_CMD_NONE;
 
 static const char * const spectre_v2_user_strings[] = {
@@ -1978,7 +1978,7 @@ static const char * const spectre_v2_user_strings[] = {
 	[SPECTRE_V2_USER_SECCOMP]		= "User space: Mitigation: STIBP via seccomp and prctl",
 };
 
-static int __init spectre_v2_user_parse_cmdline(char *str)
+static int spectre_v2_user_parse_cmdline(char *str)
 {
 	if (!str)
 		return -EINVAL;
@@ -2009,7 +2009,7 @@ static inline bool spectre_v2_in_ibrs_mode(enum spectre_v2_mitigation mode)
 	return spectre_v2_in_eibrs_mode(mode) || mode == SPECTRE_V2_IBRS;
 }
 
-static void __init spectre_v2_user_select_mitigation(void)
+static void spectre_v2_user_select_mitigation(void)
 {
 	if (!boot_cpu_has(X86_FEATURE_IBPB) && !boot_cpu_has(X86_FEATURE_STIBP))
 		return;
@@ -2070,7 +2070,7 @@ static void __init spectre_v2_user_select_mitigation(void)
 		spectre_v2_user_stibp = SPECTRE_V2_USER_NONE;
 }
 
-static void __init spectre_v2_user_update_mitigation(void)
+static void spectre_v2_user_update_mitigation(void)
 {
 	if (!boot_cpu_has(X86_FEATURE_IBPB) && !boot_cpu_has(X86_FEATURE_STIBP))
 		return;
@@ -2115,7 +2115,7 @@ static void __init spectre_v2_user_update_mitigation(void)
 	pr_info("%s\n", spectre_v2_user_strings[spectre_v2_user_stibp]);
 }
 
-static void __init spectre_v2_user_apply_mitigation(void)
+static void spectre_v2_user_apply_mitigation(void)
 {
 	/* Initialize Indirect Branch Prediction Barrier */
 	if (spectre_v2_user_ibpb != SPECTRE_V2_USER_NONE) {
@@ -2161,9 +2161,9 @@ static const char * const spectre_v2_strings[] = {
 	[SPECTRE_V2_IBRS]			= "Mitigation: IBRS",
 };
 
-static bool nospectre_v2 __ro_after_init;
+static bool nospectre_v2;
 
-static int __init nospectre_v2_parse_cmdline(char *str)
+static int nospectre_v2_parse_cmdline(char *str)
 {
 	nospectre_v2 = true;
 	spectre_v2_cmd = SPECTRE_V2_CMD_NONE;
@@ -2171,7 +2171,7 @@ static int __init nospectre_v2_parse_cmdline(char *str)
 }
 early_param("nospectre_v2", nospectre_v2_parse_cmdline);
 
-static int __init spectre_v2_parse_cmdline(char *str)
+static int spectre_v2_parse_cmdline(char *str)
 {
 	if (!str)
 		return -EINVAL;
@@ -2210,7 +2210,7 @@ static int __init spectre_v2_parse_cmdline(char *str)
 }
 early_param("spectre_v2", spectre_v2_parse_cmdline);
 
-static enum spectre_v2_mitigation __init spectre_v2_select_retpoline(void)
+static enum spectre_v2_mitigation spectre_v2_select_retpoline(void)
 {
 	if (!IS_ENABLED(CONFIG_MITIGATION_RETPOLINE)) {
 		pr_err("Kernel not compiled with retpoline; no mitigation available!");
@@ -2220,10 +2220,10 @@ static enum spectre_v2_mitigation __init spectre_v2_select_retpoline(void)
 	return SPECTRE_V2_RETPOLINE;
 }
 
-static bool __ro_after_init rrsba_disabled;
+static bool rrsba_disabled;
 
 /* Disable in-kernel use of non-RSB RET predictors */
-static void __init spec_ctrl_disable_kernel_rrsba(void)
+static void spec_ctrl_disable_kernel_rrsba(void)
 {
 	if (rrsba_disabled)
 		return;
@@ -2241,7 +2241,7 @@ static void __init spec_ctrl_disable_kernel_rrsba(void)
 	rrsba_disabled = true;
 }
 
-static void __init spectre_v2_select_rsb_mitigation(enum spectre_v2_mitigation mode)
+static void spectre_v2_select_rsb_mitigation(enum spectre_v2_mitigation mode)
 {
 	/*
 	 * WARNING! There are many subtleties to consider when changing *any*
@@ -2295,7 +2295,7 @@ static void __init spectre_v2_select_rsb_mitigation(enum spectre_v2_mitigation m
  * Set BHI_DIS_S to prevent indirect branches in kernel to be influenced by
  * branch history in userspace. Not needed if BHI_NO is set.
  */
-static bool __init spec_ctrl_bhi_dis(void)
+static bool spec_ctrl_bhi_dis(void)
 {
 	if (!boot_cpu_has(X86_FEATURE_BHI_CTRL))
 		return false;
@@ -2314,10 +2314,10 @@ enum bhi_mitigations {
 	BHI_MITIGATION_VMEXIT_ONLY,
 };
 
-static enum bhi_mitigations bhi_mitigation __ro_after_init =
+static enum bhi_mitigations bhi_mitigation =
 	IS_ENABLED(CONFIG_MITIGATION_SPECTRE_BHI) ? BHI_MITIGATION_AUTO : BHI_MITIGATION_OFF;
 
-static int __init spectre_bhi_parse_cmdline(char *str)
+static int spectre_bhi_parse_cmdline(char *str)
 {
 	if (!str)
 		return -EINVAL;
@@ -2335,7 +2335,7 @@ static int __init spectre_bhi_parse_cmdline(char *str)
 }
 early_param("spectre_bhi", spectre_bhi_parse_cmdline);
 
-static void __init bhi_select_mitigation(void)
+static void bhi_select_mitigation(void)
 {
 	if (!boot_cpu_has(X86_BUG_BHI))
 		bhi_mitigation = BHI_MITIGATION_OFF;
@@ -2353,13 +2353,13 @@ static void __init bhi_select_mitigation(void)
 	}
 }
 
-static void __init bhi_update_mitigation(void)
+static void bhi_update_mitigation(void)
 {
 	if (spectre_v2_cmd == SPECTRE_V2_CMD_NONE)
 		bhi_mitigation = BHI_MITIGATION_OFF;
 }
 
-static void __init bhi_apply_mitigation(void)
+static void bhi_apply_mitigation(void)
 {
 	if (bhi_mitigation == BHI_MITIGATION_OFF)
 		return;
@@ -2401,7 +2401,7 @@ static void bhi_reset_mitigation(void)
 }
 #endif
 
-static void __init spectre_v2_select_mitigation(void)
+static void spectre_v2_select_mitigation(void)
 {
 	if ((spectre_v2_cmd == SPECTRE_V2_CMD_RETPOLINE ||
 	     spectre_v2_cmd == SPECTRE_V2_CMD_RETPOLINE_LFENCE ||
@@ -2501,7 +2501,7 @@ static void __init spectre_v2_select_mitigation(void)
 	}
 }
 
-static void __init spectre_v2_update_mitigation(void)
+static void spectre_v2_update_mitigation(void)
 {
 	if (spectre_v2_cmd == SPECTRE_V2_CMD_AUTO &&
 	    !spectre_v2_in_eibrs_mode(spectre_v2_enabled)) {
@@ -2519,7 +2519,7 @@ static void __init spectre_v2_update_mitigation(void)
 		pr_info("%s\n", spectre_v2_strings[spectre_v2_enabled]);
 }
 
-static void __init spectre_v2_apply_mitigation(void)
+static void spectre_v2_apply_mitigation(void)
 {
 	if (spectre_v2_enabled == SPECTRE_V2_EIBRS && unprivileged_ebpf_enabled())
 		pr_err(SPECTRE_V2_EIBRS_EBPF_MSG);
@@ -2675,7 +2675,7 @@ static void update_mds_branch_idle(void)
 #undef pr_fmt
 #define pr_fmt(fmt)	"Speculative Store Bypass: " fmt
 
-static enum ssb_mitigation ssb_mode __ro_after_init =
+static enum ssb_mitigation ssb_mode =
 	IS_ENABLED(CONFIG_MITIGATION_SSB) ? SPEC_STORE_BYPASS_AUTO : SPEC_STORE_BYPASS_NONE;
 
 static const char * const ssb_strings[] = {
@@ -2685,9 +2685,9 @@ static const char * const ssb_strings[] = {
 	[SPEC_STORE_BYPASS_SECCOMP]	= "Mitigation: Speculative Store Bypass disabled via prctl and seccomp",
 };
 
-static bool nossb __ro_after_init;
+static bool nossb;
 
-static int __init nossb_parse_cmdline(char *str)
+static int nossb_parse_cmdline(char *str)
 {
 	nossb = true;
 	ssb_mode = SPEC_STORE_BYPASS_NONE;
@@ -2695,7 +2695,7 @@ static int __init nossb_parse_cmdline(char *str)
 }
 early_param("nospec_store_bypass_disable", nossb_parse_cmdline);
 
-static int __init ssb_parse_cmdline(char *str)
+static int ssb_parse_cmdline(char *str)
 {
 	if (!str)
 		return -EINVAL;
@@ -2722,7 +2722,7 @@ static int __init ssb_parse_cmdline(char *str)
 }
 early_param("spec_store_bypass_disable", ssb_parse_cmdline);
 
-static void __init ssb_select_mitigation(void)
+static void ssb_select_mitigation(void)
 {
 	if (!boot_cpu_has_bug(X86_BUG_SPEC_STORE_BYPASS)) {
 		ssb_mode = SPEC_STORE_BYPASS_NONE;
@@ -2742,7 +2742,7 @@ static void __init ssb_select_mitigation(void)
 	pr_info("%s\n", ssb_strings[ssb_mode]);
 }
 
-static void __init ssb_apply_mitigation(void)
+static void ssb_apply_mitigation(void)
 {
 	/*
 	 * We have three CPU feature flags that are in play here:
@@ -3032,7 +3032,7 @@ EXPORT_SYMBOL_GPL(itlb_multihit_kvm_mitigation);
 #define pr_fmt(fmt)	"L1TF: " fmt
 
 /* Default mitigation for L1TF-affected CPUs */
-enum l1tf_mitigations l1tf_mitigation __ro_after_init =
+enum l1tf_mitigations l1tf_mitigation =
 	IS_ENABLED(CONFIG_MITIGATION_L1TF) ? L1TF_MITIGATION_AUTO : L1TF_MITIGATION_OFF;
 #if IS_ENABLED(CONFIG_KVM_INTEL)
 EXPORT_SYMBOL_GPL(l1tf_mitigation);
@@ -3079,7 +3079,7 @@ static void override_cache_bits(struct cpuinfo_x86 *c)
 	}
 }
 
-static void __init l1tf_select_mitigation(void)
+static void l1tf_select_mitigation(void)
 {
 	if (!boot_cpu_has_bug(X86_BUG_L1TF)) {
 		l1tf_mitigation = L1TF_MITIGATION_OFF;
@@ -3100,7 +3100,7 @@ static void __init l1tf_select_mitigation(void)
 		l1tf_mitigation = L1TF_MITIGATION_FLUSH;
 }
 
-static void __init l1tf_apply_mitigation(void)
+static void l1tf_apply_mitigation(void)
 {
 	u64 half_pa;
 
@@ -3152,7 +3152,7 @@ static void l1tf_reset_mitigation(void)
 }
 #endif
 
-static int __init l1tf_cmdline(char *str)
+static int l1tf_cmdline(char *str)
 {
 	if (!boot_cpu_has_bug(X86_BUG_L1TF))
 		return 0;
@@ -3192,7 +3192,7 @@ static const char * const srso_strings[] = {
 	[SRSO_MITIGATION_BP_SPEC_REDUCE]	= "Mitigation: Reduced Speculation"
 };
 
-static int __init srso_parse_cmdline(char *str)
+static int srso_parse_cmdline(char *str)
 {
 	if (!str)
 		return -EINVAL;
@@ -3216,7 +3216,7 @@ early_param("spec_rstack_overflow", srso_parse_cmdline);
 
 #define SRSO_NOTICE "WARNING: See https://kernel.org/doc/html/latest/admin-guide/hw-vuln/srso.html for mitigation options."
 
-static void __init srso_select_mitigation(void)
+static void srso_select_mitigation(void)
 {
 	if (!boot_cpu_has_bug(X86_BUG_SRSO)) {
 		srso_mitigation = SRSO_MITIGATION_NONE;
@@ -3295,7 +3295,7 @@ static void __init srso_select_mitigation(void)
 	}
 }
 
-static void __init srso_update_mitigation(void)
+static void srso_update_mitigation(void)
 {
 	if (!boot_cpu_has_bug(X86_BUG_SRSO))
 		return;
@@ -3308,7 +3308,7 @@ static void __init srso_update_mitigation(void)
 	pr_info("%s\n", srso_strings[srso_mitigation]);
 }
 
-static void __init srso_apply_mitigation(void)
+static void srso_apply_mitigation(void)
 {
 	/*
 	 * Clear the feature flag if this mitigation is not selected as that
@@ -3397,10 +3397,10 @@ static const char * const vmscape_strings[] = {
 	[VMSCAPE_MITIGATION_IBPB_ON_VMEXIT]	= "Mitigation: IBPB on VMEXIT",
 };
 
-static enum vmscape_mitigations vmscape_mitigation __ro_after_init =
+static enum vmscape_mitigations vmscape_mitigation =
 	IS_ENABLED(CONFIG_MITIGATION_VMSCAPE) ? VMSCAPE_MITIGATION_AUTO : VMSCAPE_MITIGATION_NONE;
 
-static int __init vmscape_parse_cmdline(char *str)
+static int vmscape_parse_cmdline(char *str)
 {
 	if (!str)
 		return -EINVAL;
@@ -3420,7 +3420,7 @@ static int __init vmscape_parse_cmdline(char *str)
 }
 early_param("vmscape", vmscape_parse_cmdline);
 
-static void __init vmscape_select_mitigation(void)
+static void vmscape_select_mitigation(void)
 {
 	if (!boot_cpu_has_bug(X86_BUG_VMSCAPE) ||
 	    !boot_cpu_has(X86_FEATURE_IBPB)) {
@@ -3436,7 +3436,7 @@ static void __init vmscape_select_mitigation(void)
 	}
 }
 
-static void __init vmscape_update_mitigation(void)
+static void vmscape_update_mitigation(void)
 {
 	if (!boot_cpu_has_bug(X86_BUG_VMSCAPE))
 		return;
@@ -3448,7 +3448,7 @@ static void __init vmscape_update_mitigation(void)
 	pr_info("%s\n", vmscape_strings[vmscape_mitigation]);
 }
 
-static void __init vmscape_apply_mitigation(void)
+static void vmscape_apply_mitigation(void)
 {
 	if (vmscape_mitigation == VMSCAPE_MITIGATION_IBPB_EXIT_TO_USER)
 		setup_force_cpu_cap(X86_FEATURE_IBPB_EXIT_TO_USER);
-- 
2.34.1

[RFC PATCH 21/56] x86/callthunks: Move logic out of .init

[David Kaplan]

With dynamic re-patching, these functions may be called at runtime so
move them out of the .init section.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 arch/x86/kernel/callthunks.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/arch/x86/kernel/callthunks.c b/arch/x86/kernel/callthunks.c
index a951333c5995..758e655f36a8 100644
--- a/arch/x86/kernel/callthunks.c
+++ b/arch/x86/kernel/callthunks.c
@@ -56,7 +56,7 @@ struct core_text {
 	const char	*name;
 };
 
-static bool thunks_initialized __ro_after_init;
+static bool thunks_initialized;
 
 static const struct core_text builtin_coretext = {
 	.base = (unsigned long)_text,
@@ -151,7 +151,7 @@ static bool skip_addr(void *dest)
 	return false;
 }
 
-static __init_or_module void *call_get_dest(void *addr)
+static void *call_get_dest(void *addr)
 {
 	struct insn insn;
 	void *dest;
@@ -204,7 +204,7 @@ static void *patch_dest(void *dest, bool direct)
 	return pad;
 }
 
-static __init_or_module void patch_call(void *addr, const struct core_text *ct)
+static void patch_call(void *addr, const struct core_text *ct)
 {
 	void *pad, *dest;
 	u8 bytes[8];
@@ -229,7 +229,7 @@ static __init_or_module void patch_call(void *addr, const struct core_text *ct)
 	text_poke_early(addr, bytes, CALL_INSN_SIZE);
 }
 
-static __init_or_module void
+static void
 patch_call_sites(s32 *start, s32 *end, const struct core_text *ct)
 {
 	s32 *s;
@@ -238,7 +238,7 @@ patch_call_sites(s32 *start, s32 *end, const struct core_text *ct)
 		patch_call((void *)s + *s, ct);
 }
 
-static __init_or_module void
+static void
 callthunks_setup(struct callthunk_sites *cs, const struct core_text *ct)
 {
 	prdbg("Patching call sites %s\n", ct->name);
@@ -246,7 +246,7 @@ callthunks_setup(struct callthunk_sites *cs, const struct core_text *ct)
 	prdbg("Patching call sites done%s\n", ct->name);
 }
 
-void __init callthunks_patch_builtin_calls(void)
+void callthunks_patch_builtin_calls(void)
 {
 	struct callthunk_sites cs = {
 		.call_start	= __call_sites,
-- 
2.34.1

[RFC PATCH 22/56] cpu: Move mitigation logic out of .init

[David Kaplan]

The functions and data related to the global mitigations= option may
need to be called at runtime to support dynamic re-patching.  Move them
out of the .init section.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 kernel/cpu.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/kernel/cpu.c b/kernel/cpu.c
index 910249f6217a..33289405af30 100644
--- a/kernel/cpu.c
+++ b/kernel/cpu.c
@@ -3180,7 +3180,7 @@ void __init boot_cpu_hotplug_init(void)
  * Guest-to-Host and Guest-to-Guest vectors are only needed if KVM support is
  * present.
  */
-static bool attack_vectors[NR_CPU_ATTACK_VECTORS] __ro_after_init = {
+static bool attack_vectors[NR_CPU_ATTACK_VECTORS] = {
 	[CPU_MITIGATE_USER_KERNEL] = true,
 	[CPU_MITIGATE_USER_USER] = true,
 	[CPU_MITIGATE_GUEST_HOST] = IS_ENABLED(CONFIG_KVM),
@@ -3221,8 +3221,8 @@ enum {
 	NR_VECTOR_PARAMS,
 };
 
-enum smt_mitigations smt_mitigations __ro_after_init = SMT_MITIGATIONS_AUTO;
-static enum cpu_mitigations cpu_mitigations __ro_after_init = CPU_MITIGATIONS_AUTO;
+enum smt_mitigations smt_mitigations = SMT_MITIGATIONS_AUTO;
+static enum cpu_mitigations cpu_mitigations = CPU_MITIGATIONS_AUTO;
 
 static const match_table_t global_mitigations = {
 	{ CPU_MITIGATIONS_AUTO_NOSMT,	"auto,nosmt"},
@@ -3239,7 +3239,7 @@ static const match_table_t vector_mitigations = {
 	{ NR_VECTOR_PARAMS,	NULL},
 };
 
-static int __init mitigations_parse_global_opt(char *arg)
+static int mitigations_parse_global_opt(char *arg)
 {
 	int i;
 
@@ -3255,7 +3255,7 @@ static int __init mitigations_parse_global_opt(char *arg)
 	return 0;
 }
 
-static int __init mitigations_parse_cmdline(char *arg)
+static int mitigations_parse_cmdline(char *arg)
 {
 	char *s, *p;
 	int len;
@@ -3320,7 +3320,7 @@ bool cpu_mitigations_auto_nosmt(void)
 }
 EXPORT_SYMBOL_GPL(cpu_mitigations_auto_nosmt);
 #else
-static int __init mitigations_parse_cmdline(char *arg)
+static int mitigations_parse_cmdline(char *arg)
 {
 	pr_crit("Kernel compiled without mitigations, ignoring 'mitigations'; system may still be vulnerable\n");
 	return 0;
-- 
2.34.1

[RFC PATCH 23/56] x86/vmlinux.lds: Move alternative sections

[David Kaplan]

Move the sections containing information relevant to alternative
replacements to the beginning of the .init section.  This information will
need to be kept around to support dynamic mitigations, and putting it at
the start of the section makes it easier to conditionally free it.

No functional change.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 arch/x86/kernel/vmlinux.lds.S | 98 +++++++++++++++++------------------
 1 file changed, 49 insertions(+), 49 deletions(-)

diff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S
index d7af4a64c211..e0db3f4c97df 100644
--- a/arch/x86/kernel/vmlinux.lds.S
+++ b/arch/x86/kernel/vmlinux.lds.S
@@ -211,6 +211,55 @@ SECTIONS
 		__init_begin = .; /* paired with __init_end */
 	}
 
+#ifdef CONFIG_MITIGATION_RETPOLINE
+	/*
+	 * List of instructions that call/jmp/jcc to retpoline thunks
+	 * __x86_indirect_thunk_*(). These instructions can be patched along
+	 * with alternatives, after which the section can be freed.
+	 */
+	. = ALIGN(8);
+	.retpoline_sites : AT(ADDR(.retpoline_sites) - LOAD_OFFSET) {
+		__retpoline_sites = .;
+		*(.retpoline_sites)
+		__retpoline_sites_end = .;
+	}
+
+	. = ALIGN(8);
+	.return_sites : AT(ADDR(.return_sites) - LOAD_OFFSET) {
+		__return_sites = .;
+		*(.return_sites)
+		__return_sites_end = .;
+	}
+
+	. = ALIGN(8);
+	.call_sites : AT(ADDR(.call_sites) - LOAD_OFFSET) {
+		__call_sites = .;
+		*(.call_sites)
+		__call_sites_end = .;
+	}
+#endif
+	/*
+	 * struct alt_inst entries. From the header (alternative.h):
+	 * "Alternative instructions for different CPU types or capabilities"
+	 * Think locking instructions on spinlocks.
+	 */
+	. = ALIGN(8);
+	.altinstructions : AT(ADDR(.altinstructions) - LOAD_OFFSET) {
+		__alt_instructions = .;
+		*(.altinstructions)
+		__alt_instructions_end = .;
+	}
+
+	/*
+	 * And here are the replacement instructions. The linker sticks
+	 * them as binary blobs. The .altinstructions has enough data to
+	 * get the address and the length of them to patch the kernel safely.
+	 */
+	.altinstr_replacement : AT(ADDR(.altinstr_replacement) - LOAD_OFFSET) {
+		*(.altinstr_replacement)
+	}
+
+
 	INIT_TEXT_SECTION(PAGE_SIZE)
 
 	/*
@@ -243,34 +292,6 @@ SECTIONS
 	}
 #endif
 
-#ifdef CONFIG_MITIGATION_RETPOLINE
-	/*
-	 * List of instructions that call/jmp/jcc to retpoline thunks
-	 * __x86_indirect_thunk_*(). These instructions can be patched along
-	 * with alternatives, after which the section can be freed.
-	 */
-	. = ALIGN(8);
-	.retpoline_sites : AT(ADDR(.retpoline_sites) - LOAD_OFFSET) {
-		__retpoline_sites = .;
-		*(.retpoline_sites)
-		__retpoline_sites_end = .;
-	}
-
-	. = ALIGN(8);
-	.return_sites : AT(ADDR(.return_sites) - LOAD_OFFSET) {
-		__return_sites = .;
-		*(.return_sites)
-		__return_sites_end = .;
-	}
-
-	. = ALIGN(8);
-	.call_sites : AT(ADDR(.call_sites) - LOAD_OFFSET) {
-		__call_sites = .;
-		*(.call_sites)
-		__call_sites_end = .;
-	}
-#endif
-
 #ifdef CONFIG_X86_KERNEL_IBT
 	. = ALIGN(8);
 	.ibt_endbr_seal : AT(ADDR(.ibt_endbr_seal) - LOAD_OFFSET) {
@@ -289,27 +310,6 @@ SECTIONS
 	}
 #endif
 
-	/*
-	 * struct alt_inst entries. From the header (alternative.h):
-	 * "Alternative instructions for different CPU types or capabilities"
-	 * Think locking instructions on spinlocks.
-	 */
-	. = ALIGN(8);
-	.altinstructions : AT(ADDR(.altinstructions) - LOAD_OFFSET) {
-		__alt_instructions = .;
-		*(.altinstructions)
-		__alt_instructions_end = .;
-	}
-
-	/*
-	 * And here are the replacement instructions. The linker sticks
-	 * them as binary blobs. The .altinstructions has enough data to
-	 * get the address and the length of them to patch the kernel safely.
-	 */
-	.altinstr_replacement : AT(ADDR(.altinstr_replacement) - LOAD_OFFSET) {
-		*(.altinstr_replacement)
-	}
-
 	. = ALIGN(8);
 	.apicdrivers : AT(ADDR(.apicdrivers) - LOAD_OFFSET) {
 		__apicdrivers = .;
-- 
2.34.1

[RFC PATCH 24/56] x86/vmlinux.lds: Move altinstr_aux conditionally

[David Kaplan]

The altinstr_aux section contains code that must remain when dynamic
mitigations are supported.  Move it into the normal .text section so it is
not free'd when init memory is free'd.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 arch/x86/kernel/vmlinux.lds.S | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S
index e0db3f4c97df..c4a0bb13d095 100644
--- a/arch/x86/kernel/vmlinux.lds.S
+++ b/arch/x86/kernel/vmlinux.lds.S
@@ -143,6 +143,9 @@ SECTIONS
 		 */
 		. = srso_alias_untrain_ret | (1 << 2) | (1 << 8) | (1 << 14) | (1 << 20);
 		*(.text..__x86.rethunk_safe)
+#endif
+#ifdef CONFIG_DYNAMIC_MITIGATIONS
+		*(.altinstr_aux)
 #endif
 		ALIGN_ENTRY_TEXT_END
 
@@ -262,6 +265,7 @@ SECTIONS
 
 	INIT_TEXT_SECTION(PAGE_SIZE)
 
+#ifndef CONFIG_DYNAMIC_MITIGATIONS
 	/*
 	 * Section for code used exclusively before alternatives are run. All
 	 * references to such code must be patched out by alternatives, normally
@@ -272,8 +276,9 @@ SECTIONS
 	.altinstr_aux : AT(ADDR(.altinstr_aux) - LOAD_OFFSET) {
 		*(.altinstr_aux)
 		. = ALIGN(PAGE_SIZE);
-		__inittext_end = .;
 	}
+#endif
+	__inittext_end = .;
 
 	INIT_DATA_SECTION(16)
 
-- 
2.34.1

[RFC PATCH 25/56] x86/vmlinux.lds: Define __init_alt_end

[David Kaplan]

With the alternative related information at the beginning of the .init
section, __init_alt_end defines the end of the alternative sections.  The
memory from __init_begin to __init_alt_end must stick around to support
re-patching.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 arch/x86/kernel/vmlinux.lds.S | 5 +++++
 arch/x86/tools/relocs.c       | 1 +
 2 files changed, 6 insertions(+)

diff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S
index c4a0bb13d095..e02044da9971 100644
--- a/arch/x86/kernel/vmlinux.lds.S
+++ b/arch/x86/kernel/vmlinux.lds.S
@@ -262,6 +262,11 @@ SECTIONS
 		*(.altinstr_replacement)
 	}
 
+	/* Everything from here until __init_end is always freed */
+	. = ALIGN(PAGE_SIZE);
+	.init.alt.end : AT(ADDR(.init.alt.end) - LOAD_OFFSET) {
+		__init_alt_end = .;
+	}
 
 	INIT_TEXT_SECTION(PAGE_SIZE)
 
diff --git a/arch/x86/tools/relocs.c b/arch/x86/tools/relocs.c
index e5a2b9a912d1..2054ffbf5fd4 100644
--- a/arch/x86/tools/relocs.c
+++ b/arch/x86/tools/relocs.c
@@ -71,6 +71,7 @@ static const char * const	sym_regex_kernel[S_NSYMTYPES] = {
  */
 	[S_REL] =
 	"^(__init_(begin|end)|"
+	"__init_alt_end|"
 	"__x86_cpu_dev_(start|end)|"
 	"__alt_instructions(_end)?|"
 	"(__iommu_table|__apicdrivers|__smp_locks)(_end)?|"
-- 
2.34.1

[RFC PATCH 26/56] module: Save module ELF info

[David Kaplan]

Re-patching of module alternatives at runtime requires saving the ELF
information about where those alternatives (and returns, retpolines, etc.)
exist.  This information is already saved for livepatch modules, so save it
as well if we may be doing dynamic mitigation reconfiguration.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 kernel/module/main.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kernel/module/main.c b/kernel/module/main.c
index c66b26184936..088f9399af11 100644
--- a/kernel/module/main.c
+++ b/kernel/module/main.c
@@ -3491,7 +3491,7 @@ static int load_module(struct load_info *info, const char __user *uargs,
 	if (err < 0)
 		goto coming_cleanup;
 
-	if (is_livepatch_module(mod)) {
+	if (IS_ENABLED(CONFIG_DYNAMIC_MITIGATIONS) || is_livepatch_module(mod)) {
 		err = copy_module_elf(mod, info);
 		if (err < 0)
 			goto sysfs_cleanup;
-- 
2.34.1

[RFC PATCH 27/56] x86/mm: Conditionally free alternative sections

[David Kaplan]

Keep the alternative related sections, located at the beginning of the
.init section, around if needed.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 arch/x86/mm/init.c        | 12 ++++++++++--
 arch/x86/mm/mm_internal.h |  2 ++
 2 files changed, 12 insertions(+), 2 deletions(-)

diff --git a/arch/x86/mm/init.c b/arch/x86/mm/init.c
index 8bf6ad4b9400..8dfde4889a09 100644
--- a/arch/x86/mm/init.c
+++ b/arch/x86/mm/init.c
@@ -977,8 +977,16 @@ void __ref free_initmem(void)
 
 	mem_encrypt_free_decrypted_mem();
 
-	free_kernel_image_pages("unused kernel image (initmem)",
-				&__init_begin, &__init_end);
+	/*
+	 * __init_alt_end is after the alternative sections in case we need to
+	 * keep that around to support runtime patching.
+	 */
+	if (IS_ENABLED(CONFIG_DYNAMIC_MITIGATIONS))
+		free_kernel_image_pages("unused kernel image (initmem)",
+					&__init_alt_end, &__init_end);
+	else
+		free_kernel_image_pages("unused kernel image (initmem)",
+					&__init_begin, &__init_end);
 }
 
 #ifdef CONFIG_BLK_DEV_INITRD
diff --git a/arch/x86/mm/mm_internal.h b/arch/x86/mm/mm_internal.h
index 097aadc250f7..e961f2257009 100644
--- a/arch/x86/mm/mm_internal.h
+++ b/arch/x86/mm/mm_internal.h
@@ -29,4 +29,6 @@ extern unsigned long tlb_single_page_flush_ceiling;
 void __init x86_numa_init(void);
 #endif
 
+extern void *__init_alt_end;
+
 #endif	/* __X86_MM_INTERNAL_H */
-- 
2.34.1

[RFC PATCH 28/56] stop_machine: Add stop_machine_nmi()

[David Kaplan]

stop_machine_nmi() is a flavor of stop_machine() that runs the specified
function in NMI context.  This is useful for flows that cannot tolerate any
risk of interruption even due to an NMI.  Arch-specific code must handle
sending the actual NMI and running the stop_machine_nmi_handler().

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 include/linux/stop_machine.h | 32 +++++++++++++++
 kernel/stop_machine.c        | 79 ++++++++++++++++++++++++++++++++++--
 2 files changed, 107 insertions(+), 4 deletions(-)

diff --git a/include/linux/stop_machine.h b/include/linux/stop_machine.h
index 72820503514c..452c45640012 100644
--- a/include/linux/stop_machine.h
+++ b/include/linux/stop_machine.h
@@ -141,6 +141,29 @@ int stop_machine(cpu_stop_fn_t fn, void *data, const struct cpumask *cpus);
  */
 int stop_machine_cpuslocked(cpu_stop_fn_t fn, void *data, const struct cpumask *cpus);
 
+/**
+ * stop_machine_nmi: freeze the machine and run this function in NMI context
+ * @fn: the function to run
+ * @data: the data ptr for the @fn()
+ * @cpus: the cpus to run the @fn() on (NULL = any online cpu)
+ *
+ * Like stop_machine() but runs the function in NMI context to avoid any risk of
+ * interruption due to NMIs.
+ *
+ * Protects against CPU hotplug.
+ */
+int stop_machine_nmi(cpu_stop_fn_t fn, void *data, const struct cpumask *cpus);
+
+/**
+ * stop_machine_cpuslocked_nmi: freeze and run this function in NMI context
+ * @fn: the function to run
+ * @data: the data ptr for the @fn()
+ * @cpus: the cpus to run the @fn() on (NULL = any online cpu)
+ *
+ * Same as above. Must be called from within a cpus_read_lock() protected
+ * region. Avoids nested calls to cpus_read_lock().
+ */
+int stop_machine_cpuslocked_nmi(cpu_stop_fn_t fn, void *data, const struct cpumask *cpus);
 /**
  * stop_core_cpuslocked: - stop all threads on just one core
  * @cpu: any cpu in the targeted core
@@ -160,6 +183,15 @@ int stop_core_cpuslocked(unsigned int cpu, cpu_stop_fn_t fn, void *data);
 
 int stop_machine_from_inactive_cpu(cpu_stop_fn_t fn, void *data,
 				   const struct cpumask *cpus);
+
+bool noinstr stop_machine_nmi_handler(void);
+void arch_send_self_nmi(void);
+DECLARE_STATIC_KEY_FALSE(stop_machine_nmi_handler_enable);
+static __always_inline bool stop_machine_nmi_handler_enabled(void)
+{
+	return static_branch_unlikely(&stop_machine_nmi_handler_enable);
+}
+
 #else	/* CONFIG_SMP || CONFIG_HOTPLUG_CPU */
 
 static __always_inline int stop_machine_cpuslocked(cpu_stop_fn_t fn, void *data,
diff --git a/kernel/stop_machine.c b/kernel/stop_machine.c
index 3fe6b0c99f3d..d135f32528e8 100644
--- a/kernel/stop_machine.c
+++ b/kernel/stop_machine.c
@@ -174,6 +174,8 @@ struct multi_stop_data {
 
 	enum multi_stop_state	state;
 	atomic_t		thread_ack;
+
+	bool			use_nmi;
 };
 
 static void set_state(struct multi_stop_data *msdata,
@@ -197,6 +199,42 @@ notrace void __weak stop_machine_yield(const struct cpumask *cpumask)
 	cpu_relax();
 }
 
+struct stop_machine_nmi_ctrl {
+	bool nmi_enabled;
+	struct multi_stop_data *msdata;
+	int err;
+};
+
+DEFINE_STATIC_KEY_FALSE(stop_machine_nmi_handler_enable);
+static DEFINE_PER_CPU(struct stop_machine_nmi_ctrl, stop_machine_nmi_ctrl);
+
+static void enable_nmi_handler(struct multi_stop_data *msdata)
+{
+	this_cpu_write(stop_machine_nmi_ctrl.msdata, msdata);
+	this_cpu_write(stop_machine_nmi_ctrl.nmi_enabled, true);
+}
+
+void __weak arch_send_self_nmi(void)
+{
+	/* Arch code must implement this to support stop_machine_nmi() */
+}
+
+bool noinstr stop_machine_nmi_handler(void)
+{
+	struct multi_stop_data *msdata;
+	int err;
+
+	if (!raw_cpu_read(stop_machine_nmi_ctrl.nmi_enabled))
+		return false;
+
+	raw_cpu_write(stop_machine_nmi_ctrl.nmi_enabled, false);
+
+	msdata = raw_cpu_read(stop_machine_nmi_ctrl.msdata);
+	err = msdata->fn(msdata->data);
+	raw_cpu_write(stop_machine_nmi_ctrl.err, err);
+	return true;
+}
+
 /* This is the cpu_stop function which stops the CPU. */
 static int multi_cpu_stop(void *data)
 {
@@ -234,8 +272,15 @@ static int multi_cpu_stop(void *data)
 				hard_irq_disable();
 				break;
 			case MULTI_STOP_RUN:
-				if (is_active)
-					err = msdata->fn(msdata->data);
+				if (is_active) {
+					if (msdata->use_nmi) {
+						enable_nmi_handler(msdata);
+						arch_send_self_nmi();
+						err = raw_cpu_read(stop_machine_nmi_ctrl.err);
+					} else {
+						err = msdata->fn(msdata->data);
+					}
+				}
 				break;
 			default:
 				break;
@@ -584,14 +629,15 @@ static int __init cpu_stop_init(void)
 }
 early_initcall(cpu_stop_init);
 
-int stop_machine_cpuslocked(cpu_stop_fn_t fn, void *data,
-			    const struct cpumask *cpus)
+static int __stop_machine_cpuslocked(cpu_stop_fn_t fn, void *data,
+			    const struct cpumask *cpus, bool use_nmi)
 {
 	struct multi_stop_data msdata = {
 		.fn = fn,
 		.data = data,
 		.num_threads = num_online_cpus(),
 		.active_cpus = cpus,
+		.use_nmi = use_nmi,
 	};
 
 	lockdep_assert_cpus_held();
@@ -620,6 +666,18 @@ int stop_machine_cpuslocked(cpu_stop_fn_t fn, void *data,
 	return stop_cpus(cpu_online_mask, multi_cpu_stop, &msdata);
 }
 
+int stop_machine_cpuslocked(cpu_stop_fn_t fn, void *data,
+			    const struct cpumask *cpus)
+{
+	return __stop_machine_cpuslocked(fn, data, cpus, false);
+}
+
+int stop_machine_cpuslocked_nmi(cpu_stop_fn_t fn, void *data,
+				const struct cpumask *cpus)
+{
+	return __stop_machine_cpuslocked(fn, data, cpus, true);
+}
+
 int stop_machine(cpu_stop_fn_t fn, void *data, const struct cpumask *cpus)
 {
 	int ret;
@@ -632,6 +690,19 @@ int stop_machine(cpu_stop_fn_t fn, void *data, const struct cpumask *cpus)
 }
 EXPORT_SYMBOL_GPL(stop_machine);
 
+int stop_machine_nmi(cpu_stop_fn_t fn, void *data, const struct cpumask *cpus)
+{
+	int ret;
+
+	cpus_read_lock();
+	static_branch_enable_cpuslocked(&stop_machine_nmi_handler_enable);
+	ret = stop_machine_cpuslocked_nmi(fn, data, cpus);
+	static_branch_disable_cpuslocked(&stop_machine_nmi_handler_enable);
+	cpus_read_unlock();
+	return ret;
+}
+EXPORT_SYMBOL_GPL(stop_machine_nmi);
+
 #ifdef CONFIG_SCHED_SMT
 int stop_core_cpuslocked(unsigned int cpu, cpu_stop_fn_t fn, void *data)
 {
-- 
2.34.1

[RFC PATCH 29/56] x86/apic: Add self-NMI support

[David Kaplan]

Add function to send an NMI-to-self which is needed for stop_machine_nmi().
Note that send_IPI_self() cannot be used to send an NMI so send_IPI() is
used.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 arch/x86/kernel/apic/ipi.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/arch/x86/kernel/apic/ipi.c b/arch/x86/kernel/apic/ipi.c
index 98a57cb4aa86..6d4e5aa27529 100644
--- a/arch/x86/kernel/apic/ipi.c
+++ b/arch/x86/kernel/apic/ipi.c
@@ -4,6 +4,7 @@
 #include <linux/delay.h>
 #include <linux/smp.h>
 #include <linux/string_choices.h>
+#include <linux/stop_machine.h>
 
 #include <asm/io_apic.h>
 
@@ -248,6 +249,12 @@ void default_send_IPI_self(int vector)
 	__default_send_IPI_shortcut(APIC_DEST_SELF, vector);
 }
 
+/* Self-NMI is used in stop_machine_nmi() */
+void arch_send_self_nmi(void)
+{
+	apic->send_IPI(smp_processor_id(), NMI_VECTOR);
+}
+
 #ifdef CONFIG_X86_32
 void default_send_IPI_mask_sequence_logical(const struct cpumask *mask, int vector)
 {
-- 
2.34.1

[RFC PATCH 30/56] x86/nmi: Add support for stop_machine_nmi()

[David Kaplan]

Call the stop_machine_nmi_handler() if the appropriate static branch is
enabled in order to support stop_machine_nmi().

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 arch/x86/kernel/nmi.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/arch/x86/kernel/nmi.c b/arch/x86/kernel/nmi.c
index be93ec7255bf..c0d8e80029a0 100644
--- a/arch/x86/kernel/nmi.c
+++ b/arch/x86/kernel/nmi.c
@@ -24,6 +24,7 @@
 #include <linux/export.h>
 #include <linux/atomic.h>
 #include <linux/sched/clock.h>
+#include <linux/stop_machine.h>
 
 #include <asm/cpu_entry_area.h>
 #include <asm/traps.h>
@@ -381,6 +382,9 @@ static noinstr void default_do_nmi(struct pt_regs *regs)
 
 	instrumentation_begin();
 
+	if (stop_machine_nmi_handler_enabled() && stop_machine_nmi_handler())
+		goto out;
+
 	if (microcode_nmi_handler_enabled() && microcode_nmi_handler())
 		goto out;
 
-- 
2.34.1

[RFC PATCH 31/56] x86/alternative: Prepend nops with retpolines

[David Kaplan]

When patching retpolines, nops may be required for padding such as when
turning a 5-byte direct call into a 2-byte indirect call.  Previously,
these were appended at the end so the code becomes "call *reg;nop;nop;nop"
for example.  This was fine because it's always going from a larger
instruction to a smaller one.

But this is a problem if the sequence is transformed from a 2-byte indirect
to the 5-byte direct call version at runtime because when the called
function returns, it will be in the middle of the 5-byte call instruction.

To fix this, prepend the nops instead of appending them.  Consequently, the
return site of the called function is always the same.

For indirect jmps this is potentially slightly less efficient compared to
appending nops, but indirect jmps are so rare this hardly seems worth
optimizing.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 arch/x86/kernel/alternative.c | 27 ++++++++++++++++++++++++---
 1 file changed, 24 insertions(+), 3 deletions(-)

diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c
index 8ee5ff547357..7a1f17078581 100644
--- a/arch/x86/kernel/alternative.c
+++ b/arch/x86/kernel/alternative.c
@@ -854,6 +854,21 @@ static bool cpu_wants_indirect_its_thunk_at(unsigned long addr, int reg)
 
 #endif /* CONFIG_MITIGATION_ITS */
 
+static void prepend_nops(u8 *bytes, int curlen, int neededlen)
+{
+	u8 newbytes[16];
+	int pad = neededlen - curlen;
+
+	/* Fill padding bytes with NOP. */
+	memset(newbytes, BYTES_NOP1, pad);
+
+	/* Copy the new instruction in. */
+	memcpy(newbytes + pad, bytes, curlen);
+
+	/* And write the final result back out to bytes. */
+	memcpy(bytes, newbytes, neededlen);
+}
+
 /*
  * Rewrite the compiler generated retpoline thunk calls.
  *
@@ -942,10 +957,16 @@ static int patch_retpoline(void *addr, struct insn *insn, u8 *bytes)
 		return ret;
 	i += ret;
 
-	for (; i < insn->length;)
-		bytes[i++] = BYTES_NOP1;
+	/*
+	 * Prepend the instruction with NOPs.  These are prepended, instead of
+	 * appended so the return site does not change.  This is necessary when
+	 * re-patching retpolines at runtime, such as via
+	 * CONFIG_DYNAMIC_MITIGATIONS, but do it always since the performance is
+	 * the same either way (other than for JMP, but those are very rare).
+	 */
+	prepend_nops(bytes, i, insn->length);
 
-	return i;
+	return insn->length;
 }
 
 /*
-- 
2.34.1

[RFC PATCH 32/56] x86/alternative: Add module param

[David Kaplan]

Add a struct module* param to the apply_alternatives(), apply_returns(),
and apply_retpolines() functions.  This will later be used to help store
the old instruction bytes to support re-patching.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 arch/x86/entry/vdso/vma.c          |  2 +-
 arch/x86/include/asm/alternative.h | 11 ++++++-----
 arch/x86/kernel/alternative.c      | 19 ++++++++++---------
 arch/x86/kernel/module.c           |  6 +++---
 4 files changed, 20 insertions(+), 18 deletions(-)

diff --git a/arch/x86/entry/vdso/vma.c b/arch/x86/entry/vdso/vma.c
index afe105b2f907..2451a8007854 100644
--- a/arch/x86/entry/vdso/vma.c
+++ b/arch/x86/entry/vdso/vma.c
@@ -42,7 +42,7 @@ int __init init_vdso_image(const struct vdso_image *image)
 
 	apply_alternatives((struct alt_instr *)(image->data + image->alt),
 			   (struct alt_instr *)(image->data + image->alt +
-						image->alt_len));
+						image->alt_len), NULL);
 
 	return 0;
 }
diff --git a/arch/x86/include/asm/alternative.h b/arch/x86/include/asm/alternative.h
index 15bc07a5ebb3..73fd8ebdf8d9 100644
--- a/arch/x86/include/asm/alternative.h
+++ b/arch/x86/include/asm/alternative.h
@@ -95,16 +95,17 @@ extern s32 __smp_locks[],	__smp_locks_end[];
  */
 extern int alternatives_patched;
 
+struct module;
+
 extern void alternative_instructions(void);
-extern void apply_alternatives(struct alt_instr *start, struct alt_instr *end);
-extern void apply_retpolines(s32 *start, s32 *end);
-extern void apply_returns(s32 *start, s32 *end);
+extern void apply_alternatives(struct alt_instr *start, struct alt_instr *end,
+		struct module *mod);
+extern void apply_retpolines(s32 *start, s32 *end, struct module *mod);
+extern void apply_returns(s32 *start, s32 *end, struct module *mod);
 extern void apply_seal_endbr(s32 *start, s32 *end);
 extern void apply_fineibt(s32 *start_retpoline, s32 *end_retpoine,
 			  s32 *start_cfi, s32 *end_cfi);
 
-struct module;
-
 struct callthunk_sites {
 	s32				*call_start, *call_end;
 };
diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c
index 7a1f17078581..9fbec219e98e 100644
--- a/arch/x86/kernel/alternative.c
+++ b/arch/x86/kernel/alternative.c
@@ -615,7 +615,8 @@ static inline u8 * instr_va(struct alt_instr *i)
  * to refetch changed I$ lines.
  */
 void __init_or_module noinline apply_alternatives(struct alt_instr *start,
-						  struct alt_instr *end)
+						  struct alt_instr *end,
+						  struct module *mod)
 {
 	u8 insn_buff[MAX_PATCH_LEN];
 	u8 *instr, *replacement;
@@ -972,7 +973,7 @@ static int patch_retpoline(void *addr, struct insn *insn, u8 *bytes)
 /*
  * Generated by 'objtool --retpoline'.
  */
-void __init_or_module noinline apply_retpolines(s32 *start, s32 *end)
+void __init_or_module noinline apply_retpolines(s32 *start, s32 *end, struct module *mod)
 {
 	s32 *s;
 
@@ -1076,7 +1077,7 @@ static int patch_return(void *addr, struct insn *insn, u8 *bytes)
 	return i;
 }
 
-void __init_or_module noinline apply_returns(s32 *start, s32 *end)
+void __init_or_module noinline apply_returns(s32 *start, s32 *end, struct module *mod)
 {
 	s32 *s;
 
@@ -1117,13 +1118,13 @@ void __init_or_module noinline apply_returns(s32 *start, s32 *end)
 	}
 }
 #else /* !CONFIG_MITIGATION_RETHUNK: */
-void __init_or_module noinline apply_returns(s32 *start, s32 *end) { }
+void __init_or_module noinline apply_returns(s32 *start, s32 *end, struct module *mod) { }
 #endif /* !CONFIG_MITIGATION_RETHUNK */
 
 #else /* !CONFIG_MITIGATION_RETPOLINE || !CONFIG_OBJTOOL */
 
-void __init_or_module noinline apply_retpolines(s32 *start, s32 *end) { }
-void __init_or_module noinline apply_returns(s32 *start, s32 *end) { }
+void __init_or_module noinline apply_retpolines(s32 *start, s32 *end, struct module *mod) { }
+void __init_or_module noinline apply_returns(s32 *start, s32 *end, struct module *mod) { }
 
 #endif /* !CONFIG_MITIGATION_RETPOLINE || !CONFIG_OBJTOOL */
 
@@ -2407,8 +2408,8 @@ void __init alternative_instructions(void)
 	 * Rewrite the retpolines, must be done before alternatives since
 	 * those can rewrite the retpoline thunks.
 	 */
-	apply_retpolines(__retpoline_sites, __retpoline_sites_end);
-	apply_returns(__return_sites, __return_sites_end);
+	apply_retpolines(__retpoline_sites, __retpoline_sites_end, NULL);
+	apply_returns(__return_sites, __return_sites_end, NULL);
 
 	its_fini_core();
 
@@ -2418,7 +2419,7 @@ void __init alternative_instructions(void)
 	 */
 	callthunks_patch_builtin_calls();
 
-	apply_alternatives(__alt_instructions, __alt_instructions_end);
+	apply_alternatives(__alt_instructions, __alt_instructions_end, NULL);
 
 	/*
 	 * Seal all functions that do not have their address taken.
diff --git a/arch/x86/kernel/module.c b/arch/x86/kernel/module.c
index 0ffbae902e2f..0765d2360a33 100644
--- a/arch/x86/kernel/module.c
+++ b/arch/x86/kernel/module.c
@@ -286,14 +286,14 @@ int module_finalize(const Elf_Ehdr *hdr,
 	}
 	if (retpolines) {
 		void *rseg = (void *)retpolines->sh_addr;
-		apply_retpolines(rseg, rseg + retpolines->sh_size);
+		apply_retpolines(rseg, rseg + retpolines->sh_size, me);
 	}
 
 	its_fini_mod(me);
 
 	if (returns) {
 		void *rseg = (void *)returns->sh_addr;
-		apply_returns(rseg, rseg + returns->sh_size);
+		apply_returns(rseg, rseg + returns->sh_size, me);
 	}
 	if (calls) {
 		struct callthunk_sites cs = {};
@@ -306,7 +306,7 @@ int module_finalize(const Elf_Ehdr *hdr,
 	if (alt) {
 		/* patch .altinstructions */
 		void *aseg = (void *)alt->sh_addr;
-		apply_alternatives(aseg, aseg + alt->sh_size);
+		apply_alternatives(aseg, aseg + alt->sh_size, me);
 	}
 	if (ibt_endbr) {
 		void *iseg = (void *)ibt_endbr->sh_addr;
-- 
2.34.1

[RFC PATCH 33/56] x86/alternative: Avoid re-patching init code

[David Kaplan]

During boot, all sites should be patched but during re-patch, ignore any
sites that are in the init section of the kernel or a module.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 arch/x86/kernel/alternative.c | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)

diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c
index 9fbec219e98e..a821ea37fc29 100644
--- a/arch/x86/kernel/alternative.c
+++ b/arch/x86/kernel/alternative.c
@@ -298,6 +298,27 @@ u8 *its_static_thunk(int reg)
 static inline void its_fini_core(void) {}
 #endif /* CONFIG_MITIGATION_ITS */
 
+static bool __maybe_unused repatch_in_progress;
+
+#ifdef CONFIG_DYNAMIC_MITIGATIONS
+/* Do not patch __init text addresses when repatching */
+static bool should_patch(void *addr, struct module *mod)
+{
+	if (!repatch_in_progress)
+		return true;
+
+	if (is_kernel_text((u64) addr))
+		return true;
+
+	return mod && within_module_core((u64) addr, mod);
+}
+#else
+static bool should_patch(void *addr, struct module *mod)
+{
+	return true;
+}
+#endif
+
 /*
  * Nomenclature for variable names to simplify and clarify this code and ease
  * any potential staring at it:
@@ -658,6 +679,10 @@ void __init_or_module noinline apply_alternatives(struct alt_instr *start,
 		}
 
 		instr = instr_va(a);
+
+		if (!should_patch(instr, mod))
+			continue;
+
 		replacement = (u8 *)&a->repl_offset + a->repl_offset;
 		BUG_ON(a->instrlen > sizeof(insn_buff));
 		BUG_ON(a->cpuid >= (NCAPINTS + NBUGINTS) * 32);
@@ -986,6 +1011,10 @@ void __init_or_module noinline apply_retpolines(s32 *start, s32 *end, struct mod
 		u8 *dest;
 
 		ret = insn_decode_kernel(&insn, addr);
+
+		if (!should_patch(addr, mod))
+			continue;
+
 		if (WARN_ON_ONCE(ret < 0))
 			continue;
 
@@ -1092,6 +1121,10 @@ void __init_or_module noinline apply_returns(s32 *start, s32 *end, struct module
 		u8 op;
 
 		ret = insn_decode_kernel(&insn, addr);
+
+		if (!should_patch(addr, mod))
+			continue;
+
 		if (WARN_ON_ONCE(ret < 0))
 			continue;
 
-- 
2.34.1

[RFC PATCH 34/56] x86/alternative: Save old bytes for alternatives

[David Kaplan]

Save the existing instruction bytes at each alternative site when patching.
This is only done the first time, and these will be used later to help
restore the code back to its original form.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 arch/x86/include/asm/alternative.h |  5 ++++
 arch/x86/include/asm/module.h      |  3 +++
 arch/x86/kernel/alternative.c      | 37 +++++++++++++++++++++++++++++-
 3 files changed, 44 insertions(+), 1 deletion(-)

diff --git a/arch/x86/include/asm/alternative.h b/arch/x86/include/asm/alternative.h
index 73fd8ebdf8d9..3ee781d61927 100644
--- a/arch/x86/include/asm/alternative.h
+++ b/arch/x86/include/asm/alternative.h
@@ -97,6 +97,11 @@ extern int alternatives_patched;
 
 struct module;
 
+struct alt_site {
+	u8 *pbytes;
+	u8 len;
+};
+
 extern void alternative_instructions(void);
 extern void apply_alternatives(struct alt_instr *start, struct alt_instr *end,
 		struct module *mod);
diff --git a/arch/x86/include/asm/module.h b/arch/x86/include/asm/module.h
index 3c2de4ce3b10..2bb602f99154 100644
--- a/arch/x86/include/asm/module.h
+++ b/arch/x86/include/asm/module.h
@@ -19,6 +19,9 @@ struct mod_arch_specific {
 	struct orc_entry *orc_unwind;
 #endif
 	struct its_array its_pages;
+#ifdef CONFIG_DYNAMIC_MITIGATIONS
+	struct alt_site *alt_sites;
+#endif
 };
 
 #endif /* _ASM_X86_MODULE_H */
diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c
index a821ea37fc29..8037076e9301 100644
--- a/arch/x86/kernel/alternative.c
+++ b/arch/x86/kernel/alternative.c
@@ -301,6 +301,8 @@ static inline void its_fini_core(void) {}
 static bool __maybe_unused repatch_in_progress;
 
 #ifdef CONFIG_DYNAMIC_MITIGATIONS
+static struct alt_site *alt_sites;
+
 /* Do not patch __init text addresses when repatching */
 static bool should_patch(void *addr, struct module *mod)
 {
@@ -642,6 +644,27 @@ void __init_or_module noinline apply_alternatives(struct alt_instr *start,
 	u8 insn_buff[MAX_PATCH_LEN];
 	u8 *instr, *replacement;
 	struct alt_instr *a, *b;
+	u32 idx = 0;
+	struct alt_site *save_site = NULL;
+
+#ifdef CONFIG_DYNAMIC_MITIGATIONS
+	u32 size = ((u64)end - (u64)start)/sizeof(struct alt_instr);
+
+	/* Main kernel text alternatives */
+	if (!mod && !alt_sites) {
+		alt_sites =  kcalloc(size, sizeof(struct alt_site), GFP_KERNEL);
+		if (WARN_ON(!alt_sites))
+			return;
+
+		save_site = alt_sites;
+	} else if (mod && !mod->arch.alt_sites) {
+		mod->arch.alt_sites =  kcalloc(size, sizeof(struct alt_site), GFP_KERNEL);
+		if (WARN_ON(!mod->arch.alt_sites))
+			return;
+
+		save_site = mod->arch.alt_sites;
+	}
+#endif
 
 	DPRINTK(ALT, "alt table %px, -> %px", start, end);
 
@@ -664,7 +687,7 @@ void __init_or_module noinline apply_alternatives(struct alt_instr *start,
 	 * So be careful if you want to change the scan order to any other
 	 * order.
 	 */
-	for (a = start; a < end; a++) {
+	for (a = start; a < end; a++, idx++) {
 		int insn_buff_sz = 0;
 
 		/*
@@ -687,6 +710,18 @@ void __init_or_module noinline apply_alternatives(struct alt_instr *start,
 		BUG_ON(a->instrlen > sizeof(insn_buff));
 		BUG_ON(a->cpuid >= (NCAPINTS + NBUGINTS) * 32);
 
+		if (IS_ENABLED(CONFIG_DYNAMIC_MITIGATIONS) && save_site) {
+			/* Only save the original bytes for each location */
+			if (a == start || (instr_va(a) != instr_va(a-1))) {
+				save_site[idx].len = a->instrlen;
+				save_site[idx].pbytes = kmalloc(a->instrlen, GFP_KERNEL);
+				if (WARN_ON(!save_site[idx].pbytes))
+					return;
+
+				memcpy(save_site[idx].pbytes, instr, a->instrlen);
+			}
+		}
+
 		/*
 		 * Patch if either:
 		 * - feature is present
-- 
2.34.1

[RFC PATCH 35/56] x86/alternative: Save old bytes for retpolines

[David Kaplan]

Similar to alternatives, save the original bytes when patching retpolines
the first time.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 arch/x86/include/asm/alternative.h |  5 ++++
 arch/x86/include/asm/module.h      |  2 ++
 arch/x86/kernel/alternative.c      | 37 +++++++++++++++++++++++++++++-
 3 files changed, 43 insertions(+), 1 deletion(-)

diff --git a/arch/x86/include/asm/alternative.h b/arch/x86/include/asm/alternative.h
index 3ee781d61927..24a4afbf163b 100644
--- a/arch/x86/include/asm/alternative.h
+++ b/arch/x86/include/asm/alternative.h
@@ -102,6 +102,11 @@ struct alt_site {
 	u8 len;
 };
 
+struct retpoline_site {
+	u8 bytes[6];
+	u8 len;
+} __packed;
+
 extern void alternative_instructions(void);
 extern void apply_alternatives(struct alt_instr *start, struct alt_instr *end,
 		struct module *mod);
diff --git a/arch/x86/include/asm/module.h b/arch/x86/include/asm/module.h
index 2bb602f99154..d0c39b921408 100644
--- a/arch/x86/include/asm/module.h
+++ b/arch/x86/include/asm/module.h
@@ -21,6 +21,8 @@ struct mod_arch_specific {
 	struct its_array its_pages;
 #ifdef CONFIG_DYNAMIC_MITIGATIONS
 	struct alt_site *alt_sites;
+	struct retpoline_site *retpoline_sites;
+	int num_retpoline_sites;
 #endif
 };
 
diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c
index 8037076e9301..a02dc6bfb696 100644
--- a/arch/x86/kernel/alternative.c
+++ b/arch/x86/kernel/alternative.c
@@ -302,6 +302,8 @@ static bool __maybe_unused repatch_in_progress;
 
 #ifdef CONFIG_DYNAMIC_MITIGATIONS
 static struct alt_site *alt_sites;
+static struct retpoline_site *retpoline_sites;
+static int num_retpoline_sites;
 
 /* Do not patch __init text addresses when repatching */
 static bool should_patch(void *addr, struct module *mod)
@@ -1036,8 +1038,36 @@ static int patch_retpoline(void *addr, struct insn *insn, u8 *bytes)
 void __init_or_module noinline apply_retpolines(s32 *start, s32 *end, struct module *mod)
 {
 	s32 *s;
+	u32 idx = 0;
+	struct retpoline_site *save_site = NULL;
 
-	for (s = start; s < end; s++) {
+#ifdef CONFIG_DYNAMIC_MITIGATIONS
+	u32 size = ((u64)end - (u64)start)/4;
+
+	/* ITS code needs the save_site pointer even on re-patch. */
+	if (!mod) {
+		if (!retpoline_sites) {
+			retpoline_sites = kcalloc(size,	sizeof(struct retpoline_site), GFP_KERNEL);
+			if (WARN_ON(!retpoline_sites))
+				return;
+		}
+
+		save_site = retpoline_sites;
+		num_retpoline_sites = size;
+	} else {
+		if (!mod->arch.retpoline_sites) {
+			mod->arch.retpoline_sites = kcalloc(size, sizeof(struct retpoline_site),
+							    GFP_KERNEL);
+			if (WARN_ON(!mod->arch.retpoline_sites))
+				return;
+		}
+
+		save_site = mod->arch.retpoline_sites;
+		mod->arch.num_retpoline_sites = size;
+	}
+#endif
+
+	for (s = start; s < end; s++, idx++) {
 		void *addr = (void *)s + *s;
 		struct insn insn;
 		int len, ret;
@@ -1085,6 +1115,11 @@ void __init_or_module noinline apply_retpolines(s32 *start, s32 *end, struct mod
 			addr, addr, insn.length,
 			addr + insn.length + insn.immediate.value);
 
+		if (IS_ENABLED(CONFIG_DYNAMIC_MITIGATIONS) && save_site) {
+			save_site[idx].len = insn.length;
+			memcpy(save_site[idx].bytes, addr, insn.length);
+		}
+
 		len = patch_retpoline(addr, &insn, bytes);
 		if (len == insn.length) {
 			optimize_nops(addr, bytes, len);
-- 
2.34.1

[RFC PATCH 36/56] x86/alternative: Do not recompute len on re-patch

[David Kaplan]

Multiple alternatives may be defined at the same va, and the logic will
compute the maximum size of any alternative and update all of them with
the new max size.  This only needs to be done once, and must be skipped
on re-patch because the memory containing the alternative information is
read-only at that point.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 arch/x86/kernel/alternative.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c
index a02dc6bfb696..17b93763d1be 100644
--- a/arch/x86/kernel/alternative.c
+++ b/arch/x86/kernel/alternative.c
@@ -298,7 +298,7 @@ u8 *its_static_thunk(int reg)
 static inline void its_fini_core(void) {}
 #endif /* CONFIG_MITIGATION_ITS */
 
-static bool __maybe_unused repatch_in_progress;
+static bool repatch_in_progress;
 
 #ifdef CONFIG_DYNAMIC_MITIGATIONS
 static struct alt_site *alt_sites;
@@ -697,8 +697,12 @@ void __init_or_module noinline apply_alternatives(struct alt_instr *start,
 		 * add more padding. To ensure consistent patching find the max
 		 * padding for all alt_instr entries for this site (nested
 		 * alternatives result in consecutive entries).
+		 *
+		 * Ignore this on repatching because this has already been done
+		 * and because the alt_instr may be in read-only memory.
 		 */
-		for (b = a+1; b < end && instr_va(b) == instr_va(a); b++) {
+		for (b = a+1; b < end && instr_va(b) == instr_va(a) &&
+				!repatch_in_progress; b++) {
 			u8 len = max(a->instrlen, b->instrlen);
 			a->instrlen = b->instrlen = len;
 		}
-- 
2.34.1

[RFC PATCH 37/56] x86/alternative: Reset alternatives

[David Kaplan]

These functions reset the kernel code back to the original form it was at
boot time.  Retpoline and alternative bytes were stored when those were
first patched on boot.  For returns, all returns are simply a jmp to
__x86_return_thunk so patch that in instead.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 arch/x86/include/asm/alternative.h |  7 +++
 arch/x86/kernel/alternative.c      | 76 ++++++++++++++++++++++++++++++
 2 files changed, 83 insertions(+)

diff --git a/arch/x86/include/asm/alternative.h b/arch/x86/include/asm/alternative.h
index 24a4afbf163b..936e555c13ce 100644
--- a/arch/x86/include/asm/alternative.h
+++ b/arch/x86/include/asm/alternative.h
@@ -97,6 +97,13 @@ extern int alternatives_patched;
 
 struct module;
 
+#ifdef CONFIG_DYNAMIC_MITIGATIONS
+extern void reset_retpolines(s32 *start, s32 *end, struct module *mod);
+extern void reset_returns(s32 *start, s32 *end, struct module *mod);
+extern void reset_alternatives(struct alt_instr *start, struct alt_instr *end,
+		struct module *mod);
+#endif
+
 struct alt_site {
 	u8 *pbytes;
 	u8 len;
diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c
index 17b93763d1be..b67116ae883c 100644
--- a/arch/x86/kernel/alternative.c
+++ b/arch/x86/kernel/alternative.c
@@ -3292,3 +3292,79 @@ void __ref smp_text_poke_single(void *addr, const void *opcode, size_t len, cons
 	smp_text_poke_batch_add(addr, opcode, len, emulate);
 	smp_text_poke_batch_finish();
 }
+
+#ifdef CONFIG_DYNAMIC_MITIGATIONS
+void reset_retpolines(s32 *start, s32 *end, struct module *mod)
+{
+	s32 *s;
+	u32 idx = 0;
+	struct retpoline_site *sites;
+
+	if (!mod)
+		sites = retpoline_sites;
+	else
+		sites = mod->arch.retpoline_sites;
+
+	if (WARN_ON(!sites))
+		return;
+
+	for (s = start; s < end; s++, idx++) {
+		void *addr = (void *)s + *s;
+
+		if (!should_patch(addr, mod))
+			continue;
+		/*
+		 * This indirect might have been removed due to a static call
+		 * transform.  If so, ignore it.
+		 */
+		if (*(u8 *)addr == INT3_INSN_OPCODE)
+			continue;
+
+		if (sites[idx].len)
+			text_poke_early(addr, sites[idx].bytes, sites[idx].len);
+	}
+}
+
+void reset_returns(s32 *start, s32 *end, struct module *mod)
+{
+	s32 *s;
+
+	for (s = start; s < end; s++) {
+		void *addr = (void *)s + *s;
+		u8 bytes[JMP32_INSN_SIZE];
+
+		if (!should_patch(addr, mod))
+			continue;
+
+		/* Generate jmp __x86_return_thunk */
+		__text_gen_insn(bytes, JMP32_INSN_OPCODE, addr,
+				&__x86_return_thunk, JMP32_INSN_SIZE);
+		text_poke_early(addr, bytes, JMP32_INSN_SIZE);
+	}
+}
+
+void reset_alternatives(struct alt_instr *start, struct alt_instr *end, struct module *mod)
+{
+	struct alt_instr *s;
+	u32 idx = 0;
+	struct alt_site *sites;
+
+	if (!mod)
+		sites = alt_sites;
+	else
+		sites = mod->arch.alt_sites;
+
+	if (WARN_ON(!sites))
+		return;
+
+	for (s = start; s < end; s++, idx++) {
+		u8 *addr = instr_va(s);
+
+		if (!should_patch(addr, mod))
+			continue;
+
+		if (sites[idx].len)
+			text_poke_early(addr, sites[idx].pbytes, sites[idx].len);
+	}
+}
+#endif
-- 
2.34.1

[RFC PATCH 38/56] x86/callthunks: Reset callthunks

[David Kaplan]

Define functions to restore call sites back to their original bytes.
This is done by checking if each annotated call is pointing to the
expected thunk and if so, adjusting the call target to point back at the
original destination.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 arch/x86/include/asm/alternative.h |  7 +++
 arch/x86/include/asm/module.h      |  1 +
 arch/x86/kernel/callthunks.c       | 73 ++++++++++++++++++++++++++++++
 3 files changed, 81 insertions(+)

diff --git a/arch/x86/include/asm/alternative.h b/arch/x86/include/asm/alternative.h
index 936e555c13ce..00e60195d768 100644
--- a/arch/x86/include/asm/alternative.h
+++ b/arch/x86/include/asm/alternative.h
@@ -133,6 +133,10 @@ extern void callthunks_patch_module_calls(struct callthunk_sites *sites,
 					  struct module *mod);
 extern void *callthunks_translate_call_dest(void *dest);
 extern int x86_call_depth_emit_accounting(u8 **pprog, void *func, void *ip);
+#ifdef CONFIG_DYNAMIC_MITIGATIONS
+extern void reset_builtin_callthunks(void);
+extern void reset_module_callthunks(struct callthunk_sites *cs, struct module *mod);
+#endif
 #else
 static __always_inline void callthunks_patch_builtin_calls(void) {}
 static __always_inline void
@@ -147,6 +151,9 @@ static __always_inline int x86_call_depth_emit_accounting(u8 **pprog,
 {
 	return 0;
 }
+static __always_inline void reset_builtin_callthunks(void) {}
+static __always_inline void reset_module_callthunks(struct callthunk_sites *cs,
+						    struct module *mod) {}
 #endif
 
 #ifdef CONFIG_MITIGATION_ITS
diff --git a/arch/x86/include/asm/module.h b/arch/x86/include/asm/module.h
index d0c39b921408..58d7f1017a14 100644
--- a/arch/x86/include/asm/module.h
+++ b/arch/x86/include/asm/module.h
@@ -23,6 +23,7 @@ struct mod_arch_specific {
 	struct alt_site *alt_sites;
 	struct retpoline_site *retpoline_sites;
 	int num_retpoline_sites;
+	bool callthunks_initialized;
 #endif
 };
 
diff --git a/arch/x86/kernel/callthunks.c b/arch/x86/kernel/callthunks.c
index 758e655f36a8..3e6f00e19814 100644
--- a/arch/x86/kernel/callthunks.c
+++ b/arch/x86/kernel/callthunks.c
@@ -336,6 +336,10 @@ void noinline callthunks_patch_module_calls(struct callthunk_sites *cs,
 	mutex_lock(&text_mutex);
 	callthunks_setup(cs, &ct);
 	mutex_unlock(&text_mutex);
+
+#ifdef CONFIG_DYNAMIC_MITIGATIONS
+	mod->arch.callthunks_initialized = true;
+#endif
 }
 #endif /* CONFIG_MODULES */
 
@@ -381,3 +385,72 @@ static int __init callthunks_debugfs_init(void)
 }
 __initcall(callthunks_debugfs_init);
 #endif
+
+#ifdef CONFIG_DYNAMIC_MITIGATIONS
+static void reset_call_sites(s32 *start, s32 *end, const struct core_text *ct)
+{
+	s32 *s;
+
+	for (s = start; s < end; s++) {
+		void *dest;
+		u8 bytes[8];
+		u8 insn_buff[MAX_PATCH_LEN];
+		void *addr = (void *)s + *s;
+
+		if (!within_coretext(ct, addr))
+			continue;
+
+		dest = call_get_dest(addr);
+		if (!dest || WARN_ON_ONCE(IS_ERR(dest)))
+			continue;
+
+		memcpy(insn_buff, skl_call_thunk_template, SKL_TMPL_SIZE);
+		text_poke_apply_relocation(insn_buff, dest, SKL_TMPL_SIZE,
+					skl_call_thunk_template, SKL_TMPL_SIZE);
+		/* Check for the thunk */
+		if (bcmp(dest, insn_buff, SKL_TMPL_SIZE))
+			continue;
+
+		/* Set new destination to be after the thunk */
+		dest += SKL_TMPL_SIZE;
+		__text_gen_insn(bytes, CALL_INSN_OPCODE, addr, dest, CALL_INSN_SIZE);
+		text_poke_early(addr, bytes, CALL_INSN_SIZE);
+	}
+}
+
+static void callthunks_reset(struct callthunk_sites *cs, const struct core_text *ct)
+{
+	prdbg("Resetting call sites %s\n", ct->name);
+	reset_call_sites(cs->call_start, cs->call_end, ct);
+	prdbg("Resetting call sites done %s\n", ct->name);
+}
+
+void reset_builtin_callthunks(void)
+{
+	struct callthunk_sites cs = {
+		.call_start	= __call_sites,
+		.call_end	= __call_sites_end,
+	};
+
+	if (!thunks_initialized)
+		return;
+
+	callthunks_reset(&cs, &builtin_coretext);
+	thunks_initialized = false;
+}
+
+void reset_module_callthunks(struct callthunk_sites *cs, struct module *mod)
+{
+	struct core_text ct = {
+		.base = (unsigned long)mod->mem[MOD_TEXT].base,
+		.end  = (unsigned long)mod->mem[MOD_TEXT].base + mod->mem[MOD_TEXT].size,
+		.name = mod->name,
+	};
+
+	if (!mod->arch.callthunks_initialized)
+		return;
+
+	callthunks_reset(cs, &ct);
+	mod->arch.callthunks_initialized = false;
+}
+#endif
-- 
2.34.1

[RFC PATCH 39/56] x86/sync_core: Add sync_core_nmi_safe()

[David Kaplan]

As noted in the existing comment, sync_core() is not NMI-safe due to the
use of IRET.  sync_core_nmi_safe() uses MOV-CR2 which can be safely used in
NMI context.  This is needed when modifying kernel code within an NMI
handler.

IRET was initially chosen because it works even under environments like Xen
PV.  But Xen PV will not support CONFIG_DYNAMIC_MITIGATIONS and the need
for NMI-based kernel patching.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 arch/x86/include/asm/sync_core.h | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/arch/x86/include/asm/sync_core.h b/arch/x86/include/asm/sync_core.h
index 96bda43538ee..f4e2f868d71a 100644
--- a/arch/x86/include/asm/sync_core.h
+++ b/arch/x86/include/asm/sync_core.h
@@ -88,6 +88,20 @@ static __always_inline void sync_core(void)
 	iret_to_self();
 }
 
+/*
+ * NMI safe version of sync_core()
+ *
+ * sync_core() may do iret_to_self() which will unmask NMI.
+ * sync_core_nmi_safe() uses MOV-to-CR2 and is safe to use in NMI context.
+ *
+ * As noted in the comments above, this sequence may fault at CPL3 (i.e. Xen
+ * PV).  Therefore it should only be used if outside of those environments.
+ */
+static inline void sync_core_nmi_safe(void)
+{
+	 native_read_cr2();
+}
+
 /*
  * Ensure that a core serializing instruction is issued before returning
  * to user-mode. x86 implements return to user-space through sysexit,
-- 
2.34.1

[RFC PATCH 40/56] x86/alternative: Use sync_core_nmi_safe()

[David Kaplan]

Re-patching is done under NMI context so the NMI-safe version of
sync_core() must be used.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 arch/x86/kernel/alternative.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c
index b67116ae883c..2d48d750d4d9 100644
--- a/arch/x86/kernel/alternative.c
+++ b/arch/x86/kernel/alternative.c
@@ -2585,7 +2585,11 @@ void __init_or_module text_poke_early(void *addr, const void *opcode,
 	} else {
 		local_irq_save(flags);
 		memcpy(addr, opcode, len);
-		sync_core();
+		/* Re-patching occurs in NMI context so we can't do IRET. */
+		if (repatch_in_progress)
+			sync_core_nmi_safe();
+		else
+			sync_core();
 		local_irq_restore(flags);
 
 		/*
-- 
2.34.1

[RFC PATCH 41/56] static_call: Add update_all_static_calls()

[David Kaplan]

The update_all_static_calls() function re-scans all static call sites and
re-patches them.  This is used during re-patching.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 arch/x86/kernel/static_call.c |  3 ++-
 include/linux/static_call.h   |  2 ++
 kernel/static_call_inline.c   | 22 ++++++++++++++++++++++
 3 files changed, 26 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kernel/static_call.c b/arch/x86/kernel/static_call.c
index 378c388d1b31..2d16d69b17d0 100644
--- a/arch/x86/kernel/static_call.c
+++ b/arch/x86/kernel/static_call.c
@@ -105,7 +105,8 @@ static void __ref __static_call_transform(void *insn, enum insn_type type,
 	if (memcmp(insn, code, size) == 0)
 		return;
 
-	if (system_state == SYSTEM_BOOTING || modinit)
+	/* alternatives_patched is false if we are doing dynamic re-patching. */
+	if (system_state == SYSTEM_BOOTING || modinit || !alternatives_patched)
 		return text_poke_early(insn, code, size);
 
 	smp_text_poke_single(insn, code, size, emulate);
diff --git a/include/linux/static_call.h b/include/linux/static_call.h
index 78a77a4ae0ea..cc5f28a04539 100644
--- a/include/linux/static_call.h
+++ b/include/linux/static_call.h
@@ -181,6 +181,8 @@ struct static_call_tramp_key {
 extern void __static_call_update(struct static_call_key *key, void *tramp, void *func);
 extern int static_call_mod_init(struct module *mod);
 extern int static_call_text_reserved(void *start, void *end);
+extern void update_all_static_calls(struct static_call_site *start,
+		struct static_call_site *stop, struct module *mod);
 
 extern long __static_call_return0(void);
 
diff --git a/kernel/static_call_inline.c b/kernel/static_call_inline.c
index 269683d41aa9..504b69496711 100644
--- a/kernel/static_call_inline.c
+++ b/kernel/static_call_inline.c
@@ -9,6 +9,7 @@
 #include <linux/cpu.h>
 #include <linux/processor.h>
 #include <asm/sections.h>
+#include <linux/kallsyms.h>
 
 extern struct static_call_site __start_static_call_sites[],
 			       __stop_static_call_sites[];
@@ -492,6 +493,27 @@ int static_call_text_reserved(void *start, void *end)
 	return __static_call_mod_text_reserved(start, end);
 }
 
+void update_all_static_calls(struct static_call_site *start,
+			     struct static_call_site *stop,
+			     struct module *mod)
+{
+	struct static_call_site *site;
+	struct static_call_key *key;
+
+	for (site = start; site < stop; site++) {
+		void *site_addr = static_call_addr(site);
+
+		/* All init code is gone when this function is called. */
+		if (is_kernel_text((u64) site_addr) ||
+		    (mod &&
+		     within_module_mem_type((u64) site_addr, mod, MOD_TEXT))) {
+			key = static_call_key(site);
+			arch_static_call_transform(site_addr, NULL, key->func,
+					static_call_is_tail(site));
+		}
+	}
+}
+
 int __init static_call_init(void)
 {
 	int ret;
-- 
2.34.1

[RFC PATCH 42/56] module: Make memory writeable for re-patching

[David Kaplan]

The text code for a module must be made writeable to support
re-patching, and can then be marked read-only again after patching is
complete.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 include/linux/module.h |  5 +++++
 kernel/module/main.c   | 34 ++++++++++++++++++++++++++++++++++
 2 files changed, 39 insertions(+)

diff --git a/include/linux/module.h b/include/linux/module.h
index e135cc79acee..2d8c34cb961f 100644
--- a/include/linux/module.h
+++ b/include/linux/module.h
@@ -1020,4 +1020,9 @@ static inline unsigned long find_kallsyms_symbol_value(struct module *mod,
 /* Define __free(module_put) macro for struct module *. */
 DEFINE_FREE(module_put, struct module *, if (_T) module_put(_T))
 
+#ifdef CONFIG_DYNAMIC_MITIGATIONS
+void modules_prepare_repatch(void);
+void modules_post_repatch(void);
+#endif
+
 #endif /* _LINUX_MODULE_H */
diff --git a/kernel/module/main.c b/kernel/module/main.c
index 088f9399af11..0525b1c6d5b9 100644
--- a/kernel/module/main.c
+++ b/kernel/module/main.c
@@ -3910,3 +3910,37 @@ static int module_debugfs_init(void)
 }
 module_init(module_debugfs_init);
 #endif
+
+#ifdef CONFIG_DYNAMIC_MITIGATIONS
+static void change_mod_mem_perm(struct module *mod, enum mod_mem_type type,
+		bool writeable)
+{
+	unsigned long base, size;
+
+	base = (unsigned long) mod->mem[type].base;
+	size = mod->mem[type].size;
+
+	if (writeable)
+		set_memory_rw(base, PFN_UP(size));
+	else
+		set_memory_ro(base, PFN_UP(size));
+}
+
+void modules_prepare_repatch(void)
+{
+	struct module *mod;
+
+	list_for_each_entry(mod, &modules, list) {
+		change_mod_mem_perm(mod, MOD_TEXT, true);
+	}
+}
+
+void modules_post_repatch(void)
+{
+	struct module *mod;
+
+	list_for_each_entry(mod, &modules, list) {
+		change_mod_mem_perm(mod, MOD_TEXT, false);
+	}
+}
+#endif
-- 
2.34.1

[RFC PATCH 43/56] module: Update alternatives

[David Kaplan]

Update alternatives in modules by making the module text writeable and
then calling the arch-specific update alternatives function.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 include/linux/module.h |  6 ++++++
 kernel/module/main.c   | 42 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 48 insertions(+)

diff --git a/include/linux/module.h b/include/linux/module.h
index 2d8c34cb961f..f29974cae6bc 100644
--- a/include/linux/module.h
+++ b/include/linux/module.h
@@ -1023,6 +1023,12 @@ DEFINE_FREE(module_put, struct module *, if (_T) module_put(_T))
 #ifdef CONFIG_DYNAMIC_MITIGATIONS
 void modules_prepare_repatch(void);
 void modules_post_repatch(void);
+void modules_update_alternatives(void);
+void arch_module_update_alternatives(struct module *mod);
+void arch_module_pre_update_alternatives(struct module *mod);
+void arch_module_post_update_alternatives(struct module *mod);
+void modules_pre_update_alternatives(void);
+void modules_post_update_alternatives(void);
 #endif
 
 #endif /* _LINUX_MODULE_H */
diff --git a/kernel/module/main.c b/kernel/module/main.c
index 0525b1c6d5b9..fa46ce4285dd 100644
--- a/kernel/module/main.c
+++ b/kernel/module/main.c
@@ -3943,4 +3943,46 @@ void modules_post_repatch(void)
 		change_mod_mem_perm(mod, MOD_TEXT, false);
 	}
 }
+
+void __weak arch_module_update_alternatives(struct module *mod)
+{
+}
+
+void modules_update_alternatives(void)
+{
+	struct module *mod;
+
+	list_for_each_entry(mod, &modules, list) {
+		arch_module_update_alternatives(mod);
+		update_all_static_calls(mod->static_call_sites,
+					mod->static_call_sites +
+					mod->num_static_call_sites, mod);
+	}
+}
+
+void __weak arch_module_pre_update_alternatives(struct module *mod)
+{
+}
+
+void __weak arch_module_post_update_alternatives(struct module *mod)
+{
+}
+
+void modules_pre_update_alternatives(void)
+{
+	struct module *mod;
+
+	list_for_each_entry(mod, &modules, list) {
+		arch_module_pre_update_alternatives(mod);
+	}
+}
+
+void modules_post_update_alternatives(void)
+{
+	struct module *mod;
+
+	list_for_each_entry(mod, &modules, list) {
+		arch_module_post_update_alternatives(mod);
+	}
+}
 #endif
-- 
2.34.1

[RFC PATCH 44/56] x86/module: Update alternatives

[David Kaplan]

Support resetting and re-configuring retpolines, returns, callthunks,
and alternatives for modules.  The ELF information is kept from when the
module was first loaded.  Static calls are non-arch specific and handled
in the generic module code.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 arch/x86/include/asm/module.h |  4 +++
 arch/x86/kernel/module.c      | 58 +++++++++++++++++++++++++++++++++++
 2 files changed, 62 insertions(+)

diff --git a/arch/x86/include/asm/module.h b/arch/x86/include/asm/module.h
index 58d7f1017a14..f72359b5120d 100644
--- a/arch/x86/include/asm/module.h
+++ b/arch/x86/include/asm/module.h
@@ -27,4 +27,8 @@ struct mod_arch_specific {
 #endif
 };
 
+#ifdef CONFIG_DYNAMIC_MITIGATIONS
+void arch_module_update_alternatives(struct module *mod);
+#endif
+
 #endif /* _ASM_X86_MODULE_H */
diff --git a/arch/x86/kernel/module.c b/arch/x86/kernel/module.c
index 0765d2360a33..b6beb2b3469c 100644
--- a/arch/x86/kernel/module.c
+++ b/arch/x86/kernel/module.c
@@ -333,3 +333,61 @@ void module_arch_cleanup(struct module *mod)
 	alternatives_smp_module_del(mod);
 	its_free_mod(mod);
 }
+
+#ifdef CONFIG_DYNAMIC_MITIGATIONS
+void arch_module_update_alternatives(struct module *mod)
+{
+	const Elf_Ehdr *hdr;
+	const Elf_Shdr *sechdrs;
+	const Elf_Shdr *s, *alt = NULL, *retpolines = NULL, *returns = NULL, *calls = NULL;
+	char *secstrings;
+
+	if (!mod->klp_info) {
+		pr_warn("No module livepatch info, unable to update alternatives\n");
+		return;
+	}
+
+	hdr = &mod->klp_info->hdr;
+	sechdrs = mod->klp_info->sechdrs;
+	secstrings = mod->klp_info->secstrings;
+
+	for (s = sechdrs; s < sechdrs + hdr->e_shnum; s++) {
+		if (!strcmp(".altinstructions", secstrings + s->sh_name))
+			alt = s;
+		if (!strcmp(".retpoline_sites", secstrings + s->sh_name))
+			retpolines = s;
+		if (!strcmp(".return_sites", secstrings + s->sh_name))
+			returns = s;
+		if (!strcmp(".call_sites", secstrings + s->sh_name))
+			calls = s;
+	}
+
+	if (retpolines) {
+		void *rseg = (void *)retpolines->sh_addr;
+
+		reset_retpolines(rseg, rseg + retpolines->sh_size, mod);
+		apply_retpolines(rseg, rseg + retpolines->sh_size, mod);
+	}
+	if (returns) {
+		void *rseg = (void *)returns->sh_addr;
+
+		reset_returns(rseg, rseg + returns->sh_size, mod);
+		apply_returns(rseg, rseg + returns->sh_size, mod);
+	}
+	if (calls) {
+		struct callthunk_sites cs = {};
+
+		cs.call_start = (void *)calls->sh_addr;
+		cs.call_end = (void *)calls->sh_addr + calls->sh_size;
+
+		reset_module_callthunks(&cs, mod);
+		callthunks_patch_module_calls(&cs, mod);
+	}
+	if (alt) {
+		void *aseg = (void *)alt->sh_addr;
+
+		reset_alternatives(aseg, aseg + alt->sh_size, mod);
+		apply_alternatives(aseg, aseg + alt->sh_size, mod);
+	}
+}
+#endif
-- 
2.34.1

[RFC PATCH 45/56] x86/alternative: Use boot_cpu_has in ITS code

[David Kaplan]

Replace cpu_feature_enabled with boot_cpu_has because under dynamic
re-patching, this code may be called after features have changed but before
alternatives are patched.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 arch/x86/kernel/alternative.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c
index 2d48d750d4d9..5a543ffca10d 100644
--- a/arch/x86/kernel/alternative.c
+++ b/arch/x86/kernel/alternative.c
@@ -187,7 +187,7 @@ static void its_fini_core(void)
 #ifdef CONFIG_MODULES
 void its_init_mod(struct module *mod)
 {
-	if (!cpu_feature_enabled(X86_FEATURE_INDIRECT_THUNK_ITS))
+	if (!boot_cpu_has(X86_FEATURE_INDIRECT_THUNK_ITS))
 		return;
 
 	mutex_lock(&text_mutex);
@@ -197,7 +197,7 @@ void its_init_mod(struct module *mod)
 
 void its_fini_mod(struct module *mod)
 {
-	if (!cpu_feature_enabled(X86_FEATURE_INDIRECT_THUNK_ITS))
+	if (!boot_cpu_has(X86_FEATURE_INDIRECT_THUNK_ITS))
 		return;
 
 	WARN_ON_ONCE(its_mod != mod);
@@ -901,7 +901,7 @@ static int emit_its_trampoline(void *addr, struct insn *insn, int reg, u8 *bytes
 /* Check if an indirect branch is at ITS-unsafe address */
 static bool cpu_wants_indirect_its_thunk_at(unsigned long addr, int reg)
 {
-	if (!cpu_feature_enabled(X86_FEATURE_INDIRECT_THUNK_ITS))
+	if (!boot_cpu_has(X86_FEATURE_INDIRECT_THUNK_ITS))
 		return false;
 
 	/* Indirect branch opcode is 2 or 3 bytes depending on reg */
@@ -967,9 +967,9 @@ static int patch_retpoline(void *addr, struct insn *insn, u8 *bytes)
 	/* If anyone ever does: CALL/JMP *%rsp, we're in deep trouble. */
 	BUG_ON(reg == 4);
 
-	if (cpu_feature_enabled(X86_FEATURE_RETPOLINE) &&
-	    !cpu_feature_enabled(X86_FEATURE_RETPOLINE_LFENCE)) {
-		if (cpu_feature_enabled(X86_FEATURE_CALL_DEPTH))
+	if (boot_cpu_has(X86_FEATURE_RETPOLINE) &&
+	    !boot_cpu_has(X86_FEATURE_RETPOLINE_LFENCE)) {
+		if (boot_cpu_has(X86_FEATURE_CALL_DEPTH))
 			return emit_call_track_retpoline(addr, insn, reg, bytes);
 
 		return -1;
@@ -1004,7 +1004,7 @@ static int patch_retpoline(void *addr, struct insn *insn, u8 *bytes)
 	/*
 	 * For RETPOLINE_LFENCE: prepend the indirect CALL/JMP with an LFENCE.
 	 */
-	if (cpu_feature_enabled(X86_FEATURE_RETPOLINE_LFENCE)) {
+	if (boot_cpu_has(X86_FEATURE_RETPOLINE_LFENCE)) {
 		bytes[i++] = 0x0f;
 		bytes[i++] = 0xae;
 		bytes[i++] = 0xe8; /* LFENCE */
-- 
2.34.1

[RFC PATCH 46/56] x86/alternative: Add ITS re-patching support

[David Kaplan]

Non-atomic memory allocations from execmem and memory permission changes
cannot be done under NMI context.  Therefore, pages needed for ITS
trampolines are "pre-allocated" before actual re-patching is done.  The
address for each unique trampoline is saved in the corresponding
retpoline_site structure for use later during actual re-patching.

Pre-allocation is only needed when ITS thunks were not being used but will
be used after the re-patch.  Otherwise we can re-use existing memory that
has been allocated and configured.

If ITS thunks are no longer required, they can be free'd after the re-patch
has taken place.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 arch/x86/include/asm/alternative.h |  10 +++
 arch/x86/kernel/alternative.c      | 111 +++++++++++++++++++++++++++--
 2 files changed, 114 insertions(+), 7 deletions(-)

diff --git a/arch/x86/include/asm/alternative.h b/arch/x86/include/asm/alternative.h
index 00e60195d768..61ce8a4b1aa6 100644
--- a/arch/x86/include/asm/alternative.h
+++ b/arch/x86/include/asm/alternative.h
@@ -102,6 +102,13 @@ extern void reset_retpolines(s32 *start, s32 *end, struct module *mod);
 extern void reset_returns(s32 *start, s32 *end, struct module *mod);
 extern void reset_alternatives(struct alt_instr *start, struct alt_instr *end,
 		struct module *mod);
+#ifdef CONFIG_MITIGATION_ITS
+extern void its_prealloc(s32 *start, s32 *end, struct module *mod);
+extern void its_free_all(struct module *mod);
+#else
+void its_prealloc(s32 *start, s32 *end, struct module *mod) {}
+static __always_inline void its_free_all(struct module *mod) {}
+#endif
 #endif
 
 struct alt_site {
@@ -112,6 +119,9 @@ struct alt_site {
 struct retpoline_site {
 	u8 bytes[6];
 	u8 len;
+#ifdef CONFIG_MITIGATION_ITS
+	u8 *its_thunk;
+#endif
 } __packed;
 
 extern void alternative_instructions(void);
diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c
index 5a543ffca10d..23bb3386ec5e 100644
--- a/arch/x86/kernel/alternative.c
+++ b/arch/x86/kernel/alternative.c
@@ -181,6 +181,11 @@ static void its_fini_core(void)
 {
 	if (IS_ENABLED(CONFIG_STRICT_KERNEL_RWX))
 		its_pages_protect(&its_pages);
+
+	/* Need to keep the list of pages around in case we re-patch later. */
+	if (IS_ENABLED(CONFIG_DYNAMIC_MITIGATIONS))
+		return;
+
 	kfree(its_pages.pages);
 }
 
@@ -887,13 +892,22 @@ static int emit_call_track_retpoline(void *addr, struct insn *insn, int reg, u8
 }
 
 #ifdef CONFIG_MITIGATION_ITS
-static int emit_its_trampoline(void *addr, struct insn *insn, int reg, u8 *bytes)
+static int emit_its_trampoline(void *addr, struct insn *insn, int reg, u8 *bytes,
+				struct retpoline_site *this_site)
 {
 	u8 *thunk = __x86_indirect_its_thunk_array[reg];
-	u8 *tmp = its_allocate_thunk(reg);
+	u8 *tmp;
 
-	if (tmp)
-		thunk = tmp;
+	if (!this_site || !this_site->its_thunk) {
+		tmp = its_allocate_thunk(reg);
+		if (tmp)
+			thunk = tmp;
+
+		if (this_site)
+			this_site->its_thunk = thunk;
+	} else {
+		thunk = this_site->its_thunk;
+	}
 
 	return __emit_trampoline(addr, insn, bytes, thunk, thunk);
 }
@@ -952,7 +966,8 @@ static void prepend_nops(u8 *bytes, int curlen, int neededlen)
  *
  * It also tries to inline spectre_v2=retpoline,lfence when size permits.
  */
-static int patch_retpoline(void *addr, struct insn *insn, u8 *bytes)
+static int patch_retpoline(void *addr, struct insn *insn, u8 *bytes,
+				struct retpoline_site *this_site)
 {
 	retpoline_thunk_t *target;
 	int reg, ret, i = 0;
@@ -1016,7 +1031,7 @@ static int patch_retpoline(void *addr, struct insn *insn, u8 *bytes)
 	 * lower-half of the cacheline. Such branches need ITS mitigation.
 	 */
 	if (cpu_wants_indirect_its_thunk_at((unsigned long)addr + i, reg))
-		return emit_its_trampoline(addr, insn, reg, bytes);
+		return emit_its_trampoline(addr, insn, reg, bytes, this_site);
 #endif
 
 	ret = emit_indirect(op, reg, bytes + i, insn->length - i);
@@ -1124,7 +1139,7 @@ void __init_or_module noinline apply_retpolines(s32 *start, s32 *end, struct mod
 			memcpy(save_site[idx].bytes, addr, insn.length);
 		}
 
-		len = patch_retpoline(addr, &insn, bytes);
+		len = patch_retpoline(addr, &insn, bytes, &save_site[idx]);
 		if (len == insn.length) {
 			optimize_nops(addr, bytes, len);
 			DUMP_BYTES(RETPOLINE, ((u8*)addr),  len, "%px: orig: ", addr);
@@ -3371,4 +3386,86 @@ void reset_alternatives(struct alt_instr *start, struct alt_instr *end, struct m
 			text_poke_early(addr, sites[idx].pbytes, sites[idx].len);
 	}
 }
+
+#ifdef CONFIG_MITIGATION_ITS
+void its_prealloc(s32 *start, s32 *end, struct module *mod)
+{
+	s32 *s;
+	u32 idx = 0;
+	struct retpoline_site *sites;
+
+	/* If its_page!=NULL it means that we already have ITS pages ready. */
+	if (!boot_cpu_has(X86_FEATURE_INDIRECT_THUNK_ITS) || its_page)
+		return;
+
+	if (!mod)
+		sites = retpoline_sites;
+	else
+		sites = mod->arch.retpoline_sites;
+
+	if (WARN_ON(!sites))
+		return;
+
+	for (s = start; s < end; s++, idx++) {
+		void *addr = (void *)s + *s;
+		struct insn insn;
+		int ret;
+		u8 bytes[16];
+
+		if (!should_patch(addr, mod))
+			continue;
+
+		/* We are decoding the original (compile-time) instruction. */
+		ret = insn_decode_kernel(&insn, sites[idx].bytes);
+
+		if (WARN_ON_ONCE(ret < 0))
+			continue;
+
+		/*
+		 * This function prepares for patching but doesn't actually
+		 * change any kernel text.  We are using it to allocate and
+		 * prepare the its pages, which will be used later during
+		 * re-patching.
+		 */
+		patch_retpoline(addr, &insn, bytes, &sites[idx]);
+	}
+}
+
+/* Free all ITS pages if no longer needed. */
+void its_free_all(struct module *mod)
+{
+	int i;
+	int num_sites;
+	struct retpoline_site *sites;
+	struct its_array *pages;
+
+	if (cpu_feature_enabled(X86_FEATURE_INDIRECT_THUNK_ITS))
+		return;
+
+	if (!mod) {
+		pages = &its_pages;
+		sites = retpoline_sites;
+		num_sites = num_retpoline_sites;
+	} else {
+		pages = &mod->arch.its_pages;
+		sites = mod->arch.retpoline_sites;
+		num_sites = mod->arch.num_retpoline_sites;
+	}
+
+	for (i = 0; i < num_sites; i++)
+		sites[i].its_thunk = NULL;
+
+	for (i = 0; i < pages->num; i++) {
+		void *page = pages->pages[i];
+
+		execmem_free(page);
+	}
+
+	kfree(pages->pages);
+	its_offset = 0;
+	pages->pages = NULL;
+	pages->num = 0;
+	its_page = NULL;
+}
+#endif
 #endif
-- 
2.34.1

[RFC PATCH 47/56] x86/module: Add ITS re-patch support for modules

[David Kaplan]

When re-patching modules, call the appropriate functions related to ITS
mitigation support.

The ITS mitigation is unique because it requires memory operations that are
not possible under NMI context.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 arch/x86/kernel/module.c | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)

diff --git a/arch/x86/kernel/module.c b/arch/x86/kernel/module.c
index b6beb2b3469c..12a934755097 100644
--- a/arch/x86/kernel/module.c
+++ b/arch/x86/kernel/module.c
@@ -390,4 +390,36 @@ void arch_module_update_alternatives(struct module *mod)
 		apply_alternatives(aseg, aseg + alt->sh_size, mod);
 	}
 }
+
+void arch_module_pre_update_alternatives(struct module *mod)
+{
+	const Elf_Ehdr *hdr;
+	const Elf_Shdr *sechdrs;
+	const Elf_Shdr *s;
+	char *secstrings;
+
+	if (!mod->klp_info) {
+		pr_warn("No module livepatch info, unable to update alternatives\n");
+		return;
+	}
+
+	hdr = &mod->klp_info->hdr;
+	sechdrs = mod->klp_info->sechdrs;
+	secstrings = mod->klp_info->secstrings;
+
+	for (s = sechdrs; s < sechdrs + hdr->e_shnum; s++) {
+		if (!strcmp(".retpoline_sites", secstrings + s->sh_name)) {
+			void *rseg = (void *)s->sh_addr;
+
+			its_init_mod(mod);
+			its_prealloc(rseg, rseg + s->sh_size, mod);
+			its_fini_mod(mod);
+		}
+	}
+}
+
+void arch_module_post_update_alternatives(struct module *mod)
+{
+	its_free_all(mod);
+}
 #endif
-- 
2.34.1

[RFC PATCH 48/56] x86/bugs: Move code for updating speculation MSRs

[David Kaplan]

Move the code for updating speculation related MSRs to bugs.c.  Besides
belonging more naturally there, this allows for making the 3 functions it
calls static.  Also this prepares the code for when re-patching will also
need to call this same routine.

No functional change.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 arch/x86/include/asm/bugs.h  |  1 +
 arch/x86/kernel/cpu/bugs.c   | 14 +++++++++++---
 arch/x86/kernel/cpu/common.c |  6 +-----
 arch/x86/kernel/cpu/cpu.h    |  4 ----
 4 files changed, 13 insertions(+), 12 deletions(-)

diff --git a/arch/x86/include/asm/bugs.h b/arch/x86/include/asm/bugs.h
index e43b9412645e..2e1a7d282e51 100644
--- a/arch/x86/include/asm/bugs.h
+++ b/arch/x86/include/asm/bugs.h
@@ -12,5 +12,6 @@ static inline int ppro_with_ram_bug(void) { return 0; }
 
 extern void cpu_bugs_smt_update(void);
 void arch_cpu_reset_mitigations(void);
+void cpu_bugs_update_speculation_msrs(void);
 
 #endif /* _ASM_X86_BUGS_H */
diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
index 06061bcb08b8..2f82261d033d 100644
--- a/arch/x86/kernel/cpu/bugs.c
+++ b/arch/x86/kernel/cpu/bugs.c
@@ -925,7 +925,7 @@ static const char * const srbds_strings[] = {
 
 static bool srbds_off;
 
-void update_srbds_msr(void)
+static void update_srbds_msr(void)
 {
 	u64 mcu_ctrl;
 
@@ -1085,7 +1085,7 @@ bool gds_ucode_mitigated(void)
 }
 EXPORT_SYMBOL_GPL(gds_ucode_mitigated);
 
-void update_gds_msr(void)
+static void update_gds_msr(void)
 {
 	u64 mcu_ctrl_after;
 	u64 mcu_ctrl;
@@ -3014,7 +3014,7 @@ int arch_prctl_spec_ctrl_get(struct task_struct *task, unsigned long which)
 	}
 }
 
-void x86_spec_ctrl_setup_ap(void)
+static void x86_spec_ctrl_setup_ap(void)
 {
 	if (boot_cpu_has(X86_FEATURE_MSR_SPEC_CTRL))
 		update_spec_ctrl(x86_spec_ctrl_base);
@@ -4010,3 +4010,11 @@ void arch_cpu_reset_mitigations(void)
 	vmscape_reset_mitigation();
 }
 #endif
+
+void cpu_bugs_update_speculation_msrs(void)
+{
+	x86_spec_ctrl_setup_ap();
+	update_srbds_msr();
+	if (boot_cpu_has_bug(X86_BUG_GDS))
+		update_gds_msr();
+}
diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
index c7d3512914ca..c37ff92aaec2 100644
--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -2138,11 +2138,7 @@ void identify_secondary_cpu(unsigned int cpu)
 #ifdef CONFIG_X86_32
 	enable_sep_cpu();
 #endif
-	x86_spec_ctrl_setup_ap();
-	update_srbds_msr();
-	if (boot_cpu_has_bug(X86_BUG_GDS))
-		update_gds_msr();
-
+	cpu_bugs_update_speculation_msrs();
 	tsx_ap_init();
 	c->initialized = true;
 }
diff --git a/arch/x86/kernel/cpu/cpu.h b/arch/x86/kernel/cpu/cpu.h
index bc38b2d56f26..48ecc6dbaaa1 100644
--- a/arch/x86/kernel/cpu/cpu.h
+++ b/arch/x86/kernel/cpu/cpu.h
@@ -87,10 +87,6 @@ static inline struct amd_northbridge *amd_init_l3_cache(int index)
 unsigned int aperfmperf_get_khz(int cpu);
 void cpu_select_mitigations(void);
 
-extern void x86_spec_ctrl_setup_ap(void);
-extern void update_srbds_msr(void);
-extern void update_gds_msr(void);
-
 extern enum spectre_v2_mitigation spectre_v2_enabled;
 
 static inline bool spectre_v2_in_eibrs_mode(enum spectre_v2_mitigation mode)
-- 
2.34.1

[RFC PATCH 49/56] x86/fpu: Qualify warning in os_xsave

[David Kaplan]

The warning about using FPU before alternatives are patched is only
relevant during boot.  Ignore the warning unless the system is booting.
This suppresses the erroneous warning that can occur during alternative
re-patching at runtime.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 arch/x86/kernel/fpu/xstate.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/x86/kernel/fpu/xstate.h b/arch/x86/kernel/fpu/xstate.h
index 52ce19289989..924552a20db1 100644
--- a/arch/x86/kernel/fpu/xstate.h
+++ b/arch/x86/kernel/fpu/xstate.h
@@ -220,7 +220,7 @@ static inline void os_xsave(struct fpstate *fpstate)
 	u32 hmask = mask >> 32;
 	int err;
 
-	WARN_ON_FPU(!alternatives_patched);
+	WARN_ON_FPU(!alternatives_patched && system_state == SYSTEM_BOOTING);
 	xfd_validate_state(fpstate, mask, false);
 
 	XSTATE_XSAVE(&fpstate->regs.xsave, lmask, hmask, err);
-- 
2.34.1

[RFC PATCH 50/56] x86/alternative: Add re-patch support

[David Kaplan]

Updating alternatives is done under the biggest hammers possible.  The
freezer is used to freeze all processes and kernel threads at safe
points to ensure they are not in the middle of a sequence we're about to
patch.  Then stop_machine_nmi() synchronizes all CPUs and puts them into
a tight spin loop while re-patching occurs.  The actual patching is done
using simple memcpy, just like during boot.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 arch/x86/include/asm/alternative.h |   6 ++
 arch/x86/kernel/alternative.c      | 131 +++++++++++++++++++++++++++++
 2 files changed, 137 insertions(+)

diff --git a/arch/x86/include/asm/alternative.h b/arch/x86/include/asm/alternative.h
index 61ce8a4b1aa6..f0b863292c3c 100644
--- a/arch/x86/include/asm/alternative.h
+++ b/arch/x86/include/asm/alternative.h
@@ -19,6 +19,7 @@
 #ifndef __ASSEMBLER__
 
 #include <linux/stddef.h>
+#include <linux/static_call_types.h>
 
 /*
  * Alternative inline assembly for SMP.
@@ -89,6 +90,9 @@ extern s32 __cfi_sites[],	__cfi_sites_end[];
 extern s32 __ibt_endbr_seal[],	__ibt_endbr_seal_end[];
 extern s32 __smp_locks[],	__smp_locks_end[];
 
+extern struct static_call_site __start_static_call_sites[],
+			       __stop_static_call_sites[];
+
 /*
  * Debug flag that can be tested to see whether alternative
  * instructions were patched in already:
@@ -98,6 +102,8 @@ extern int alternatives_patched;
 struct module;
 
 #ifdef CONFIG_DYNAMIC_MITIGATIONS
+extern void cpu_update_alternatives(void);
+extern void cpu_prepare_repatch_alternatives(void);
 extern void reset_retpolines(s32 *start, s32 *end, struct module *mod);
 extern void reset_returns(s32 *start, s32 *end, struct module *mod);
 extern void reset_alternatives(struct alt_instr *start, struct alt_instr *end,
diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c
index 23bb3386ec5e..613cb645bd9f 100644
--- a/arch/x86/kernel/alternative.c
+++ b/arch/x86/kernel/alternative.c
@@ -6,12 +6,15 @@
 #include <linux/vmalloc.h>
 #include <linux/memory.h>
 #include <linux/execmem.h>
+#include <linux/stop_machine.h>
+#include <linux/freezer.h>
 
 #include <asm/text-patching.h>
 #include <asm/insn.h>
 #include <asm/ibt.h>
 #include <asm/set_memory.h>
 #include <asm/nmi.h>
+#include <asm/bugs.h>
 
 int __read_mostly alternatives_patched;
 
@@ -3468,4 +3471,132 @@ void its_free_all(struct module *mod)
 	its_page = NULL;
 }
 #endif
+static atomic_t thread_ack;
+
+/*
+ * This function is called by ALL online CPUs but only CPU0 will do the
+ * re-patching.  It is important that all other cores spin in the tight loop
+ * below (and not in multi_cpu_stop) because they cannot safely do return
+ * instructions while returns are being patched.  Therefore, spin them here
+ * (with interrupts disabled) until CPU0 has finished its work.
+ */
+static int __cpu_update_alternatives(void *__unused)
+{
+	if (smp_processor_id()) {
+		atomic_dec(&thread_ack);
+		while (!READ_ONCE(alternatives_patched))
+			cpu_relax();
+
+		cpu_bugs_update_speculation_msrs();
+	} else {
+		repatch_in_progress = true;
+
+		/* Wait for all cores to enter this function. */
+		while (atomic_read(&thread_ack))
+			cpu_relax();
+
+		/* These must be un-done in the opposite order in which they were applied. */
+		reset_alternatives(__alt_instructions, __alt_instructions_end, NULL);
+		reset_builtin_callthunks();
+		reset_returns(__return_sites, __return_sites_end, NULL);
+		reset_retpolines(__retpoline_sites, __retpoline_sites_end, NULL);
+
+		apply_retpolines(__retpoline_sites, __retpoline_sites_end, NULL);
+		apply_returns(__return_sites, __return_sites_end, NULL);
+		callthunks_patch_builtin_calls();
+		apply_alternatives(__alt_instructions, __alt_instructions_end, NULL);
+
+		update_all_static_calls(__start_static_call_sites,
+				__stop_static_call_sites, NULL);
+		modules_update_alternatives();
+		cpu_bugs_update_speculation_msrs();
+		repatch_in_progress = false;
+
+		/* This will wake the other CPUs. */
+		WRITE_ONCE(alternatives_patched, 1);
+	}
+	return 0;
+}
+
+void cpu_prepare_repatch_alternatives(void)
+{
+	alternatives_patched = 0;
+	/* Reset the synchronization barrier. */
+	atomic_set(&thread_ack, num_online_cpus() - 1);
+}
+
+static void make_all_text_writeable(void)
+{
+	unsigned long start = PFN_ALIGN(_text);
+	unsigned long end = PFN_ALIGN(_etext);
+
+	set_memory_rw(start, (end - start) >> PAGE_SHIFT);
+	modules_prepare_repatch();
+}
+
+static void make_all_text_readonly(void)
+{
+	unsigned long start = PFN_ALIGN(_text);
+	unsigned long end = PFN_ALIGN(_etext);
+
+	set_memory_ro(start, (end - start) >> PAGE_SHIFT);
+	modules_post_repatch();
+}
+
+void cpu_update_alternatives(void)
+{
+	/*
+	 * Re-patching is not supported under Xen PV because it uses MOV-CR2
+	 * for synchronization (see sync_core_nmi_safe()).
+	 */
+	if (cpu_feature_enabled(X86_FEATURE_XENPV)) {
+		pr_err("Xen PV does not support dynamic mitigations!\n");
+		alternatives_patched = 1;
+		return;
+	}
+
+	pr_info("Re-patching alternatives\n");
+
+	/*
+	 * ITS mitigation requires dynamic memory allocation and changing memory
+	 * permissions.  These are not possible under NMI context.
+	 *
+	 * Therefore, pre-allocate ITS pages if needed.  If previous ITS pages
+	 * exist, those will be used instead.
+	 */
+	its_prealloc(__retpoline_sites, __retpoline_sites_end, NULL);
+	modules_pre_update_alternatives();
+
+	/*
+	 * Freeze everything because we cannot have a thread be in the middle of
+	 * something we're about to change when we issue stop_machine.
+	 *
+	 * Therefore, use the freezer to get all tasks to a safe place before we
+	 * re-patch.
+	 */
+	if (freeze_processes()) {
+		pr_err("Unable to freeze processes for re-patching!\n");
+		return;
+	}
+
+	if (freeze_kernel_threads()) {
+		pr_err("Unable to freeze tasks for re-patching!\n");
+		thaw_processes();
+		return;
+	}
+
+	make_all_text_writeable();
+	stop_machine_nmi(__cpu_update_alternatives, NULL, cpu_online_mask);
+	make_all_text_readonly();
+
+	cpu_bugs_smt_update();
+
+	/* Free un-needed ITS pages.  This cannot happen in NMI context. */
+	its_free_all(NULL);
+	modules_post_update_alternatives();
+
+	thaw_kernel_threads();
+	thaw_processes();
+	pr_info("Finished re-patching alternatives\n");
+}
 #endif
-- 
2.34.1

[RFC PATCH 51/56] cpu: Parse string of mitigation options

[David Kaplan]

Support runtime parsing of a cmdline-like string of mitigation options.
The global 'mitigations' parameter is processed in generic code and all
other parsing is done via the arch-specific function.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 include/linux/cpu.h |  4 ++++
 kernel/cpu.c        | 32 ++++++++++++++++++++++++++++++++
 2 files changed, 36 insertions(+)

diff --git a/include/linux/cpu.h b/include/linux/cpu.h
index 3da629a76a49..6f69344465f1 100644
--- a/include/linux/cpu.h
+++ b/include/linux/cpu.h
@@ -210,6 +210,10 @@ enum smt_mitigations {
 
 void cpu_reset_mitigations(void);
 void arch_cpu_reset_mitigations(void);
+bool cpu_is_mitigation_opt(char *param);
+bool arch_is_mitigation_opt(char *param);
+int cpu_parse_mitigation_options(const char *str);
+int arch_parse_mitigation_opt(char *param, char *val);
 
 #ifdef CONFIG_CPU_MITIGATIONS
 extern bool cpu_mitigations_off(void);
diff --git a/kernel/cpu.c b/kernel/cpu.c
index 33289405af30..942a200398c9 100644
--- a/kernel/cpu.c
+++ b/kernel/cpu.c
@@ -3343,4 +3343,36 @@ void cpu_reset_mitigations(void)
 	attack_vectors[CPU_MITIGATE_GUEST_GUEST] = IS_ENABLED(CONFIG_KVM);
 	arch_cpu_reset_mitigations();
 }
+
+int __weak arch_parse_mitigation_opt(char *param, char *val) { return 0; }
+bool __weak arch_is_mitigation_opt(char *param) { return false; }
+
+bool cpu_is_mitigation_opt(char *param)
+{
+	if (parameq(param, "mitigations"))
+		return true;
+	else
+		return arch_is_mitigation_opt(param);
+}
+
+static int __cpu_parse_mitigation_options(char *param, char *val,
+		const char *unused, void *arg)
+{
+	if (parameq(param, "mitigations"))
+		return mitigations_parse_cmdline(val);
+	else
+		return arch_parse_mitigation_opt(param, val);
+}
+
+int cpu_parse_mitigation_options(const char *str)
+{
+	char *tmpstr;
+
+	/* Copy the provided string because parse_args will mangle it. */
+	tmpstr = kstrdup(str, GFP_KERNEL);
+	parse_args("dynamic mitigations", tmpstr, NULL, 0, 0, 0, NULL,
+			__cpu_parse_mitigation_options);
+	kfree(tmpstr);
+	return 0;
+}
 #endif
-- 
2.34.1

[RFC PATCH 52/56] x86/bugs: Support parsing mitigation options

[David Kaplan]

Add arch-specific function for determining if an option is related to a
mitigation and parsing it.  These will be used for parsing a string of
options for re-evaluating cpu mitigations.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 arch/x86/include/asm/bugs.h |  2 ++
 arch/x86/kernel/cpu/bugs.c  | 56 +++++++++++++++++++++++++++++++++++++
 2 files changed, 58 insertions(+)

diff --git a/arch/x86/include/asm/bugs.h b/arch/x86/include/asm/bugs.h
index 2e1a7d282e51..1e142a676335 100644
--- a/arch/x86/include/asm/bugs.h
+++ b/arch/x86/include/asm/bugs.h
@@ -13,5 +13,7 @@ static inline int ppro_with_ram_bug(void) { return 0; }
 extern void cpu_bugs_smt_update(void);
 void arch_cpu_reset_mitigations(void);
 void cpu_bugs_update_speculation_msrs(void);
+bool arch_is_mitigation_opt(char *param);
+int arch_parse_mitigation_opt(char *param, char *val);
 
 #endif /* _ASM_X86_BUGS_H */
diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
index 2f82261d033d..26ceb42e0cfb 100644
--- a/arch/x86/kernel/cpu/bugs.c
+++ b/arch/x86/kernel/cpu/bugs.c
@@ -3991,6 +3991,62 @@ void __warn_thunk(void)
 }
 
 #ifdef CONFIG_DYNAMIC_MITIGATIONS
+struct mitigation_info {
+	char *param;
+	int (*parse)(char *str);
+};
+
+static struct mitigation_info mitigation_parsers[] = {
+	{"mds", mds_cmdline},
+	{"tsx_async_abort", tsx_async_abort_parse_cmdline},
+	{"mmio_stale_data", mmio_stale_data_parse_cmdline},
+	{"reg_file_data_sampling", rfds_parse_cmdline},
+	{"srbds", srbds_parse_cmdline},
+	{"gather_data_sampling", gds_parse_cmdline},
+	{"nospectre_v1", nospectre_v1_cmdline},
+	{"retbleed", retbleed_parse_cmdline},
+	{"indirect_target_selection", its_parse_cmdline},
+	{"spectre_v2_user", spectre_v2_user_parse_cmdline},
+	{"nospectre_v2", nospectre_v2_parse_cmdline},
+	{"spectre_v2", spectre_v2_parse_cmdline},
+	{"spectre_bhi", spectre_bhi_parse_cmdline},
+	{"nospec_store_bypass_disable", nossb_parse_cmdline},
+	{"spec_store_bypass_disable", ssb_parse_cmdline},
+	{"l1tf", l1tf_cmdline},
+	{"spec_rstack_overflow", srso_parse_cmdline},
+	{"tsa", tsa_parse_cmdline},
+	{"vmscape", vmscape_parse_cmdline}
+};
+
+static struct mitigation_info *get_mitigation_info(char *param)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(mitigation_parsers); i++) {
+		if (parameq(param, mitigation_parsers[i].param))
+			return &mitigation_parsers[i];
+	}
+
+	return NULL;
+}
+
+bool arch_is_mitigation_opt(char *param)
+{
+	return get_mitigation_info(param);
+}
+
+int arch_parse_mitigation_opt(char *param, char *val)
+{
+	struct mitigation_info *info = get_mitigation_info(param);
+
+	if (!info) {
+		pr_warn("Ignoring non-mitigation option %s\n", param);
+		return 0;
+	}
+
+	return info->parse(val);
+}
+
 void arch_cpu_reset_mitigations(void)
 {
 	spectre_v1_reset_mitigation();
-- 
2.34.1

[RFC PATCH 53/56] drivers/cpu: Re-patch mitigations through sysfs

[David Kaplan]

Create a new file at /sys/devices/system/cpu/mitigations that prints the
current set of mitigation options and can be written to in order to
re-select mitigations.

Only options related to mitigations are handled, with the file initially
returning the relevant options from the command line.  When the file is
written, any existing selections are discarded and the new options are
evaluated.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 .../ABI/testing/sysfs-devices-system-cpu      |   8 ++
 drivers/base/cpu.c                            | 113 ++++++++++++++++++
 include/linux/cpu.h                           |   3 +
 3 files changed, 124 insertions(+)

diff --git a/Documentation/ABI/testing/sysfs-devices-system-cpu b/Documentation/ABI/testing/sysfs-devices-system-cpu
index 8aed6d94c4cd..5a0d4372e739 100644
--- a/Documentation/ABI/testing/sysfs-devices-system-cpu
+++ b/Documentation/ABI/testing/sysfs-devices-system-cpu
@@ -777,3 +777,11 @@ Date:		Nov 2022
 Contact:	Linux kernel mailing list <linux-kernel@vger.kernel.org>
 Description:
 		(RO) the list of CPUs that can be brought online.
+
+What:		/sys/devices/system/cpu/mitigations
+Date:		Oct 2025
+Contact:	Linux kernel mailing list <linux-kernel@vger.kernel.org>
+Description:
+		Read/write interface to control CPU mitigations.
+
+		See also: Documentation/admin-guide/hw-vuln/dynamic_mitigations.rst
diff --git a/drivers/base/cpu.c b/drivers/base/cpu.c
index fa0a2eef93ac..3f9410dee67c 100644
--- a/drivers/base/cpu.c
+++ b/drivers/base/cpu.c
@@ -21,6 +21,7 @@
 #include <linux/pm_qos.h>
 #include <linux/delay.h>
 #include <linux/sched/isolation.h>
+#include <linux/filter.h>
 
 #include "base.h"
 
@@ -605,6 +606,113 @@ CPU_SHOW_VULN_FALLBACK(indirect_target_selection);
 CPU_SHOW_VULN_FALLBACK(tsa);
 CPU_SHOW_VULN_FALLBACK(vmscape);
 
+#ifdef CONFIG_DYNAMIC_MITIGATIONS
+static char *saved_opts;
+
+static int __cpu_filter_mitigation_opts(char *param, char *val,
+		const char *unused, void *arg)
+{
+	char *opt;
+	/* Extra byte allocated for the ' ' at the end. */
+	int len = strlen(param) + 2;
+
+	if (!cpu_is_mitigation_opt(param))
+		return 0;
+
+	/* Extra byte allocated for the '='. */
+	if (val)
+		len += strlen(val) + 1;
+
+	opt = kmalloc(len, GFP_KERNEL);
+	if (WARN_ON(!opt))
+		return 0;
+
+	if (val)
+		sprintf(opt, "%s=%s ", param, val);
+	else
+		sprintf(opt, "%s ", param);
+
+	strcat(saved_opts, opt);
+	kfree(opt);
+
+	return 0;
+}
+
+/* Only save options related to mitigations. */
+static void cpu_filter_mitigation_opts(const char *str)
+{
+	char *tmpstr;
+	char *newline = "\n";
+
+	kfree(saved_opts);
+
+	/*
+	 * 2 extra bytes, one for the final NULL and one for the space we add
+	 * between each option.
+	 */
+	saved_opts = kmalloc(strlen(str)+2, GFP_KERNEL);
+	tmpstr = kstrdup(str, GFP_KERNEL);
+
+	if (WARN_ON(!saved_opts) || WARN_ON(!tmpstr))
+		return;
+
+	saved_opts[0] = '\0';
+
+	parse_args("mitigation filter", tmpstr, NULL, 0, 0, 0, NULL,
+			__cpu_filter_mitigation_opts);
+
+	strcat(saved_opts, newline);
+	kfree(tmpstr);
+}
+
+ssize_t cpu_show_mitigation_options(struct device *dev, struct device_attribute *attr, char *buf);
+ssize_t cpu_show_mitigation_options(struct device *dev, struct device_attribute *attr, char *buf)
+{
+	return sysfs_emit(buf, saved_opts);
+}
+
+ssize_t cpu_write_mitigation_options(struct device *dev,
+				struct device_attribute *attr,
+				const char *buf, size_t count);
+
+void __weak cpu_prepare_repatch_alternatives(void)
+{
+}
+
+void __weak cpu_update_alternatives(void)
+{
+}
+
+void __weak cpu_select_mitigations(void)
+{
+}
+
+ssize_t cpu_write_mitigation_options(struct device *dev,
+				struct device_attribute *attr,
+				const char *buf, size_t count)
+{
+	/* Save and filter the provided options. */
+	cpu_filter_mitigation_opts(buf);
+
+	/* Reset and re-select all mitigations */
+	cpu_prepare_repatch_alternatives();
+	cpu_reset_mitigations();
+
+	if (cpu_parse_mitigation_options(buf))
+		pr_warn("Error parsing new options %s\n", buf);
+
+	cpu_select_mitigations();
+	cpu_update_alternatives();
+
+	if (ebpf_jit_enabled())
+		pr_warn("WARNING! EBPF JIT is enabled.  See Documentation/admin-guide/hw-vuln/dynamic_mitigations.rst\n");
+
+	return count;
+
+}
+static DEVICE_ATTR(mitigations, 0644, cpu_show_mitigation_options, cpu_write_mitigation_options);
+#endif
+
 static DEVICE_ATTR(meltdown, 0444, cpu_show_meltdown, NULL);
 static DEVICE_ATTR(spectre_v1, 0444, cpu_show_spectre_v1, NULL);
 static DEVICE_ATTR(spectre_v2, 0444, cpu_show_spectre_v2, NULL);
@@ -660,6 +768,11 @@ static void __init cpu_register_vulnerabilities(void)
 	if (dev) {
 		if (sysfs_create_group(&dev->kobj, &cpu_root_vulnerabilities_group))
 			pr_err("Unable to register CPU vulnerabilities\n");
+#ifdef CONFIG_DYNAMIC_MITIGATIONS
+		if (sysfs_create_file(&dev->kobj, &dev_attr_mitigations.attr))
+			pr_err("Unable to register CPU vulnerability controls\n");
+		cpu_filter_mitigation_opts(saved_command_line);
+#endif
 		put_device(dev);
 	}
 }
diff --git a/include/linux/cpu.h b/include/linux/cpu.h
index 6f69344465f1..99e4b08ff36d 100644
--- a/include/linux/cpu.h
+++ b/include/linux/cpu.h
@@ -210,6 +210,9 @@ enum smt_mitigations {
 
 void cpu_reset_mitigations(void);
 void arch_cpu_reset_mitigations(void);
+void cpu_select_mitigations(void);
+void cpu_prepare_repatch_alternatives(void);
+void cpu_update_alternatives(void);
 bool cpu_is_mitigation_opt(char *param);
 bool arch_is_mitigation_opt(char *param);
 int cpu_parse_mitigation_options(const char *str);
-- 
2.34.1

[RFC PATCH 54/56] x86/debug: Create debugfs interface to x86_capabilities

[David Kaplan]

Create two debugfs files under x86/x86_cabilities that enable access to the
X86_FEATURE_* and X86_BUG_* bits.  The X86_FEATURE_* bits are read-only
while the X86_BUG_* bits are read-write.  This interface will allow for
user-space test programs to verify mitigation settings are selecting the
correct feature bits.  It will also allow for testing mitigations even on
hardware that may not have a particular bug.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 arch/x86/kernel/cpu/common.c | 59 ++++++++++++++++++++++++++++++++++++
 1 file changed, 59 insertions(+)

diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
index c37ff92aaec2..1755f91a5643 100644
--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -27,6 +27,7 @@
 #include <linux/stackprotector.h>
 #include <linux/utsname.h>
 #include <linux/efi.h>
+#include <linux/debugfs.h>
 
 #include <asm/alternative.h>
 #include <asm/cmdline.h>
@@ -2608,3 +2609,61 @@ void __init arch_cpu_finalize_init(void)
 	 */
 	mem_encrypt_init();
 }
+
+#ifdef CONFIG_DYNAMIC_MITIGATIONS
+/*
+ * The boot_cpu_data.x86_capability[] array has NCAPINTS of u32 for X86_FEATURE
+ * bits followed by NBUGINTS of u32 for X86_BUG bits.
+ *
+ * The debugfs interface allows reading the X86_FEATURE_* bits and read/writing
+ * the X86_BUG_* bits.  Note that updates to the X86_BUG_* bits may not be
+ * visible to the entire kernel until alternatives have been re-patched through
+ * the dynamic mitigation interface.
+ */
+static ssize_t bug_read(struct file *file, char __user *user_buf,
+			size_t count, loff_t *ppos)
+{
+	return simple_read_from_buffer(user_buf, count, ppos,
+				       &boot_cpu_data.x86_capability[NCAPINTS],
+				       NBUGINTS * sizeof(u32));
+}
+
+static ssize_t bug_write(struct file *file, const char __user *user_buf,
+			 size_t count, loff_t *ppos)
+{
+	return simple_write_to_buffer(&boot_cpu_data.x86_capability[NCAPINTS],
+				      NBUGINTS * sizeof(u32), ppos, user_buf,
+				      count);
+}
+
+static ssize_t feature_read(struct file *file, char __user *user_buf,
+			size_t count, loff_t *ppos)
+{
+	return simple_read_from_buffer(user_buf, count, ppos,
+				       boot_cpu_data.x86_capability,
+				       NCAPINTS * sizeof(u32));
+}
+
+static const struct file_operations dfs_bug_ops = {
+	.read		= bug_read,
+	.write		= bug_write,
+	.llseek		= default_llseek,
+};
+
+static const struct file_operations dfs_feature_ops = {
+	.read		= feature_read,
+	.llseek		= default_llseek,
+};
+
+static int __init x86_caps_debugfs_init(void)
+{
+	struct dentry *dir;
+
+	dir = debugfs_create_dir("x86_capabilities", arch_debugfs_dir);
+	debugfs_create_file("bugs", 0600, dir, NULL, &dfs_bug_ops);
+	debugfs_create_file("features", 0400, dir, NULL, &dfs_feature_ops);
+
+	return 0;
+}
+late_initcall(x86_caps_debugfs_init);
+#endif
-- 
2.34.1

[RFC PATCH 55/56] x86/debug: Show return thunk in debugfs

[David Kaplan]

Make the value of x86_return_thunk visible in debugfs to support user-space
testing.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 arch/x86/kernel/cpu/bugs.c | 44 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 44 insertions(+)

diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
index 26ceb42e0cfb..8365448b3aef 100644
--- a/arch/x86/kernel/cpu/bugs.c
+++ b/arch/x86/kernel/cpu/bugs.c
@@ -16,6 +16,7 @@
 #include <linux/sched/smt.h>
 #include <linux/pgtable.h>
 #include <linux/bpf.h>
+#include <linux/debugfs.h>
 
 #include <asm/spec-ctrl.h>
 #include <asm/cmdline.h>
@@ -4065,6 +4066,49 @@ void arch_cpu_reset_mitigations(void)
 	tsa_reset_mitigation();
 	vmscape_reset_mitigation();
 }
+
+static int rethunk_debug_show(struct seq_file *m, void *p)
+{
+	if (x86_return_thunk == __x86_return_thunk)
+		seq_puts(m, "__x86_return_thunk\n");
+	else if (x86_return_thunk == retbleed_return_thunk)
+		seq_puts(m, "retbleed_return_thunk\n");
+	else if (x86_return_thunk == call_depth_return_thunk)
+		seq_puts(m, "call_depth_return_thunk\n");
+	else if (x86_return_thunk == its_return_thunk)
+		seq_puts(m, "its_return_thunk\n");
+	else if (x86_return_thunk == srso_alias_return_thunk)
+		seq_puts(m, "srso_alias_return_thunk\n");
+	else if (x86_return_thunk == srso_return_thunk)
+		seq_puts(m, "srso_return_thunk\n");
+	else
+		seq_puts(m, "unknown\n");
+
+	return 0;
+}
+
+static int rethunk_debug_open(struct inode *inode, struct file *file)
+{
+	return single_open(file, rethunk_debug_show, inode->i_private);
+}
+
+static const struct file_operations dfs_thunk_ops = {
+	.open		= rethunk_debug_open,
+	.read		= seq_read,
+	.llseek		= seq_lseek,
+	.release	= single_release,
+};
+
+static int __init mitigations_debugfs_init(void)
+{
+	struct dentry *dir;
+
+	dir = debugfs_create_dir("mitigations", arch_debugfs_dir);
+	debugfs_create_file("x86_return_thunk", 0400, dir, NULL, &dfs_thunk_ops);
+
+	return 0;
+}
+late_initcall(mitigations_debugfs_init);
 #endif
 
 void cpu_bugs_update_speculation_msrs(void)
-- 
2.34.1

[RFC PATCH 56/56] x86/debug: Show static branch config in debugfs

[David Kaplan]

Create a debugfs interface showing the current enablement status of the
static branches related to mitigations.  This will be used by user-space
testing tools to verify mitigation configuration.

Signed-off-by: David Kaplan <david.kaplan@amd.com>
---
 arch/x86/kernel/cpu/bugs.c | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)

diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
index 8365448b3aef..eeb7d50332cf 100644
--- a/arch/x86/kernel/cpu/bugs.c
+++ b/arch/x86/kernel/cpu/bugs.c
@@ -4099,12 +4099,44 @@ static const struct file_operations dfs_thunk_ops = {
 	.release	= single_release,
 };
 
+static int static_branch_debug_show(struct seq_file *m, void *p)
+{
+	if (static_key_enabled((struct static_key *)m->private))
+		seq_puts(m, "enabled\n");
+	else
+		seq_puts(m, "disabled\n");
+
+	return 0;
+}
+
+static int static_branch_debug_open(struct inode *inode, struct file *file)
+{
+	return single_open(file, static_branch_debug_show, inode->i_private);
+}
+
+static const struct file_operations dfs_static_branch_ops = {
+	.open		= static_branch_debug_open,
+	.read		= seq_read,
+	.llseek		= seq_lseek,
+	.release	= single_release,
+};
+
 static int __init mitigations_debugfs_init(void)
 {
 	struct dentry *dir;
 
 	dir = debugfs_create_dir("mitigations", arch_debugfs_dir);
 	debugfs_create_file("x86_return_thunk", 0400, dir, NULL, &dfs_thunk_ops);
+	debugfs_create_file("switch_mm_cond_ibpb", 0400, dir,
+			&switch_mm_cond_ibpb, &dfs_static_branch_ops);
+	debugfs_create_file("switch_mm_always_ibpb", 0400, dir,
+			&switch_mm_always_ibpb, &dfs_static_branch_ops);
+	debugfs_create_file("switch_vcpu_ibpb", 0400, dir,
+			&switch_vcpu_ibpb, &dfs_static_branch_ops);
+	debugfs_create_file("cpu_buf_idle_clear", 0400, dir,
+			&cpu_buf_idle_clear, &dfs_static_branch_ops);
+	debugfs_create_file("cpu_buf_vm_clear", 0400, dir,
+			&cpu_buf_vm_clear, &dfs_static_branch_ops);
 
 	return 0;
 }
-- 
2.34.1

Re: [RFC PATCH 00/56] Dynamic mitigations

[Josh Poimboeuf]

On Mon, Oct 13, 2025 at 09:33:48AM -0500, David Kaplan wrote:
> Dynamic mitigations enables changing the kernel CPU security mitigations at
> runtime without a reboot/kexec.
> 
> Previously, mitigation choices had to be made on the kernel cmdline.  With
> this feature an administrator can select new mitigation choices by writing
> a sysfs file, after which the kernel will re-patch itself based on the new
> mitigations.
> 
> As the performance cost of CPU mitigations can be significant, selecting
> the right set of mitigations is important to achieve the correct balance of
> performance/security.
> 
> Use
> ---
> As described in the supplied documentation file, new mitigations are
> selected by writing cmdline options to a new sysfs file.  Only cmdline
> options related to mitigations are recognized via this interface.  All
> previous mitigation-related cmdline options are ignored and selections are
> done based on the new options.
> 
> Examples:
>    echo "mitigations=off" > /sys/devices/system/cpu/mitigations
>    echo "spectre_v2=retpoline tsa=off" > /sys/devices/system/cpu/mitigations
> 
> 
> There are several use cases that will benefit from dynamic mitigations:
> 
> Use Cases
> ---------
> 1. Runtime Policy
> 
> Some workflows rely on booting a generic kernel before customizing the system.
> cloud-init is a popular example of this where a VM is started typically with
> default settings and then is customized based on a customer-provided
> configuration file.

I'm not really a fan of this.  It adds complexity to some areas that are
already struggling with too much complexity.

IMO this would need some REALLY strong justification, more than just
"hey, this makes things more convenient."

The mitigations should be a "set it and forget it" thing.  I don't see
anything here which justifies the considerable maintenance burden this
would add for all existing and future mitigations.

-- 
Josh

RE: [RFC PATCH 00/56] Dynamic mitigations

[Kaplan, David]

[AMD Official Use Only - AMD Internal Distribution Only]

> -----Original Message-----
> From: Josh Poimboeuf <jpoimboe@kernel.org>
> Sent: Tuesday, October 14, 2025 11:29 AM
> To: Kaplan, David <David.Kaplan@amd.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>; Borislav Petkov <bp@alien8.de>; Peter
> Zijlstra <peterz@infradead.org>; Pawan Gupta
> <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar <mingo@redhat.com>; Dave
> Hansen <dave.hansen@linux.intel.com>; x86@kernel.org; H . Peter Anvin
> <hpa@zytor.com>; Alexander Graf <graf@amazon.com>; Boris Ostrovsky
> <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
> Subject: Re: [RFC PATCH 00/56] Dynamic mitigations
>
> Caution: This message originated from an External Source. Use proper caution
> when opening attachments, clicking links, or responding.
>
>
> On Mon, Oct 13, 2025 at 09:33:48AM -0500, David Kaplan wrote:
> > Dynamic mitigations enables changing the kernel CPU security mitigations at
> > runtime without a reboot/kexec.
> >
> > Previously, mitigation choices had to be made on the kernel cmdline.  With
> > this feature an administrator can select new mitigation choices by writing
> > a sysfs file, after which the kernel will re-patch itself based on the new
> > mitigations.
> >
> > As the performance cost of CPU mitigations can be significant, selecting
> > the right set of mitigations is important to achieve the correct balance of
> > performance/security.
> >
> > Use
> > ---
> > As described in the supplied documentation file, new mitigations are
> > selected by writing cmdline options to a new sysfs file.  Only cmdline
> > options related to mitigations are recognized via this interface.  All
> > previous mitigation-related cmdline options are ignored and selections are
> > done based on the new options.
> >
> > Examples:
> >    echo "mitigations=off" > /sys/devices/system/cpu/mitigations
> >    echo "spectre_v2=retpoline tsa=off" > /sys/devices/system/cpu/mitigations
> >
> >
> > There are several use cases that will benefit from dynamic mitigations:
> >
> > Use Cases
> > ---------
> > 1. Runtime Policy
> >
> > Some workflows rely on booting a generic kernel before customizing the system.
> > cloud-init is a popular example of this where a VM is started typically with
> > default settings and then is customized based on a customer-provided
> > configuration file.
>
> I'm not really a fan of this.  It adds complexity to some areas that are
> already struggling with too much complexity.
>
> IMO this would need some REALLY strong justification, more than just
> "hey, this makes things more convenient."
>
> The mitigations should be a "set it and forget it" thing.  I don't see
> anything here which justifies the considerable maintenance burden this
> would add for all existing and future mitigations.
>

The problem is there are environments like the one outlined where you can't just 'set it and forget it' because the kernel needs it set at boot-time, but in these environments you don't know how to configure the system until much later in boot.  So you end up running with the default settings all the time, even if you don't need them.  And the default settings can have significant performance impacts in many cases.

The cloud guys on this thread may be able to offer some additional color here since I believe that's where you're most likely to have this situation.

--David Kaplan

Re: [RFC PATCH 04/56] x86/bugs: Reset spectre_v1 mitigations

[Dave Hansen]

On 10/13/25 07:33, David Kaplan wrote:
...
> +#ifdef CONFIG_DYNAMIC_MITIGATIONS
> +static void spectre_v1_reset_mitigation(void)
> +{
> +	setup_clear_cpu_cap(X86_FEATURE_FENCE_SWAPGS_USER);
> +	setup_clear_cpu_cap(X86_FEATURE_FENCE_SWAPGS_KERNEL);
> +	spectre_v1_mitigation = SPECTRE_V1_MITIGATION_AUTO;
> +}
> +#endif
> +
>  static int __init nospectre_v1_cmdline(char *str)
>  {
>  	spectre_v1_mitigation = SPECTRE_V1_MITIGATION_NONE;
> @@ -3794,3 +3805,10 @@ void __warn_thunk(void)
>  {
>  	WARN_ONCE(1, "Unpatched return thunk in use. This should not happen!\n");
>  }
> +
> +#ifdef CONFIG_DYNAMIC_MITIGATIONS
> +void arch_cpu_reset_mitigations(void)
> +{
> +	spectre_v1_reset_mitigation();
> +}
> +#endif

Are all the #ifdefs necessary? For instance, a single:

 void arch_cpu_reset_mitigations(void)
 {
+	if (!IS_ENABLED(CONFIG_DYNAMIC_MITIGATIONS))
+		return;
...

and removing *all* the #ifdefs might be enough to tell the compiler that
all the *_reset_mitigation() functions are unreachable.

But the larger concern through all of this is that this is all *new*
code. The reset concept is completely new and the reset functions look
to be ad-hoc. They follow a few patterns, but nothing super consistent.

Also, this series is going to need to be broken up. I'd suggest going
after the dynamic alternatives, first. I'm sure there's a need for that
_somewhere_ other than for vulnerabilities.

But, in the end, (IMNHO) this series really just adds complexity. It
doesn't do enough refactoring or complexity reduction to go in with this
approach.

RE: [RFC PATCH 04/56] x86/bugs: Reset spectre_v1 mitigations

[Kaplan, David]

[AMD Official Use Only - AMD Internal Distribution Only]

> -----Original Message-----
> From: Dave Hansen <dave.hansen@intel.com>
> Sent: Tuesday, October 14, 2025 1:38 PM
> To: Kaplan, David <David.Kaplan@amd.com>; Thomas Gleixner
> <tglx@linutronix.de>; Borislav Petkov <bp@alien8.de>; Peter Zijlstra
> <peterz@infradead.org>; Josh Poimboeuf <jpoimboe@kernel.org>; Pawan Gupta
> <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar <mingo@redhat.com>; Dave
> Hansen <dave.hansen@linux.intel.com>; x86@kernel.org; H . Peter Anvin
> <hpa@zytor.com>
> Cc: Alexander Graf <graf@amazon.com>; Boris Ostrovsky
> <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
> Subject: Re: [RFC PATCH 04/56] x86/bugs: Reset spectre_v1 mitigations
>
> Caution: This message originated from an External Source. Use proper caution
> when opening attachments, clicking links, or responding.
>
>
> On 10/13/25 07:33, David Kaplan wrote:
> ...
> > +#ifdef CONFIG_DYNAMIC_MITIGATIONS
> > +static void spectre_v1_reset_mitigation(void)
> > +{
> > +     setup_clear_cpu_cap(X86_FEATURE_FENCE_SWAPGS_USER);
> > +     setup_clear_cpu_cap(X86_FEATURE_FENCE_SWAPGS_KERNEL);
> > +     spectre_v1_mitigation = SPECTRE_V1_MITIGATION_AUTO;
> > +}
> > +#endif
> > +
> >  static int __init nospectre_v1_cmdline(char *str)
> >  {
> >       spectre_v1_mitigation = SPECTRE_V1_MITIGATION_NONE;
> > @@ -3794,3 +3805,10 @@ void __warn_thunk(void)
> >  {
> >       WARN_ONCE(1, "Unpatched return thunk in use. This should not
> happen!\n");
> >  }
> > +
> > +#ifdef CONFIG_DYNAMIC_MITIGATIONS
> > +void arch_cpu_reset_mitigations(void)
> > +{
> > +     spectre_v1_reset_mitigation();
> > +}
> > +#endif
>
> Are all the #ifdefs necessary? For instance, a single:
>
>  void arch_cpu_reset_mitigations(void)
>  {
> +       if (!IS_ENABLED(CONFIG_DYNAMIC_MITIGATIONS))
> +               return;
> ...
>
> and removing *all* the #ifdefs might be enough to tell the compiler that
> all the *_reset_mitigation() functions are unreachable.

Oh, hmm yeah that could be worth trying instead.

>
> But the larger concern through all of this is that this is all *new*
> code. The reset concept is completely new and the reset functions look
> to be ad-hoc. They follow a few patterns, but nothing super consistent.

The reset logic is basically 'undo'ing the apply_mitigation() functions.  So any feature flag, static branch, etc. that may get touched in the apply function are reset in the reset function.

That's not the only way this could be structured...there could be a single function that resets any and all mitigation-related things used by any of the mitigation (there is certainly overlap in many of them).  When writing it, I found the chosen approach to be pretty straightforward because it was easy enough to model the reset function after the apply function (and that's why it typically follows the apply function in the file).

>
> Also, this series is going to need to be broken up. I'd suggest going
> after the dynamic alternatives, first. I'm sure there's a need for that
> _somewhere_ other than for vulnerabilities.

Ok.  I'm certainly open to ideas of what else could benefit from this type of functionality.

I think to properly re-patch vulnerabilities, we need all 3 things (alternatives, returns, indirects).  We don't want to create some intermediate thing where the kernel is partially patched in one way and partially patched in another way.  Part of my goal here was that after re-patching, things should look like you had originally booted under those same options.

But if there was another use case for say just re-patching alternatives, that support could presumably be eventually leveraged to patch mitigations.

>
> But, in the end, (IMNHO) this series really just adds complexity. It
> doesn't do enough refactoring or complexity reduction to go in with this
> approach.

Fwiw, this series does actually help a lot with mitigation testing.  In fact, as part of the development I wrote a fuzzer to test various combinations of X86_BUG bits and command line options.  That found several bugs in upstream in fact (which I submitted separate patches for).  Given the number of different pieces of hardware that exist, various cmdline options (and their interactions), I see being able to better test the mitigation logic as a positive thing.  Obviously that must be balanced against the overall complexity, but I think it's worth noting that there may be some maintainability benefits too of runtime re-configuration.

--David Kaplan

Re: [RFC PATCH 00/56] Dynamic mitigations

[Aaron Rainbolt]

[-- Attachment #1: Type: text/plain, Size: 12282 bytes --]

On Mon, 13 Oct 2025 09:33:48 -0500
David Kaplan <david.kaplan@amd.com> wrote:

> Dynamic mitigations enables changing the kernel CPU security
> mitigations at runtime without a reboot/kexec.
> 
> Previously, mitigation choices had to be made on the kernel cmdline.
> With this feature an administrator can select new mitigation choices
> by writing a sysfs file, after which the kernel will re-patch itself
> based on the new mitigations.
> 
> As the performance cost of CPU mitigations can be significant,
> selecting the right set of mitigations is important to achieve the
> correct balance of performance/security.
> 
> Use
> ---
> As described in the supplied documentation file, new mitigations are
> selected by writing cmdline options to a new sysfs file.  Only cmdline
> options related to mitigations are recognized via this interface.  All
> previous mitigation-related cmdline options are ignored and
> selections are done based on the new options.
> 
> Examples:
>    echo "mitigations=off" > /sys/devices/system/cpu/mitigations
>    echo "spectre_v2=retpoline tsa=off" >
> /sys/devices/system/cpu/mitigations

If `root` is capable of setting `mitigations=off` via this interface,
doesn't that somewhat defeat the purpose of denying `/proc/kcore`
access in lockdown confidentiality mode? Assuming one is running on a
CPU with some form of side-channel memory read vulnerability (which they
very likely are), they can turn off all mitigations, then read kernel
memory via one of those exploits.

There should be a one-way switch to allow denying all further writes to
this interface, so that once the system's mitigations are set properly,
any further attempts to change them until the next reboot can be
prevented.

--
Aaron

> 
> There are several use cases that will benefit from dynamic
> mitigations:
> 
> Use Cases
> ---------
> 1. Runtime Policy
> 
> Some workflows rely on booting a generic kernel before customizing
> the system. cloud-init is a popular example of this where a VM is
> started typically with default settings and then is customized based
> on a customer-provided configuration file.
> 
> As flows like this rely on configuring the system after boot, they
> currently cannot customize the mitigation policy.  With dynamic
> mitigations, this configuration information can be augmented to
> include security policy information.
> 
> For example, a cloud VM which runs only trusted workloads likely does
> not need any CPU security mitigations applied.  But as this policy
> information is not known at boot time, the kernel will be booted with
> unnecessary mitigations enabled.  With dynamic mitigations, these
> mitigations can be disabled during boot after policy information is
> retrieved, improving performance.
> 
> 2. Mitigation Changes
> 
> Sometimes there are needs to change the mitigation settings in light
> of new security findings.  For example, AMD-SB-1036 advised of a
> security issue with a spectre v2 mitigation and advised using a
> different one instead.
> 
> With dynamic mitigations, such changes can be made without a
> reboot/kexec which minimizes disruption in environments which cannot
> easily tolerate such an event.
> 
> 3. Mitigation Testing
> 
> Being able to quickly change between different mitigation settings
> without having to restart applications is beneficial when conducting
> mitigation development and testing.
> 
> Note that some bugs have multiple mitigation options, which may have
> varying performance impacts.  Being able to quickly switch between
> them makes evaluating such options easier.
> 
> 
> Implementation Details
> ----------------------
> Re-patching the kernel is expected to be a very rare operation and is
> done under very big hammers.  All tasks are put into the freezer and
> the re-patching is then done under the (new) stop_machine_nmi()
> routine.
> 
> To re-patch the kernel, it is first reverted back to its compile-time
> state.  The original bytes from alternatives, retpolines, etc. are
> saved during boot so they can later be used to restore the original
> kernel image. After that, the kernel is patched based on the new
> feature flags.
> 
> This simplifies the re-patch process as restoring the original kernel
> image is relatively straightforward.  In other words, instead of
> having to re-patch from mitigation A to mitigation B directly, we
> first restore the original image and then patch from that to
> mitigation B, similar to if the system had booted with mitigation B
> selected originally.
> 
> 
> Performance
> -----------
> Testing so far has demonstrated that re-patching takes ~50ms on an
> AMD EPYC 7713 running a typical Ubuntu kernel with around 100 modules
> loaded.
> 
> Guide to Patch Series
> ---------------------
> As this series is rather lengthy, this may help with understanding it:
> 
> Patches 3-18 focus on "resetting" mitigations.  Every bug that may
> set feature flags, MSRs, static branches, etc. now has matching
> "reset" functions that will undo all these changes.  This is used at
> the beginning of the re-patch flow.
> 
> Patches 20-22 move various functions and values out of the .init
> section.  Most of the existing mitigation logic was marked as __init
> and the mitigation settings as __ro_after_init but now these can be
> changed at runtime.  The __ro_after_init marking functioned as a
> defense-in-depth measure but is arguably of limited meaningful
> security value as an attacker who can modify kernel data can do a lot
> worse than change some speculation settings.  As re-patching requires
> being able to modify these settings, it was simplest to remove them
> from that section.
> 
> Patches 23-27 involve linker and related modifications to keep
> alternative information around at runtime instead of free'ing it
> after boot.  This does result in slightly higher runtime memory
> consumption which is one reason why this feature is behind a Kconfig
> option.  On a typical kernel, this was measured at around 2MB of
> extra kernel memory usage.
> 
> Patches 28-30 focus on the new stop_machine_nmi() which behaves like
> stop_machine() but runs the handler in NMI context, thus ensuring
> that even NMIs cannot interrupt the handler.  As dynamic mitigations
> involves re-patching functions used by NMI entry code, this is
> required for safety.
> 
> Patches 31-40 focus on support for restoring the kernel text at
> runtime.  This involves saving the original kernel bytes when patched
> the first time and adding support to then restore those later.
> 
> Patches 41-44 start building support for updating code, in particular
> module code at runtime.
> 
> Patches 45-47 focus on support for the Indirect Target Selection
> mitigation which is particularly challenging because it requires
> runtime memory allocations and permission changes which are not
> possible in NMI context.  As a result, ITS memory is pre-allocated
> before entering NMI context.
> 
> Patch 50 adds the complete function for resetting and re-patching the
> kernel.
> 
> Patches 51-53 build the sysfs interface for re-patching and support
> for parsing the new options provided.
> 
> Patches 54-56 add debugfs interfaces to values which are important for
> mitigations.  These are useful for userspace test utilities to be
> able to force a CPU to appear to be vulnerable or immune to certain
> bugs as well as being able to help verify if the kernel is correctly
> mitigating various vulnerabilities.
> 
> David Kaplan (56):
>   Documentation/admin-guide: Add documentation
>   x86/Kconfig: Add CONFIG_DYNAMIC_MITIGATIONS
>   cpu: Reset global mitigations
>   x86/bugs: Reset spectre_v1 mitigations
>   x86/bugs: Reset spectre_v2 mitigations
>   x86/bugs: Reset retbleed mitigations
>   x86/bugs: Reset spectre_v2_user mitigations
>   x86/bugs: Reset SSB mitigations
>   x86/bugs: Reset L1TF mitigations
>   x86/bugs: Reset MDS mitigations
>   x86/bugs: Reset MMIO mitigations
>   x86/bugs: Reset SRBDS mitigations
>   x86/bugs: Reset SRSO mitigations
>   x86/bugs: Reset GDS mitigations
>   x86/bugs: Reset BHI mitigations
>   x86/bugs: Reset ITS mitigation
>   x86/bugs: Reset TSA mitigations
>   x86/bugs: Reset VMSCAPE mitigations
>   x86/bugs: Define bugs_smt_disable()
>   x86/bugs: Move bugs.c logic out of .init section
>   x86/callthunks: Move logic out of .init
>   cpu: Move mitigation logic out of .init
>   x86/vmlinux.lds: Move alternative sections
>   x86/vmlinux.lds: Move altinstr_aux conditionally
>   x86/vmlinux.lds: Define __init_alt_end
>   module: Save module ELF info
>   x86/mm: Conditionally free alternative sections
>   stop_machine: Add stop_machine_nmi()
>   x86/apic: Add self-NMI support
>   x86/nmi: Add support for stop_machine_nmi()
>   x86/alternative: Prepend nops with retpolines
>   x86/alternative: Add module param
>   x86/alternative: Avoid re-patching init code
>   x86/alternative: Save old bytes for alternatives
>   x86/alternative: Save old bytes for retpolines
>   x86/alternative: Do not recompute len on re-patch
>   x86/alternative: Reset alternatives
>   x86/callthunks: Reset callthunks
>   x86/sync_core: Add sync_core_nmi_safe()
>   x86/alternative: Use sync_core_nmi_safe()
>   static_call: Add update_all_static_calls()
>   module: Make memory writeable for re-patching
>   module: Update alternatives
>   x86/module: Update alternatives
>   x86/alternative: Use boot_cpu_has in ITS code
>   x86/alternative: Add ITS re-patching support
>   x86/module: Add ITS re-patch support for modules
>   x86/bugs: Move code for updating speculation MSRs
>   x86/fpu: Qualify warning in os_xsave
>   x86/alternative: Add re-patch support
>   cpu: Parse string of mitigation options
>   x86/bugs: Support parsing mitigation options
>   drivers/cpu: Re-patch mitigations through sysfs
>   x86/debug: Create debugfs interface to x86_capabilities
>   x86/debug: Show return thunk in debugfs
>   x86/debug: Show static branch config in debugfs
> 
>  .../ABI/testing/sysfs-devices-system-cpu      |   8 +
>  .../hw-vuln/dynamic_mitigations.rst           |  75 ++
>  Documentation/admin-guide/hw-vuln/index.rst   |   1 +
>  arch/x86/Kconfig                              |  12 +
>  arch/x86/entry/vdso/vma.c                     |   2 +-
>  arch/x86/include/asm/alternative.h            |  51 +-
>  arch/x86/include/asm/bugs.h                   |   4 +
>  arch/x86/include/asm/module.h                 |  10 +
>  arch/x86/include/asm/sync_core.h              |  14 +
>  arch/x86/kernel/alternative.c                 | 497 ++++++++++++-
>  arch/x86/kernel/apic/ipi.c                    |   7 +
>  arch/x86/kernel/callthunks.c                  |  85 ++-
>  arch/x86/kernel/cpu/bugs.c                    | 686
> +++++++++++++----- arch/x86/kernel/cpu/common.c                  |
> 65 +- arch/x86/kernel/cpu/cpu.h                     |   4 -
>  arch/x86/kernel/fpu/xstate.h                  |   2 +-
>  arch/x86/kernel/module.c                      |  96 ++-
>  arch/x86/kernel/nmi.c                         |   4 +
>  arch/x86/kernel/static_call.c                 |   3 +-
>  arch/x86/kernel/vmlinux.lds.S                 | 110 +--
>  arch/x86/mm/init.c                            |  12 +-
>  arch/x86/mm/mm_internal.h                     |   2 +
>  arch/x86/tools/relocs.c                       |   1 +
>  drivers/base/cpu.c                            | 113 +++
>  include/linux/cpu.h                           |  10 +
>  include/linux/module.h                        |  11 +
>  include/linux/static_call.h                   |   2 +
>  include/linux/stop_machine.h                  |  32 +
>  kernel/cpu.c                                  |  62 +-
>  kernel/module/main.c                          |  78 +-
>  kernel/static_call_inline.c                   |  22 +
>  kernel/stop_machine.c                         |  79 +-
>  32 files changed, 1876 insertions(+), 284 deletions(-)
>  create mode 100644
> Documentation/admin-guide/hw-vuln/dynamic_mitigations.rst
> 
> 
> base-commit: a5652f0f2a69fadcfb2f687a11a737a57f15b28e


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [RFC PATCH 00/56] Dynamic mitigations

[Alexander Graf]

On 14.10.25 20:06, Kaplan, David wrote:
> [AMD Official Use Only - AMD Internal Distribution Only]
>
>> -----Original Message-----
>> From: Josh Poimboeuf <jpoimboe@kernel.org>
>> Sent: Tuesday, October 14, 2025 11:29 AM
>> To: Kaplan, David <David.Kaplan@amd.com>
>> Cc: Thomas Gleixner <tglx@linutronix.de>; Borislav Petkov <bp@alien8.de>; Peter
>> Zijlstra <peterz@infradead.org>; Pawan Gupta
>> <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar <mingo@redhat.com>; Dave
>> Hansen <dave.hansen@linux.intel.com>; x86@kernel.org; H . Peter Anvin
>> <hpa@zytor.com>; Alexander Graf <graf@amazon.com>; Boris Ostrovsky
>> <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
>> Subject: Re: [RFC PATCH 00/56] Dynamic mitigations
>>
>> Caution: This message originated from an External Source. Use proper caution
>> when opening attachments, clicking links, or responding.
>>
>>
>> On Mon, Oct 13, 2025 at 09:33:48AM -0500, David Kaplan wrote:
>>> Dynamic mitigations enables changing the kernel CPU security mitigations at
>>> runtime without a reboot/kexec.
>>>
>>> Previously, mitigation choices had to be made on the kernel cmdline.  With
>>> this feature an administrator can select new mitigation choices by writing
>>> a sysfs file, after which the kernel will re-patch itself based on the new
>>> mitigations.
>>>
>>> As the performance cost of CPU mitigations can be significant, selecting
>>> the right set of mitigations is important to achieve the correct balance of
>>> performance/security.
>>>
>>> Use
>>> ---
>>> As described in the supplied documentation file, new mitigations are
>>> selected by writing cmdline options to a new sysfs file.  Only cmdline
>>> options related to mitigations are recognized via this interface.  All
>>> previous mitigation-related cmdline options are ignored and selections are
>>> done based on the new options.
>>>
>>> Examples:
>>>     echo "mitigations=off" > /sys/devices/system/cpu/mitigations
>>>     echo "spectre_v2=retpoline tsa=off" > /sys/devices/system/cpu/mitigations
>>>
>>>
>>> There are several use cases that will benefit from dynamic mitigations:
>>>
>>> Use Cases
>>> ---------
>>> 1. Runtime Policy
>>>
>>> Some workflows rely on booting a generic kernel before customizing the system.
>>> cloud-init is a popular example of this where a VM is started typically with
>>> default settings and then is customized based on a customer-provided
>>> configuration file.
>> I'm not really a fan of this.  It adds complexity to some areas that are
>> already struggling with too much complexity.
>>
>> IMO this would need some REALLY strong justification, more than just
>> "hey, this makes things more convenient."
>>
>> The mitigations should be a "set it and forget it" thing.  I don't see
>> anything here which justifies the considerable maintenance burden this
>> would add for all existing and future mitigations.
>>
> The problem is there are environments like the one outlined where you can't just 'set it and forget it' because the kernel needs it set at boot-time, but in these environments you don't know how to configure the system until much later in boot.  So you end up running with the default settings all the time, even if you don't need them.  And the default settings can have significant performance impacts in many cases.
>
> The cloud guys on this thread may be able to offer some additional color here since I believe that's where you're most likely to have this situation.


The crux of the problem here is that the kernel command line is 
difficult to influence in most cloud environments.

In the cloud, you typically start from a generic base image which then 
boots, talks to a configuration mechanism (IMDS in EC2) which then 
contains all of the actual customization. For most customization, that 
is perfectly fine: You can install packages, run scripts, launch 
services, etc. But there is no simple way to modify the kernel command 
line. The story gets even worse when you try to abstract the cloud 
environment itself by using configuration layers on top like puppet, 
ansible, salt, etc. because you could not do a pre-boot environment hack 
even if you wanted to.

Users could in theory have a bootup script which checks the current 
command line, modifies the boot loader configuration, regenerates boot 
loader config files (If they even can. Signed UKIs make that difficult), 
and then reboot/kexec into the new environment. So we would punt a *lot* 
of complexity onto users and still degrade their experience by 
prolonging the launch phase.

I'm all ears for alternatives, but runtime setting seems like the most 
natural way to allow bootup / configuration scripts to actually instill 
policy.


Alex




Amazon Web Services Development Center Germany GmbH
Tamara-Danz-Str. 13
10243 Berlin
Geschaeftsfuehrung: Christian Schlaeger
Eingetragen am Amtsgericht Charlottenburg unter HRB 257764 B
Sitz: Berlin
Ust-ID: DE 365 538 597

Re: [RFC PATCH 34/56] x86/alternative: Save old bytes for alternatives

[Juergen Gross]

[-- Attachment #1.1.1: Type: text/plain, Size: 658 bytes --]

On 13.10.25 16:34, David Kaplan wrote:
> Save the existing instruction bytes at each alternative site when patching.
> This is only done the first time, and these will be used later to help
> restore the code back to its original form.
> 
> Signed-off-by: David Kaplan <david.kaplan@amd.com>

Instead of saving the original instructions at runtime, why don't you
expand struct alt_instr to have an additional offset to a saved copy
of the original instruction, located in .altinstr_replacement?

The new field should be guarded with #ifdef CONFIG_DYNAMIC_MITIGATIONS,
of course, like the added handling in the ALTERNATIVE() macros.


Juergen

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3743 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

RE: [RFC PATCH 34/56] x86/alternative: Save old bytes for alternatives

[Kaplan, David]

[AMD Official Use Only - AMD Internal Distribution Only]

> -----Original Message-----
> From: Juergen Gross <jgross@suse.com>
> Sent: Wednesday, October 15, 2025 5:38 AM
> To: Kaplan, David <David.Kaplan@amd.com>; Thomas Gleixner
> <tglx@linutronix.de>; Borislav Petkov <bp@alien8.de>; Peter Zijlstra
> <peterz@infradead.org>; Josh Poimboeuf <jpoimboe@kernel.org>; Pawan Gupta
> <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar <mingo@redhat.com>; Dave
> Hansen <dave.hansen@linux.intel.com>; x86@kernel.org; H . Peter Anvin
> <hpa@zytor.com>
> Cc: Alexander Graf <graf@amazon.com>; Boris Ostrovsky
> <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
> Subject: Re: [RFC PATCH 34/56] x86/alternative: Save old bytes for alternatives
>
> On 13.10.25 16:34, David Kaplan wrote:
> > Save the existing instruction bytes at each alternative site when patching.
> > This is only done the first time, and these will be used later to help
> > restore the code back to its original form.
> >
> > Signed-off-by: David Kaplan <david.kaplan@amd.com>
>
> Instead of saving the original instructions at runtime, why don't you
> expand struct alt_instr to have an additional offset to a saved copy
> of the original instruction, located in .altinstr_replacement?
>
> The new field should be guarded with #ifdef CONFIG_DYNAMIC_MITIGATIONS,
> of course, like the added handling in the ALTERNATIVE() macros.
>

That's an interesting idea, I think that could work.  That would make the kernel image on disk (slightly) larger though, as the original bytes will essentially be duplicated (at the original location and in .altinstr_replacement).  I'm not sure which is the better trade-off (kernel image bytes on disk vs runtime memory usage).  Although I think we're talking about a relatively small amount of memory regardless.  Most of the runtime overhead of dynamic mitigations comes from remembering the call sites/returns.

Thanks
--David Kaplan

RE: [RFC PATCH 00/56] Dynamic mitigations

[Kaplan, David]

[AMD Official Use Only - AMD Internal Distribution Only]

> -----Original Message-----
> From: Aaron Rainbolt <arraybolt3@gmail.com>
> Sent: Tuesday, October 14, 2025 11:11 PM
> To: Kaplan, David <David.Kaplan@amd.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>; Borislav Petkov <bp@alien8.de>; Peter
> Zijlstra <peterz@infradead.org>; Josh Poimboeuf <jpoimboe@kernel.org>; Pawan
> Gupta <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar <mingo@redhat.com>;
> Dave Hansen <dave.hansen@linux.intel.com>; x86@kernel.org; H . Peter Anvin
> <hpa@zytor.com>; Alexander Graf <graf@amazon.com>; Boris Ostrovsky
> <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
> Subject: Re: [RFC PATCH 00/56] Dynamic mitigations
>
> On Mon, 13 Oct 2025 09:33:48 -0500
> David Kaplan <david.kaplan@amd.com> wrote:
>
> > Dynamic mitigations enables changing the kernel CPU security
> > mitigations at runtime without a reboot/kexec.
> >
> > Previously, mitigation choices had to be made on the kernel cmdline.
> > With this feature an administrator can select new mitigation choices
> > by writing a sysfs file, after which the kernel will re-patch itself
> > based on the new mitigations.
> >
> > As the performance cost of CPU mitigations can be significant,
> > selecting the right set of mitigations is important to achieve the
> > correct balance of performance/security.
> >
> > Use
> > ---
> > As described in the supplied documentation file, new mitigations are
> > selected by writing cmdline options to a new sysfs file.  Only cmdline
> > options related to mitigations are recognized via this interface.  All
> > previous mitigation-related cmdline options are ignored and
> > selections are done based on the new options.
> >
> > Examples:
> >    echo "mitigations=off" > /sys/devices/system/cpu/mitigations
> >    echo "spectre_v2=retpoline tsa=off" >
> > /sys/devices/system/cpu/mitigations
>
> If `root` is capable of setting `mitigations=off` via this interface,
> doesn't that somewhat defeat the purpose of denying `/proc/kcore`
> access in lockdown confidentiality mode? Assuming one is running on a
> CPU with some form of side-channel memory read vulnerability (which they
> very likely are), they can turn off all mitigations, then read kernel
> memory via one of those exploits.
>
> There should be a one-way switch to allow denying all further writes to
> this interface, so that once the system's mitigations are set properly,
> any further attempts to change them until the next reboot can be
> prevented.
>

That's a good idea, there could be a separate mitigation_lock file perhaps that once written to 1 denies any further changes.

Thanks --David Kaplan

Re: [RFC PATCH 00/56] Dynamic mitigations

[Josh Poimboeuf]

On Wed, Oct 15, 2025 at 01:53:31PM +0000, Kaplan, David wrote:
> > If `root` is capable of setting `mitigations=off` via this interface,
> > doesn't that somewhat defeat the purpose of denying `/proc/kcore`
> > access in lockdown confidentiality mode? Assuming one is running on a
> > CPU with some form of side-channel memory read vulnerability (which they
> > very likely are), they can turn off all mitigations, then read kernel
> > memory via one of those exploits.
> >
> > There should be a one-way switch to allow denying all further writes to
> > this interface, so that once the system's mitigations are set properly,
> > any further attempts to change them until the next reboot can be
> > prevented.
> >
> 
> That's a good idea, there could be a separate mitigation_lock file
> perhaps that once written to 1 denies any further changes.

Wouldn't the enablement of lockdown mode effectively function as that
one way switch?

-- 
Josh

RE: [RFC PATCH 00/56] Dynamic mitigations

[Kaplan, David]

[AMD Official Use Only - AMD Internal Distribution Only]

> -----Original Message-----
> From: Josh Poimboeuf <jpoimboe@kernel.org>
> Sent: Wednesday, October 15, 2025 10:43 AM
> To: Kaplan, David <David.Kaplan@amd.com>
> Cc: Aaron Rainbolt <arraybolt3@gmail.com>; Thomas Gleixner
> <tglx@linutronix.de>; Borislav Petkov <bp@alien8.de>; Peter Zijlstra
> <peterz@infradead.org>; Pawan Gupta <pawan.kumar.gupta@linux.intel.com>;
> Ingo Molnar <mingo@redhat.com>; Dave Hansen <dave.hansen@linux.intel.com>;
> x86@kernel.org; H . Peter Anvin <hpa@zytor.com>; Alexander Graf
> <graf@amazon.com>; Boris Ostrovsky <boris.ostrovsky@oracle.com>; linux-
> kernel@vger.kernel.org
> Subject: Re: [RFC PATCH 00/56] Dynamic mitigations
>
> Caution: This message originated from an External Source. Use proper caution
> when opening attachments, clicking links, or responding.
>
>
> On Wed, Oct 15, 2025 at 01:53:31PM +0000, Kaplan, David wrote:
> > > If `root` is capable of setting `mitigations=off` via this interface,
> > > doesn't that somewhat defeat the purpose of denying `/proc/kcore`
> > > access in lockdown confidentiality mode? Assuming one is running on a
> > > CPU with some form of side-channel memory read vulnerability (which they
> > > very likely are), they can turn off all mitigations, then read kernel
> > > memory via one of those exploits.
> > >
> > > There should be a one-way switch to allow denying all further writes to
> > > this interface, so that once the system's mitigations are set properly,
> > > any further attempts to change them until the next reboot can be
> > > prevented.
> > >
> >
> > That's a good idea, there could be a separate mitigation_lock file
> > perhaps that once written to 1 denies any further changes.
>
> Wouldn't the enablement of lockdown mode effectively function as that
> one way switch?
>

I'm not too familiar with lockdown mode, but that gets enabled (with right cmdline options) during boot right?  I guess the question is would we want to allow any window for userspace to reconfigure things and then lock things down, or say that if you enable lockdown then this interface is completely disabled and you need to specify your mitigation options on the cmdline only.

--David Kaplan

Re: [RFC PATCH 00/56] Dynamic mitigations

[Josh Poimboeuf]

On Wed, Oct 15, 2025 at 03:51:01PM +0000, Kaplan, David wrote:
> > On Wed, Oct 15, 2025 at 01:53:31PM +0000, Kaplan, David wrote:
> > > > If `root` is capable of setting `mitigations=off` via this interface,
> > > > doesn't that somewhat defeat the purpose of denying `/proc/kcore`
> > > > access in lockdown confidentiality mode? Assuming one is running on a
> > > > CPU with some form of side-channel memory read vulnerability (which they
> > > > very likely are), they can turn off all mitigations, then read kernel
> > > > memory via one of those exploits.
> > > >
> > > > There should be a one-way switch to allow denying all further writes to
> > > > this interface, so that once the system's mitigations are set properly,
> > > > any further attempts to change them until the next reboot can be
> > > > prevented.
> > > >
> > >
> > > That's a good idea, there could be a separate mitigation_lock file
> > > perhaps that once written to 1 denies any further changes.
> >
> > Wouldn't the enablement of lockdown mode effectively function as that
> > one way switch?
> >
> 
> I'm not too familiar with lockdown mode, but that gets enabled (with
> right cmdline options) during boot right?  I guess the question is
> would we want to allow any window for userspace to reconfigure things
> and then lock things down, or say that if you enable lockdown then
> this interface is completely disabled and you need to specify your
> mitigation options on the cmdline only.

Yeah, I would say the latter, otherwise it defeats the point of lockdown
mode.  Note that lockdown mode can also be enabled at runtime.

-- 
Josh

RE: [RFC PATCH 00/56] Dynamic mitigations

[Kaplan, David]

[AMD Official Use Only - AMD Internal Distribution Only]

> -----Original Message-----
> From: Josh Poimboeuf <jpoimboe@kernel.org>
> Sent: Wednesday, October 15, 2025 11:02 AM
> To: Kaplan, David <David.Kaplan@amd.com>
> Cc: Aaron Rainbolt <arraybolt3@gmail.com>; Thomas Gleixner
> <tglx@linutronix.de>; Borislav Petkov <bp@alien8.de>; Peter Zijlstra
> <peterz@infradead.org>; Pawan Gupta <pawan.kumar.gupta@linux.intel.com>;
> Ingo Molnar <mingo@redhat.com>; Dave Hansen <dave.hansen@linux.intel.com>;
> x86@kernel.org; H . Peter Anvin <hpa@zytor.com>; Alexander Graf
> <graf@amazon.com>; Boris Ostrovsky <boris.ostrovsky@oracle.com>; linux-
> kernel@vger.kernel.org
> Subject: Re: [RFC PATCH 00/56] Dynamic mitigations
>
> Caution: This message originated from an External Source. Use proper caution
> when opening attachments, clicking links, or responding.
>
>
> On Wed, Oct 15, 2025 at 03:51:01PM +0000, Kaplan, David wrote:
> > > On Wed, Oct 15, 2025 at 01:53:31PM +0000, Kaplan, David wrote:
> > > > > If `root` is capable of setting `mitigations=off` via this interface,
> > > > > doesn't that somewhat defeat the purpose of denying `/proc/kcore`
> > > > > access in lockdown confidentiality mode? Assuming one is running on a
> > > > > CPU with some form of side-channel memory read vulnerability (which they
> > > > > very likely are), they can turn off all mitigations, then read kernel
> > > > > memory via one of those exploits.
> > > > >
> > > > > There should be a one-way switch to allow denying all further writes to
> > > > > this interface, so that once the system's mitigations are set properly,
> > > > > any further attempts to change them until the next reboot can be
> > > > > prevented.
> > > > >
> > > >
> > > > That's a good idea, there could be a separate mitigation_lock file
> > > > perhaps that once written to 1 denies any further changes.
> > >
> > > Wouldn't the enablement of lockdown mode effectively function as that
> > > one way switch?
> > >
> >
> > I'm not too familiar with lockdown mode, but that gets enabled (with
> > right cmdline options) during boot right?  I guess the question is
> > would we want to allow any window for userspace to reconfigure things
> > and then lock things down, or say that if you enable lockdown then
> > this interface is completely disabled and you need to specify your
> > mitigation options on the cmdline only.
>
> Yeah, I would say the latter, otherwise it defeats the point of lockdown
> mode.  Note that lockdown mode can also be enabled at runtime.
>

Ok.  So that's using the security_locked_down() API presumably.  And what reason would we want to check for?  Should it be LOCKDOWN_DEV_MEM (since we're potentially leaking arbitrary memory)?  Or a new lockdown reason?

--David Kaplan

Re: [RFC PATCH 00/56] Dynamic mitigations

[Boris Ostrovsky]

On 10/14/25 2:06 PM, Kaplan, David wrote:
> [AMD Official Use Only - AMD Internal Distribution Only]
> 
>> -----Original Message-----
>> From: Josh Poimboeuf <jpoimboe@kernel.org>
>> Sent: Tuesday, October 14, 2025 11:29 AM
>> To: Kaplan, David <David.Kaplan@amd.com>
>> Cc: Thomas Gleixner <tglx@linutronix.de>; Borislav Petkov <bp@alien8.de>; Peter
>> Zijlstra <peterz@infradead.org>; Pawan Gupta
>> <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar <mingo@redhat.com>; Dave
>> Hansen <dave.hansen@linux.intel.com>; x86@kernel.org; H . Peter Anvin
>> <hpa@zytor.com>; Alexander Graf <graf@amazon.com>; Boris Ostrovsky
>> <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
>> Subject: Re: [RFC PATCH 00/56] Dynamic mitigations


>>
>> I'm not really a fan of this.  It adds complexity to some areas that are
>> already struggling with too much complexity.
>>
>> IMO this would need some REALLY strong justification, more than just
>> "hey, this makes things more convenient."
>>
>> The mitigations should be a "set it and forget it" thing.  I don't see
>> anything here which justifies the considerable maintenance burden this
>> would add for all existing and future mitigations.
>>
> 
> The problem is there are environments like the one outlined where you can't just 'set it and forget it' because the kernel needs it set at boot-time, but in these environments you don't know how to configure the system until much later in boot.  So you end up running with the default settings all the time, even if you don't need them.  And the default settings can have significant performance impacts in many cases.
> 
> The cloud guys on this thread may be able to offer some additional color here since I believe that's where you're most likely to have this situation.


We've had quite a few cases where initially selected mitigation would 
turn out to be wrong --- either because we discovered that performance 
characteristics were not what we expected or that mitigation had some 
issues (i.e. bugs). Once could argue that this could be true of any code 
but this area has been particularly troublesome.

There are ways to get around this such as live migration or 
ksplice/kpatch but these are not always possible or practical.


-boris

Re: [RFC PATCH 00/56] Dynamic mitigations

[Nicolas Bouchinet]

Hi David,

> > > > On Wed, Oct 15, 2025 at 01:53:31PM +0000, Kaplan, David wrote:
> > > > > > If `root` is capable of setting `mitigations=off` via this interface,
> > > > > > doesn't that somewhat defeat the purpose of denying `/proc/kcore`
> > > > > > access in lockdown confidentiality mode? Assuming one is running on a
> > > > > > CPU with some form of side-channel memory read vulnerability (which they
> > > > > > very likely are), they can turn off all mitigations, then read kernel
> > > > > > memory via one of those exploits.
> > > > > >
> > > > > > There should be a one-way switch to allow denying all further writes to
> > > > > > this interface, so that once the system's mitigations are set properly,
> > > > > > any further attempts to change them until the next reboot can be
> > > > > > prevented.
> > > > > >
> > > > >
> > > > > That's a good idea, there could be a separate mitigation_lock file
> > > > > perhaps that once written to 1 denies any further changes.
> > > >
> > > > Wouldn't the enablement of lockdown mode effectively function as that
> > > > one way switch?
> > > >
> > >
> > > I'm not too familiar with lockdown mode, but that gets enabled (with
> > > right cmdline options) during boot right?  I guess the question is
> > > would we want to allow any window for userspace to reconfigure things
> > > and then lock things down, or say that if you enable lockdown then
> > > this interface is completely disabled and you need to specify your
> > > mitigation options on the cmdline only.
> >
> > Yeah, I would say the latter, otherwise it defeats the point of lockdown
> > mode.  Note that lockdown mode can also be enabled at runtime.
> >
> 
> Ok.  So that's using the security_locked_down() API presumably.  And what reason would we want to check for?  Should it be LOCKDOWN_DEV_MEM (since we're potentially leaking arbitrary memory)?  Or a new lockdown reason?
> 
> --David Kaplan

LOCKDOWN_DEV_MEM is an integrity reason and should not be used for this
scenario.
I'd rather like to add a new Lockdown reason in the confidentiality set,
maybe LOCKDOWN_CPU_MITIGATION ?

Thank's

Nicolas

Re: [RFC PATCH 31/56] x86/alternative: Prepend nops with retpolines

[Peter Zijlstra]

On Mon, Oct 13, 2025 at 09:34:19AM -0500, David Kaplan wrote:
> When patching retpolines, nops may be required for padding such as when
> turning a 5-byte direct call into a 2-byte indirect call.  Previously,
> these were appended at the end so the code becomes "call *reg;nop;nop;nop"
> for example.  This was fine because it's always going from a larger
> instruction to a smaller one.
> 
> But this is a problem if the sequence is transformed from a 2-byte indirect
> to the 5-byte direct call version at runtime because when the called
> function returns, it will be in the middle of the 5-byte call instruction.
> 
> To fix this, prepend the nops instead of appending them.  Consequently, the
> return site of the called function is always the same.
> 
> For indirect jmps this is potentially slightly less efficient compared to
> appending nops, but indirect jmps are so rare this hardly seems worth
> optimizing.

Durr.. 

So Retpoline sites can be 5 or 6 bytes, depending on the register. 

Also, I suppose at this point I would prefer prefix stuffing
over multiple instructions. The prefix stuffing ensures it stays a
single instruction.

Alas, some micro-archs have significant decode penalties for >3
prefixes, and filling 6 bytes will need 4 prefixes :-(

Re: [RFC PATCH 40/56] x86/alternative: Use sync_core_nmi_safe()

[Peter Zijlstra]

On Mon, Oct 13, 2025 at 09:34:28AM -0500, David Kaplan wrote:
> Re-patching is done under NMI context so the NMI-safe version of
> sync_core() must be used.
> 
> Signed-off-by: David Kaplan <david.kaplan@amd.com>
> ---
>  arch/x86/kernel/alternative.c | 6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c
> index b67116ae883c..2d48d750d4d9 100644
> --- a/arch/x86/kernel/alternative.c
> +++ b/arch/x86/kernel/alternative.c
> @@ -2585,7 +2585,11 @@ void __init_or_module text_poke_early(void *addr, const void *opcode,
>  	} else {
>  		local_irq_save(flags);
>  		memcpy(addr, opcode, len);
> -		sync_core();
> +		/* Re-patching occurs in NMI context so we can't do IRET. */
> +		if (repatch_in_progress)
> +			sync_core_nmi_safe();
> +		else
> +			sync_core();
>  		local_irq_restore(flags);
>  

Can we please keep this in sync_core()? Something like:

static __always_inline void sync_core(void)
{
	if (static_cpu_has(X86_FEATURE_SERIALIZE)) {
		serialize();
		return;
	}

+	if (repatch_in_progress) {
+		sync_core_nmi_safe();
+		return;
+	}
+
	iret_to_self();
}

That way all the modern stuff that has SERIALIZE will still use that.

Re: [RFC PATCH 31/56] x86/alternative: Prepend nops with retpolines

[Peter Zijlstra]

On Mon, Oct 13, 2025 at 09:34:19AM -0500, David Kaplan wrote:
> When patching retpolines, nops may be required for padding such as when
> turning a 5-byte direct call into a 2-byte indirect call.  Previously,
> these were appended at the end so the code becomes "call *reg;nop;nop;nop"
> for example.  This was fine because it's always going from a larger
> instruction to a smaller one.
> 
> But this is a problem if the sequence is transformed from a 2-byte indirect
> to the 5-byte direct call version at runtime because when the called
> function returns, it will be in the middle of the 5-byte call instruction.
> 
> To fix this, prepend the nops instead of appending them.  Consequently, the
> return site of the called function is always the same.
> 

So this results in:

NOP3; call *%r11

And you're saying a task can be on the other side of that call and then
return lines up. But what if the task is preempted right after that
NOP3?

Same for all the alternative patching; what ensures no task is currently
having a register state that is in the middle of things?

Re: [RFC PATCH 31/56] x86/alternative: Prepend nops with retpolines

[Peter Zijlstra]

On Thu, Oct 16, 2025 at 12:32:33PM +0200, Peter Zijlstra wrote:

> Alas, some micro-archs have significant decode penalties for >3
> prefixes, and filling 6 bytes will need 4 prefixes :-(

N/m this, because if it has 6 bytes, then the indirect call will need 3
bytes and we're back to needing 3 prefixes.

Re: [RFC PATCH 31/56] x86/alternative: Prepend nops with retpolines

[Peter Zijlstra]

On Thu, Oct 16, 2025 at 01:07:17PM +0200, Peter Zijlstra wrote:
> On Mon, Oct 13, 2025 at 09:34:19AM -0500, David Kaplan wrote:
> > When patching retpolines, nops may be required for padding such as when
> > turning a 5-byte direct call into a 2-byte indirect call.  Previously,
> > these were appended at the end so the code becomes "call *reg;nop;nop;nop"
> > for example.  This was fine because it's always going from a larger
> > instruction to a smaller one.
> > 
> > But this is a problem if the sequence is transformed from a 2-byte indirect
> > to the 5-byte direct call version at runtime because when the called
> > function returns, it will be in the middle of the 5-byte call instruction.
> > 
> > To fix this, prepend the nops instead of appending them.  Consequently, the
> > return site of the called function is always the same.
> > 
> 
> So this results in:
> 
> NOP3; call *%r11

Also possible:

  lfence; call *r11

(which is why we needed 6 bytes for reg>8)

> And you're saying a task can be on the other side of that call and then
> return lines up. But what if the task is preempted right after that
> NOP3?
> 
> Same for all the alternative patching; what ensures no task is currently
> having a register state that is in the middle of things?

Re: [RFC PATCH 31/56] x86/alternative: Prepend nops with retpolines

[Peter Zijlstra]

On Thu, Oct 16, 2025 at 01:07:17PM +0200, Peter Zijlstra wrote:
> On Mon, Oct 13, 2025 at 09:34:19AM -0500, David Kaplan wrote:
> > When patching retpolines, nops may be required for padding such as when
> > turning a 5-byte direct call into a 2-byte indirect call.  Previously,
> > these were appended at the end so the code becomes "call *reg;nop;nop;nop"
> > for example.  This was fine because it's always going from a larger
> > instruction to a smaller one.
> > 
> > But this is a problem if the sequence is transformed from a 2-byte indirect
> > to the 5-byte direct call version at runtime because when the called
> > function returns, it will be in the middle of the 5-byte call instruction.
> > 
> > To fix this, prepend the nops instead of appending them.  Consequently, the
> > return site of the called function is always the same.
> > 
> 
> So this results in:
> 
> NOP3; call *%r11
> 
> And you're saying a task can be on the other side of that call and then
> return lines up. But what if the task is preempted right after that
> NOP3?
> 
> Same for all the alternative patching; what ensures no task is currently
> having a register state that is in the middle of things?

Ah, I found it, you freeze everything, which puts it at safe points.

Re: [RFC PATCH 00/56] Dynamic mitigations

[Brendan Jackman]

On Tue Oct 14, 2025 at 6:06 PM UTC, David Kaplan wrote:
>> From: Josh Poimboeuf <jpoimboe@kernel.org>
>> > There are several use cases that will benefit from dynamic mitigations:
>> >
>> > Use Cases
>> > ---------
>> > 1. Runtime Policy
>> >
>> > Some workflows rely on booting a generic kernel before customizing the system.
>> > cloud-init is a popular example of this where a VM is started typically with
>> > default settings and then is customized based on a customer-provided
>> > configuration file.
>>
>> I'm not really a fan of this.  It adds complexity to some areas that are
>> already struggling with too much complexity.
>>
>> IMO this would need some REALLY strong justification, more than just
>> "hey, this makes things more convenient."
>>
>> The mitigations should be a "set it and forget it" thing.  I don't see
>> anything here which justifies the considerable maintenance burden this
>> would add for all existing and future mitigations.
>>
>
> The problem is there are environments like the one outlined where you can't just 'set it and forget it' because the kernel needs it set at boot-time, but in these environments you don't know how to configure the system until much later in boot.  So you end up running with the default settings all the time, even if you don't need them.  And the default settings can have significant performance impacts in many cases.
>
> The cloud guys on this thread may be able to offer some additional color here since I believe that's where you're most likely to have this situation.
>
> --David Kaplan

There's definitely a desire for more dynamic control at Google too,
similar to what Boris said.

Getting the config right is tricky and let's not forget it's
fundamentally a moving target as David noted. Even without dependence on
cloud-init type stuff, it's very useful to be able to change mitigations
on a timescale that's faster than we can reboot hosts. 

Re: [RFC PATCH 20/56] x86/bugs: Move bugs.c logic out of .init section

[Brendan Jackman]

On Mon Oct 13, 2025 at 2:34 PM UTC, David Kaplan wrote:
> If dynamic mitigations are supported, all the mitigation selection
> functions and mitigation choices may change at runtime.  Therefore, none of
> the functions may exist in .init and the data must not be read-only.

I suppose we should have something akin to
__meminit/__init_or_module/__net_init here (like __init, but conditional
on Kconfig), so that users with CONFIG_DYNAMIC_MITIGATIONS=n can still
get the RAM back?

Re: [RFC PATCH 07/56] x86/bugs: Reset spectre_v2_user mitigations

[Brendan Jackman]

On Mon Oct 13, 2025 at 2:33 PM UTC, David Kaplan wrote:
> Add function to reset spectre_v2_user mitigations back to their boot-time
> defaults.
>
> Signed-off-by: David Kaplan <david.kaplan@amd.com>
> ---
>  arch/x86/kernel/cpu/bugs.c | 13 +++++++++++++
>  1 file changed, 13 insertions(+)
>
> diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
> index 1f56ccb5f641..4ca46f58e384 100644
> --- a/arch/x86/kernel/cpu/bugs.c
> +++ b/arch/x86/kernel/cpu/bugs.c
> @@ -2056,6 +2056,18 @@ static void __init spectre_v2_user_apply_mitigation(void)
>  	}
>  }
>  
> +#ifdef CONFIG_DYNAMIC_MITIGATIONS
> +static void spectre_v2_user_reset_mitigation(void)
> +{
> +	static_branch_disable(&switch_vcpu_ibpb);
> +	static_branch_disable(&switch_mm_always_ibpb);
> +	static_branch_disable(&switch_mm_cond_ibpb);
> +	spectre_v2_user_stibp = SPECTRE_V2_USER_NONE;
> +	spectre_v2_user_ibpb = SPECTRE_V2_USER_NONE;
> +	spectre_v2_user_cmd = SPECTRE_V2_USER_CMD_AUTO;
> +}
> +#endif
> +
>  static const char * const spectre_v2_strings[] = {
>  	[SPECTRE_V2_NONE]			= "Vulnerable",
>  	[SPECTRE_V2_RETPOLINE]			= "Mitigation: Retpolines",
> @@ -3844,5 +3856,6 @@ void arch_cpu_reset_mitigations(void)
>  	spectre_v1_reset_mitigation();
>  	spectre_v2_reset_mitigation();
>  	retbleed_reset_mitigation();
> +	spectre_v2_user_reset_mitigation();
>  }
>  #endif

I think this might be failing to account for task state? E.g. if a
user boots with spectre_v2=off then we ignore the PR_SPEC_DISABLE calls
that would enable IBPB-on-context-switch for that task. Then if they
enable it via this dynamic interface they probably expect their
PR_SPEC_DISABLE to take effect retroactively. I don't think it will with
the current code, do I have that right?

RE: [RFC PATCH 31/56] x86/alternative: Prepend nops with retpolines

[Kaplan, David]

[AMD Official Use Only - AMD Internal Distribution Only]

> -----Original Message-----
> From: Peter Zijlstra <peterz@infradead.org>
> Sent: Thursday, October 16, 2025 6:23 AM
> To: Kaplan, David <David.Kaplan@amd.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>; Borislav Petkov <bp@alien8.de>; Josh
> Poimboeuf <jpoimboe@kernel.org>; Pawan Gupta
> <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar <mingo@redhat.com>; Dave
> Hansen <dave.hansen@linux.intel.com>; x86@kernel.org; H . Peter Anvin
> <hpa@zytor.com>; Alexander Graf <graf@amazon.com>; Boris Ostrovsky
> <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
> Subject: Re: [RFC PATCH 31/56] x86/alternative: Prepend nops with retpolines
>
> Caution: This message originated from an External Source. Use proper caution
> when opening attachments, clicking links, or responding.
>
>
> On Thu, Oct 16, 2025 at 01:07:17PM +0200, Peter Zijlstra wrote:
> > On Mon, Oct 13, 2025 at 09:34:19AM -0500, David Kaplan wrote:
> > > When patching retpolines, nops may be required for padding such as when
> > > turning a 5-byte direct call into a 2-byte indirect call.  Previously,
> > > these were appended at the end so the code becomes "call *reg;nop;nop;nop"
> > > for example.  This was fine because it's always going from a larger
> > > instruction to a smaller one.
> > >
> > > But this is a problem if the sequence is transformed from a 2-byte indirect
> > > to the 5-byte direct call version at runtime because when the called
> > > function returns, it will be in the middle of the 5-byte call instruction.
> > >
> > > To fix this, prepend the nops instead of appending them.  Consequently, the
> > > return site of the called function is always the same.
> > >
> >
> > So this results in:
> >
> > NOP3; call *%r11
> >
> > And you're saying a task can be on the other side of that call and then
> > return lines up. But what if the task is preempted right after that
> > NOP3?
> >
> > Same for all the alternative patching; what ensures no task is currently
> > having a register state that is in the middle of things?
>
> Ah, I found it, you freeze everything, which puts it at safe points.

Yes.  In fact, I think you were the one who pointed me in that direction :)

Despite the freezer though, this patch is necessary in particular because stop_machine_nmi() uses an indirect branch to run the handler.  Which means that while patching is going on, all cores are inside a function which is going to return to after the indirect call site.  And so that needs to be the end of the 5 (or 6) byte sequence.

--David Kaplan

RE: [RFC PATCH 00/56] Dynamic mitigations

[Kaplan, David]

[AMD Official Use Only - AMD Internal Distribution Only]

> -----Original Message-----
> From: Nicolas Bouchinet <nicolas.bouchinet@oss.cyber.gouv.fr>
> Sent: Thursday, October 16, 2025 5:00 AM
> To: Kaplan, David <David.Kaplan@amd.com>
> Cc: Josh Poimboeuf <jpoimboe@kernel.org>; Aaron Rainbolt
> <arraybolt3@gmail.com>; Thomas Gleixner <tglx@linutronix.de>; Borislav Petkov
> <bp@alien8.de>; Peter Zijlstra <peterz@infradead.org>; Pawan Gupta
> <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar <mingo@redhat.com>; Dave
> Hansen <dave.hansen@linux.intel.com>; x86@kernel.org; H . Peter Anvin
> <hpa@zytor.com>; Alexander Graf <graf@amazon.com>; Boris Ostrovsky
> <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org; Xiujianfeng
> <xiujianfeng@huawei.com>; xiujianfeng@huaweicloud.com
> Subject: Re: [RFC PATCH 00/56] Dynamic mitigations
>
> Caution: This message originated from an External Source. Use proper caution
> when opening attachments, clicking links, or responding.
>
>
> Hi David,
>
> > > > > On Wed, Oct 15, 2025 at 01:53:31PM +0000, Kaplan, David wrote:
> > > > > > > If `root` is capable of setting `mitigations=off` via this interface,
> > > > > > > doesn't that somewhat defeat the purpose of denying `/proc/kcore`
> > > > > > > access in lockdown confidentiality mode? Assuming one is running on a
> > > > > > > CPU with some form of side-channel memory read vulnerability (which
> they
> > > > > > > very likely are), they can turn off all mitigations, then read kernel
> > > > > > > memory via one of those exploits.
> > > > > > >
> > > > > > > There should be a one-way switch to allow denying all further writes to
> > > > > > > this interface, so that once the system's mitigations are set properly,
> > > > > > > any further attempts to change them until the next reboot can be
> > > > > > > prevented.
> > > > > > >
> > > > > >
> > > > > > That's a good idea, there could be a separate mitigation_lock file
> > > > > > perhaps that once written to 1 denies any further changes.
> > > > >
> > > > > Wouldn't the enablement of lockdown mode effectively function as that
> > > > > one way switch?
> > > > >
> > > >
> > > > I'm not too familiar with lockdown mode, but that gets enabled (with
> > > > right cmdline options) during boot right?  I guess the question is
> > > > would we want to allow any window for userspace to reconfigure things
> > > > and then lock things down, or say that if you enable lockdown then
> > > > this interface is completely disabled and you need to specify your
> > > > mitigation options on the cmdline only.
> > >
> > > Yeah, I would say the latter, otherwise it defeats the point of lockdown
> > > mode.  Note that lockdown mode can also be enabled at runtime.
> > >
> >
> > Ok.  So that's using the security_locked_down() API presumably.  And what
> reason would we want to check for?  Should it be LOCKDOWN_DEV_MEM (since
> we're potentially leaking arbitrary memory)?  Or a new lockdown reason?
> >
> > --David Kaplan
>
> LOCKDOWN_DEV_MEM is an integrity reason and should not be used for this
> scenario.
> I'd rather like to add a new Lockdown reason in the confidentiality set,
> maybe LOCKDOWN_CPU_MITIGATION ?
>

Ok, that makes sense.  Just to clarify, would that mean something like the below:

diff --git a/drivers/base/cpu.c b/drivers/base/cpu.c
index 3f9410dee67c..9b4864f84146 100644
--- a/drivers/base/cpu.c
+++ b/drivers/base/cpu.c
@@ -691,6 +691,9 @@ ssize_t cpu_write_mitigation_options(struct device *dev,
                                struct device_attribute *attr,
                                const char *buf, size_t count)
 {
+       if (security_locked_down(LOCKDOWN_CPU_MITIGATIONS))
+               return -EPERM;
+
        /* Save and filter the provided options. */
        cpu_filter_mitigation_opts(buf);

diff --git a/include/linux/security.h b/include/linux/security.h
index 92ac3f27b973..81cb52cf2111 100644
--- a/include/linux/security.h
+++ b/include/linux/security.h
@@ -153,6 +153,7 @@ enum lockdown_reason {
        LOCKDOWN_TRACEFS,
        LOCKDOWN_XMON_RW,
        LOCKDOWN_XFRM_SECRET,
+       LOCKDOWN_CPU_MITIGATIONS,
        LOCKDOWN_CONFIDENTIALITY_MAX,
 };

RE: [RFC PATCH 20/56] x86/bugs: Move bugs.c logic out of .init section

[Kaplan, David]

[AMD Official Use Only - AMD Internal Distribution Only]

> -----Original Message-----
> From: Brendan Jackman <jackmanb@google.com>
> Sent: Thursday, October 16, 2025 7:32 AM
> To: Kaplan, David <David.Kaplan@amd.com>; Thomas Gleixner
> <tglx@linutronix.de>; Borislav Petkov <bp@alien8.de>; Peter Zijlstra
> <peterz@infradead.org>; Josh Poimboeuf <jpoimboe@kernel.org>; Pawan Gupta
> <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar <mingo@redhat.com>; Dave
> Hansen <dave.hansen@linux.intel.com>; x86@kernel.org; H . Peter Anvin
> <hpa@zytor.com>
> Cc: Alexander Graf <graf@amazon.com>; Boris Ostrovsky
> <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
> Subject: Re: [RFC PATCH 20/56] x86/bugs: Move bugs.c logic out of .init section
>
> Caution: This message originated from an External Source. Use proper caution
> when opening attachments, clicking links, or responding.
>
>
> On Mon Oct 13, 2025 at 2:34 PM UTC, David Kaplan wrote:
> > If dynamic mitigations are supported, all the mitigation selection
> > functions and mitigation choices may change at runtime.  Therefore, none of
> > the functions may exist in .init and the data must not be read-only.
>
> I suppose we should have something akin to
> __meminit/__init_or_module/__net_init here (like __init, but conditional
> on Kconfig), so that users with CONFIG_DYNAMIC_MITIGATIONS=n can still
> get the RAM back?

Yeah, that's an option (maybe like __bugs_init).  Initial feedback I got was just to remove them from .init since they're now sometimes used later, but we could create a separate attribute tag.  Similar for the __ro_after_init ones.

Thanks
--David Kaplan

Re: [RFC PATCH 00/56] Dynamic mitigations

[Nicolas Bouchinet]

[snip]
> > LOCKDOWN_DEV_MEM is an integrity reason and should not be used for this
> > scenario.
> > I'd rather like to add a new Lockdown reason in the confidentiality set,
> > maybe LOCKDOWN_CPU_MITIGATION ?
> >
> 
> Ok, that makes sense.  Just to clarify, would that mean something like the below:
> 
> diff --git a/drivers/base/cpu.c b/drivers/base/cpu.c
> index 3f9410dee67c..9b4864f84146 100644
> --- a/drivers/base/cpu.c
> +++ b/drivers/base/cpu.c
> @@ -691,6 +691,9 @@ ssize_t cpu_write_mitigation_options(struct device *dev,
>                                 struct device_attribute *attr,
>                                 const char *buf, size_t count)
>  {
> +       if (security_locked_down(LOCKDOWN_CPU_MITIGATIONS))
> +               return -EPERM;
> +
>         /* Save and filter the provided options. */
>         cpu_filter_mitigation_opts(buf);
> 
> diff --git a/include/linux/security.h b/include/linux/security.h
> index 92ac3f27b973..81cb52cf2111 100644
> --- a/include/linux/security.h
> +++ b/include/linux/security.h
> @@ -153,6 +153,7 @@ enum lockdown_reason {
>         LOCKDOWN_TRACEFS,
>         LOCKDOWN_XMON_RW,
>         LOCKDOWN_XFRM_SECRET,
> +       LOCKDOWN_CPU_MITIGATIONS,
>         LOCKDOWN_CONFIDENTIALITY_MAX,
>  };

You should also add an entry to the lockdown_reasons array in
`security/security.c` with a description of the reason.

Nicolas

RE: [RFC PATCH 00/56] Dynamic mitigations

[Kaplan, David]

[AMD Official Use Only - AMD Internal Distribution Only]

> -----Original Message-----
> From: Nicolas Bouchinet <nicolas.bouchinet@oss.cyber.gouv.fr>
> Sent: Thursday, October 16, 2025 8:55 AM
> To: Kaplan, David <David.Kaplan@amd.com>
> Cc: Josh Poimboeuf <jpoimboe@kernel.org>; Aaron Rainbolt
> <arraybolt3@gmail.com>; Thomas Gleixner <tglx@linutronix.de>; Borislav Petkov
> <bp@alien8.de>; Peter Zijlstra <peterz@infradead.org>; Pawan Gupta
> <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar <mingo@redhat.com>; Dave
> Hansen <dave.hansen@linux.intel.com>; x86@kernel.org; H . Peter Anvin
> <hpa@zytor.com>; Alexander Graf <graf@amazon.com>; Boris Ostrovsky
> <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org; Xiujianfeng
> <xiujianfeng@huawei.com>; xiujianfeng@huaweicloud.com
> Subject: Re: [RFC PATCH 00/56] Dynamic mitigations
>
> Caution: This message originated from an External Source. Use proper caution
> when opening attachments, clicking links, or responding.
>
>
> [snip]
> > > LOCKDOWN_DEV_MEM is an integrity reason and should not be used for this
> > > scenario.
> > > I'd rather like to add a new Lockdown reason in the confidentiality set,
> > > maybe LOCKDOWN_CPU_MITIGATION ?
> > >
> >
> > Ok, that makes sense.  Just to clarify, would that mean something like the below:
> >
> > diff --git a/drivers/base/cpu.c b/drivers/base/cpu.c
> > index 3f9410dee67c..9b4864f84146 100644
> > --- a/drivers/base/cpu.c
> > +++ b/drivers/base/cpu.c
> > @@ -691,6 +691,9 @@ ssize_t cpu_write_mitigation_options(struct device *dev,
> >                                 struct device_attribute *attr,
> >                                 const char *buf, size_t count)
> >  {
> > +       if (security_locked_down(LOCKDOWN_CPU_MITIGATIONS))
> > +               return -EPERM;
> > +
> >         /* Save and filter the provided options. */
> >         cpu_filter_mitigation_opts(buf);
> >
> > diff --git a/include/linux/security.h b/include/linux/security.h
> > index 92ac3f27b973..81cb52cf2111 100644
> > --- a/include/linux/security.h
> > +++ b/include/linux/security.h
> > @@ -153,6 +153,7 @@ enum lockdown_reason {
> >         LOCKDOWN_TRACEFS,
> >         LOCKDOWN_XMON_RW,
> >         LOCKDOWN_XFRM_SECRET,
> > +       LOCKDOWN_CPU_MITIGATIONS,
> >         LOCKDOWN_CONFIDENTIALITY_MAX,
> >  };
>
> You should also add an entry to the lockdown_reasons array in
> `security/security.c` with a description of the reason.
>

Ah right.  Thanks!

--David Kaplan

RE: [RFC PATCH 07/56] x86/bugs: Reset spectre_v2_user mitigations

[Kaplan, David]

[AMD Official Use Only - AMD Internal Distribution Only]

> -----Original Message-----
> From: Brendan Jackman <jackmanb@google.com>
> Sent: Thursday, October 16, 2025 7:54 AM
> To: Kaplan, David <David.Kaplan@amd.com>; Thomas Gleixner
> <tglx@linutronix.de>; Borislav Petkov <bp@alien8.de>; Peter Zijlstra
> <peterz@infradead.org>; Josh Poimboeuf <jpoimboe@kernel.org>; Pawan Gupta
> <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar <mingo@redhat.com>; Dave
> Hansen <dave.hansen@linux.intel.com>; x86@kernel.org; H . Peter Anvin
> <hpa@zytor.com>
> Cc: Alexander Graf <graf@amazon.com>; Boris Ostrovsky
> <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
> Subject: Re: [RFC PATCH 07/56] x86/bugs: Reset spectre_v2_user mitigations
>
> Caution: This message originated from an External Source. Use proper caution
> when opening attachments, clicking links, or responding.
>
>
> On Mon Oct 13, 2025 at 2:33 PM UTC, David Kaplan wrote:
> > Add function to reset spectre_v2_user mitigations back to their boot-time
> > defaults.
> >
> > Signed-off-by: David Kaplan <david.kaplan@amd.com>
> > ---
> >  arch/x86/kernel/cpu/bugs.c | 13 +++++++++++++
> >  1 file changed, 13 insertions(+)
> >
> > diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
> > index 1f56ccb5f641..4ca46f58e384 100644
> > --- a/arch/x86/kernel/cpu/bugs.c
> > +++ b/arch/x86/kernel/cpu/bugs.c
> > @@ -2056,6 +2056,18 @@ static void __init
> spectre_v2_user_apply_mitigation(void)
> >       }
> >  }
> >
> > +#ifdef CONFIG_DYNAMIC_MITIGATIONS
> > +static void spectre_v2_user_reset_mitigation(void)
> > +{
> > +     static_branch_disable(&switch_vcpu_ibpb);
> > +     static_branch_disable(&switch_mm_always_ibpb);
> > +     static_branch_disable(&switch_mm_cond_ibpb);
> > +     spectre_v2_user_stibp = SPECTRE_V2_USER_NONE;
> > +     spectre_v2_user_ibpb = SPECTRE_V2_USER_NONE;
> > +     spectre_v2_user_cmd = SPECTRE_V2_USER_CMD_AUTO;
> > +}
> > +#endif
> > +
> >  static const char * const spectre_v2_strings[] = {
> >       [SPECTRE_V2_NONE]                       = "Vulnerable",
> >       [SPECTRE_V2_RETPOLINE]                  = "Mitigation: Retpolines",
> > @@ -3844,5 +3856,6 @@ void arch_cpu_reset_mitigations(void)
> >       spectre_v1_reset_mitigation();
> >       spectre_v2_reset_mitigation();
> >       retbleed_reset_mitigation();
> > +     spectre_v2_user_reset_mitigation();
> >  }
> >  #endif
>
> I think this might be failing to account for task state? E.g. if a
> user boots with spectre_v2=off then we ignore the PR_SPEC_DISABLE calls
> that would enable IBPB-on-context-switch for that task. Then if they
> enable it via this dynamic interface they probably expect their
> PR_SPEC_DISABLE to take effect retroactively. I don't think it will with
> the current code, do I have that right?

If I'm reading the logic correct, if a process tries to do PR_SPEC_DISABLE say for indirects but spectre_v2_user=off then they'll get -EPERM, so we don't ignore it.

But there could be a case where spectre_v2=on (aka force), when PR_SPEC_DISABLE does get ignored.  And then if spectre_v2 is changed to something else like prctl then the relevant task flags weren't set.

Not sure the best way to handle this...even if we were to always set the task flags in this case, there could be other cases where the process might think it could set this flag and then get surprised with an -EPERM.  Open to ideas here.

--David Kaplan

Re: [RFC PATCH 31/56] x86/alternative: Prepend nops with retpolines

[Peter Zijlstra]

On Thu, Oct 16, 2025 at 01:27:53PM +0000, Kaplan, David wrote:

> > Ah, I found it, you freeze everything, which puts it at safe points.
> 
> Yes.  In fact, I think you were the one who pointed me in that direction :)

Heh, yeah, I remembered talking to you about this, but had forgotten all
details. I quickly checked the cover letter, but found insufficient
detail there, so I went and asked. Then later memory started coming back
and I went looking for it in later patches.

> Despite the freezer though, this patch is necessary in particular
> because stop_machine_nmi() uses an indirect branch to run the handler.
> Which means that while patching is going on, all cores are inside a
> function which is going to return to after the indirect call site.
> And so that needs to be the end of the 5 (or 6) byte sequence.

Yeah, makes sense. But I like I wrote, I think prefix stuffing might be
a better option.

RE: [RFC PATCH 31/56] x86/alternative: Prepend nops with retpolines

[Kaplan, David]

[AMD Official Use Only - AMD Internal Distribution Only]

> -----Original Message-----
> From: Peter Zijlstra <peterz@infradead.org>
> Sent: Thursday, October 16, 2025 9:08 AM
> To: Kaplan, David <David.Kaplan@amd.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>; Borislav Petkov <bp@alien8.de>; Josh
> Poimboeuf <jpoimboe@kernel.org>; Pawan Gupta
> <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar <mingo@redhat.com>; Dave
> Hansen <dave.hansen@linux.intel.com>; x86@kernel.org; H . Peter Anvin
> <hpa@zytor.com>; Alexander Graf <graf@amazon.com>; Boris Ostrovsky
> <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
> Subject: Re: [RFC PATCH 31/56] x86/alternative: Prepend nops with retpolines
>
> Caution: This message originated from an External Source. Use proper caution
> when opening attachments, clicking links, or responding.
>
>
> On Thu, Oct 16, 2025 at 01:27:53PM +0000, Kaplan, David wrote:
>
> > > Ah, I found it, you freeze everything, which puts it at safe points.
> >
> > Yes.  In fact, I think you were the one who pointed me in that direction :)
>
> Heh, yeah, I remembered talking to you about this, but had forgotten all
> details. I quickly checked the cover letter, but found insufficient
> detail there, so I went and asked. Then later memory started coming back
> and I went looking for it in later patches.
>
> > Despite the freezer though, this patch is necessary in particular
> > because stop_machine_nmi() uses an indirect branch to run the handler.
> > Which means that while patching is going on, all cores are inside a
> > function which is going to return to after the indirect call site.
> > And so that needs to be the end of the 5 (or 6) byte sequence.
>
> Yeah, makes sense. But I like I wrote, I think prefix stuffing might be
> a better option.

Ok, and that's because we need at most 3 prefixes as Intel uarch's don't have significant penalties at 3 prefixes, only at 4+?

I'll need to check on this on the AMD side too.

--David Kaplan

Re: [RFC PATCH 31/56] x86/alternative: Prepend nops with retpolines

[Peter Zijlstra]

On Thu, Oct 16, 2025 at 02:16:04PM +0000, Kaplan, David wrote:

> > Yeah, makes sense. But I like I wrote, I think prefix stuffing might be
> > a better option.
> 
> Ok, and that's because we need at most 3 prefixes as Intel uarch's
> don't have significant penalties at 3 prefixes, only at 4+?

Yeah, IIRC 3 was the magic number. Sadly there isn't much public
information on this. The best summary on the subject I could find was
here:

  https://reviews.llvm.org/D75945

> I'll need to check on this on the AMD side too.

The above, quoting Agner's optimization guide, says Bulldozer is
affected.

Re: [RFC PATCH 20/56] x86/bugs: Move bugs.c logic out of .init section

[Brendan Jackman]

On Thu Oct 16, 2025 at 1:46 PM UTC, David Kaplan wrote:
> [AMD Official Use Only - AMD Internal Distribution Only]
>
>> -----Original Message-----
>> From: Brendan Jackman <jackmanb@google.com>
>> Sent: Thursday, October 16, 2025 7:32 AM
>> To: Kaplan, David <David.Kaplan@amd.com>; Thomas Gleixner
>> <tglx@linutronix.de>; Borislav Petkov <bp@alien8.de>; Peter Zijlstra
>> <peterz@infradead.org>; Josh Poimboeuf <jpoimboe@kernel.org>; Pawan Gupta
>> <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar <mingo@redhat.com>; Dave
>> Hansen <dave.hansen@linux.intel.com>; x86@kernel.org; H . Peter Anvin
>> <hpa@zytor.com>
>> Cc: Alexander Graf <graf@amazon.com>; Boris Ostrovsky
>> <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
>> Subject: Re: [RFC PATCH 20/56] x86/bugs: Move bugs.c logic out of .init section
>>
>> Caution: This message originated from an External Source. Use proper caution
>> when opening attachments, clicking links, or responding.
>>
>>
>> On Mon Oct 13, 2025 at 2:34 PM UTC, David Kaplan wrote:
>> > If dynamic mitigations are supported, all the mitigation selection
>> > functions and mitigation choices may change at runtime.  Therefore, none of
>> > the functions may exist in .init and the data must not be read-only.
>>
>> I suppose we should have something akin to
>> __meminit/__init_or_module/__net_init here (like __init, but conditional
>> on Kconfig), so that users with CONFIG_DYNAMIC_MITIGATIONS=n can still
>> get the RAM back?
>
> Yeah, that's an option (maybe like __bugs_init).  Initial feedback I got was just to remove them from .init since they're now sometimes used later, but we could create a separate attribute tag.  Similar for the __ro_after_init ones.

OK, if you'd already thought of it and people didn't want the
maintenance burden, it's definitely not a big deal for me.

RE: [RFC PATCH 40/56] x86/alternative: Use sync_core_nmi_safe()

[Kaplan, David]

[AMD Official Use Only - AMD Internal Distribution Only]

> -----Original Message-----
> From: Peter Zijlstra <peterz@infradead.org>
> Sent: Thursday, October 16, 2025 5:36 AM
> To: Kaplan, David <David.Kaplan@amd.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>; Borislav Petkov <bp@alien8.de>; Josh
> Poimboeuf <jpoimboe@kernel.org>; Pawan Gupta
> <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar <mingo@redhat.com>; Dave
> Hansen <dave.hansen@linux.intel.com>; x86@kernel.org; H . Peter Anvin
> <hpa@zytor.com>; Alexander Graf <graf@amazon.com>; Boris Ostrovsky
> <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
> Subject: Re: [RFC PATCH 40/56] x86/alternative: Use sync_core_nmi_safe()
>
> Caution: This message originated from an External Source. Use proper caution
> when opening attachments, clicking links, or responding.
>
>
> On Mon, Oct 13, 2025 at 09:34:28AM -0500, David Kaplan wrote:
> > Re-patching is done under NMI context so the NMI-safe version of
> > sync_core() must be used.
> >
> > Signed-off-by: David Kaplan <david.kaplan@amd.com>
> > ---
> >  arch/x86/kernel/alternative.c | 6 +++++-
> >  1 file changed, 5 insertions(+), 1 deletion(-)
> >
> > diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c
> > index b67116ae883c..2d48d750d4d9 100644
> > --- a/arch/x86/kernel/alternative.c
> > +++ b/arch/x86/kernel/alternative.c
> > @@ -2585,7 +2585,11 @@ void __init_or_module text_poke_early(void *addr,
> const void *opcode,
> >       } else {
> >               local_irq_save(flags);
> >               memcpy(addr, opcode, len);
> > -             sync_core();
> > +             /* Re-patching occurs in NMI context so we can't do IRET. */
> > +             if (repatch_in_progress)
> > +                     sync_core_nmi_safe();
> > +             else
> > +                     sync_core();
> >               local_irq_restore(flags);
> >
>
> Can we please keep this in sync_core()? Something like:
>
> static __always_inline void sync_core(void)
> {
>         if (static_cpu_has(X86_FEATURE_SERIALIZE)) {
>                 serialize();
>                 return;
>         }
>
> +       if (repatch_in_progress) {
> +               sync_core_nmi_safe();
> +               return;
> +       }
> +
>         iret_to_self();
> }
>
> That way all the modern stuff that has SERIALIZE will still use that.

Hmm, I can't quite do that because sync_core() is used in a number of other places too (unless we make repatch_in_progress a true global).

I wonder though if it'd be ok to have sync_core() check IS_ENABLED(CONFIG_DYNAMIC_MITIGATIONS) and then always use the mov-cr2 version?  It might also have to check X86_FEATURE_XENPV and use IRET in that case but otherwise I'd think it's safe for machines that could support dynamic mitigations.

--David Kaplan

Re: [RFC PATCH 40/56] x86/alternative: Use sync_core_nmi_safe()

[Peter Zijlstra]

On Thu, Oct 16, 2025 at 02:40:51PM +0000, Kaplan, David wrote:

> > Can we please keep this in sync_core()? Something like:
> >
> > static __always_inline void sync_core(void)
> > {
> >         if (static_cpu_has(X86_FEATURE_SERIALIZE)) {
> >                 serialize();
> >                 return;
> >         }
> >
> > +       if (repatch_in_progress) {
> > +               sync_core_nmi_safe();
> > +               return;
> > +       }
> > +
> >         iret_to_self();
> > }
> >
> > That way all the modern stuff that has SERIALIZE will still use that.
> 
> Hmm, I can't quite do that because sync_core() is used in a number of
> other places too (unless we make repatch_in_progress a true global).

We could just out-of-line the thing; nothing using this should care
about cycles -- all of this is quite expensive.

> I wonder though if it'd be ok to have sync_core() check
> IS_ENABLED(CONFIG_DYNAMIC_MITIGATIONS) and then always use the mov-cr2
> version?  It might also have to check X86_FEATURE_XENPV and use IRET
> in that case but otherwise I'd think it's safe for machines that could
> support dynamic mitigations.

Yeah, dunno.. I'm not well versed in the virt thing.

BTW, will AMD do that SERIALIZE instruction?

Re: [RFC PATCH 07/56] x86/bugs: Reset spectre_v2_user mitigations

[Brendan Jackman]

On Thu Oct 16, 2025 at 2:06 PM UTC, David Kaplan wrote:
> [AMD Official Use Only - AMD Internal Distribution Only]
>
>> -----Original Message-----
>> From: Brendan Jackman <jackmanb@google.com>
>> Sent: Thursday, October 16, 2025 7:54 AM
>> To: Kaplan, David <David.Kaplan@amd.com>; Thomas Gleixner
>> <tglx@linutronix.de>; Borislav Petkov <bp@alien8.de>; Peter Zijlstra
>> <peterz@infradead.org>; Josh Poimboeuf <jpoimboe@kernel.org>; Pawan Gupta
>> <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar <mingo@redhat.com>; Dave
>> Hansen <dave.hansen@linux.intel.com>; x86@kernel.org; H . Peter Anvin
>> <hpa@zytor.com>
>> Cc: Alexander Graf <graf@amazon.com>; Boris Ostrovsky
>> <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
>> Subject: Re: [RFC PATCH 07/56] x86/bugs: Reset spectre_v2_user mitigations
>>
>> Caution: This message originated from an External Source. Use proper caution
>> when opening attachments, clicking links, or responding.
>>
>>
>> On Mon Oct 13, 2025 at 2:33 PM UTC, David Kaplan wrote:
>> > Add function to reset spectre_v2_user mitigations back to their boot-time
>> > defaults.
>> >
>> > Signed-off-by: David Kaplan <david.kaplan@amd.com>
>> > ---
>> >  arch/x86/kernel/cpu/bugs.c | 13 +++++++++++++
>> >  1 file changed, 13 insertions(+)
>> >
>> > diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
>> > index 1f56ccb5f641..4ca46f58e384 100644
>> > --- a/arch/x86/kernel/cpu/bugs.c
>> > +++ b/arch/x86/kernel/cpu/bugs.c
>> > @@ -2056,6 +2056,18 @@ static void __init
>> spectre_v2_user_apply_mitigation(void)
>> >       }
>> >  }
>> >
>> > +#ifdef CONFIG_DYNAMIC_MITIGATIONS
>> > +static void spectre_v2_user_reset_mitigation(void)
>> > +{
>> > +     static_branch_disable(&switch_vcpu_ibpb);
>> > +     static_branch_disable(&switch_mm_always_ibpb);
>> > +     static_branch_disable(&switch_mm_cond_ibpb);
>> > +     spectre_v2_user_stibp = SPECTRE_V2_USER_NONE;
>> > +     spectre_v2_user_ibpb = SPECTRE_V2_USER_NONE;
>> > +     spectre_v2_user_cmd = SPECTRE_V2_USER_CMD_AUTO;
>> > +}
>> > +#endif
>> > +
>> >  static const char * const spectre_v2_strings[] = {
>> >       [SPECTRE_V2_NONE]                       = "Vulnerable",
>> >       [SPECTRE_V2_RETPOLINE]                  = "Mitigation: Retpolines",
>> > @@ -3844,5 +3856,6 @@ void arch_cpu_reset_mitigations(void)
>> >       spectre_v1_reset_mitigation();
>> >       spectre_v2_reset_mitigation();
>> >       retbleed_reset_mitigation();
>> > +     spectre_v2_user_reset_mitigation();
>> >  }
>> >  #endif
>>
>> I think this might be failing to account for task state? E.g. if a
>> user boots with spectre_v2=off then we ignore the PR_SPEC_DISABLE calls
>> that would enable IBPB-on-context-switch for that task. Then if they
>> enable it via this dynamic interface they probably expect their
>> PR_SPEC_DISABLE to take effect retroactively. I don't think it will with
>> the current code, do I have that right?
>
> If I'm reading the logic correct, if a process tries to do PR_SPEC_DISABLE say for indirects but spectre_v2_user=off then they'll get -EPERM, so we don't ignore it.
>
> But there could be a case where spectre_v2=on (aka force), when PR_SPEC_DISABLE does get ignored.  And then if spectre_v2 is changed to something else like prctl then the relevant task flags weren't set.

Er yeah good point, my example was wrong but the issue exists to some
extent.

> Not sure the best way to handle this...even if we were to always set the task flags in this case, there could be other cases where the process might think it could set this flag and then get surprised with an -EPERM.  Open to ideas here.

Hm, isn't the issue of surprise-EPERM orthogonal to the task state
issue? I suspect I'm doing a bit of motivated reasoning here, but it
feels to me like a userspace bug for someone to infer statically whether
they should expect -EPERM here. Like, the docs don't say that much, but
they do say something about what the prctls do, and that doesn't include
anything about the _reason_ you got -EPERM.

(BTW, it's not relevant here but I actually think -EPERM is a bug [0])

[0] https://lore.kernel.org/all/DDJU0415JEBQ.H2SD942NMDWX@google.com/

RE: [RFC PATCH 07/56] x86/bugs: Reset spectre_v2_user mitigations

[Kaplan, David]

[AMD Official Use Only - AMD Internal Distribution Only]

> -----Original Message-----
> From: Brendan Jackman <jackmanb@google.com>
> Sent: Thursday, October 16, 2025 9:57 AM
> To: Kaplan, David <David.Kaplan@amd.com>; Brendan Jackman
> <jackmanb@google.com>; Thomas Gleixner <tglx@linutronix.de>; Borislav Petkov
> <bp@alien8.de>; Peter Zijlstra <peterz@infradead.org>; Josh Poimboeuf
> <jpoimboe@kernel.org>; Pawan Gupta <pawan.kumar.gupta@linux.intel.com>;
> Ingo Molnar <mingo@redhat.com>; Dave Hansen <dave.hansen@linux.intel.com>;
> x86@kernel.org; H . Peter Anvin <hpa@zytor.com>
> Cc: Alexander Graf <graf@amazon.com>; Boris Ostrovsky
> <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
> Subject: Re: [RFC PATCH 07/56] x86/bugs: Reset spectre_v2_user mitigations
>
> Caution: This message originated from an External Source. Use proper caution
> when opening attachments, clicking links, or responding.
>
>
> On Thu Oct 16, 2025 at 2:06 PM UTC, David Kaplan wrote:
> > [AMD Official Use Only - AMD Internal Distribution Only]
> >
> >> -----Original Message-----
> >> From: Brendan Jackman <jackmanb@google.com>
> >> Sent: Thursday, October 16, 2025 7:54 AM
> >> To: Kaplan, David <David.Kaplan@amd.com>; Thomas Gleixner
> >> <tglx@linutronix.de>; Borislav Petkov <bp@alien8.de>; Peter Zijlstra
> >> <peterz@infradead.org>; Josh Poimboeuf <jpoimboe@kernel.org>; Pawan
> Gupta
> >> <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar <mingo@redhat.com>;
> Dave
> >> Hansen <dave.hansen@linux.intel.com>; x86@kernel.org; H . Peter Anvin
> >> <hpa@zytor.com>
> >> Cc: Alexander Graf <graf@amazon.com>; Boris Ostrovsky
> >> <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
> >> Subject: Re: [RFC PATCH 07/56] x86/bugs: Reset spectre_v2_user mitigations
> >>
> >> Caution: This message originated from an External Source. Use proper caution
> >> when opening attachments, clicking links, or responding.
> >>
> >>
> >> On Mon Oct 13, 2025 at 2:33 PM UTC, David Kaplan wrote:
> >> > Add function to reset spectre_v2_user mitigations back to their boot-time
> >> > defaults.
> >> >
> >> > Signed-off-by: David Kaplan <david.kaplan@amd.com>
> >> > ---
> >> >  arch/x86/kernel/cpu/bugs.c | 13 +++++++++++++
> >> >  1 file changed, 13 insertions(+)
> >> >
> >> > diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
> >> > index 1f56ccb5f641..4ca46f58e384 100644
> >> > --- a/arch/x86/kernel/cpu/bugs.c
> >> > +++ b/arch/x86/kernel/cpu/bugs.c
> >> > @@ -2056,6 +2056,18 @@ static void __init
> >> spectre_v2_user_apply_mitigation(void)
> >> >       }
> >> >  }
> >> >
> >> > +#ifdef CONFIG_DYNAMIC_MITIGATIONS
> >> > +static void spectre_v2_user_reset_mitigation(void)
> >> > +{
> >> > +     static_branch_disable(&switch_vcpu_ibpb);
> >> > +     static_branch_disable(&switch_mm_always_ibpb);
> >> > +     static_branch_disable(&switch_mm_cond_ibpb);
> >> > +     spectre_v2_user_stibp = SPECTRE_V2_USER_NONE;
> >> > +     spectre_v2_user_ibpb = SPECTRE_V2_USER_NONE;
> >> > +     spectre_v2_user_cmd = SPECTRE_V2_USER_CMD_AUTO;
> >> > +}
> >> > +#endif
> >> > +
> >> >  static const char * const spectre_v2_strings[] = {
> >> >       [SPECTRE_V2_NONE]                       = "Vulnerable",
> >> >       [SPECTRE_V2_RETPOLINE]                  = "Mitigation: Retpolines",
> >> > @@ -3844,5 +3856,6 @@ void arch_cpu_reset_mitigations(void)
> >> >       spectre_v1_reset_mitigation();
> >> >       spectre_v2_reset_mitigation();
> >> >       retbleed_reset_mitigation();
> >> > +     spectre_v2_user_reset_mitigation();
> >> >  }
> >> >  #endif
> >>
> >> I think this might be failing to account for task state? E.g. if a
> >> user boots with spectre_v2=off then we ignore the PR_SPEC_DISABLE calls
> >> that would enable IBPB-on-context-switch for that task. Then if they
> >> enable it via this dynamic interface they probably expect their
> >> PR_SPEC_DISABLE to take effect retroactively. I don't think it will with
> >> the current code, do I have that right?
> >
> > If I'm reading the logic correct, if a process tries to do PR_SPEC_DISABLE say
> for indirects but spectre_v2_user=off then they'll get -EPERM, so we don't ignore it.
> >
> > But there could be a case where spectre_v2=on (aka force), when
> PR_SPEC_DISABLE does get ignored.  And then if spectre_v2 is changed to
> something else like prctl then the relevant task flags weren't set.
>
> Er yeah good point, my example was wrong but the issue exists to some
> extent.
>
> > Not sure the best way to handle this...even if we were to always set the task flags
> in this case, there could be other cases where the process might think it could set
> this flag and then get surprised with an -EPERM.  Open to ideas here.
>
> Hm, isn't the issue of surprise-EPERM orthogonal to the task state
> issue? I suspect I'm doing a bit of motivated reasoning here, but it
> feels to me like a userspace bug for someone to infer statically whether
> they should expect -EPERM here. Like, the docs don't say that much, but
> they do say something about what the prctls do, and that doesn't include
> anything about the _reason_ you got -EPERM.
>
> (BTW, it's not relevant here but I actually think -EPERM is a bug [0])
>
> [0] https://lore.kernel.org/all/DDJU0415JEBQ.H2SD942NMDWX@google.com/

Well, if you do PR_GET_SPECULATION_CTRL, and it says PR_SPEC_PRCTL=1 that means you can control it per-task.  But then if you later get an error message when you try to set it, that would seem weird.

I don't know if anyone is actually using these controls today but I am nervous about causing confusion (even if the API is rather unclear).

One potential option here could be to always return PR_SPEC_ENABLE if dynamic mitigations are enabled (telling userspace they cannot control the mitigation and they should assume the speculation is enabled).  Or maybe create new options that allow a process to say 'try to give me this speculation control' but without any promise that they'll actually get it.

Assuming that dynamic mitigations get locked down late in boot as part of kernel lockdown, there may only be a relatively small period of time where there is a risk of things changing underneath a task so maybe telling userspace they can't control these until things are locked is ok?

--David Kaplan

RE: [RFC PATCH 40/56] x86/alternative: Use sync_core_nmi_safe()

[Kaplan, David]

[AMD Official Use Only - AMD Internal Distribution Only]

> -----Original Message-----
> From: Peter Zijlstra <peterz@infradead.org>
> Sent: Thursday, October 16, 2025 9:48 AM
> To: Kaplan, David <David.Kaplan@amd.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>; Borislav Petkov <bp@alien8.de>; Josh
> Poimboeuf <jpoimboe@kernel.org>; Pawan Gupta
> <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar <mingo@redhat.com>; Dave
> Hansen <dave.hansen@linux.intel.com>; x86@kernel.org; H . Peter Anvin
> <hpa@zytor.com>; Alexander Graf <graf@amazon.com>; Boris Ostrovsky
> <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
> Subject: Re: [RFC PATCH 40/56] x86/alternative: Use sync_core_nmi_safe()
>
> Caution: This message originated from an External Source. Use proper caution
> when opening attachments, clicking links, or responding.
>
>
> On Thu, Oct 16, 2025 at 02:40:51PM +0000, Kaplan, David wrote:
>
> > > Can we please keep this in sync_core()? Something like:
> > >
> > > static __always_inline void sync_core(void)
> > > {
> > >         if (static_cpu_has(X86_FEATURE_SERIALIZE)) {
> > >                 serialize();
> > >                 return;
> > >         }
> > >
> > > +       if (repatch_in_progress) {
> > > +               sync_core_nmi_safe();
> > > +               return;
> > > +       }
> > > +
> > >         iret_to_self();
> > > }
> > >
> > > That way all the modern stuff that has SERIALIZE will still use that.
> >
> > Hmm, I can't quite do that because sync_core() is used in a number of
> > other places too (unless we make repatch_in_progress a true global).
>
> We could just out-of-line the thing; nothing using this should care
> about cycles -- all of this is quite expensive.

I guess I could also just add the SERIALIZE check into sync_core_nmi_safe() and prefer that over mov-cr2.  Although as you said, cycles shouldn't matter...

>
> > I wonder though if it'd be ok to have sync_core() check
> > IS_ENABLED(CONFIG_DYNAMIC_MITIGATIONS) and then always use the
> mov-cr2
> > version?  It might also have to check X86_FEATURE_XENPV and use IRET
> > in that case but otherwise I'd think it's safe for machines that could
> > support dynamic mitigations.
>
> Yeah, dunno.. I'm not well versed in the virt thing.
>
> BTW, will AMD do that SERIALIZE instruction?

It is not supported on current parts, I can't comment on future ones.

I will note that on AMD, MFENCE has the required serializing properties (like SERIALIZE).  On AMD, we could use MFENCE in sync_core() which is probably faster that iret_to_self().  But again...do we really care.

--David Kaplan

Re: [RFC PATCH 07/56] x86/bugs: Reset spectre_v2_user mitigations

[Brendan Jackman]

On Thu Oct 16, 2025 at 3:26 PM UTC, David Kaplan wrote:
> [AMD Official Use Only - AMD Internal Distribution Only]
>
>> -----Original Message-----
>> From: Brendan Jackman <jackmanb@google.com>
>> Sent: Thursday, October 16, 2025 9:57 AM
>> To: Kaplan, David <David.Kaplan@amd.com>; Brendan Jackman
>> <jackmanb@google.com>; Thomas Gleixner <tglx@linutronix.de>; Borislav Petkov
>> <bp@alien8.de>; Peter Zijlstra <peterz@infradead.org>; Josh Poimboeuf
>> <jpoimboe@kernel.org>; Pawan Gupta <pawan.kumar.gupta@linux.intel.com>;
>> Ingo Molnar <mingo@redhat.com>; Dave Hansen <dave.hansen@linux.intel.com>;
>> x86@kernel.org; H . Peter Anvin <hpa@zytor.com>
>> Cc: Alexander Graf <graf@amazon.com>; Boris Ostrovsky
>> <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
>> Subject: Re: [RFC PATCH 07/56] x86/bugs: Reset spectre_v2_user mitigations
>>
>> Caution: This message originated from an External Source. Use proper caution
>> when opening attachments, clicking links, or responding.
>>
>>
>> On Thu Oct 16, 2025 at 2:06 PM UTC, David Kaplan wrote:
>> > [AMD Official Use Only - AMD Internal Distribution Only]
>> >
>> >> -----Original Message-----
>> >> From: Brendan Jackman <jackmanb@google.com>
>> >> Sent: Thursday, October 16, 2025 7:54 AM
>> >> To: Kaplan, David <David.Kaplan@amd.com>; Thomas Gleixner
>> >> <tglx@linutronix.de>; Borislav Petkov <bp@alien8.de>; Peter Zijlstra
>> >> <peterz@infradead.org>; Josh Poimboeuf <jpoimboe@kernel.org>; Pawan
>> Gupta
>> >> <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar <mingo@redhat.com>;
>> Dave
>> >> Hansen <dave.hansen@linux.intel.com>; x86@kernel.org; H . Peter Anvin
>> >> <hpa@zytor.com>
>> >> Cc: Alexander Graf <graf@amazon.com>; Boris Ostrovsky
>> >> <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
>> >> Subject: Re: [RFC PATCH 07/56] x86/bugs: Reset spectre_v2_user mitigations
>> >>
>> >> Caution: This message originated from an External Source. Use proper caution
>> >> when opening attachments, clicking links, or responding.
>> >>
>> >>
>> >> On Mon Oct 13, 2025 at 2:33 PM UTC, David Kaplan wrote:
>> >> > Add function to reset spectre_v2_user mitigations back to their boot-time
>> >> > defaults.
>> >> >
>> >> > Signed-off-by: David Kaplan <david.kaplan@amd.com>
>> >> > ---
>> >> >  arch/x86/kernel/cpu/bugs.c | 13 +++++++++++++
>> >> >  1 file changed, 13 insertions(+)
>> >> >
>> >> > diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
>> >> > index 1f56ccb5f641..4ca46f58e384 100644
>> >> > --- a/arch/x86/kernel/cpu/bugs.c
>> >> > +++ b/arch/x86/kernel/cpu/bugs.c
>> >> > @@ -2056,6 +2056,18 @@ static void __init
>> >> spectre_v2_user_apply_mitigation(void)
>> >> >       }
>> >> >  }
>> >> >
>> >> > +#ifdef CONFIG_DYNAMIC_MITIGATIONS
>> >> > +static void spectre_v2_user_reset_mitigation(void)
>> >> > +{
>> >> > +     static_branch_disable(&switch_vcpu_ibpb);
>> >> > +     static_branch_disable(&switch_mm_always_ibpb);
>> >> > +     static_branch_disable(&switch_mm_cond_ibpb);
>> >> > +     spectre_v2_user_stibp = SPECTRE_V2_USER_NONE;
>> >> > +     spectre_v2_user_ibpb = SPECTRE_V2_USER_NONE;
>> >> > +     spectre_v2_user_cmd = SPECTRE_V2_USER_CMD_AUTO;
>> >> > +}
>> >> > +#endif
>> >> > +
>> >> >  static const char * const spectre_v2_strings[] = {
>> >> >       [SPECTRE_V2_NONE]                       = "Vulnerable",
>> >> >       [SPECTRE_V2_RETPOLINE]                  = "Mitigation: Retpolines",
>> >> > @@ -3844,5 +3856,6 @@ void arch_cpu_reset_mitigations(void)
>> >> >       spectre_v1_reset_mitigation();
>> >> >       spectre_v2_reset_mitigation();
>> >> >       retbleed_reset_mitigation();
>> >> > +     spectre_v2_user_reset_mitigation();
>> >> >  }
>> >> >  #endif
>> >>
>> >> I think this might be failing to account for task state? E.g. if a
>> >> user boots with spectre_v2=off then we ignore the PR_SPEC_DISABLE calls
>> >> that would enable IBPB-on-context-switch for that task. Then if they
>> >> enable it via this dynamic interface they probably expect their
>> >> PR_SPEC_DISABLE to take effect retroactively. I don't think it will with
>> >> the current code, do I have that right?
>> >
>> > If I'm reading the logic correct, if a process tries to do PR_SPEC_DISABLE say
>> for indirects but spectre_v2_user=off then they'll get -EPERM, so we don't ignore it.
>> >
>> > But there could be a case where spectre_v2=on (aka force), when
>> PR_SPEC_DISABLE does get ignored.  And then if spectre_v2 is changed to
>> something else like prctl then the relevant task flags weren't set.
>>
>> Er yeah good point, my example was wrong but the issue exists to some
>> extent.
>>
>> > Not sure the best way to handle this...even if we were to always set the task flags
>> in this case, there could be other cases where the process might think it could set
>> this flag and then get surprised with an -EPERM.  Open to ideas here.
>>
>> Hm, isn't the issue of surprise-EPERM orthogonal to the task state
>> issue? I suspect I'm doing a bit of motivated reasoning here, but it
>> feels to me like a userspace bug for someone to infer statically whether
>> they should expect -EPERM here. Like, the docs don't say that much, but
>> they do say something about what the prctls do, and that doesn't include
>> anything about the _reason_ you got -EPERM.
>>
>> (BTW, it's not relevant here but I actually think -EPERM is a bug [0])
>>
>> [0] https://lore.kernel.org/all/DDJU0415JEBQ.H2SD942NMDWX@google.com/
>
> Well, if you do PR_GET_SPECULATION_CTRL, and it says PR_SPEC_PRCTL=1 that means you can control it per-task.  But then if you later get an error message when you try to set it, that would seem weird.

Yeah I see, I didn't notice there was a case where we _explicitly_ say
it can be controlled (I was just thinking of -EPERM/-ENXIO). It does
seem weird to suddenly countermand that one.

> I don't know if anyone is actually using these controls today but I am nervous about causing confusion (even if the API is rather unclear).

I do know of some users of this API.

> One potential option here could be to always return PR_SPEC_ENABLE if dynamic mitigations are enabled (telling userspace they cannot control the mitigation and they should assume the speculation is enabled).  Or maybe create new options that allow a process to say 'try to give me this speculation control' but without any promise that they'll actually get it.
>
> Assuming that dynamic mitigations get locked down late in boot as part of kernel lockdown, there may only be a relatively small period of time where there is a risk of things changing underneath a task so maybe telling userspace they can't control these until things are locked is ok?

I don't think this should be designed under the assumption that everyone
is using lockdown. It's correct that lockdown disables this feature but
that doesn't mean lockdown is the right security decision for all users.
It would be a shame if it didn't actually support the use case of "oh
no, we found out our assumptions were wrong, we want to change our CPU
posture without rebooting everything". And I think if we just always
returned PR_SPEC_ENABLE that would essentially just mean you can't have
both dynamic control and efficient per-process control.

For PR_SPEC_ENABLE/DISABLE in the return value of
PR_GET_SPECULATION_CTRL, I think we should just return the current state
accurately. The fact that this can now change below userspace's feet
seems kinda above their pay-grade to me.

For PR_SPEC_PRCTl, I guess we could leave it clear and add a new bit to
say "I'm dynamic, you can try the PRCTL but I might randomly fail"? Then
code that doesn't know about the new bit doesn't get surprise errors, it
just has to do whatever fallbacks it would do on unsupported hardware.
Then new code can be updated to deal with random failures gracefully.

This seems a bit over-complicated though, in practice it seems pretty
likely that just hitting people with the unexpected errors is sort of
OK? At least for the usecase I'm seeing everything through the lens of,
I think it would be. 

Re: [RFC PATCH 40/56] x86/alternative: Use sync_core_nmi_safe()

[Dave Hansen]

On 10/16/25 08:34, Kaplan, David wrote:
>> BTW, will AMD do that SERIALIZE instruction?
> It is not supported on current parts, I can't comment on future ones.

I think that was Peter's nice way of asking AMD to go implement
SERIALIZE. ;)

Re: [RFC PATCH 40/56] x86/alternative: Use sync_core_nmi_safe()

[Borislav Petkov]

On Thu, Oct 16, 2025 at 09:15:32AM -0700, Dave Hansen wrote:
> On 10/16/25 08:34, Kaplan, David wrote:
> >> BTW, will AMD do that SERIALIZE instruction?
> > It is not supported on current parts, I can't comment on future ones.
> 
> I think that was Peter's nice way of asking AMD to go implement
> SERIALIZE. ;)

Why does it matter?

:-P

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: [RFC PATCH 40/56] x86/alternative: Use sync_core_nmi_safe()

[Peter Zijlstra]

On Thu, Oct 16, 2025 at 03:34:14PM +0000, Kaplan, David wrote:

> I will note that on AMD, MFENCE has the required serializing
> properties (like SERIALIZE).  On AMD, we could use MFENCE in
> sync_core() which is probably faster that iret_to_self().  But
> again...do we really care.

About faster, no. But MFENCE has the benefit of not causing VMEXITs and
also not being IRET, so I'm not opposed to you using that as an AMD
version of SERIALIZE for the time being.

RE: [RFC PATCH 40/56] x86/alternative: Use sync_core_nmi_safe()

[Kaplan, David]

[AMD Official Use Only - AMD Internal Distribution Only]

> -----Original Message-----
> From: Peter Zijlstra <peterz@infradead.org>
> Sent: Thursday, October 16, 2025 1:53 PM
> To: Kaplan, David <David.Kaplan@amd.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>; Borislav Petkov <bp@alien8.de>; Josh
> Poimboeuf <jpoimboe@kernel.org>; Pawan Gupta
> <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar <mingo@redhat.com>; Dave
> Hansen <dave.hansen@linux.intel.com>; x86@kernel.org; H . Peter Anvin
> <hpa@zytor.com>; Alexander Graf <graf@amazon.com>; Boris Ostrovsky
> <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
> Subject: Re: [RFC PATCH 40/56] x86/alternative: Use sync_core_nmi_safe()
>
> Caution: This message originated from an External Source. Use proper caution
> when opening attachments, clicking links, or responding.
>
>
> On Thu, Oct 16, 2025 at 03:34:14PM +0000, Kaplan, David wrote:
>
> > I will note that on AMD, MFENCE has the required serializing
> > properties (like SERIALIZE).  On AMD, we could use MFENCE in
> > sync_core() which is probably faster that iret_to_self().  But
> > again...do we really care.
>
> About faster, no. But MFENCE has the benefit of not causing VMEXITs and
> also not being IRET, so I'm not opposed to you using that as an AMD
> version of SERIALIZE for the time being.

Ok.  Btw, how long has Intel supported SERIALIZE?

Do we even need a 'mov-cr2' version of sync_core or could we say that dynamic mitigations requires a CPU capable of either a serializing MFENCE or SERIALIZE.

--David Kaplan

Re: [RFC PATCH 40/56] x86/alternative: Use sync_core_nmi_safe()

[Peter Zijlstra]

On Thu, Oct 16, 2025 at 06:56:41PM +0000, Kaplan, David wrote:

> > About faster, no. But MFENCE has the benefit of not causing VMEXITs and
> > also not being IRET, so I'm not opposed to you using that as an AMD
> > version of SERIALIZE for the time being.
> 
> Ok.  Btw, how long has Intel supported SERIALIZE?

I'm not sure.. its fairly new, so

> Do we even need a 'mov-cr2' version of sync_core or could we say that
> dynamic mitigations requires a CPU capable of either a serializing
> MFENCE or SERIALIZE.

don't think we can get away with that, alas.

Re: [RFC PATCH 02/56] x86/Kconfig: Add CONFIG_DYNAMIC_MITIGATIONS

[Josh Poimboeuf]

On Mon, Oct 13, 2025 at 09:33:50AM -0500, David Kaplan wrote:
> CONFIG_DYNAMIC_MITIGATIONS enables support for runtime re-patching of the
> kernel when mitigation selections are changed.  It depends on
> CONFIG_LIVEPATCH because it needs modules to preserve all their ELF
> information for later re-patching.  It also depends on CONFIG_FREEZER
> because re-patching must be done while all tasks are in the freezer to
> avoid race conditions.

The LIVEPATCH dependency seems a bit arbitrary, can we have an
underlying CONFIG_MODULE_PRESERVE_ELF, which LIVEPATCH and
DYNAMIC_MITIGATIONS both select?

-- 
Josh

Re: [RFC PATCH 01/56] Documentation/admin-guide: Add documentation

[Josh Poimboeuf]

On Mon, Oct 13, 2025 at 09:33:49AM -0500, David Kaplan wrote:
> +Runtime Limitations
> +^^^^^^^^^^^^^^^^^^^
> +
> +There are a few mitigations that cannot be toggled at runtime due to the way
> +they are structured.  Specifically, kernel PTI (page table isolation) cannot be
> +toggled because of the complexity of this mitigation.  Additionally, SMT cannot
> +be disabled at runtime.  Therefore, if a bug mitigation requires disabling SMT,
> +a warning message will be printed.

Is there a particular reason SMT can't be disabled?  There's definitely
a way to do it, see /sys/devices/system/cpu/smt/{active,control}.

-- 
Josh

Re: [RFC PATCH 03/56] cpu: Reset global mitigations

[Josh Poimboeuf]

On Mon, Oct 13, 2025 at 09:33:51AM -0500, David Kaplan wrote:
> +#ifdef CONFIG_DYNAMIC_MITIGATIONS
> +void __weak arch_cpu_reset_mitigations(void)
> +{
> +}
> +
> +void cpu_reset_mitigations(void)
> +{
> +	smt_mitigations = SMT_MITIGATIONS_AUTO;
> +	cpu_mitigations = CPU_MITIGATIONS_AUTO;
> +	attack_vectors[CPU_MITIGATE_USER_KERNEL] = true;
> +	attack_vectors[CPU_MITIGATE_USER_USER] = true;
> +	attack_vectors[CPU_MITIGATE_GUEST_HOST] = IS_ENABLED(CONFIG_KVM);
> +	attack_vectors[CPU_MITIGATE_GUEST_GUEST] = IS_ENABLED(CONFIG_KVM);
> +	arch_cpu_reset_mitigations();
> +}
> +#endif

Considering this will have no effect on other arches (or even on x86 at
this point in the series), should CONFIG_DYNAMIC_MITIGATIONS depend on
an arch-specific CONFIG_HAVE_DYNAMIC_MITIGATIONS?

Then the weak function can be removed (and weak functions should be
avoided anyway, IMO).

-- 
Josh

Re: [RFC PATCH 40/56] x86/alternative: Use sync_core_nmi_safe()

[Andrew Cooper]

>> Ok. Btw, how long has Intel supported SERIALIZE?
> I'm not sure.. its fairly new, so

Serialise was introduced in GLC/GMT, so AlderLake and Sapphire Rapids. 
It's not even 4 years old.

~Andrew

RE: [RFC PATCH 02/56] x86/Kconfig: Add CONFIG_DYNAMIC_MITIGATIONS

[Kaplan, David]

[AMD Official Use Only - AMD Internal Distribution Only]

> -----Original Message-----
> From: Josh Poimboeuf <jpoimboe@kernel.org>
> Sent: Thursday, October 16, 2025 4:20 PM
> To: Kaplan, David <David.Kaplan@amd.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>; Borislav Petkov <bp@alien8.de>; Peter
> Zijlstra <peterz@infradead.org>; Pawan Gupta
> <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar <mingo@redhat.com>; Dave
> Hansen <dave.hansen@linux.intel.com>; x86@kernel.org; H . Peter Anvin
> <hpa@zytor.com>; Alexander Graf <graf@amazon.com>; Boris Ostrovsky
> <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
> Subject: Re: [RFC PATCH 02/56] x86/Kconfig: Add
> CONFIG_DYNAMIC_MITIGATIONS
>
> Caution: This message originated from an External Source. Use proper caution
> when opening attachments, clicking links, or responding.
>
>
> On Mon, Oct 13, 2025 at 09:33:50AM -0500, David Kaplan wrote:
> > CONFIG_DYNAMIC_MITIGATIONS enables support for runtime re-patching of the
> > kernel when mitigation selections are changed.  It depends on
> > CONFIG_LIVEPATCH because it needs modules to preserve all their ELF
> > information for later re-patching.  It also depends on CONFIG_FREEZER
> > because re-patching must be done while all tasks are in the freezer to
> > avoid race conditions.
>
> The LIVEPATCH dependency seems a bit arbitrary, can we have an
> underlying CONFIG_MODULE_PRESERVE_ELF, which LIVEPATCH and
> DYNAMIC_MITIGATIONS both select?
>

Ah, that's a good idea.  I'll look into that.

Thanks
--David Kaplan

RE: [RFC PATCH 01/56] Documentation/admin-guide: Add documentation

[Kaplan, David]

[AMD Official Use Only - AMD Internal Distribution Only]

> -----Original Message-----
> From: Josh Poimboeuf <jpoimboe@kernel.org>
> Sent: Thursday, October 16, 2025 4:25 PM
> To: Kaplan, David <David.Kaplan@amd.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>; Borislav Petkov <bp@alien8.de>; Peter
> Zijlstra <peterz@infradead.org>; Pawan Gupta
> <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar <mingo@redhat.com>; Dave
> Hansen <dave.hansen@linux.intel.com>; x86@kernel.org; H . Peter Anvin
> <hpa@zytor.com>; Alexander Graf <graf@amazon.com>; Boris Ostrovsky
> <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
> Subject: Re: [RFC PATCH 01/56] Documentation/admin-guide: Add documentation
>
> Caution: This message originated from an External Source. Use proper caution
> when opening attachments, clicking links, or responding.
>
>
> On Mon, Oct 13, 2025 at 09:33:49AM -0500, David Kaplan wrote:
> > +Runtime Limitations
> > +^^^^^^^^^^^^^^^^^^^
> > +
> > +There are a few mitigations that cannot be toggled at runtime due to the way
> > +they are structured.  Specifically, kernel PTI (page table isolation) cannot be
> > +toggled because of the complexity of this mitigation.  Additionally, SMT cannot
> > +be disabled at runtime.  Therefore, if a bug mitigation requires disabling SMT,
> > +a warning message will be printed.
>
> Is there a particular reason SMT can't be disabled?  There's definitely
> a way to do it, see /sys/devices/system/cpu/smt/{active,control}.
>

Reason was I didn't realize you could do that :)

The existing bugs.c logic used cpu_smt_disable() which is an __init function.  But now I see that if you have CONFIG_HOTPLUG_SMT then cpuhp_smt_enable()/cpuhp_smt_disable() look like they should be able to handle this.

I will look into supporting this, if CONFIG_HOTPLUG_SMT is enabled.

Thanks
--David Kaplan

RE: [RFC PATCH 03/56] cpu: Reset global mitigations

[Kaplan, David]

[AMD Official Use Only - AMD Internal Distribution Only]

> -----Original Message-----
> From: Josh Poimboeuf <jpoimboe@kernel.org>
> Sent: Thursday, October 16, 2025 4:34 PM
> To: Kaplan, David <David.Kaplan@amd.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>; Borislav Petkov <bp@alien8.de>; Peter
> Zijlstra <peterz@infradead.org>; Pawan Gupta
> <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar <mingo@redhat.com>; Dave
> Hansen <dave.hansen@linux.intel.com>; x86@kernel.org; H . Peter Anvin
> <hpa@zytor.com>; Alexander Graf <graf@amazon.com>; Boris Ostrovsky
> <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
> Subject: Re: [RFC PATCH 03/56] cpu: Reset global mitigations
>
> Caution: This message originated from an External Source. Use proper caution
> when opening attachments, clicking links, or responding.
>
>
> On Mon, Oct 13, 2025 at 09:33:51AM -0500, David Kaplan wrote:
> > +#ifdef CONFIG_DYNAMIC_MITIGATIONS
> > +void __weak arch_cpu_reset_mitigations(void)
> > +{
> > +}
> > +
> > +void cpu_reset_mitigations(void)
> > +{
> > +     smt_mitigations = SMT_MITIGATIONS_AUTO;
> > +     cpu_mitigations = CPU_MITIGATIONS_AUTO;
> > +     attack_vectors[CPU_MITIGATE_USER_KERNEL] = true;
> > +     attack_vectors[CPU_MITIGATE_USER_USER] = true;
> > +     attack_vectors[CPU_MITIGATE_GUEST_HOST] =
> IS_ENABLED(CONFIG_KVM);
> > +     attack_vectors[CPU_MITIGATE_GUEST_GUEST] =
> IS_ENABLED(CONFIG_KVM);
> > +     arch_cpu_reset_mitigations();
> > +}
> > +#endif
>
> Considering this will have no effect on other arches (or even on x86 at
> this point in the series), should CONFIG_DYNAMIC_MITIGATIONS depend on
> an arch-specific CONFIG_HAVE_DYNAMIC_MITIGATIONS?
>
> Then the weak function can be removed (and weak functions should be
> avoided anyway, IMO).
>

Ok.  I agree, the feature doesn't make sense without arch-specific support anyway, so that seems reasonable.

Thanks --David Kaplan

RE: [RFC PATCH 03/56] cpu: Reset global mitigations

[Kaplan, David]

[AMD Official Use Only - AMD Internal Distribution Only]

> -----Original Message-----
> From: Kaplan, David
> Sent: Friday, October 17, 2025 9:06 AM
> To: Josh Poimboeuf <jpoimboe@kernel.org>
> Cc: Thomas Gleixner <tglx@linutronix.de>; Borislav Petkov <bp@alien8.de>; Peter
> Zijlstra <peterz@infradead.org>; Pawan Gupta
> <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar <mingo@redhat.com>; Dave
> Hansen <dave.hansen@linux.intel.com>; x86@kernel.org; H . Peter Anvin
> <hpa@zytor.com>; Alexander Graf <graf@amazon.com>; Boris Ostrovsky
> <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
> Subject: RE: [RFC PATCH 03/56] cpu: Reset global mitigations
>
>
>
> > -----Original Message-----
> > From: Josh Poimboeuf <jpoimboe@kernel.org>
> > Sent: Thursday, October 16, 2025 4:34 PM
> > To: Kaplan, David <David.Kaplan@amd.com>
> > Cc: Thomas Gleixner <tglx@linutronix.de>; Borislav Petkov <bp@alien8.de>;
> Peter
> > Zijlstra <peterz@infradead.org>; Pawan Gupta
> > <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar <mingo@redhat.com>; Dave
> > Hansen <dave.hansen@linux.intel.com>; x86@kernel.org; H . Peter Anvin
> > <hpa@zytor.com>; Alexander Graf <graf@amazon.com>; Boris Ostrovsky
> > <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
> > Subject: Re: [RFC PATCH 03/56] cpu: Reset global mitigations
> >
> > Caution: This message originated from an External Source. Use proper caution
> > when opening attachments, clicking links, or responding.
> >
> >
> > On Mon, Oct 13, 2025 at 09:33:51AM -0500, David Kaplan wrote:
> > > +#ifdef CONFIG_DYNAMIC_MITIGATIONS
> > > +void __weak arch_cpu_reset_mitigations(void)
> > > +{
> > > +}
> > > +
> > > +void cpu_reset_mitigations(void)
> > > +{
> > > +     smt_mitigations = SMT_MITIGATIONS_AUTO;
> > > +     cpu_mitigations = CPU_MITIGATIONS_AUTO;
> > > +     attack_vectors[CPU_MITIGATE_USER_KERNEL] = true;
> > > +     attack_vectors[CPU_MITIGATE_USER_USER] = true;
> > > +     attack_vectors[CPU_MITIGATE_GUEST_HOST] =
> > IS_ENABLED(CONFIG_KVM);
> > > +     attack_vectors[CPU_MITIGATE_GUEST_GUEST] =
> > IS_ENABLED(CONFIG_KVM);
> > > +     arch_cpu_reset_mitigations();
> > > +}
> > > +#endif
> >
> > Considering this will have no effect on other arches (or even on x86 at
> > this point in the series), should CONFIG_DYNAMIC_MITIGATIONS depend on
> > an arch-specific CONFIG_HAVE_DYNAMIC_MITIGATIONS?
> >
> > Then the weak function can be removed (and weak functions should be
> > avoided anyway, IMO).
> >
>
> Ok.  I agree, the feature doesn't make sense without arch-specific support anyway,
> so that seems reasonable.

Well, so right now CONFIG_DYNAMIC_MITIGATIONS is only defined in arch/x86/Kconfig so it's already arch-specific.  It's part of the CPU mitigations menu there.

So I guess these weak functions aren't actually needed in the first place.

Alternatively I suppose I could make CONFIG_DYNAMIC_MITIGATIONS a generic feature (which requires an arch to support it).

--David Kaplan

Re: [RFC PATCH 08/56] x86/bugs: Reset SSB mitigations

[Nikolay Borisov]

On 10/13/25 17:33, David Kaplan wrote:
> Add function to reset SSB mitigations back to their boot-time defaults.
> 
> Signed-off-by: David Kaplan <david.kaplan@amd.com>
> ---
>   arch/x86/kernel/cpu/bugs.c | 24 ++++++++++++++++++++++++
>   1 file changed, 24 insertions(+)
> 
> diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
> index 4ca46f58e384..cc7b1b67d22d 100644
> --- a/arch/x86/kernel/cpu/bugs.c
> +++ b/arch/x86/kernel/cpu/bugs.c
> @@ -380,6 +380,16 @@ static void x86_amd_ssb_disable(void)
>   		wrmsrq(MSR_AMD64_LS_CFG, msrval);
>   }
>   
> +static void x86_amd_ssb_enable(void)
> +{
> +	u64 msrval = x86_amd_ls_cfg_base;
> +
> +	if (boot_cpu_has(X86_FEATURE_VIRT_SSBD))
> +		wrmsrl(MSR_AMD64_VIRT_SPEC_CTRL, 0);
> +	else if (boot_cpu_has(X86_FEATURE_LS_CFG_SSBD))
> +		wrmsrl(MSR_AMD64_LS_CFG, msrval);

nit: No need for the local msrval variable, just pass x86_amd_ls_cfg_base.

> +}
> +
>   #undef pr_fmt
>   #define pr_fmt(fmt)	"MDS: " fmt
>   
> @@ -2672,6 +2682,17 @@ static void __init ssb_apply_mitigation(void)
>   	}
>   }
>   
> +#ifdef CONFIG_DYNAMIC_MITIGATIONS
> +static void ssb_reset_mitigation(void)
> +{
> +	setup_clear_cpu_cap(X86_FEATURE_SPEC_STORE_BYPASS_DISABLE);
> +	x86_spec_ctrl_base &= ~SPEC_CTRL_SSBD;
> +	nossb = false;
> +	ssb_mode = IS_ENABLED(CONFIG_MITIGATION_SSB) ?
> +		SPEC_STORE_BYPASS_AUTO : SPEC_STORE_BYPASS_NONE;
> +}
> +#endif
> +
>   #undef pr_fmt
>   #define pr_fmt(fmt)     "Speculation prctl: " fmt
>   
> @@ -2916,6 +2937,8 @@ void x86_spec_ctrl_setup_ap(void)
>   
>   	if (ssb_mode == SPEC_STORE_BYPASS_DISABLE)
>   		x86_amd_ssb_disable();
> +	else
> +		x86_amd_ssb_enable();

Does it mean SSB hasn't been working correctly up until now since 
_enable function has been missing?

<snip>

RE: [RFC PATCH 08/56] x86/bugs: Reset SSB mitigations

[Kaplan, David]

[AMD Official Use Only - AMD Internal Distribution Only]

> -----Original Message-----
> From: Nikolay Borisov <nik.borisov@suse.com>
> Sent: Friday, October 17, 2025 10:14 AM
> To: Kaplan, David <David.Kaplan@amd.com>; Thomas Gleixner
> <tglx@linutronix.de>; Borislav Petkov <bp@alien8.de>; Peter Zijlstra
> <peterz@infradead.org>; Josh Poimboeuf <jpoimboe@kernel.org>; Pawan Gupta
> <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar <mingo@redhat.com>; Dave
> Hansen <dave.hansen@linux.intel.com>; x86@kernel.org; H . Peter Anvin
> <hpa@zytor.com>
> Cc: Alexander Graf <graf@amazon.com>; Boris Ostrovsky
> <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
> Subject: Re: [RFC PATCH 08/56] x86/bugs: Reset SSB mitigations
>
> Caution: This message originated from an External Source. Use proper caution
> when opening attachments, clicking links, or responding.
>
>
> On 10/13/25 17:33, David Kaplan wrote:
> > Add function to reset SSB mitigations back to their boot-time defaults.
> >
> > Signed-off-by: David Kaplan <david.kaplan@amd.com>
> > ---
> >   arch/x86/kernel/cpu/bugs.c | 24 ++++++++++++++++++++++++
> >   1 file changed, 24 insertions(+)
> >
> > diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
> > index 4ca46f58e384..cc7b1b67d22d 100644
> > --- a/arch/x86/kernel/cpu/bugs.c
> > +++ b/arch/x86/kernel/cpu/bugs.c
> > @@ -380,6 +380,16 @@ static void x86_amd_ssb_disable(void)
> >               wrmsrq(MSR_AMD64_LS_CFG, msrval);
> >   }
> >
> > +static void x86_amd_ssb_enable(void)
> > +{
> > +     u64 msrval = x86_amd_ls_cfg_base;
> > +
> > +     if (boot_cpu_has(X86_FEATURE_VIRT_SSBD))
> > +             wrmsrl(MSR_AMD64_VIRT_SPEC_CTRL, 0);
> > +     else if (boot_cpu_has(X86_FEATURE_LS_CFG_SSBD))
> > +             wrmsrl(MSR_AMD64_LS_CFG, msrval);
>
> nit: No need for the local msrval variable, just pass x86_amd_ls_cfg_base.

Ok

>
> > +}
> > +
> >   #undef pr_fmt
> >   #define pr_fmt(fmt) "MDS: " fmt
> >
> > @@ -2672,6 +2682,17 @@ static void __init ssb_apply_mitigation(void)
> >       }
> >   }
> >
> > +#ifdef CONFIG_DYNAMIC_MITIGATIONS
> > +static void ssb_reset_mitigation(void)
> > +{
> > +     setup_clear_cpu_cap(X86_FEATURE_SPEC_STORE_BYPASS_DISABLE);
> > +     x86_spec_ctrl_base &= ~SPEC_CTRL_SSBD;
> > +     nossb = false;
> > +     ssb_mode = IS_ENABLED(CONFIG_MITIGATION_SSB) ?
> > +             SPEC_STORE_BYPASS_AUTO : SPEC_STORE_BYPASS_NONE;
> > +}
> > +#endif
> > +
> >   #undef pr_fmt
> >   #define pr_fmt(fmt)     "Speculation prctl: " fmt
> >
> > @@ -2916,6 +2937,8 @@ void x86_spec_ctrl_setup_ap(void)
> >
> >       if (ssb_mode == SPEC_STORE_BYPASS_DISABLE)
> >               x86_amd_ssb_disable();
> > +     else
> > +             x86_amd_ssb_enable();
>
> Does it mean SSB hasn't been working correctly up until now since
> _enable function has been missing?
>

I think it's been ok...SSB is enabled by default out of reset, so the kernel only had to deal with potentially disabling it (setting SSBD) in this part of the logic.

--David Kaplan

Re: [RFC PATCH 03/56] cpu: Reset global mitigations

[Josh Poimboeuf]

On Fri, Oct 17, 2025 at 02:19:43PM +0000, Kaplan, David wrote:
> > > Considering this will have no effect on other arches (or even on x86 at
> > > this point in the series), should CONFIG_DYNAMIC_MITIGATIONS depend on
> > > an arch-specific CONFIG_HAVE_DYNAMIC_MITIGATIONS?
> > >
> > > Then the weak function can be removed (and weak functions should be
> > > avoided anyway, IMO).
> > >
> >
> > Ok.  I agree, the feature doesn't make sense without arch-specific support anyway,
> > so that seems reasonable.
> 
> Well, so right now CONFIG_DYNAMIC_MITIGATIONS is only defined in
> arch/x86/Kconfig so it's already arch-specific.  It's part of the CPU
> mitigations menu there.
> 
> So I guess these weak functions aren't actually needed in the first
> place.
> 
> Alternatively I suppose I could make CONFIG_DYNAMIC_MITIGATIONS a
> generic feature (which requires an arch to support it).

I'd say generic is probably the way to go, as the sysfs files and
mitigations= interfaces are already generic, and users might want this
on other arches eventually.

-- 
Josh

Re: [RFC PATCH 03/56] cpu: Reset global mitigations

[Borislav Petkov]

On Fri, Oct 17, 2025 at 09:03:08AM -0700, Josh Poimboeuf wrote:
> I'd say generic is probably the way to go, as the sysfs files and
> mitigations= interfaces are already generic, and users might want this
> on other arches eventually.

Then users will do the move then. What's the point of making some Kconfig glue
generic if there's no other-arch implementation in the works?

And if other arch does that, they can do those minor Kconfig modifications
then. We don't do "someone might want this" code design anyway...

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: [RFC PATCH 01/56] Documentation/admin-guide: Add documentation

[Borislav Petkov]

On Mon, Oct 13, 2025 at 09:33:49AM -0500, David Kaplan wrote:
> +Move Policy To Userspace
> +^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +Mitigation choices are related to the security policy and posture of the system.
> +Most mitigations are only necessary on shared, multi-user systems if untrusted
> +code may be run on the system, such as through untrusted userspace or untrusted
> +virtual machines.  The kernel may not know how the system will be used on boot,
									^^^^^^^^^^

"after it has been booted" I'd say.

> +and therefore must adopt a strong security posture for safety.
> +
> +With dynamic mitigations, userspace can re-select mitigations once the needs of
> +the system can be determined and more policy information is available.
> +
> +Mitigation Testing
> +^^^^^^^^^^^^^^^^^^
> +
> +Dynamic mitigation support makes it easy to toggle individual mitigations or
> +choose between different mitigation options without the expense of a reboot or
> +kexec.  This may be useful when evaluating the performance of various
> +mitigation options.  It can also be useful for performing bug fixes without a

"for fixing bugs in the mitigations themselves" - simpler

> +reboot, in case a particular mitigation is undesired or buggy.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

RE: [RFC PATCH 01/56] Documentation/admin-guide: Add documentation

[Kaplan, David]

[AMD Official Use Only - AMD Internal Distribution Only]

> -----Original Message-----
> From: Borislav Petkov <bp@alien8.de>
> Sent: Saturday, October 18, 2025 8:39 AM
> To: Kaplan, David <David.Kaplan@amd.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>; Peter Zijlstra <peterz@infradead.org>;
> Josh Poimboeuf <jpoimboe@kernel.org>; Pawan Gupta
> <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar <mingo@redhat.com>; Dave
> Hansen <dave.hansen@linux.intel.com>; x86@kernel.org; H . Peter Anvin
> <hpa@zytor.com>; Alexander Graf <graf@amazon.com>; Boris Ostrovsky
> <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
> Subject: Re: [RFC PATCH 01/56] Documentation/admin-guide: Add documentation
>
> Caution: This message originated from an External Source. Use proper caution
> when opening attachments, clicking links, or responding.
>
>
> On Mon, Oct 13, 2025 at 09:33:49AM -0500, David Kaplan wrote:
> > +Move Policy To Userspace
> > +^^^^^^^^^^^^^^^^^^^^^^^^
> > +
> > +Mitigation choices are related to the security policy and posture of the system.
> > +Most mitigations are only necessary on shared, multi-user systems if untrusted
> > +code may be run on the system, such as through untrusted userspace or
> untrusted
> > +virtual machines.  The kernel may not know how the system will be used on
> boot,
>                                                                         ^^^^^^^^^^
>
> "after it has been booted" I'd say.

Ack

>
> > +and therefore must adopt a strong security posture for safety.
> > +
> > +With dynamic mitigations, userspace can re-select mitigations once the needs of
> > +the system can be determined and more policy information is available.
> > +
> > +Mitigation Testing
> > +^^^^^^^^^^^^^^^^^^
> > +
> > +Dynamic mitigation support makes it easy to toggle individual mitigations or
> > +choose between different mitigation options without the expense of a reboot or
> > +kexec.  This may be useful when evaluating the performance of various
> > +mitigation options.  It can also be useful for performing bug fixes without a
>
> "for fixing bugs in the mitigations themselves" - simpler
>
> > +reboot, in case a particular mitigation is undesired or buggy.
>

Ack.  Although to be clear, all it can do is select a different mitigation.  For instance, if there was a bug in safe-RET, it can't fix that bug.  But it could allow selecting a different SRSO mitigation.

Thanks
--David Kaplan

RE: [RFC PATCH 40/56] x86/alternative: Use sync_core_nmi_safe()

[Kaplan, David]

[AMD Official Use Only - AMD Internal Distribution Only]

> -----Original Message-----
> From: Peter Zijlstra <peterz@infradead.org>
> Sent: Thursday, October 16, 2025 9:48 AM
> To: Kaplan, David <David.Kaplan@amd.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>; Borislav Petkov <bp@alien8.de>; Josh
> Poimboeuf <jpoimboe@kernel.org>; Pawan Gupta
> <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar <mingo@redhat.com>; Dave
> Hansen <dave.hansen@linux.intel.com>; x86@kernel.org; H . Peter Anvin
> <hpa@zytor.com>; Alexander Graf <graf@amazon.com>; Boris Ostrovsky
> <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
> Subject: Re: [RFC PATCH 40/56] x86/alternative: Use sync_core_nmi_safe()
>
> Caution: This message originated from an External Source. Use proper caution
> when opening attachments, clicking links, or responding.
>
>
> On Thu, Oct 16, 2025 at 02:40:51PM +0000, Kaplan, David wrote:
>
> > > Can we please keep this in sync_core()? Something like:
> > >
> > > static __always_inline void sync_core(void)
> > > {
> > >         if (static_cpu_has(X86_FEATURE_SERIALIZE)) {
> > >                 serialize();
> > >                 return;
> > >         }
> > >
> > > +       if (repatch_in_progress) {
> > > +               sync_core_nmi_safe();
> > > +               return;
> > > +       }
> > > +
> > >         iret_to_self();
> > > }
> > >
> > > That way all the modern stuff that has SERIALIZE will still use that.
> >
> > Hmm, I can't quite do that because sync_core() is used in a number of
> > other places too (unless we make repatch_in_progress a true global).
>
> We could just out-of-line the thing; nothing using this should care
> about cycles -- all of this is quite expensive.
>

Coming back to this, are you thinking we should just create something like 'text_poke_sync_core()' inside alternative.c and that can use:
   1. SERIALIZE (if available)
   2. MOV-CR2 (if re-patching)
   3. Else, IRET

And maybe someday we put MFENCE into there too for AMD parts.

Right now, of course this is the only logic that would care about an NMI-safe sync_core().  So maybe this makes sense vs creating a generic version that nobody else is using?

--David Kaplan

Re: [RFC PATCH 40/56] x86/alternative: Use sync_core_nmi_safe()

[Peter Zijlstra]

On Mon, Oct 20, 2025 at 02:49:56PM +0000, Kaplan, David wrote:

> Coming back to this, are you thinking we should just create something
> like 'text_poke_sync_core()' inside alternative.c and that can use:
>    1. SERIALIZE (if available)
>    2. MOV-CR2 (if re-patching)
>    3. Else, IRET
> 
> And maybe someday we put MFENCE into there too for AMD parts.
> 
> Right now, of course this is the only logic that would care about an
> NMI-safe sync_core().  So maybe this makes sense vs creating a generic
> version that nobody else is using?

I was thinking something fairly straight forward like the below. Yes,
there are a few more sync_core() callers out there, git tells me:

arch/x86/kernel/alternative.c:          sync_core();
arch/x86/kernel/alternative.c:noinstr void sync_core(void)
arch/x86/kernel/alternative.c:  sync_core();
arch/x86/kernel/cpu/mce/core.c: sync_core();
arch/x86/kernel/cpu/mce/core.c:         sync_core();
arch/x86/kernel/static_call.c:  sync_core();
drivers/misc/sgi-gru/grufault.c:                sync_core();
drivers/misc/sgi-gru/grufault.c:                sync_core();            /* make sure we are have current data */
drivers/misc/sgi-gru/gruhandles.c:      sync_core();
drivers/misc/sgi-gru/gruhandles.c:      sync_core();
drivers/misc/sgi-gru/grukservices.c:    sync_core();

But none of that seems like it cares about an extra few cycles, and why
complicate matters with another sync_core variant and all that.


diff --git a/arch/x86/include/asm/sync_core.h b/arch/x86/include/asm/sync_core.h
index 96bda43538ee..ef4508a03800 100644
--- a/arch/x86/include/asm/sync_core.h
+++ b/arch/x86/include/asm/sync_core.h
@@ -7,86 +7,7 @@
 #include <asm/cpufeature.h>
 #include <asm/special_insns.h>
 
-#ifdef CONFIG_X86_32
-static __always_inline void iret_to_self(void)
-{
-	asm volatile (
-		"pushfl\n\t"
-		"pushl %%cs\n\t"
-		"pushl $1f\n\t"
-		"iret\n\t"
-		"1:"
-		: ASM_CALL_CONSTRAINT : : "memory");
-}
-#else
-static __always_inline void iret_to_self(void)
-{
-	unsigned int tmp;
-
-	asm volatile (
-		"mov %%ss, %0\n\t"
-		"pushq %q0\n\t"
-		"pushq %%rsp\n\t"
-		"addq $8, (%%rsp)\n\t"
-		"pushfq\n\t"
-		"mov %%cs, %0\n\t"
-		"pushq %q0\n\t"
-		"pushq $1f\n\t"
-		"iretq\n\t"
-		"1:"
-		: "=&r" (tmp), ASM_CALL_CONSTRAINT : : "cc", "memory");
-}
-#endif /* CONFIG_X86_32 */
-
-/*
- * This function forces the icache and prefetched instruction stream to
- * catch up with reality in two very specific cases:
- *
- *  a) Text was modified using one virtual address and is about to be executed
- *     from the same physical page at a different virtual address.
- *
- *  b) Text was modified on a different CPU, may subsequently be
- *     executed on this CPU, and you want to make sure the new version
- *     gets executed.  This generally means you're calling this in an IPI.
- *
- * If you're calling this for a different reason, you're probably doing
- * it wrong.
- *
- * Like all of Linux's memory ordering operations, this is a
- * compiler barrier as well.
- */
-static __always_inline void sync_core(void)
-{
-	/*
-	 * The SERIALIZE instruction is the most straightforward way to
-	 * do this, but it is not universally available.
-	 */
-	if (static_cpu_has(X86_FEATURE_SERIALIZE)) {
-		serialize();
-		return;
-	}
-
-	/*
-	 * For all other processors, there are quite a few ways to do this.
-	 * IRET-to-self is nice because it works on every CPU, at any CPL
-	 * (so it's compatible with paravirtualization), and it never exits
-	 * to a hypervisor.  The only downsides are that it's a bit slow
-	 * (it seems to be a bit more than 2x slower than the fastest
-	 * options) and that it unmasks NMIs.  The "push %cs" is needed,
-	 * because in paravirtual environments __KERNEL_CS may not be a
-	 * valid CS value when we do IRET directly.
-	 *
-	 * In case NMI unmasking or performance ever becomes a problem,
-	 * the next best option appears to be MOV-to-CR2 and an
-	 * unconditional jump.  That sequence also works on all CPUs,
-	 * but it will fault at CPL3 (i.e. Xen PV).
-	 *
-	 * CPUID is the conventional way, but it's nasty: it doesn't
-	 * exist on some 486-like CPUs, and it usually exits to a
-	 * hypervisor.
-	 */
-	iret_to_self();
-}
+extern void sync_core(void);
 
 /*
  * Ensure that a core serializing instruction is issued before returning
diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c
index e377b06e70e3..2a5daae3626b 100644
--- a/arch/x86/kernel/alternative.c
+++ b/arch/x86/kernel/alternative.c
@@ -2687,6 +2687,87 @@ void *text_poke_set(void *addr, int c, size_t len)
 	return addr;
 }
 
+#ifdef CONFIG_X86_32
+static __always_inline void iret_to_self(void)
+{
+	asm volatile (
+		"pushfl\n\t"
+		"pushl %%cs\n\t"
+		"pushl $1f\n\t"
+		"iret\n\t"
+		"1:"
+		: ASM_CALL_CONSTRAINT : : "memory");
+}
+#else
+static __always_inline void iret_to_self(void)
+{
+	unsigned int tmp;
+
+	asm volatile (
+		"mov %%ss, %0\n\t"
+		"pushq %q0\n\t"
+		"pushq %%rsp\n\t"
+		"addq $8, (%%rsp)\n\t"
+		"pushfq\n\t"
+		"mov %%cs, %0\n\t"
+		"pushq %q0\n\t"
+		"pushq $1f\n\t"
+		"iretq\n\t"
+		"1:"
+		: "=&r" (tmp), ASM_CALL_CONSTRAINT : : "cc", "memory");
+}
+#endif /* CONFIG_X86_32 */
+
+/*
+ * This function forces the icache and prefetched instruction stream to
+ * catch up with reality in two very specific cases:
+ *
+ *  a) Text was modified using one virtual address and is about to be executed
+ *     from the same physical page at a different virtual address.
+ *
+ *  b) Text was modified on a different CPU, may subsequently be
+ *     executed on this CPU, and you want to make sure the new version
+ *     gets executed.  This generally means you're calling this in an IPI.
+ *
+ * If you're calling this for a different reason, you're probably doing
+ * it wrong.
+ *
+ * Like all of Linux's memory ordering operations, this is a
+ * compiler barrier as well.
+ */
+noinstr void sync_core(void)
+{
+	/*
+	 * The SERIALIZE instruction is the most straightforward way to
+	 * do this, but it is not universally available.
+	 */
+	if (static_cpu_has(X86_FEATURE_SERIALIZE)) {
+		serialize();
+		return;
+	}
+
+	/*
+	 * For all other processors, there are quite a few ways to do this.
+	 * IRET-to-self is nice because it works on every CPU, at any CPL
+	 * (so it's compatible with paravirtualization), and it never exits
+	 * to a hypervisor.  The only downsides are that it's a bit slow
+	 * (it seems to be a bit more than 2x slower than the fastest
+	 * options) and that it unmasks NMIs.  The "push %cs" is needed,
+	 * because in paravirtual environments __KERNEL_CS may not be a
+	 * valid CS value when we do IRET directly.
+	 *
+	 * In case NMI unmasking or performance ever becomes a problem,
+	 * the next best option appears to be MOV-to-CR2 and an
+	 * unconditional jump.  That sequence also works on all CPUs,
+	 * but it will fault at CPL3 (i.e. Xen PV).
+	 *
+	 * CPUID is the conventional way, but it's nasty: it doesn't
+	 * exist on some 486-like CPUs, and it usually exits to a
+	 * hypervisor.
+	 */
+	iret_to_self();
+}
+
 static void do_sync_core(void *info)
 {
 	sync_core();

RE: [RFC PATCH 40/56] x86/alternative: Use sync_core_nmi_safe()

[H. Peter Anvin]

On October 20, 2025 7:49:56 AM PDT, "Kaplan, David" <David.Kaplan@amd.com> wrote:
>[AMD Official Use Only - AMD Internal Distribution Only]
>
>> -----Original Message-----
>> From: Peter Zijlstra <peterz@infradead.org>
>> Sent: Thursday, October 16, 2025 9:48 AM
>> To: Kaplan, David <David.Kaplan@amd.com>
>> Cc: Thomas Gleixner <tglx@linutronix.de>; Borislav Petkov <bp@alien8.de>; Josh
>> Poimboeuf <jpoimboe@kernel.org>; Pawan Gupta
>> <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar <mingo@redhat.com>; Dave
>> Hansen <dave.hansen@linux.intel.com>; x86@kernel.org; H . Peter Anvin
>> <hpa@zytor.com>; Alexander Graf <graf@amazon.com>; Boris Ostrovsky
>> <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
>> Subject: Re: [RFC PATCH 40/56] x86/alternative: Use sync_core_nmi_safe()
>>
>> Caution: This message originated from an External Source. Use proper caution
>> when opening attachments, clicking links, or responding.
>>
>>
>> On Thu, Oct 16, 2025 at 02:40:51PM +0000, Kaplan, David wrote:
>>
>> > > Can we please keep this in sync_core()? Something like:
>> > >
>> > > static __always_inline void sync_core(void)
>> > > {
>> > >         if (static_cpu_has(X86_FEATURE_SERIALIZE)) {
>> > >                 serialize();
>> > >                 return;
>> > >         }
>> > >
>> > > +       if (repatch_in_progress) {
>> > > +               sync_core_nmi_safe();
>> > > +               return;
>> > > +       }
>> > > +
>> > >         iret_to_self();
>> > > }
>> > >
>> > > That way all the modern stuff that has SERIALIZE will still use that.
>> >
>> > Hmm, I can't quite do that because sync_core() is used in a number of
>> > other places too (unless we make repatch_in_progress a true global).
>>
>> We could just out-of-line the thing; nothing using this should care
>> about cycles -- all of this is quite expensive.
>>
>
>Coming back to this, are you thinking we should just create something like 'text_poke_sync_core()' inside alternative.c and that can use:
>   1. SERIALIZE (if available)
>   2. MOV-CR2 (if re-patching)
>   3. Else, IRET
>
>And maybe someday we put MFENCE into there too for AMD parts.
>
>Right now, of course this is the only logic that would care about an NMI-safe sync_core().  So maybe this makes sense vs creating a generic version that nobody else is using?
>
>--David Kaplan
>

This is something that can be done with a function call... not fast no matter how to slice it.

Re: [RFC PATCH 31/56] x86/alternative: Prepend nops with retpolines

[David Laight]

On Thu, 16 Oct 2025 13:27:53 +0000
"Kaplan, David" <David.Kaplan@amd.com> wrote:

> [AMD Official Use Only - AMD Internal Distribution Only]
> 
> > -----Original Message-----
> > From: Peter Zijlstra <peterz@infradead.org>
> > Sent: Thursday, October 16, 2025 6:23 AM
> > To: Kaplan, David <David.Kaplan@amd.com>
> > Cc: Thomas Gleixner <tglx@linutronix.de>; Borislav Petkov <bp@alien8.de>; Josh
> > Poimboeuf <jpoimboe@kernel.org>; Pawan Gupta
> > <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar <mingo@redhat.com>; Dave
> > Hansen <dave.hansen@linux.intel.com>; x86@kernel.org; H . Peter Anvin
> > <hpa@zytor.com>; Alexander Graf <graf@amazon.com>; Boris Ostrovsky
> > <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
> > Subject: Re: [RFC PATCH 31/56] x86/alternative: Prepend nops with retpolines
> >
> > Caution: This message originated from an External Source. Use proper caution
> > when opening attachments, clicking links, or responding.
> >
> >
> > On Thu, Oct 16, 2025 at 01:07:17PM +0200, Peter Zijlstra wrote:  
> > > On Mon, Oct 13, 2025 at 09:34:19AM -0500, David Kaplan wrote:  
> > > > When patching retpolines, nops may be required for padding such as when
> > > > turning a 5-byte direct call into a 2-byte indirect call.  Previously,
> > > > these were appended at the end so the code becomes "call *reg;nop;nop;nop"
> > > > for example.  This was fine because it's always going from a larger
> > > > instruction to a smaller one.
> > > >
> > > > But this is a problem if the sequence is transformed from a 2-byte indirect
> > > > to the 5-byte direct call version at runtime because when the called
> > > > function returns, it will be in the middle of the 5-byte call instruction.
> > > >
> > > > To fix this, prepend the nops instead of appending them.  Consequently, the
> > > > return site of the called function is always the same.
> > > >  
> > >
> > > So this results in:
> > >
> > > NOP3; call *%r11
> > >
> > > And you're saying a task can be on the other side of that call and then
> > > return lines up. But what if the task is preempted right after that
> > > NOP3?
> > >
> > > Same for all the alternative patching; what ensures no task is currently
> > > having a register state that is in the middle of things?  
> >
> > Ah, I found it, you freeze everything, which puts it at safe points.  
> 
> Yes.  In fact, I think you were the one who pointed me in that direction :)

Does that help?
It'll stop the cpu prefetch queue containing garbage and let you flush the I-cache,
but I don't see how it can stop the return address after the NOP3 being on the
stack from an earlier interrupt, or even the nmi entry itself.

I'm not sure, but if the kernel is pre-emptable could a sleeping thread have
a stack that includes the address after the NOP3 - eg if an interrupt at
that point is what caused the reschedule.

Clearly using multiple prefixes doesn't have this problem.

	David 


> 
> Despite the freezer though, this patch is necessary in particular because stop_machine_nmi() uses an indirect branch to run the handler.  Which means that while patching is going on, all cores are inside a function which is going to return to after the indirect call site.  And so that needs to be the end of the 5 (or 6) byte sequence.
> 
> --David Kaplan
> 

Re: [RFC PATCH 31/56] x86/alternative: Prepend nops with retpolines

[Peter Zijlstra]

On Wed, Oct 22, 2025 at 09:41:05AM +0100, David Laight wrote:

> > > Ah, I found it, you freeze everything, which puts it at safe points.  
> > 
> > Yes.  In fact, I think you were the one who pointed me in that direction :)
> 
> Does that help?
> It'll stop the cpu prefetch queue containing garbage and let you flush the I-cache,
> but I don't see how it can stop the return address after the NOP3 being on the
> stack from an earlier interrupt, or even the nmi entry itself.
> 
> I'm not sure, but if the kernel is pre-emptable could a sleeping thread have
> a stack that includes the address after the NOP3 - eg if an interrupt at
> that point is what caused the reschedule.

The thing is that freezing is only done at known safe points;
specifically schedule() calls that have TASK_FREEZABLE set.

Typically this is the return to userspace point and for kernel threads
somewhere in their main event loop.

This ensures tasks are not preempted at random points like in the middle
of an alternative.

Re: [RFC PATCH 01/56] Documentation/admin-guide: Add documentation

[Borislav Petkov]

On Mon, Oct 20, 2025 at 01:53:11PM +0000, Kaplan, David wrote:
> Ack.  Although to be clear, all it can do is select a different mitigation.
> For instance, if there was a bug in safe-RET, it can't fix that bug.  But it
> could allow selecting a different SRSO mitigation.

Yeah, let's spell it out as this would be read by users too, I hope.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

RE: [RFC PATCH 40/56] x86/alternative: Use sync_core_nmi_safe()

[Kaplan, David]

[AMD Official Use Only - AMD Internal Distribution Only]

> -----Original Message-----
> From: Peter Zijlstra <peterz@infradead.org>
> Sent: Monday, October 20, 2025 10:02 AM
> To: Kaplan, David <David.Kaplan@amd.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>; Borislav Petkov <bp@alien8.de>; Josh
> Poimboeuf <jpoimboe@kernel.org>; Pawan Gupta
> <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar <mingo@redhat.com>; Dave
> Hansen <dave.hansen@linux.intel.com>; x86@kernel.org; H . Peter Anvin
> <hpa@zytor.com>; Alexander Graf <graf@amazon.com>; Boris Ostrovsky
> <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
> Subject: Re: [RFC PATCH 40/56] x86/alternative: Use sync_core_nmi_safe()
>
> Caution: This message originated from an External Source. Use proper caution
> when opening attachments, clicking links, or responding.
>
>
> On Mon, Oct 20, 2025 at 02:49:56PM +0000, Kaplan, David wrote:
>
> > Coming back to this, are you thinking we should just create something
> > like 'text_poke_sync_core()' inside alternative.c and that can use:
> >    1. SERIALIZE (if available)
> >    2. MOV-CR2 (if re-patching)
> >    3. Else, IRET
> >
> > And maybe someday we put MFENCE into there too for AMD parts.
> >
> > Right now, of course this is the only logic that would care about an
> > NMI-safe sync_core().  So maybe this makes sense vs creating a generic
> > version that nobody else is using?
>
> I was thinking something fairly straight forward like the below. Yes,
> there are a few more sync_core() callers out there, git tells me:
>
> arch/x86/kernel/alternative.c:          sync_core();
> arch/x86/kernel/alternative.c:noinstr void sync_core(void)
> arch/x86/kernel/alternative.c:  sync_core();
> arch/x86/kernel/cpu/mce/core.c: sync_core();
> arch/x86/kernel/cpu/mce/core.c:         sync_core();
> arch/x86/kernel/static_call.c:  sync_core();
> drivers/misc/sgi-gru/grufault.c:                sync_core();
> drivers/misc/sgi-gru/grufault.c:                sync_core();            /* make sure we are
> have current data */
> drivers/misc/sgi-gru/gruhandles.c:      sync_core();
> drivers/misc/sgi-gru/gruhandles.c:      sync_core();
> drivers/misc/sgi-gru/grukservices.c:    sync_core();
>
> But none of that seems like it cares about an extra few cycles, and why
> complicate matters with another sync_core variant and all that.
>
>
> diff --git a/arch/x86/include/asm/sync_core.h b/arch/x86/include/asm/sync_core.h
> index 96bda43538ee..ef4508a03800 100644
> --- a/arch/x86/include/asm/sync_core.h
> +++ b/arch/x86/include/asm/sync_core.h
> @@ -7,86 +7,7 @@
>  #include <asm/cpufeature.h>
>  #include <asm/special_insns.h>
>
> -#ifdef CONFIG_X86_32
> -static __always_inline void iret_to_self(void)
> -{
> -       asm volatile (
> -               "pushfl\n\t"
> -               "pushl %%cs\n\t"
> -               "pushl $1f\n\t"
> -               "iret\n\t"
> -               "1:"
> -               : ASM_CALL_CONSTRAINT : : "memory");
> -}
> -#else
> -static __always_inline void iret_to_self(void)
> -{
> -       unsigned int tmp;
> -
> -       asm volatile (
> -               "mov %%ss, %0\n\t"
> -               "pushq %q0\n\t"
> -               "pushq %%rsp\n\t"
> -               "addq $8, (%%rsp)\n\t"
> -               "pushfq\n\t"
> -               "mov %%cs, %0\n\t"
> -               "pushq %q0\n\t"
> -               "pushq $1f\n\t"
> -               "iretq\n\t"
> -               "1:"
> -               : "=&r" (tmp), ASM_CALL_CONSTRAINT : : "cc", "memory");
> -}
> -#endif /* CONFIG_X86_32 */
> -
> -/*
> - * This function forces the icache and prefetched instruction stream to
> - * catch up with reality in two very specific cases:
> - *
> - *  a) Text was modified using one virtual address and is about to be executed
> - *     from the same physical page at a different virtual address.
> - *
> - *  b) Text was modified on a different CPU, may subsequently be
> - *     executed on this CPU, and you want to make sure the new version
> - *     gets executed.  This generally means you're calling this in an IPI.
> - *
> - * If you're calling this for a different reason, you're probably doing
> - * it wrong.
> - *
> - * Like all of Linux's memory ordering operations, this is a
> - * compiler barrier as well.
> - */
> -static __always_inline void sync_core(void)
> -{
> -       /*
> -        * The SERIALIZE instruction is the most straightforward way to
> -        * do this, but it is not universally available.
> -        */
> -       if (static_cpu_has(X86_FEATURE_SERIALIZE)) {
> -               serialize();
> -               return;
> -       }
> -
> -       /*
> -        * For all other processors, there are quite a few ways to do this.
> -        * IRET-to-self is nice because it works on every CPU, at any CPL
> -        * (so it's compatible with paravirtualization), and it never exits
> -        * to a hypervisor.  The only downsides are that it's a bit slow
> -        * (it seems to be a bit more than 2x slower than the fastest
> -        * options) and that it unmasks NMIs.  The "push %cs" is needed,
> -        * because in paravirtual environments __KERNEL_CS may not be a
> -        * valid CS value when we do IRET directly.
> -        *
> -        * In case NMI unmasking or performance ever becomes a problem,
> -        * the next best option appears to be MOV-to-CR2 and an
> -        * unconditional jump.  That sequence also works on all CPUs,
> -        * but it will fault at CPL3 (i.e. Xen PV).
> -        *
> -        * CPUID is the conventional way, but it's nasty: it doesn't
> -        * exist on some 486-like CPUs, and it usually exits to a
> -        * hypervisor.
> -        */
> -       iret_to_self();
> -}
> +extern void sync_core(void);
>
>  /*
>   * Ensure that a core serializing instruction is issued before returning
> diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c
> index e377b06e70e3..2a5daae3626b 100644
> --- a/arch/x86/kernel/alternative.c
> +++ b/arch/x86/kernel/alternative.c
> @@ -2687,6 +2687,87 @@ void *text_poke_set(void *addr, int c, size_t len)
>         return addr;
>  }
>
> +#ifdef CONFIG_X86_32
> +static __always_inline void iret_to_self(void)
> +{
> +       asm volatile (
> +               "pushfl\n\t"
> +               "pushl %%cs\n\t"
> +               "pushl $1f\n\t"
> +               "iret\n\t"
> +               "1:"
> +               : ASM_CALL_CONSTRAINT : : "memory");
> +}
> +#else
> +static __always_inline void iret_to_self(void)
> +{
> +       unsigned int tmp;
> +
> +       asm volatile (
> +               "mov %%ss, %0\n\t"
> +               "pushq %q0\n\t"
> +               "pushq %%rsp\n\t"
> +               "addq $8, (%%rsp)\n\t"
> +               "pushfq\n\t"
> +               "mov %%cs, %0\n\t"
> +               "pushq %q0\n\t"
> +               "pushq $1f\n\t"
> +               "iretq\n\t"
> +               "1:"
> +               : "=&r" (tmp), ASM_CALL_CONSTRAINT : : "cc", "memory");
> +}
> +#endif /* CONFIG_X86_32 */
> +
> +/*
> + * This function forces the icache and prefetched instruction stream to
> + * catch up with reality in two very specific cases:
> + *
> + *  a) Text was modified using one virtual address and is about to be executed
> + *     from the same physical page at a different virtual address.
> + *
> + *  b) Text was modified on a different CPU, may subsequently be
> + *     executed on this CPU, and you want to make sure the new version
> + *     gets executed.  This generally means you're calling this in an IPI.
> + *
> + * If you're calling this for a different reason, you're probably doing
> + * it wrong.
> + *
> + * Like all of Linux's memory ordering operations, this is a
> + * compiler barrier as well.
> + */
> +noinstr void sync_core(void)
> +{
> +       /*
> +        * The SERIALIZE instruction is the most straightforward way to
> +        * do this, but it is not universally available.
> +        */
> +       if (static_cpu_has(X86_FEATURE_SERIALIZE)) {
> +               serialize();
> +               return;
> +       }
> +
> +       /*
> +        * For all other processors, there are quite a few ways to do this.
> +        * IRET-to-self is nice because it works on every CPU, at any CPL
> +        * (so it's compatible with paravirtualization), and it never exits
> +        * to a hypervisor.  The only downsides are that it's a bit slow
> +        * (it seems to be a bit more than 2x slower than the fastest
> +        * options) and that it unmasks NMIs.  The "push %cs" is needed,
> +        * because in paravirtual environments __KERNEL_CS may not be a
> +        * valid CS value when we do IRET directly.
> +        *
> +        * In case NMI unmasking or performance ever becomes a problem,
> +        * the next best option appears to be MOV-to-CR2 and an
> +        * unconditional jump.  That sequence also works on all CPUs,
> +        * but it will fault at CPL3 (i.e. Xen PV).
> +        *
> +        * CPUID is the conventional way, but it's nasty: it doesn't
> +        * exist on some 486-like CPUs, and it usually exits to a
> +        * hypervisor.
> +        */
> +       iret_to_self();
> +}
> +
>  static void do_sync_core(void *info)
>  {
>         sync_core();

Ah, makes sense.  Thanks for spelling it out :)

--David Kaplan

Re: [RFC PATCH 40/56] x86/alternative: Use sync_core_nmi_safe()

[Andrew Cooper]

> + /* + * For all other processors, there are quite a few ways to do
> this. + * IRET-to-self is nice because it works on every CPU, at any
> CPL + * (so it's compatible with paravirtualization), and it never
> exits + * to a hypervisor.

"Never" isn't correct.  "Typically doesn't" is better in this context.

On Intel, there's the NMI_WINDOW vmexit which generally occurs after the
IRET is complete.

On AMD, prior to current generation CPUs, there was no NMI virt support,
and intercepting IRET is how the hypervisor is forced to cope.  (The
intercept is Fault-like so happens before the IRET executes, and there
are a variety of increasingly terrible ways of trying to account for this.)

~Andrew

Re: [RFC PATCH 40/56] x86/alternative: Use sync_core_nmi_safe()

[David Laight]

On Mon, 20 Oct 2025 17:01:33 +0200
Peter Zijlstra <peterz@infradead.org> wrote:

> On Mon, Oct 20, 2025 at 02:49:56PM +0000, Kaplan, David wrote:
> 
> > Coming back to this, are you thinking we should just create something
> > like 'text_poke_sync_core()' inside alternative.c and that can use:
> >    1. SERIALIZE (if available)
> >    2. MOV-CR2 (if re-patching)
> >    3. Else, IRET
> > 
> > And maybe someday we put MFENCE into there too for AMD parts.
> > 
> > Right now, of course this is the only logic that would care about an
> > NMI-safe sync_core().  So maybe this makes sense vs creating a generic
> > version that nobody else is using?  
> 
> I was thinking something fairly straight forward like the below. Yes,
> there are a few more sync_core() callers out there, git tells me:
> 
...
> But none of that seems like it cares about an extra few cycles, and why
> complicate matters with another sync_core variant and all that.
> 
...
> +	 * CPUID is the conventional way, but it's nasty: it doesn't
> +	 * exist on some 486-like CPUs, and it usually exits to a
> +	 * hypervisor.

Hasn't support for those 486 cpu been dropped now?
So the comment probably needs updating?

	David

Re: [RFC PATCH 14/56] x86/bugs: Reset GDS mitigations

[Pawan Gupta]

On Mon, Oct 13, 2025 at 09:34:02AM -0500, David Kaplan wrote:
> Add function to reset GDS mitigations back to their boot-time defaults.
> 
> Signed-off-by: David Kaplan <david.kaplan@amd.com>
> ---
>  arch/x86/kernel/cpu/bugs.c | 13 +++++++++++++
>  1 file changed, 13 insertions(+)
> 
> diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
> index 112553058ccc..e765ac0b9240 100644
> --- a/arch/x86/kernel/cpu/bugs.c
> +++ b/arch/x86/kernel/cpu/bugs.c
> @@ -1182,6 +1182,18 @@ static void __init gds_apply_mitigation(void)
>  	pr_info("%s\n", gds_strings[gds_mitigation]);
>  }
>  
> +#ifdef CONFIG_DYNAMIC_MITIGATIONS
> +static void gds_reset_mitigation(void)
> +{
> +	/* To cause the MSR bit to be cleared. */
> +	gds_mitigation = GDS_MITIGATION_OFF;
> +	if (x86_arch_cap_msr & ARCH_CAP_GDS_CTRL)
> +		update_gds_msr();

This also needs to check for GDS_MITIGATION_FULL_LOCKED, otherwise in the
locked case it would trigger this warning in update_gds_msr():

	WARN_ON_ONCE(mcu_ctrl != mcu_ctrl_after);

Re: [RFC PATCH 15/56] x86/bugs: Reset BHI mitigations

[Pawan Gupta]

On Mon, Oct 13, 2025 at 09:34:03AM -0500, David Kaplan wrote:
> Add function to reset BHI mitigations back to their boot-time defaults.
> 
> Signed-off-by: David Kaplan <david.kaplan@amd.com>
> ---
>  arch/x86/kernel/cpu/bugs.c | 12 ++++++++++++
>  1 file changed, 12 insertions(+)
> 
> diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
> index e765ac0b9240..67561e5c2154 100644
> --- a/arch/x86/kernel/cpu/bugs.c
> +++ b/arch/x86/kernel/cpu/bugs.c
> @@ -2360,6 +2360,17 @@ static void __init bhi_apply_mitigation(void)
>  	setup_force_cpu_cap(X86_FEATURE_CLEAR_BHB_VMEXIT);
>  }
>  
> +#ifdef CONFIG_DYNAMIC_MITIGATIONS
> +static void bhi_reset_mitigation(void)
> +{
> +	/* RRSBA already cleared in spectre_v2_reset_mitigation() */
> +	setup_clear_cpu_cap(X86_FEATURE_CLEAR_BHB_VMEXIT);
> +	setup_clear_cpu_cap(X86_FEATURE_CLEAR_BHB_LOOP);

Also needs to reset SPEC_CTRL_BHI_DIS_S in x86_spec_ctrl_base.

An alternative is to add spec_ctrl_reset_mitigation() that resets
x86_spec_ctrl_base for SPEC_CTRL_MITIGATIONS_MASK. To be consistent with
reset functions of other mitigations, probably also reset the MSR.

> +	bhi_mitigation = IS_ENABLED(CONFIG_MITIGATION_SPECTRE_BHI) ?
> +		BHI_MITIGATION_AUTO : BHI_MITIGATION_OFF;
> +}

Re: [RFC PATCH 00/56] Dynamic mitigations

[Pawan Gupta]

On Mon, Oct 13, 2025 at 09:33:48AM -0500, David Kaplan wrote:
> Dynamic mitigations enables changing the kernel CPU security mitigations at
> runtime without a reboot/kexec.
> 
> Previously, mitigation choices had to be made on the kernel cmdline.  With
> this feature an administrator can select new mitigation choices by writing
> a sysfs file, after which the kernel will re-patch itself based on the new
> mitigations.
> 
> As the performance cost of CPU mitigations can be significant, selecting
> the right set of mitigations is important to achieve the correct balance of
> performance/security.
> 
> Use
> ---
> As described in the supplied documentation file, new mitigations are
> selected by writing cmdline options to a new sysfs file.  Only cmdline
> options related to mitigations are recognized via this interface.  All
> previous mitigation-related cmdline options are ignored and selections are
> done based on the new options.
> 
> Examples:
>    echo "mitigations=off" > /sys/devices/system/cpu/mitigations
>    echo "spectre_v2=retpoline tsa=off" > /sys/devices/system/cpu/mitigations
> 
> 
> There are several use cases that will benefit from dynamic mitigations:
> 
> Use Cases
> ---------
> 1. Runtime Policy
> 
> Some workflows rely on booting a generic kernel before customizing the system.
> cloud-init is a popular example of this where a VM is started typically with
> default settings and then is customized based on a customer-provided
> configuration file.
> 
> As flows like this rely on configuring the system after boot, they currently
> cannot customize the mitigation policy.  With dynamic mitigations, this
> configuration information can be augmented to include security policy
> information.
> 
> For example, a cloud VM which runs only trusted workloads likely does not
> need any CPU security mitigations applied.  But as this policy information
> is not known at boot time, the kernel will be booted with unnecessary
> mitigations enabled.  With dynamic mitigations, these mitigations can be
> disabled during boot after policy information is retrieved, improving
> performance.
> 
> 2. Mitigation Changes
> 
> Sometimes there are needs to change the mitigation settings in light of new
> security findings.  For example, AMD-SB-1036 advised of a security issue
> with a spectre v2 mitigation and advised using a different one instead.
> 
> With dynamic mitigations, such changes can be made without a reboot/kexec
> which minimizes disruption in environments which cannot easily tolerate
> such an event.
> 
> 3. Mitigation Testing
> 
> Being able to quickly change between different mitigation settings without
> having to restart applications is beneficial when conducting mitigation
> development and testing.
> 
> Note that some bugs have multiple mitigation options, which may have
> varying performance impacts.  Being able to quickly switch between them
> makes evaluating such options easier.
> 
> 
> Implementation Details
> ----------------------
> Re-patching the kernel is expected to be a very rare operation and is done
> under very big hammers.  All tasks are put into the freezer and the
> re-patching is then done under the (new) stop_machine_nmi() routine.
> 
> To re-patch the kernel, it is first reverted back to its compile-time
> state.  The original bytes from alternatives, retpolines, etc. are saved
> during boot so they can later be used to restore the original kernel image.
> After that, the kernel is patched based on the new feature flags.
> 
> This simplifies the re-patch process as restoring the original kernel image
> is relatively straightforward.  In other words, instead of having to
> re-patch from mitigation A to mitigation B directly, we first restore the
> original image and then patch from that to mitigation B, similar to if the
> system had booted with mitigation B selected originally.
> 
> 
> Performance
> -----------
> Testing so far has demonstrated that re-patching takes ~50ms on an AMD EPYC
> 7713 running a typical Ubuntu kernel with around 100 modules loaded.
> 
> Guide to Patch Series
> ---------------------
> As this series is rather lengthy, this may help with understanding it:
> 
> Patches 3-18 focus on "resetting" mitigations.  Every bug that may set feature
> flags, MSRs, static branches, etc. now has matching "reset" functions that will
> undo all these changes.  This is used at the beginning of the re-patch flow.
> 
> Patches 20-22 move various functions and values out of the .init section.  Most
> of the existing mitigation logic was marked as __init and the mitigation
> settings as __ro_after_init but now these can be changed at runtime.  The
> __ro_after_init marking functioned as a defense-in-depth measure but is arguably
> of limited meaningful security value as an attacker who can modify kernel data
> can do a lot worse than change some speculation settings.  As re-patching
> requires being able to modify these settings, it was simplest to remove them
> from that section.
> 
> Patches 23-27 involve linker and related modifications to keep alternative
> information around at runtime instead of free'ing it after boot.  This does
> result in slightly higher runtime memory consumption which is one reason why
> this feature is behind a Kconfig option.  On a typical kernel, this was measured
> at around 2MB of extra kernel memory usage.
> 
> Patches 28-30 focus on the new stop_machine_nmi() which behaves like
> stop_machine() but runs the handler in NMI context, thus ensuring that even NMIs
> cannot interrupt the handler.  As dynamic mitigations involves re-patching
> functions used by NMI entry code, this is required for safety.
> 
> Patches 31-40 focus on support for restoring the kernel text at runtime.  This
> involves saving the original kernel bytes when patched the first time and adding
> support to then restore those later.
> 
> Patches 41-44 start building support for updating code, in particular module
> code at runtime.
> 
> Patches 45-47 focus on support for the Indirect Target Selection mitigation
> which is particularly challenging because it requires runtime memory allocations
> and permission changes which are not possible in NMI context.  As a result, ITS
> memory is pre-allocated before entering NMI context.
> 
> Patch 50 adds the complete function for resetting and re-patching the kernel.
> 
> Patches 51-53 build the sysfs interface for re-patching and support for parsing
> the new options provided.
> 
> Patches 54-56 add debugfs interfaces to values which are important for
> mitigations.  These are useful for userspace test utilities to be able to force
> a CPU to appear to be vulnerable or immune to certain bugs as well as being able
> to help verify if the kernel is correctly mitigating various vulnerabilities.

Although it adds some complexity, this adds a very useful feature. Thanks
for doing this series.

Just curious, for patching indirect branches, was replacing alternatives
with static_calls considered? I haven't looked at the feasibility, but
static_calls seems to be more suited for post-boot patching.

Thinking out loud, patching in something similar to suspend-to-RAM flow may
reduce some corner cases. Barring the BSP, the APs gets reinitialized in
that case.

RE: [RFC PATCH 00/56] Dynamic mitigations

[Kaplan, David]

[AMD Official Use Only - AMD Internal Distribution Only]

> -----Original Message-----
> From: Pawan Gupta <pawan.kumar.gupta@linux.intel.com>
> Sent: Friday, October 24, 2025 12:01 AM
> To: Kaplan, David <David.Kaplan@amd.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>; Borislav Petkov <bp@alien8.de>; Peter
> Zijlstra <peterz@infradead.org>; Josh Poimboeuf <jpoimboe@kernel.org>; Ingo
> Molnar <mingo@redhat.com>; Dave Hansen <dave.hansen@linux.intel.com>;
> x86@kernel.org; H . Peter Anvin <hpa@zytor.com>; Alexander Graf
> <graf@amazon.com>; Boris Ostrovsky <boris.ostrovsky@oracle.com>; linux-
> kernel@vger.kernel.org
> Subject: Re: [RFC PATCH 00/56] Dynamic mitigations
>
> Although it adds some complexity, this adds a very useful feature. Thanks
> for doing this series.
>
> Just curious, for patching indirect branches, was replacing alternatives
> with static_calls considered? I haven't looked at the feasibility, but
> static_calls seems to be more suited for post-boot patching.

Something like that may be doable for indirect branches, but I think I dismissed that general direction because I didn't see how it could work for alternatives in general.  Runtime patching is quite complex, especially with the instruction emulation that happens within the int3 handler.  While that can work for branches, I didn't think that could work as a general solution.  Given how rare mitigation re-patching was expected to be, making everything quiesce seemed safer and significantly less complex.

>
> Thinking out loud, patching in something similar to suspend-to-RAM flow may
> reduce some corner cases. Barring the BSP, the APs gets reinitialized in
> that case.

Right now, the BSP handles most of the patching.  The AP's spin during the patching and then when it's complete, they update their local MSR values for the speculation related MSRs (similar to on AP startup I think).  But that should be all they need to do I think.

Thanks
--David Kaplan

RE: [RFC PATCH 14/56] x86/bugs: Reset GDS mitigations

[Kaplan, David]

[AMD Official Use Only - AMD Internal Distribution Only]

> -----Original Message-----
> From: Pawan Gupta <pawan.kumar.gupta@linux.intel.com>
> Sent: Thursday, October 23, 2025 9:41 PM
> To: Kaplan, David <David.Kaplan@amd.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>; Borislav Petkov <bp@alien8.de>; Peter
> Zijlstra <peterz@infradead.org>; Josh Poimboeuf <jpoimboe@kernel.org>; Ingo
> Molnar <mingo@redhat.com>; Dave Hansen <dave.hansen@linux.intel.com>;
> x86@kernel.org; H . Peter Anvin <hpa@zytor.com>; Alexander Graf
> <graf@amazon.com>; Boris Ostrovsky <boris.ostrovsky@oracle.com>; linux-
> kernel@vger.kernel.org
> Subject: Re: [RFC PATCH 14/56] x86/bugs: Reset GDS mitigations
>
> Caution: This message originated from an External Source. Use proper caution
> when opening attachments, clicking links, or responding.
>
>
> On Mon, Oct 13, 2025 at 09:34:02AM -0500, David Kaplan wrote:
> > Add function to reset GDS mitigations back to their boot-time defaults.
> >
> > Signed-off-by: David Kaplan <david.kaplan@amd.com>
> > ---
> >  arch/x86/kernel/cpu/bugs.c | 13 +++++++++++++
> >  1 file changed, 13 insertions(+)
> >
> > diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
> > index 112553058ccc..e765ac0b9240 100644
> > --- a/arch/x86/kernel/cpu/bugs.c
> > +++ b/arch/x86/kernel/cpu/bugs.c
> > @@ -1182,6 +1182,18 @@ static void __init gds_apply_mitigation(void)
> >       pr_info("%s\n", gds_strings[gds_mitigation]);
> >  }
> >
> > +#ifdef CONFIG_DYNAMIC_MITIGATIONS
> > +static void gds_reset_mitigation(void)
> > +{
> > +     /* To cause the MSR bit to be cleared. */
> > +     gds_mitigation = GDS_MITIGATION_OFF;
> > +     if (x86_arch_cap_msr & ARCH_CAP_GDS_CTRL)
> > +             update_gds_msr();
>
> This also needs to check for GDS_MITIGATION_FULL_LOCKED, otherwise in the
> locked case it would trigger this warning in update_gds_msr():
>
>         WARN_ON_ONCE(mcu_ctrl != mcu_ctrl_after);

Ah, thanks.  I will qualify the call to update_gds_msr() with this.

--David Kaplan

RE: [RFC PATCH 15/56] x86/bugs: Reset BHI mitigations

[Kaplan, David]

[AMD Official Use Only - AMD Internal Distribution Only]

> -----Original Message-----
> From: Pawan Gupta <pawan.kumar.gupta@linux.intel.com>
> Sent: Thursday, October 23, 2025 9:49 PM
> To: Kaplan, David <David.Kaplan@amd.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>; Borislav Petkov <bp@alien8.de>; Peter
> Zijlstra <peterz@infradead.org>; Josh Poimboeuf <jpoimboe@kernel.org>; Ingo
> Molnar <mingo@redhat.com>; Dave Hansen <dave.hansen@linux.intel.com>;
> x86@kernel.org; H . Peter Anvin <hpa@zytor.com>; Alexander Graf
> <graf@amazon.com>; Boris Ostrovsky <boris.ostrovsky@oracle.com>; linux-
> kernel@vger.kernel.org
> Subject: Re: [RFC PATCH 15/56] x86/bugs: Reset BHI mitigations
>
> Caution: This message originated from an External Source. Use proper caution
> when opening attachments, clicking links, or responding.
>
>
> On Mon, Oct 13, 2025 at 09:34:03AM -0500, David Kaplan wrote:
> > Add function to reset BHI mitigations back to their boot-time defaults.
> >
> > Signed-off-by: David Kaplan <david.kaplan@amd.com>
> > ---
> >  arch/x86/kernel/cpu/bugs.c | 12 ++++++++++++
> >  1 file changed, 12 insertions(+)
> >
> > diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
> > index e765ac0b9240..67561e5c2154 100644
> > --- a/arch/x86/kernel/cpu/bugs.c
> > +++ b/arch/x86/kernel/cpu/bugs.c
> > @@ -2360,6 +2360,17 @@ static void __init bhi_apply_mitigation(void)
> >       setup_force_cpu_cap(X86_FEATURE_CLEAR_BHB_VMEXIT);
> >  }
> >
> > +#ifdef CONFIG_DYNAMIC_MITIGATIONS
> > +static void bhi_reset_mitigation(void)
> > +{
> > +     /* RRSBA already cleared in spectre_v2_reset_mitigation() */
> > +     setup_clear_cpu_cap(X86_FEATURE_CLEAR_BHB_VMEXIT);
> > +     setup_clear_cpu_cap(X86_FEATURE_CLEAR_BHB_LOOP);
>
> Also needs to reset SPEC_CTRL_BHI_DIS_S in x86_spec_ctrl_base.
>
> An alternative is to add spec_ctrl_reset_mitigation() that resets
> x86_spec_ctrl_base for SPEC_CTRL_MITIGATIONS_MASK. To be consistent with
> reset functions of other mitigations, probably also reset the MSR.

Actually I think it's better if none of the reset functions touch the MSR.  That only runs on whatever thread is doing the reset, which is random.  The __cpu_update_alternatives() function will later update all speculation related MSRs on all CPUs after new mitigations are selected.  This also avoids a window where the MSR setting might be insecure even though a mitigation is not actually changing.

I will add resetting SPEC_CTRL_BHI_DIS_S though, looks like I missed that one.

Thanks
--David Kaplan

Re: [RFC PATCH 52/56] x86/bugs: Support parsing mitigation options

[Nikolay Borisov]

On 10/13/25 17:34, David Kaplan wrote:
> Add arch-specific function for determining if an option is related to a
> mitigation and parsing it.  These will be used for parsing a string of
> options for re-evaluating cpu mitigations.
> 
> Signed-off-by: David Kaplan <david.kaplan@amd.com>
> ---
>   arch/x86/include/asm/bugs.h |  2 ++
>   arch/x86/kernel/cpu/bugs.c  | 56 +++++++++++++++++++++++++++++++++++++
>   2 files changed, 58 insertions(+)
> 
> diff --git a/arch/x86/include/asm/bugs.h b/arch/x86/include/asm/bugs.h
> index 2e1a7d282e51..1e142a676335 100644
> --- a/arch/x86/include/asm/bugs.h
> +++ b/arch/x86/include/asm/bugs.h
> @@ -13,5 +13,7 @@ static inline int ppro_with_ram_bug(void) { return 0; }
>   extern void cpu_bugs_smt_update(void);
>   void arch_cpu_reset_mitigations(void);
>   void cpu_bugs_update_speculation_msrs(void);
> +bool arch_is_mitigation_opt(char *param);
> +int arch_parse_mitigation_opt(char *param, char *val);
>   
>   #endif /* _ASM_X86_BUGS_H */
> diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
> index 2f82261d033d..26ceb42e0cfb 100644
> --- a/arch/x86/kernel/cpu/bugs.c
> +++ b/arch/x86/kernel/cpu/bugs.c
> @@ -3991,6 +3991,62 @@ void __warn_thunk(void)
>   }
>   
>   #ifdef CONFIG_DYNAMIC_MITIGATIONS
> +struct mitigation_info {
> +	char *param;
> +	int (*parse)(char *str);
> +};
> +
> +static struct mitigation_info mitigation_parsers[] = {
> +	{"mds", mds_cmdline},
> +	{"tsx_async_abort", tsx_async_abort_parse_cmdline},
> +	{"mmio_stale_data", mmio_stale_data_parse_cmdline},
> +	{"reg_file_data_sampling", rfds_parse_cmdline},
> +	{"srbds", srbds_parse_cmdline},
> +	{"gather_data_sampling", gds_parse_cmdline},
> +	{"nospectre_v1", nospectre_v1_cmdline},
> +	{"retbleed", retbleed_parse_cmdline},
> +	{"indirect_target_selection", its_parse_cmdline},
> +	{"spectre_v2_user", spectre_v2_user_parse_cmdline},
> +	{"nospectre_v2", nospectre_v2_parse_cmdline},
> +	{"spectre_v2", spectre_v2_parse_cmdline},
> +	{"spectre_bhi", spectre_bhi_parse_cmdline},
> +	{"nospec_store_bypass_disable", nossb_parse_cmdline},
> +	{"spec_store_bypass_disable", ssb_parse_cmdline},
> +	{"l1tf", l1tf_cmdline},
> +	{"spec_rstack_overflow", srso_parse_cmdline},
> +	{"tsa", tsa_parse_cmdline},
> +	{"vmscape", vmscape_parse_cmdline}
> +};
> +
> +static struct mitigation_info *get_mitigation_info(char *param)
> +{
> +	int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(mitigation_parsers); i++) {
> +		if (parameq(param, mitigation_parsers[i].param))
> +			return &mitigation_parsers[i];
> +	}
> +
> +	return NULL;
> +}
> +
> +bool arch_is_mitigation_opt(char *param)
> +{
> +	return get_mitigation_info(param);

nit: This has an implied conversion from a pointer to a bool, should it 
be return get_mitigation_info != NULL

It would work either ways but being explicit is better?

> +}
> +
> +int arch_parse_mitigation_opt(char *param, char *val)
> +{
> +	struct mitigation_info *info = get_mitigation_info(param);
> +
> +	if (!info) {
> +		pr_warn("Ignoring non-mitigation option %s\n", param);

nit: Do we want to be that verbose?

> +		return 0;
> +	}
> +
> +	return info->parse(val);
> +}
> +
>   void arch_cpu_reset_mitigations(void)
>   {
>   	spectre_v1_reset_mitigation();

Re: [RFC PATCH 34/56] x86/alternative: Save old bytes for alternatives

[Nikolay Borisov]

On 10/15/25 16:45, Kaplan, David wrote:
> [AMD Official Use Only - AMD Internal Distribution Only]
> 
>> -----Original Message-----
>> From: Juergen Gross <jgross@suse.com>
>> Sent: Wednesday, October 15, 2025 5:38 AM
>> To: Kaplan, David <David.Kaplan@amd.com>; Thomas Gleixner
>> <tglx@linutronix.de>; Borislav Petkov <bp@alien8.de>; Peter Zijlstra
>> <peterz@infradead.org>; Josh Poimboeuf <jpoimboe@kernel.org>; Pawan Gupta
>> <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar <mingo@redhat.com>; Dave
>> Hansen <dave.hansen@linux.intel.com>; x86@kernel.org; H . Peter Anvin
>> <hpa@zytor.com>
>> Cc: Alexander Graf <graf@amazon.com>; Boris Ostrovsky
>> <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
>> Subject: Re: [RFC PATCH 34/56] x86/alternative: Save old bytes for alternatives
>>
>> On 13.10.25 16:34, David Kaplan wrote:
>>> Save the existing instruction bytes at each alternative site when patching.
>>> This is only done the first time, and these will be used later to help
>>> restore the code back to its original form.
>>>
>>> Signed-off-by: David Kaplan <david.kaplan@amd.com>
>>
>> Instead of saving the original instructions at runtime, why don't you
>> expand struct alt_instr to have an additional offset to a saved copy
>> of the original instruction, located in .altinstr_replacement?
>>
>> The new field should be guarded with #ifdef CONFIG_DYNAMIC_MITIGATIONS,
>> of course, like the added handling in the ALTERNATIVE() macros.
>>
> 
> That's an interesting idea, I think that could work.  That would make the kernel image on disk (slightly) larger though, as the original bytes will essentially be duplicated (at the original location and in .altinstr_replacement).  I'm not sure which is the better trade-off (kernel image bytes on disk vs runtime memory usage).  Although I think we're talking about a relatively small amount of memory regardless.  Most of the runtime overhead of dynamic mitigations comes from remembering the call sites/returns.

It's not just about memory usage per-se but also memory pressure from 
allocation and the resulting fragmentation, though I'd think that 
majority of the allocation will fit into kmalloc-32 bucket, still having 
them as part of the kernel image eliminates the additional allocs.
> 
> Thanks
> --David Kaplan

Re: [RFC PATCH 53/56] drivers/cpu: Re-patch mitigations through sysfs

[Nikolay Borisov]

On 10/13/25 17:34, David Kaplan wrote:
> Create a new file at /sys/devices/system/cpu/mitigations that prints the
> current set of mitigation options and can be written to in order to
> re-select mitigations.
> 
> Only options related to mitigations are handled, with the file initially
> returning the relevant options from the command line.  When the file is
> written, any existing selections are discarded and the new options are
> evaluated.
> 
> Signed-off-by: David Kaplan <david.kaplan@amd.com>
> ---
>   .../ABI/testing/sysfs-devices-system-cpu      |   8 ++
>   drivers/base/cpu.c                            | 113 ++++++++++++++++++
>   include/linux/cpu.h                           |   3 +
>   3 files changed, 124 insertions(+)
> 

<snip>

> +
> +ssize_t cpu_show_mitigation_options(struct device *dev, struct device_attribute *attr, char *buf);
> +ssize_t cpu_show_mitigation_options(struct device *dev, struct device_attribute *attr, char *buf)

Make it static

> +{
> +	return sysfs_emit(buf, saved_opts);
> +}
> +
> +ssize_t cpu_write_mitigation_options(struct device *dev,
> +				struct device_attribute *attr,
> +				const char *buf, size_t count);
> +
> +void __weak cpu_prepare_repatch_alternatives(void)
> +{
> +}
> +
> +void __weak cpu_update_alternatives(void)
> +{
> +}
> +
> +void __weak cpu_select_mitigations(void)
> +{
> +}
> +
> +ssize_t cpu_write_mitigation_options(struct device *dev,
> +				struct device_attribute *attr,
> +				const char *buf, size_t count)
Ditto

<snip>

Re: [RFC PATCH 55/56] x86/debug: Show return thunk in debugfs

[Nikolay Borisov]

On 10/13/25 17:34, David Kaplan wrote:
> Make the value of x86_return_thunk visible in debugfs to support user-space
> testing.
> 
> Signed-off-by: David Kaplan <david.kaplan@amd.com>

Reviewed-by: Nikolay Borisov <nik.borisov@suse.com>

> ---
>   arch/x86/kernel/cpu/bugs.c | 44 ++++++++++++++++++++++++++++++++++++++
>   1 file changed, 44 insertions(+)
> 
> diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
> index 26ceb42e0cfb..8365448b3aef 100644
> --- a/arch/x86/kernel/cpu/bugs.c
> +++ b/arch/x86/kernel/cpu/bugs.c
> @@ -16,6 +16,7 @@
>   #include <linux/sched/smt.h>
>   #include <linux/pgtable.h>
>   #include <linux/bpf.h>
> +#include <linux/debugfs.h>
>   
>   #include <asm/spec-ctrl.h>
>   #include <asm/cmdline.h>
> @@ -4065,6 +4066,49 @@ void arch_cpu_reset_mitigations(void)
>   	tsa_reset_mitigation();
>   	vmscape_reset_mitigation();
>   }
> +
> +static int rethunk_debug_show(struct seq_file *m, void *p)
> +{
> +	if (x86_return_thunk == __x86_return_thunk)
> +		seq_puts(m, "__x86_return_thunk\n");
> +	else if (x86_return_thunk == retbleed_return_thunk)
> +		seq_puts(m, "retbleed_return_thunk\n");
> +	else if (x86_return_thunk == call_depth_return_thunk)
> +		seq_puts(m, "call_depth_return_thunk\n");
> +	else if (x86_return_thunk == its_return_thunk)
> +		seq_puts(m, "its_return_thunk\n");
> +	else if (x86_return_thunk == srso_alias_return_thunk)
> +		seq_puts(m, "srso_alias_return_thunk\n");
> +	else if (x86_return_thunk == srso_return_thunk)
> +		seq_puts(m, "srso_return_thunk\n");
> +	else
> +		seq_puts(m, "unknown\n");

nit: This might be better suited for a switch construct but it's fine 
either way.

<snip>

RE: [RFC PATCH 52/56] x86/bugs: Support parsing mitigation options

[Kaplan, David]

[AMD Official Use Only - AMD Internal Distribution Only]

> -----Original Message-----
> From: Nikolay Borisov <nik.borisov@suse.com>
> Sent: Monday, October 27, 2025 6:31 AM
> To: Kaplan, David <David.Kaplan@amd.com>; Thomas Gleixner
> <tglx@linutronix.de>; Borislav Petkov <bp@alien8.de>; Peter Zijlstra
> <peterz@infradead.org>; Josh Poimboeuf <jpoimboe@kernel.org>; Pawan Gupta
> <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar <mingo@redhat.com>; Dave
> Hansen <dave.hansen@linux.intel.com>; x86@kernel.org; H . Peter Anvin
> <hpa@zytor.com>
> Cc: Alexander Graf <graf@amazon.com>; Boris Ostrovsky
> <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
> Subject: Re: [RFC PATCH 52/56] x86/bugs: Support parsing mitigation options
>
> Caution: This message originated from an External Source. Use proper caution
> when opening attachments, clicking links, or responding.
>
>
> On 10/13/25 17:34, David Kaplan wrote:
> > Add arch-specific function for determining if an option is related to a
> > mitigation and parsing it.  These will be used for parsing a string of
> > options for re-evaluating cpu mitigations.
> >
> > Signed-off-by: David Kaplan <david.kaplan@amd.com>
> > ---
> >   arch/x86/include/asm/bugs.h |  2 ++
> >   arch/x86/kernel/cpu/bugs.c  | 56
> +++++++++++++++++++++++++++++++++++++
> >   2 files changed, 58 insertions(+)
> >
> > diff --git a/arch/x86/include/asm/bugs.h b/arch/x86/include/asm/bugs.h
> > index 2e1a7d282e51..1e142a676335 100644
> > --- a/arch/x86/include/asm/bugs.h
> > +++ b/arch/x86/include/asm/bugs.h
> > @@ -13,5 +13,7 @@ static inline int ppro_with_ram_bug(void) { return 0; }
> >   extern void cpu_bugs_smt_update(void);
> >   void arch_cpu_reset_mitigations(void);
> >   void cpu_bugs_update_speculation_msrs(void);
> > +bool arch_is_mitigation_opt(char *param);
> > +int arch_parse_mitigation_opt(char *param, char *val);
> >
> >   #endif /* _ASM_X86_BUGS_H */
> > diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
> > index 2f82261d033d..26ceb42e0cfb 100644
> > --- a/arch/x86/kernel/cpu/bugs.c
> > +++ b/arch/x86/kernel/cpu/bugs.c
> > @@ -3991,6 +3991,62 @@ void __warn_thunk(void)
> >   }
> >
> >   #ifdef CONFIG_DYNAMIC_MITIGATIONS
> > +struct mitigation_info {
> > +     char *param;
> > +     int (*parse)(char *str);
> > +};
> > +
> > +static struct mitigation_info mitigation_parsers[] = {
> > +     {"mds", mds_cmdline},
> > +     {"tsx_async_abort", tsx_async_abort_parse_cmdline},
> > +     {"mmio_stale_data", mmio_stale_data_parse_cmdline},
> > +     {"reg_file_data_sampling", rfds_parse_cmdline},
> > +     {"srbds", srbds_parse_cmdline},
> > +     {"gather_data_sampling", gds_parse_cmdline},
> > +     {"nospectre_v1", nospectre_v1_cmdline},
> > +     {"retbleed", retbleed_parse_cmdline},
> > +     {"indirect_target_selection", its_parse_cmdline},
> > +     {"spectre_v2_user", spectre_v2_user_parse_cmdline},
> > +     {"nospectre_v2", nospectre_v2_parse_cmdline},
> > +     {"spectre_v2", spectre_v2_parse_cmdline},
> > +     {"spectre_bhi", spectre_bhi_parse_cmdline},
> > +     {"nospec_store_bypass_disable", nossb_parse_cmdline},
> > +     {"spec_store_bypass_disable", ssb_parse_cmdline},
> > +     {"l1tf", l1tf_cmdline},
> > +     {"spec_rstack_overflow", srso_parse_cmdline},
> > +     {"tsa", tsa_parse_cmdline},
> > +     {"vmscape", vmscape_parse_cmdline}
> > +};
> > +
> > +static struct mitigation_info *get_mitigation_info(char *param)
> > +{
> > +     int i;
> > +
> > +     for (i = 0; i < ARRAY_SIZE(mitigation_parsers); i++) {
> > +             if (parameq(param, mitigation_parsers[i].param))
> > +                     return &mitigation_parsers[i];
> > +     }
> > +
> > +     return NULL;
> > +}
> > +
> > +bool arch_is_mitigation_opt(char *param)
> > +{
> > +     return get_mitigation_info(param);
>
> nit: This has an implied conversion from a pointer to a bool, should it
> be return get_mitigation_info != NULL
>
> It would work either ways but being explicit is better?

Ack

>
> > +}
> > +
> > +int arch_parse_mitigation_opt(char *param, char *val)
> > +{
> > +     struct mitigation_info *info = get_mitigation_info(param);
> > +
> > +     if (!info) {
> > +             pr_warn("Ignoring non-mitigation option %s\n", param);
>
> nit: Do we want to be that verbose?

My thinking was that the admin is writing a series of cmdline options to this interface, but the interface only recognizes a small number of specific cmdline options (the ones related to mitigation settings).  It therefore may make sense to warn them if they've written an option (thinking it will have an effect) but it is not supported by this interface.  It's also a way to notify them if they made a typo on an option.

That said, open to other ideas here.

Thanks
--David Kaplan

RE: [RFC PATCH 53/56] drivers/cpu: Re-patch mitigations through sysfs

[Kaplan, David]

[AMD Official Use Only - AMD Internal Distribution Only]

> -----Original Message-----
> From: Nikolay Borisov <nik.borisov@suse.com>
> Sent: Monday, October 27, 2025 7:25 AM
> To: Kaplan, David <David.Kaplan@amd.com>; Thomas Gleixner
> <tglx@linutronix.de>; Borislav Petkov <bp@alien8.de>; Peter Zijlstra
> <peterz@infradead.org>; Josh Poimboeuf <jpoimboe@kernel.org>; Pawan Gupta
> <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar <mingo@redhat.com>; Dave
> Hansen <dave.hansen@linux.intel.com>; x86@kernel.org; H . Peter Anvin
> <hpa@zytor.com>
> Cc: Alexander Graf <graf@amazon.com>; Boris Ostrovsky
> <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
> Subject: Re: [RFC PATCH 53/56] drivers/cpu: Re-patch mitigations through sysfs
>
> Caution: This message originated from an External Source. Use proper caution
> when opening attachments, clicking links, or responding.
>
>
> On 10/13/25 17:34, David Kaplan wrote:
> > Create a new file at /sys/devices/system/cpu/mitigations that prints the
> > current set of mitigation options and can be written to in order to
> > re-select mitigations.
> >
> > Only options related to mitigations are handled, with the file initially
> > returning the relevant options from the command line.  When the file is
> > written, any existing selections are discarded and the new options are
> > evaluated.
> >
> > Signed-off-by: David Kaplan <david.kaplan@amd.com>
> > ---
> >   .../ABI/testing/sysfs-devices-system-cpu      |   8 ++
> >   drivers/base/cpu.c                            | 113 ++++++++++++++++++
> >   include/linux/cpu.h                           |   3 +
> >   3 files changed, 124 insertions(+)
> >
>
> <snip>
>
> > +
> > +ssize_t cpu_show_mitigation_options(struct device *dev, struct device_attribute
> *attr, char *buf);
> > +ssize_t cpu_show_mitigation_options(struct device *dev, struct device_attribute
> *attr, char *buf)
>
> Make it static
>
> > +{
> > +     return sysfs_emit(buf, saved_opts);
> > +}
> > +
> > +ssize_t cpu_write_mitigation_options(struct device *dev,
> > +                             struct device_attribute *attr,
> > +                             const char *buf, size_t count);
> > +
> > +void __weak cpu_prepare_repatch_alternatives(void)
> > +{
> > +}
> > +
> > +void __weak cpu_update_alternatives(void)
> > +{
> > +}
> > +
> > +void __weak cpu_select_mitigations(void)
> > +{
> > +}
> > +
> > +ssize_t cpu_write_mitigation_options(struct device *dev,
> > +                             struct device_attribute *attr,
> > +                             const char *buf, size_t count)
> Ditto
>
> <snip>

Ack, thanks

--David Kaplan

RE: [RFC PATCH 34/56] x86/alternative: Save old bytes for alternatives

[Kaplan, David]

[AMD Official Use Only - AMD Internal Distribution Only]

> -----Original Message-----
> From: Nikolay Borisov <nik.borisov@suse.com>
> Sent: Monday, October 27, 2025 6:35 AM
> To: Kaplan, David <David.Kaplan@amd.com>; Juergen Gross <jgross@suse.com>;
> Thomas Gleixner <tglx@linutronix.de>; Borislav Petkov <bp@alien8.de>; Peter
> Zijlstra <peterz@infradead.org>; Josh Poimboeuf <jpoimboe@kernel.org>; Pawan
> Gupta <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar <mingo@redhat.com>;
> Dave Hansen <dave.hansen@linux.intel.com>; x86@kernel.org; H . Peter Anvin
> <hpa@zytor.com>
> Cc: Alexander Graf <graf@amazon.com>; Boris Ostrovsky
> <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
> Subject: Re: [RFC PATCH 34/56] x86/alternative: Save old bytes for alternatives
>
> Caution: This message originated from an External Source. Use proper caution
> when opening attachments, clicking links, or responding.
>
>
> On 10/15/25 16:45, Kaplan, David wrote:
> > [AMD Official Use Only - AMD Internal Distribution Only]
> >
> >> -----Original Message-----
> >> From: Juergen Gross <jgross@suse.com>
> >> Sent: Wednesday, October 15, 2025 5:38 AM
> >> To: Kaplan, David <David.Kaplan@amd.com>; Thomas Gleixner
> >> <tglx@linutronix.de>; Borislav Petkov <bp@alien8.de>; Peter Zijlstra
> >> <peterz@infradead.org>; Josh Poimboeuf <jpoimboe@kernel.org>; Pawan
> Gupta
> >> <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar <mingo@redhat.com>;
> Dave
> >> Hansen <dave.hansen@linux.intel.com>; x86@kernel.org; H . Peter Anvin
> >> <hpa@zytor.com>
> >> Cc: Alexander Graf <graf@amazon.com>; Boris Ostrovsky
> >> <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
> >> Subject: Re: [RFC PATCH 34/56] x86/alternative: Save old bytes for alternatives
> >>
> >> On 13.10.25 16:34, David Kaplan wrote:
> >>> Save the existing instruction bytes at each alternative site when patching.
> >>> This is only done the first time, and these will be used later to help
> >>> restore the code back to its original form.
> >>>
> >>> Signed-off-by: David Kaplan <david.kaplan@amd.com>
> >>
> >> Instead of saving the original instructions at runtime, why don't you
> >> expand struct alt_instr to have an additional offset to a saved copy
> >> of the original instruction, located in .altinstr_replacement?
> >>
> >> The new field should be guarded with #ifdef CONFIG_DYNAMIC_MITIGATIONS,
> >> of course, like the added handling in the ALTERNATIVE() macros.
> >>
> >
> > That's an interesting idea, I think that could work.  That would make the kernel
> image on disk (slightly) larger though, as the original bytes will essentially be
> duplicated (at the original location and in .altinstr_replacement).  I'm not sure which
> is the better trade-off (kernel image bytes on disk vs runtime memory usage).
> Although I think we're talking about a relatively small amount of memory regardless.
> Most of the runtime overhead of dynamic mitigations comes from remembering the
> call sites/returns.
>
> It's not just about memory usage per-se but also memory pressure from
> allocation and the resulting fragmentation, though I'd think that
> majority of the allocation will fit into kmalloc-32 bucket, still having
> them as part of the kernel image eliminates the additional allocs.

I see.  Just to understand, the issue is more with the numerous small allocations right? (that is the kmalloc at each alt_site)  And less about the single large allocation of the array?

I'm just thinking about the retpoline_site handling too.  That one also has a large dynamically allocated array, although it does not have numerous small allocations because the size of each instruction is constrained to at most 6 bytes.

Thanks
--David Kaplan

Re: [RFC PATCH 55/56] x86/debug: Show return thunk in debugfs

[David Laight]

On Mon, 27 Oct 2025 14:29:59 +0200
Nikolay Borisov <nik.borisov@suse.com> wrote:

> On 10/13/25 17:34, David Kaplan wrote:
> > Make the value of x86_return_thunk visible in debugfs to support user-space
> > testing.
> > 
> > Signed-off-by: David Kaplan <david.kaplan@amd.com>  
> 
> Reviewed-by: Nikolay Borisov <nik.borisov@suse.com>
> 
> > ---
> >   arch/x86/kernel/cpu/bugs.c | 44 ++++++++++++++++++++++++++++++++++++++
> >   1 file changed, 44 insertions(+)
> > 
> > diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
> > index 26ceb42e0cfb..8365448b3aef 100644
> > --- a/arch/x86/kernel/cpu/bugs.c
> > +++ b/arch/x86/kernel/cpu/bugs.c
> > @@ -16,6 +16,7 @@
> >   #include <linux/sched/smt.h>
> >   #include <linux/pgtable.h>
> >   #include <linux/bpf.h>
> > +#include <linux/debugfs.h>
> >   
> >   #include <asm/spec-ctrl.h>
> >   #include <asm/cmdline.h>
> > @@ -4065,6 +4066,49 @@ void arch_cpu_reset_mitigations(void)
> >   	tsa_reset_mitigation();
> >   	vmscape_reset_mitigation();
> >   }
> > +
> > +static int rethunk_debug_show(struct seq_file *m, void *p)
> > +{
> > +	if (x86_return_thunk == __x86_return_thunk)
> > +		seq_puts(m, "__x86_return_thunk\n");
> > +	else if (x86_return_thunk == retbleed_return_thunk)
> > +		seq_puts(m, "retbleed_return_thunk\n");
> > +	else if (x86_return_thunk == call_depth_return_thunk)
> > +		seq_puts(m, "call_depth_return_thunk\n");
> > +	else if (x86_return_thunk == its_return_thunk)
> > +		seq_puts(m, "its_return_thunk\n");
> > +	else if (x86_return_thunk == srso_alias_return_thunk)
> > +		seq_puts(m, "srso_alias_return_thunk\n");
> > +	else if (x86_return_thunk == srso_return_thunk)
> > +		seq_puts(m, "srso_return_thunk\n");
> > +	else
> > +		seq_puts(m, "unknown\n");  
> 
> nit: This might be better suited for a switch construct but it's fine 
> either way.

That won't work - they are not integers.
eg:
#ifdef CONFIG_MITIGATION_UNRET_ENTRY
extern void retbleed_return_thunk(void);
#else
static inline void retbleed_return_thunk(void) {}
#endif

I'm not even sure you want to be testing anything against a static inline.

As coded the compiler might generate all 7 copies of seq_puts().
So better to use temporary for the result.

	David

> 
> <snip>
> 
> 

Re: [RFC PATCH 34/56] x86/alternative: Save old bytes for alternatives

[Nikolay Borisov]

On 10/27/25 16:19, Kaplan, David wrote:
> [AMD Official Use Only - AMD Internal Distribution Only]
> 
>> -----Original Message-----
>> From: Nikolay Borisov <nik.borisov@suse.com>
>> Sent: Monday, October 27, 2025 6:35 AM
>> To: Kaplan, David <David.Kaplan@amd.com>; Juergen Gross <jgross@suse.com>;
>> Thomas Gleixner <tglx@linutronix.de>; Borislav Petkov <bp@alien8.de>; Peter
>> Zijlstra <peterz@infradead.org>; Josh Poimboeuf <jpoimboe@kernel.org>; Pawan
>> Gupta <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar <mingo@redhat.com>;
>> Dave Hansen <dave.hansen@linux.intel.com>; x86@kernel.org; H . Peter Anvin
>> <hpa@zytor.com>
>> Cc: Alexander Graf <graf@amazon.com>; Boris Ostrovsky
>> <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
>> Subject: Re: [RFC PATCH 34/56] x86/alternative: Save old bytes for alternatives
>>
>> Caution: This message originated from an External Source. Use proper caution
>> when opening attachments, clicking links, or responding.
>>
>>
>> On 10/15/25 16:45, Kaplan, David wrote:
>>> [AMD Official Use Only - AMD Internal Distribution Only]
>>>
>>>> -----Original Message-----
>>>> From: Juergen Gross <jgross@suse.com>
>>>> Sent: Wednesday, October 15, 2025 5:38 AM
>>>> To: Kaplan, David <David.Kaplan@amd.com>; Thomas Gleixner
>>>> <tglx@linutronix.de>; Borislav Petkov <bp@alien8.de>; Peter Zijlstra
>>>> <peterz@infradead.org>; Josh Poimboeuf <jpoimboe@kernel.org>; Pawan
>> Gupta
>>>> <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar <mingo@redhat.com>;
>> Dave
>>>> Hansen <dave.hansen@linux.intel.com>; x86@kernel.org; H . Peter Anvin
>>>> <hpa@zytor.com>
>>>> Cc: Alexander Graf <graf@amazon.com>; Boris Ostrovsky
>>>> <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
>>>> Subject: Re: [RFC PATCH 34/56] x86/alternative: Save old bytes for alternatives
>>>>
>>>> On 13.10.25 16:34, David Kaplan wrote:
>>>>> Save the existing instruction bytes at each alternative site when patching.
>>>>> This is only done the first time, and these will be used later to help
>>>>> restore the code back to its original form.
>>>>>
>>>>> Signed-off-by: David Kaplan <david.kaplan@amd.com>
>>>>
>>>> Instead of saving the original instructions at runtime, why don't you
>>>> expand struct alt_instr to have an additional offset to a saved copy
>>>> of the original instruction, located in .altinstr_replacement?
>>>>
>>>> The new field should be guarded with #ifdef CONFIG_DYNAMIC_MITIGATIONS,
>>>> of course, like the added handling in the ALTERNATIVE() macros.
>>>>
>>>
>>> That's an interesting idea, I think that could work.  That would make the kernel
>> image on disk (slightly) larger though, as the original bytes will essentially be
>> duplicated (at the original location and in .altinstr_replacement).  I'm not sure which
>> is the better trade-off (kernel image bytes on disk vs runtime memory usage).
>> Although I think we're talking about a relatively small amount of memory regardless.
>> Most of the runtime overhead of dynamic mitigations comes from remembering the
>> call sites/returns.
>>
>> It's not just about memory usage per-se but also memory pressure from
>> allocation and the resulting fragmentation, though I'd think that
>> majority of the allocation will fit into kmalloc-32 bucket, still having
>> them as part of the kernel image eliminates the additional allocs.
> 
> I see.  Just to understand, the issue is more with the numerous small allocations right? (that is the kmalloc at each alt_site)  And less about the single large allocation of the array?

Yep, do you have some statistics how many allocs have to be done?

> 
> I'm just thinking about the retpoline_site handling too.  That one also has a large dynamically allocated array, although it does not have numerous small allocations because the size of each instruction is constrained to at most 6 bytes.


> 
> Thanks
> --David Kaplan
> 
> 

Re: [RFC PATCH 04/56] x86/bugs: Reset spectre_v1 mitigations

[Borislav Petkov]

On Mon, Oct 13, 2025 at 09:33:52AM -0500, David Kaplan wrote:
> diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
> index 6a526ae1fe99..9d5c6a3e50e1 100644
> --- a/arch/x86/kernel/cpu/bugs.c
> +++ b/arch/x86/kernel/cpu/bugs.c
> @@ -46,6 +46,8 @@
>   *				   may want to change based on other choices
>   *				   made.  This function is optional.
>   *   <vuln>_apply_mitigation() -- Enable the selected mitigation.
> + *   <vuln>_reset_mitigation() -- Undo's the apply_mitigation step, this is used

"Undoes" no?

> + *				  with runtime mitigation patching.
>   *
>   * The compile-time mitigation in all cases should be AUTO.  An explicit
>   * command-line option can override AUTO.  If no such option is
> @@ -1247,6 +1249,15 @@ static void __init spectre_v1_apply_mitigation(void)
>  	pr_info("%s\n", spectre_v1_strings[spectre_v1_mitigation]);
>  }
>  
> +#ifdef CONFIG_DYNAMIC_MITIGATIONS
> +static void spectre_v1_reset_mitigation(void)
> +{
> +	setup_clear_cpu_cap(X86_FEATURE_FENCE_SWAPGS_USER);
> +	setup_clear_cpu_cap(X86_FEATURE_FENCE_SWAPGS_KERNEL);
> +	spectre_v1_mitigation = SPECTRE_V1_MITIGATION_AUTO;
> +}
> +#endif

Right, let's see in the end what is easier: we have the cross-dependencies
between mitigations so having separate "undo" functions might require them to
run in reverse order to the "apply" functions. I can imagine a single "undo"
function would be easier because you have everything in one place.

We'll see.

Thx.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

RE: [RFC PATCH 04/56] x86/bugs: Reset spectre_v1 mitigations

[Kaplan, David]

[AMD Official Use Only - AMD Internal Distribution Only]

> -----Original Message-----
> From: Borislav Petkov <bp@alien8.de>
> Sent: Wednesday, October 29, 2025 6:57 AM
> To: Kaplan, David <David.Kaplan@amd.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>; Peter Zijlstra <peterz@infradead.org>;
> Josh Poimboeuf <jpoimboe@kernel.org>; Pawan Gupta
> <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar <mingo@redhat.com>; Dave
> Hansen <dave.hansen@linux.intel.com>; x86@kernel.org; H . Peter Anvin
> <hpa@zytor.com>; Alexander Graf <graf@amazon.com>; Boris Ostrovsky
> <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
> Subject: Re: [RFC PATCH 04/56] x86/bugs: Reset spectre_v1 mitigations
>
> Caution: This message originated from an External Source. Use proper caution
> when opening attachments, clicking links, or responding.
>
>
> On Mon, Oct 13, 2025 at 09:33:52AM -0500, David Kaplan wrote:
> > diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
> > index 6a526ae1fe99..9d5c6a3e50e1 100644
> > --- a/arch/x86/kernel/cpu/bugs.c
> > +++ b/arch/x86/kernel/cpu/bugs.c
> > @@ -46,6 +46,8 @@
> >   *                              may want to change based on other choices
> >   *                              made.  This function is optional.
> >   *   <vuln>_apply_mitigation() -- Enable the selected mitigation.
> > + *   <vuln>_reset_mitigation() -- Undo's the apply_mitigation step, this is used
>
> "Undoes" no?

Yes, will fix

>
> > + *                             with runtime mitigation patching.
> >   *
> >   * The compile-time mitigation in all cases should be AUTO.  An explicit
> >   * command-line option can override AUTO.  If no such option is
> > @@ -1247,6 +1249,15 @@ static void __init spectre_v1_apply_mitigation(void)
> >       pr_info("%s\n", spectre_v1_strings[spectre_v1_mitigation]);
> >  }
> >
> > +#ifdef CONFIG_DYNAMIC_MITIGATIONS
> > +static void spectre_v1_reset_mitigation(void)
> > +{
> > +     setup_clear_cpu_cap(X86_FEATURE_FENCE_SWAPGS_USER);
> > +     setup_clear_cpu_cap(X86_FEATURE_FENCE_SWAPGS_KERNEL);
> > +     spectre_v1_mitigation = SPECTRE_V1_MITIGATION_AUTO;
> > +}
> > +#endif
>
> Right, let's see in the end what is easier: we have the cross-dependencies
> between mitigations so having separate "undo" functions might require them to
> run in reverse order to the "apply" functions. I can imagine a single "undo"
> function would be easier because you have everything in one place.
>
> We'll see.
>

We have cross-dependencies around the *selection* of mitigations, but not around the application of them.  There is no ordering requirement around the *_apply_mitigation() functions.  As such I would not expect (and have not observed) any ordering requirements around the reset functions.

--David Kaplan

RE: [RFC PATCH 34/56] x86/alternative: Save old bytes for alternatives

[Kaplan, David]

[AMD Official Use Only - AMD Internal Distribution Only]

> -----Original Message-----
> From: Nikolay Borisov <nik.borisov@suse.com>
> Sent: Wednesday, October 29, 2025 4:37 AM
> To: Kaplan, David <David.Kaplan@amd.com>; Juergen Gross
> <jgross@suse.com>; Thomas Gleixner <tglx@linutronix.de>; Borislav Petkov
> <bp@alien8.de>; Peter Zijlstra <peterz@infradead.org>; Josh Poimboeuf
> <jpoimboe@kernel.org>; Pawan Gupta <pawan.kumar.gupta@linux.intel.com>;
> Ingo Molnar <mingo@redhat.com>; Dave Hansen <dave.hansen@linux.intel.com>;
> x86@kernel.org; H . Peter Anvin <hpa@zytor.com>
> Cc: Alexander Graf <graf@amazon.com>; Boris Ostrovsky
> <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
> Subject: Re: [RFC PATCH 34/56] x86/alternative: Save old bytes for alternatives
>
> Caution: This message originated from an External Source. Use proper caution
> when opening attachments, clicking links, or responding.
>
>
> On 10/27/25 16:19, Kaplan, David wrote:
> > [AMD Official Use Only - AMD Internal Distribution Only]
> >
> >> -----Original Message-----
> >> From: Nikolay Borisov <nik.borisov@suse.com>
> >> Sent: Monday, October 27, 2025 6:35 AM
> >> To: Kaplan, David <David.Kaplan@amd.com>; Juergen Gross
> <jgross@suse.com>;
> >> Thomas Gleixner <tglx@linutronix.de>; Borislav Petkov <bp@alien8.de>; Peter
> >> Zijlstra <peterz@infradead.org>; Josh Poimboeuf <jpoimboe@kernel.org>;
> Pawan
> >> Gupta <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar
> <mingo@redhat.com>;
> >> Dave Hansen <dave.hansen@linux.intel.com>; x86@kernel.org; H . Peter Anvin
> >> <hpa@zytor.com>
> >> Cc: Alexander Graf <graf@amazon.com>; Boris Ostrovsky
> >> <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
> >> Subject: Re: [RFC PATCH 34/56] x86/alternative: Save old bytes for alternatives
> >>
> >> Caution: This message originated from an External Source. Use proper caution
> >> when opening attachments, clicking links, or responding.
> >>
> >>
> >> On 10/15/25 16:45, Kaplan, David wrote:
> >>> [AMD Official Use Only - AMD Internal Distribution Only]
> >>>
> >>>> -----Original Message-----
> >>>> From: Juergen Gross <jgross@suse.com>
> >>>> Sent: Wednesday, October 15, 2025 5:38 AM
> >>>> To: Kaplan, David <David.Kaplan@amd.com>; Thomas Gleixner
> >>>> <tglx@linutronix.de>; Borislav Petkov <bp@alien8.de>; Peter Zijlstra
> >>>> <peterz@infradead.org>; Josh Poimboeuf <jpoimboe@kernel.org>; Pawan
> >> Gupta
> >>>> <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar <mingo@redhat.com>;
> >> Dave
> >>>> Hansen <dave.hansen@linux.intel.com>; x86@kernel.org; H . Peter Anvin
> >>>> <hpa@zytor.com>
> >>>> Cc: Alexander Graf <graf@amazon.com>; Boris Ostrovsky
> >>>> <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
> >>>> Subject: Re: [RFC PATCH 34/56] x86/alternative: Save old bytes for
> alternatives
> >>>>
> >>>> On 13.10.25 16:34, David Kaplan wrote:
> >>>>> Save the existing instruction bytes at each alternative site when patching.
> >>>>> This is only done the first time, and these will be used later to help
> >>>>> restore the code back to its original form.
> >>>>>
> >>>>> Signed-off-by: David Kaplan <david.kaplan@amd.com>
> >>>>
> >>>> Instead of saving the original instructions at runtime, why don't you
> >>>> expand struct alt_instr to have an additional offset to a saved copy
> >>>> of the original instruction, located in .altinstr_replacement?
> >>>>
> >>>> The new field should be guarded with #ifdef
> CONFIG_DYNAMIC_MITIGATIONS,
> >>>> of course, like the added handling in the ALTERNATIVE() macros.
> >>>>
> >>>
> >>> That's an interesting idea, I think that could work.  That would make the kernel
> >> image on disk (slightly) larger though, as the original bytes will essentially be
> >> duplicated (at the original location and in .altinstr_replacement).  I'm not sure
> which
> >> is the better trade-off (kernel image bytes on disk vs runtime memory usage).
> >> Although I think we're talking about a relatively small amount of memory
> regardless.
> >> Most of the runtime overhead of dynamic mitigations comes from remembering
> the
> >> call sites/returns.
> >>
> >> It's not just about memory usage per-se but also memory pressure from
> >> allocation and the resulting fragmentation, though I'd think that
> >> majority of the allocation will fit into kmalloc-32 bucket, still having
> >> them as part of the kernel image eliminates the additional allocs.
> >
> > I see.  Just to understand, the issue is more with the numerous small allocations
> right? (that is the kmalloc at each alt_site)  And less about the single large allocation
> of the array?
>
> Yep, do you have some statistics how many allocs have to be done?
>

On a typical kernel, I'm seeing 6427 kmallocs() from this with a total size of ~36kb.

If that is too many, another option could be to go through and figure out the total size needed and then do one big allocation.

--David Kaplan

Re: [RFC PATCH 34/56] x86/alternative: Save old bytes for alternatives

[David Laight]

On Wed, 29 Oct 2025 16:26:58 +0000
"Kaplan, David" <David.Kaplan@amd.com> wrote:

> [AMD Official Use Only - AMD Internal Distribution Only]
> 
> > -----Original Message-----
> > From: Nikolay Borisov <nik.borisov@suse.com>
> > Sent: Wednesday, October 29, 2025 4:37 AM
> > To: Kaplan, David <David.Kaplan@amd.com>; Juergen Gross
> > <jgross@suse.com>; Thomas Gleixner <tglx@linutronix.de>; Borislav Petkov
> > <bp@alien8.de>; Peter Zijlstra <peterz@infradead.org>; Josh Poimboeuf
> > <jpoimboe@kernel.org>; Pawan Gupta <pawan.kumar.gupta@linux.intel.com>;
> > Ingo Molnar <mingo@redhat.com>; Dave Hansen <dave.hansen@linux.intel.com>;
> > x86@kernel.org; H . Peter Anvin <hpa@zytor.com>
> > Cc: Alexander Graf <graf@amazon.com>; Boris Ostrovsky
> > <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
> > Subject: Re: [RFC PATCH 34/56] x86/alternative: Save old bytes for alternatives
> >
> > Caution: This message originated from an External Source. Use proper caution
> > when opening attachments, clicking links, or responding.
> >
> >
> > On 10/27/25 16:19, Kaplan, David wrote:  
> > > [AMD Official Use Only - AMD Internal Distribution Only]
> > >  
> > >> -----Original Message-----
> > >> From: Nikolay Borisov <nik.borisov@suse.com>
> > >> Sent: Monday, October 27, 2025 6:35 AM
> > >> To: Kaplan, David <David.Kaplan@amd.com>; Juergen Gross  
> > <jgross@suse.com>;  
> > >> Thomas Gleixner <tglx@linutronix.de>; Borislav Petkov <bp@alien8.de>; Peter
> > >> Zijlstra <peterz@infradead.org>; Josh Poimboeuf <jpoimboe@kernel.org>;  
> > Pawan  
> > >> Gupta <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar  
> > <mingo@redhat.com>;  
> > >> Dave Hansen <dave.hansen@linux.intel.com>; x86@kernel.org; H . Peter Anvin
> > >> <hpa@zytor.com>
> > >> Cc: Alexander Graf <graf@amazon.com>; Boris Ostrovsky
> > >> <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
> > >> Subject: Re: [RFC PATCH 34/56] x86/alternative: Save old bytes for alternatives
> > >>
> > >> Caution: This message originated from an External Source. Use proper caution
> > >> when opening attachments, clicking links, or responding.
> > >>
> > >>
> > >> On 10/15/25 16:45, Kaplan, David wrote:  
> > >>> [AMD Official Use Only - AMD Internal Distribution Only]
> > >>>  
> > >>>> -----Original Message-----
> > >>>> From: Juergen Gross <jgross@suse.com>
> > >>>> Sent: Wednesday, October 15, 2025 5:38 AM
> > >>>> To: Kaplan, David <David.Kaplan@amd.com>; Thomas Gleixner
> > >>>> <tglx@linutronix.de>; Borislav Petkov <bp@alien8.de>; Peter Zijlstra
> > >>>> <peterz@infradead.org>; Josh Poimboeuf <jpoimboe@kernel.org>; Pawan  
> > >> Gupta  
> > >>>> <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar <mingo@redhat.com>;  
> > >> Dave  
> > >>>> Hansen <dave.hansen@linux.intel.com>; x86@kernel.org; H . Peter Anvin
> > >>>> <hpa@zytor.com>
> > >>>> Cc: Alexander Graf <graf@amazon.com>; Boris Ostrovsky
> > >>>> <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
> > >>>> Subject: Re: [RFC PATCH 34/56] x86/alternative: Save old bytes for  
> > alternatives  
> > >>>>
> > >>>> On 13.10.25 16:34, David Kaplan wrote:  
> > >>>>> Save the existing instruction bytes at each alternative site when patching.
> > >>>>> This is only done the first time, and these will be used later to help
> > >>>>> restore the code back to its original form.
> > >>>>>
> > >>>>> Signed-off-by: David Kaplan <david.kaplan@amd.com>  
> > >>>>
> > >>>> Instead of saving the original instructions at runtime, why don't you
> > >>>> expand struct alt_instr to have an additional offset to a saved copy
> > >>>> of the original instruction, located in .altinstr_replacement?
> > >>>>
> > >>>> The new field should be guarded with #ifdef  
> > CONFIG_DYNAMIC_MITIGATIONS,  
> > >>>> of course, like the added handling in the ALTERNATIVE() macros.
> > >>>>  
> > >>>
> > >>> That's an interesting idea, I think that could work.  That would make the kernel  
> > >> image on disk (slightly) larger though, as the original bytes will essentially be
> > >> duplicated (at the original location and in .altinstr_replacement).  I'm not sure  
> > which  
> > >> is the better trade-off (kernel image bytes on disk vs runtime memory usage).
> > >> Although I think we're talking about a relatively small amount of memory  
> > regardless.  
> > >> Most of the runtime overhead of dynamic mitigations comes from remembering  
> > the  
> > >> call sites/returns.
> > >>
> > >> It's not just about memory usage per-se but also memory pressure from
> > >> allocation and the resulting fragmentation, though I'd think that
> > >> majority of the allocation will fit into kmalloc-32 bucket, still having
> > >> them as part of the kernel image eliminates the additional allocs.  
> > >
> > > I see.  Just to understand, the issue is more with the numerous small allocations  
> > right? (that is the kmalloc at each alt_site)  And less about the single large allocation
> > of the array?
> >
> > Yep, do you have some statistics how many allocs have to be done?
> >  
> 
> On a typical kernel, I'm seeing 6427 kmallocs() from this with a total size of ~36kb.
> 
> If that is too many, another option could be to go through and figure out the total size needed and then do one big allocation.

Is there also an 8 byte pointer to each allocation? They add up as well.
Is may be worth doing multiple (say) 4k allocations in a list (or array of pointers).
Then the pointer can be replaced by an offset into the overall 'big buffer'.
Align the entries (a bit) and maybe a the 8 byte pointer can be replaced with
a 16bit index?

	David


> 
> --David Kaplan

RE: [RFC PATCH 34/56] x86/alternative: Save old bytes for alternatives

[Kaplan, David]

[AMD Official Use Only - AMD Internal Distribution Only]

> -----Original Message-----
> From: David Laight <david.laight.linux@gmail.com>
> Sent: Wednesday, October 29, 2025 5:14 PM
> To: Kaplan, David <David.Kaplan@amd.com>
> Cc: Nikolay Borisov <nik.borisov@suse.com>; Juergen Gross
> <jgross@suse.com>; Thomas Gleixner <tglx@linutronix.de>; Borislav Petkov
> <bp@alien8.de>; Peter Zijlstra <peterz@infradead.org>; Josh Poimboeuf
> <jpoimboe@kernel.org>; Pawan Gupta <pawan.kumar.gupta@linux.intel.com>;
> Ingo Molnar <mingo@redhat.com>; Dave Hansen <dave.hansen@linux.intel.com>;
> x86@kernel.org; H . Peter Anvin <hpa@zytor.com>; Alexander Graf
> <graf@amazon.com>; Boris Ostrovsky <boris.ostrovsky@oracle.com>; linux-
> kernel@vger.kernel.org
> Subject: Re: [RFC PATCH 34/56] x86/alternative: Save old bytes for alternatives
>
> Caution: This message originated from an External Source. Use proper caution
> when opening attachments, clicking links, or responding.
>
>
> On Wed, 29 Oct 2025 16:26:58 +0000
> "Kaplan, David" <David.Kaplan@amd.com> wrote:
>
> > [AMD Official Use Only - AMD Internal Distribution Only]
> >
> > > -----Original Message-----
> > > From: Nikolay Borisov <nik.borisov@suse.com>
> > > Sent: Wednesday, October 29, 2025 4:37 AM
> > > To: Kaplan, David <David.Kaplan@amd.com>; Juergen Gross
> > > <jgross@suse.com>; Thomas Gleixner <tglx@linutronix.de>; Borislav Petkov
> > > <bp@alien8.de>; Peter Zijlstra <peterz@infradead.org>; Josh Poimboeuf
> > > <jpoimboe@kernel.org>; Pawan Gupta <pawan.kumar.gupta@linux.intel.com>;
> > > Ingo Molnar <mingo@redhat.com>; Dave Hansen
> <dave.hansen@linux.intel.com>;
> > > x86@kernel.org; H . Peter Anvin <hpa@zytor.com>
> > > Cc: Alexander Graf <graf@amazon.com>; Boris Ostrovsky
> > > <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
> > > Subject: Re: [RFC PATCH 34/56] x86/alternative: Save old bytes for
> alternatives
> > >
> > > Caution: This message originated from an External Source. Use proper caution
> > > when opening attachments, clicking links, or responding.
> > >
> > >
> > > On 10/27/25 16:19, Kaplan, David wrote:
> > > > [AMD Official Use Only - AMD Internal Distribution Only]
> > > >
> > > >> -----Original Message-----
> > > >> From: Nikolay Borisov <nik.borisov@suse.com>
> > > >> Sent: Monday, October 27, 2025 6:35 AM
> > > >> To: Kaplan, David <David.Kaplan@amd.com>; Juergen Gross
> > > <jgross@suse.com>;
> > > >> Thomas Gleixner <tglx@linutronix.de>; Borislav Petkov <bp@alien8.de>;
> Peter
> > > >> Zijlstra <peterz@infradead.org>; Josh Poimboeuf <jpoimboe@kernel.org>;
> > > Pawan
> > > >> Gupta <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar
> > > <mingo@redhat.com>;
> > > >> Dave Hansen <dave.hansen@linux.intel.com>; x86@kernel.org; H . Peter
> Anvin
> > > >> <hpa@zytor.com>
> > > >> Cc: Alexander Graf <graf@amazon.com>; Boris Ostrovsky
> > > >> <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
> > > >> Subject: Re: [RFC PATCH 34/56] x86/alternative: Save old bytes for
> alternatives
> > > >>
> > > >> Caution: This message originated from an External Source. Use proper
> caution
> > > >> when opening attachments, clicking links, or responding.
> > > >>
> > > >>
> > > >> On 10/15/25 16:45, Kaplan, David wrote:
> > > >>> [AMD Official Use Only - AMD Internal Distribution Only]
> > > >>>
> > > >>>> -----Original Message-----
> > > >>>> From: Juergen Gross <jgross@suse.com>
> > > >>>> Sent: Wednesday, October 15, 2025 5:38 AM
> > > >>>> To: Kaplan, David <David.Kaplan@amd.com>; Thomas Gleixner
> > > >>>> <tglx@linutronix.de>; Borislav Petkov <bp@alien8.de>; Peter Zijlstra
> > > >>>> <peterz@infradead.org>; Josh Poimboeuf <jpoimboe@kernel.org>;
> Pawan
> > > >> Gupta
> > > >>>> <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar
> <mingo@redhat.com>;
> > > >> Dave
> > > >>>> Hansen <dave.hansen@linux.intel.com>; x86@kernel.org; H . Peter
> Anvin
> > > >>>> <hpa@zytor.com>
> > > >>>> Cc: Alexander Graf <graf@amazon.com>; Boris Ostrovsky
> > > >>>> <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
> > > >>>> Subject: Re: [RFC PATCH 34/56] x86/alternative: Save old bytes for
> > > alternatives
> > > >>>>
> > > >>>> On 13.10.25 16:34, David Kaplan wrote:
> > > >>>>> Save the existing instruction bytes at each alternative site when
> patching.
> > > >>>>> This is only done the first time, and these will be used later to help
> > > >>>>> restore the code back to its original form.
> > > >>>>>
> > > >>>>> Signed-off-by: David Kaplan <david.kaplan@amd.com>
> > > >>>>
> > > >>>> Instead of saving the original instructions at runtime, why don't you
> > > >>>> expand struct alt_instr to have an additional offset to a saved copy
> > > >>>> of the original instruction, located in .altinstr_replacement?
> > > >>>>
> > > >>>> The new field should be guarded with #ifdef
> > > CONFIG_DYNAMIC_MITIGATIONS,
> > > >>>> of course, like the added handling in the ALTERNATIVE() macros.
> > > >>>>
> > > >>>
> > > >>> That's an interesting idea, I think that could work.  That would make the
> kernel
> > > >> image on disk (slightly) larger though, as the original bytes will essentially be
> > > >> duplicated (at the original location and in .altinstr_replacement).  I'm not sure
> > > which
> > > >> is the better trade-off (kernel image bytes on disk vs runtime memory
> usage).
> > > >> Although I think we're talking about a relatively small amount of memory
> > > regardless.
> > > >> Most of the runtime overhead of dynamic mitigations comes from
> remembering
> > > the
> > > >> call sites/returns.
> > > >>
> > > >> It's not just about memory usage per-se but also memory pressure from
> > > >> allocation and the resulting fragmentation, though I'd think that
> > > >> majority of the allocation will fit into kmalloc-32 bucket, still having
> > > >> them as part of the kernel image eliminates the additional allocs.
> > > >
> > > > I see.  Just to understand, the issue is more with the numerous small
> allocations
> > > right? (that is the kmalloc at each alt_site)  And less about the single large
> allocation
> > > of the array?
> > >
> > > Yep, do you have some statistics how many allocs have to be done?
> > >
> >
> > On a typical kernel, I'm seeing 6427 kmallocs() from this with a total size of
> ~36kb.
> >
> > If that is too many, another option could be to go through and figure out the total
> size needed and then do one big allocation.
>
> Is there also an 8 byte pointer to each allocation? They add up as well.
> Is may be worth doing multiple (say) 4k allocations in a list (or array of pointers).
> Then the pointer can be replaced by an offset into the overall 'big buffer'.
> Align the entries (a bit) and maybe a the 8 byte pointer can be replaced with
> a 16bit index?
>

Yes, there is an 8B pointer to each allocation (although I didn't include that in the number above).

There's a number of ways to optimize this, doing a single 'big buffer' with perhaps a 32-bit index seems rather straightforward.  And maybe there are then further ways to squeeze this.  But I think we're really talking about a small amount of memory, especially compared to the other overhead noted above.

Thanks
--David Kaplan

Re: [RFC PATCH 34/56] x86/alternative: Save old bytes for alternatives

[Nikolay Borisov]

On 10/30/25 16:39, Kaplan, David wrote:
<snip>

>>
> 
> Yes, there is an 8B pointer to each allocation (although I didn't include that in the number above).
> 
> There's a number of ways to optimize this, doing a single 'big buffer' with perhaps a 32-bit index seems rather straightforward.  And maybe there are then further ways to squeeze this.  But I think we're really talking about a small amount of memory, especially compared to the other overhead noted above.
> 


I spoke with Vlastimil who's a lot more familiar with MM and he said 
that allocations made early in the boot are likely to fall within the 
same 2mb block so actually what we are discussing here might very well 
fall within  "premature optimisation" land.



> Thanks
> --David Kaplan

RE: [RFC PATCH 34/56] x86/alternative: Save old bytes for alternatives

[Kaplan, David]

[AMD Official Use Only - AMD Internal Distribution Only]

> -----Original Message-----
> From: Nikolay Borisov <nik.borisov@suse.com>
> Sent: Thursday, October 30, 2025 10:43 AM
> To: Kaplan, David <David.Kaplan@amd.com>; David Laight
> <david.laight.linux@gmail.com>
> Cc: Juergen Gross <jgross@suse.com>; Thomas Gleixner <tglx@linutronix.de>;
> Borislav Petkov <bp@alien8.de>; Peter Zijlstra <peterz@infradead.org>; Josh
> Poimboeuf <jpoimboe@kernel.org>; Pawan Gupta
> <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar <mingo@redhat.com>; Dave
> Hansen <dave.hansen@linux.intel.com>; x86@kernel.org; H . Peter Anvin
> <hpa@zytor.com>; Alexander Graf <graf@amazon.com>; Boris Ostrovsky
> <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org; Vlastimil Babka
> <vbabka@suse.cz>
> Subject: Re: [RFC PATCH 34/56] x86/alternative: Save old bytes for alternatives
>
> Caution: This message originated from an External Source. Use proper caution
> when opening attachments, clicking links, or responding.
>
>
> On 10/30/25 16:39, Kaplan, David wrote:
> <snip>
>
> >>
> >
> > Yes, there is an 8B pointer to each allocation (although I didn't include that in the
> number above).
> >
> > There's a number of ways to optimize this, doing a single 'big buffer' with perhaps
> a 32-bit index seems rather straightforward.  And maybe there are then further ways
> to squeeze this.  But I think we're really talking about a small amount of memory,
> especially compared to the other overhead noted above.
> >
>
>
> I spoke with Vlastimil who's a lot more familiar with MM and he said
> that allocations made early in the boot are likely to fall within the
> same 2mb block so actually what we are discussing here might very well
> fall within  "premature optimisation" land.
>

Ah, that's good to know.  I'll leave it as-is for now.

Thanks!
--David Kaplan

Re: [RFC PATCH 50/56] x86/alternative: Add re-patch support

[Nikolay Borisov]

On 10/13/25 17:34, David Kaplan wrote:
> Updating alternatives is done under the biggest hammers possible.  The
> freezer is used to freeze all processes and kernel threads at safe
> points to ensure they are not in the middle of a sequence we're about to
> patch.  Then stop_machine_nmi() synchronizes all CPUs and puts them into
> a tight spin loop while re-patching occurs.  The actual patching is done
> using simple memcpy, just like during boot.
> 
> Signed-off-by: David Kaplan <david.kaplan@amd.com>
> ---
>   arch/x86/include/asm/alternative.h |   6 ++
>   arch/x86/kernel/alternative.c      | 131 +++++++++++++++++++++++++++++
>   2 files changed, 137 insertions(+)
> 
> diff --git a/arch/x86/include/asm/alternative.h b/arch/x86/include/asm/alternative.h
> index 61ce8a4b1aa6..f0b863292c3c 100644
> --- a/arch/x86/include/asm/alternative.h
> +++ b/arch/x86/include/asm/alternative.h
> @@ -19,6 +19,7 @@
>   #ifndef __ASSEMBLER__
>   
>   #include <linux/stddef.h>
> +#include <linux/static_call_types.h>
>   
>   /*
>    * Alternative inline assembly for SMP.
> @@ -89,6 +90,9 @@ extern s32 __cfi_sites[],	__cfi_sites_end[];
>   extern s32 __ibt_endbr_seal[],	__ibt_endbr_seal_end[];
>   extern s32 __smp_locks[],	__smp_locks_end[];
>   
> +extern struct static_call_site __start_static_call_sites[],
> +			       __stop_static_call_sites[];
> +
>   /*
>    * Debug flag that can be tested to see whether alternative
>    * instructions were patched in already:
> @@ -98,6 +102,8 @@ extern int alternatives_patched;
>   struct module;
>   
>   #ifdef CONFIG_DYNAMIC_MITIGATIONS
> +extern void cpu_update_alternatives(void);
> +extern void cpu_prepare_repatch_alternatives(void);
>   extern void reset_retpolines(s32 *start, s32 *end, struct module *mod);
>   extern void reset_returns(s32 *start, s32 *end, struct module *mod);
>   extern void reset_alternatives(struct alt_instr *start, struct alt_instr *end,
> diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c
> index 23bb3386ec5e..613cb645bd9f 100644
> --- a/arch/x86/kernel/alternative.c
> +++ b/arch/x86/kernel/alternative.c
> @@ -6,12 +6,15 @@
>   #include <linux/vmalloc.h>
>   #include <linux/memory.h>
>   #include <linux/execmem.h>
> +#include <linux/stop_machine.h>
> +#include <linux/freezer.h>
>   
>   #include <asm/text-patching.h>
>   #include <asm/insn.h>
>   #include <asm/ibt.h>
>   #include <asm/set_memory.h>
>   #include <asm/nmi.h>
> +#include <asm/bugs.h>
>   
>   int __read_mostly alternatives_patched;
>   
> @@ -3468,4 +3471,132 @@ void its_free_all(struct module *mod)
>   	its_page = NULL;
>   }
>   #endif
> +static atomic_t thread_ack;
> +
> +/*
> + * This function is called by ALL online CPUs but only CPU0 will do the
> + * re-patching.  It is important that all other cores spin in the tight loop
> + * below (and not in multi_cpu_stop) because they cannot safely do return
> + * instructions while returns are being patched.  Therefore, spin them here
> + * (with interrupts disabled) until CPU0 has finished its work.
> + */
> +static int __cpu_update_alternatives(void *__unused)
> +{
> +	if (smp_processor_id()) {
> +		atomic_dec(&thread_ack);
> +		while (!READ_ONCE(alternatives_patched))
> +			cpu_relax();
> +
> +		cpu_bugs_update_speculation_msrs();
> +	} else {
> +		repatch_in_progress = true;
> +
> +		/* Wait for all cores to enter this function. */
> +		while (atomic_read(&thread_ack))
> +			cpu_relax();
> +
> +		/* These must be un-done in the opposite order in which they were applied. */
> +		reset_alternatives(__alt_instructions, __alt_instructions_end, NULL);
> +		reset_builtin_callthunks();
> +		reset_returns(__return_sites, __return_sites_end, NULL);
> +		reset_retpolines(__retpoline_sites, __retpoline_sites_end, NULL);
> +
> +		apply_retpolines(__retpoline_sites, __retpoline_sites_end, NULL);
> +		apply_returns(__return_sites, __return_sites_end, NULL);

This triggers the following splat:

[  363.467469] BUG: sleeping function called from invalid context at kernel/locking/mutex.c:575
[  363.467472] in_atomic(): 1, irqs_disabled(): 1, non_block: 0, pid: 18, name: migration/0
[  363.467472] preempt_count: 110001, expected: 0
[  363.467473] RCU nest depth: 0, expected: 0
[  363.467474] no locks held by migration/0/18.
[  363.467474] irq event stamp: 1280
[  363.467475] hardirqs last  enabled at (1279): [<ffffffff91fd1444>] _raw_spin_unlock_irq+0x24/0x50
[  363.467479] hardirqs last disabled at (1280): [<ffffffff913c98f9>] multi_cpu_stop+0x119/0x170
[  363.467482] softirqs last  enabled at (0): [<ffffffff9129eaab>] copy_process+0x7fb/0x1990
[  363.467484] softirqs last disabled at (0): [<0000000000000000>] 0x0
[  363.467485] Preemption disabled at:
[  363.467486] [<ffffffff913c8e63>] cpu_stopper_thread+0x93/0x150
[  363.467488] CPU: 0 UID: 0 PID: 18 Comm: migration/0 Not tainted 6.18.0-rc1-default+ #9 PREEMPT(none)
[  363.467490] Hardware name: QEMU Ubuntu 24.04 PC (i440FX + PIIX, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[  363.467491] Stopper: multi_cpu_stop+0x0/0x170 <- __stop_cpus.constprop.0+0x77/0xb0
[  363.467493] Call Trace:
[  363.467494]  <NMI>
[  363.467496]  dump_stack_lvl+0x62/0x90
[  363.467498]  __might_resched+0x19f/0x2b0
[  363.467501]  ? its_return_thunk+0x10/0x10
[  363.467503]  __mutex_lock+0x67/0x1060
[  363.467504]  ? look_up_lock_class+0x59/0x130
[  363.467506]  ? look_up_lock_class+0x59/0x130
[  363.467508]  ? __static_call_fixup+0x4f/0xa0
[  363.467510]  ? insn_get_prefixes+0x1a4/0x3f0
[  363.467512]  ? __SCT__tp_func_emulate_vsyscall+0x8/0x8
[  363.467513]  ? its_return_thunk+0x10/0x10
[  363.467514]  ? its_return_thunk+0x10/0x10
[  363.467516]  ? __static_call_fixup+0x4f/0xa0
[  363.467517]  __static_call_fixup+0x4f/0xa0
[  363.467518]  ? __SCT__tp_func_emulate_vsyscall+0x8/0x8
[  363.467519]  apply_returns+0x13e/0x370
[  363.467523]  ? __SCT__tp_func_emulate_vsyscall+0x8/0x8
[  363.467524]  ? __SCT__x86_pmu_disable_all+0x7/0x8
[  363.467525]  ? __SCT__x86_pmu_handle_irq+0x5/0x8
[  363.467527]  ? __copy_user_flushcache+0xf3/0x100
[  363.467528]  ? its_return_thunk+0x10/0x10
[  363.467529]  __cpu_update_alternatives+0x1e3/0x240
[  363.467531]  ? x2apic_send_IPI+0x40/0x60
[  363.467533]  stop_machine_nmi_handler+0x29/0x40
[  363.467534]  default_do_nmi+0x137/0x1a0
[  363.467536]  exc_nmi+0xef/0x120
[  363.467538]  end_repeat_nmi+0xf/0x53

[  363.467578] ================================
[  363.467578] WARNING: inconsistent lock state
[  363.467578] 6.18.0-rc1-default+ #9 Tainted: G        W
[  363.467579] --------------------------------
[  363.467579] inconsistent {INITIAL USE} -> {IN-NMI} usage.
[  363.467580] migration/0/18 [HC1[1]:SC0[0]:HE0:SE1] takes:
[  363.467581] ffffffff92668c28 (text_mutex){+.+.}-{4:4}, at: __static_call_fixup+0x4f/0xa0
[  363.467583] {INITIAL USE} state was registered at:
[  363.467584] irq event stamp: 1280
[  363.467584] hardirqs last  enabled at (1279): [<ffffffff91fd1444>] _raw_spin_unlock_irq+0x24/0x50
[  363.467586] hardirqs last disabled at (1280): [<ffffffff913c98f9>] multi_cpu_stop+0x119/0x170
[  363.467587] softirqs last  enabled at (0): [<ffffffff9129eaab>] copy_process+0x7fb/0x1990
[  363.467588] softirqs last disabled at (0): [<0000000000000000>] 0x0
[  363.467589]
                other info that might help us debug this:
[  363.467590]  Possible unsafe locking scenario:

[  363.467590]        CPU0
[  363.467590]        ----
[  363.467590]   lock(text_mutex);
[  363.467591]   <Interrupt>
[  363.467591]     lock(text_mutex);
[  363.467592]
                 *** DEADLOCK ***

[  363.467592] no locks held by migration/0/18.
[  363.467592]
                stack backtrace:
[  363.467593] CPU: 0 UID: 0 PID: 18 Comm: migration/0 Tainted: G        W           6.18.0-rc1-default+ #9 PREEMPT(none)
[  363.467594] Tainted: [W]=WARN
[  363.467595] Hardware name: QEMU Ubuntu 24.04 PC (i440FX + PIIX, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[  363.467595] Stopper: multi_cpu_stop+0x0/0x170 <- __stop_cpus.constprop.0+0x77/0xb0
[  363.467597] Call Trace:
[  363.467598]  <NMI>
[  363.467598]  dump_stack_lvl+0x62/0x90
[  363.467600]  print_usage_bug.part.0+0x22c/0x2c0
[  363.467602]  lock_acquire+0x208/0x2d0
[  363.467604]  ? __static_call_fixup+0x4f/0xa0
[  363.467605]  ? its_return_thunk+0x10/0x10
[  363.467607]  __mutex_lock+0xb3/0x1060
[  363.467607]  ? __static_call_fixup+0x4f/0xa0
[  363.467608]  ? look_up_lock_class+0x59/0x130
[  363.467610]  ? look_up_lock_class+0x59/0x130
[  363.467611]  ? __static_call_fixup+0x4f/0xa0
[  363.467613]  ? insn_get_prefixes+0x1a4/0x3f0
[  363.467614]  ? __SCT__tp_func_emulate_vsyscall+0x8/0x8
[  363.467615]  ? its_return_thunk+0x10/0x10
[  363.467617]  ? its_return_thunk+0x10/0x10
[  363.467618]  ? __static_call_fixup+0x4f/0xa0
[  363.467619]  __static_call_fixup+0x4f/0xa0
[  363.467619]  ? __SCT__tp_func_emulate_vsyscall+0x8/0x8
[  363.467621]  apply_returns+0x13e/0x370
[  363.467624]  ? __SCT__tp_func_emulate_vsyscall+0x8/0x8
[  363.467625]  ? __SCT__x86_pmu_disable_all+0x7/0x8
[  363.467626]  ? __SCT__x86_pmu_handle_irq+0x5/0x8
[  363.467627]  ? __copy_user_flushcache+0xf3/0x100
[  363.467628]  ? its_return_thunk+0x10/0x10
[  363.467630]  __cpu_update_alternatives+0x1e3/0x240
[  363.467631]  ? x2apic_send_IPI+0x40/0x60
[  363.467633]  stop_machine_nmi_handler+0x29/0x40
[  363.467634]  default_do_nmi+0x137/0x1a0
[  363.467635]  exc_nmi+0xef/0x120
[  363.467637]  end_repeat_nmi+0xf/0x53


The reason being apply_returns->__static_call_fixup acquires text_mutex from NMI context.


<snip>

Re: [RFC PATCH 04/56] x86/bugs: Reset spectre_v1 mitigations

[Borislav Petkov]

On Wed, Oct 29, 2025 at 01:48:03PM +0000, Kaplan, David wrote:
> We have cross-dependencies around the *selection* of mitigations, but not
> around the application of them.  There is no ordering requirement around the
> *_apply_mitigation() functions.  As such I would not expect (and have not
> observed) any ordering requirements around the reset functions.

This sounds like a single undo-function should be fine...

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: [RFC PATCH 05/56] x86/bugs: Reset spectre_v2 mitigations

[Borislav Petkov]

On Mon, Oct 13, 2025 at 09:33:53AM -0500, David Kaplan wrote:
> Add function to reset spectre_v2 mitigations back to their boot-time
> defaults.
> 
> Signed-off-by: David Kaplan <david.kaplan@amd.com>
> ---
>  arch/x86/kernel/cpu/bugs.c | 19 +++++++++++++++++++
>  1 file changed, 19 insertions(+)
> 
> diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
> index 9d5c6a3e50e1..0430635bb17d 100644
> --- a/arch/x86/kernel/cpu/bugs.c
> +++ b/arch/x86/kernel/cpu/bugs.c
> @@ -2477,6 +2477,24 @@ static void __init spectre_v2_apply_mitigation(void)
>  	}
>  }
>  
> +#ifdef CONFIG_DYNAMIC_MITIGATIONS
> +static void spectre_v2_reset_mitigation(void)

Yeah, the reset should definitely be a single function - this ifdeffery around
every single one is going to become yucky.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

RE: [RFC PATCH 05/56] x86/bugs: Reset spectre_v2 mitigations

[Kaplan, David]

[AMD Official Use Only - AMD Internal Distribution Only]

> -----Original Message-----
> From: Borislav Petkov <bp@alien8.de>
> Sent: Monday, November 3, 2025 1:32 PM
> To: Kaplan, David <David.Kaplan@amd.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>; Peter Zijlstra
> <peterz@infradead.org>; Josh Poimboeuf <jpoimboe@kernel.org>; Pawan
> Gupta <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar
> <mingo@redhat.com>; Dave Hansen <dave.hansen@linux.intel.com>;
> x86@kernel.org; H . Peter Anvin <hpa@zytor.com>; Alexander Graf
> <graf@amazon.com>; Boris Ostrovsky <boris.ostrovsky@oracle.com>; linux-
> kernel@vger.kernel.org
> Subject: Re: [RFC PATCH 05/56] x86/bugs: Reset spectre_v2 mitigations
>
> Caution: This message originated from an External Source. Use proper caution
> when opening attachments, clicking links, or responding.
>
>
> On Mon, Oct 13, 2025 at 09:33:53AM -0500, David Kaplan wrote:
> > Add function to reset spectre_v2 mitigations back to their boot-time
> > defaults.
> >
> > Signed-off-by: David Kaplan <david.kaplan@amd.com>
> > ---
> >  arch/x86/kernel/cpu/bugs.c | 19 +++++++++++++++++++
> >  1 file changed, 19 insertions(+)
> >
> > diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
> > index 9d5c6a3e50e1..0430635bb17d 100644
> > --- a/arch/x86/kernel/cpu/bugs.c
> > +++ b/arch/x86/kernel/cpu/bugs.c
> > @@ -2477,6 +2477,24 @@ static void __init
> spectre_v2_apply_mitigation(void)
> >       }
> >  }
> >
> > +#ifdef CONFIG_DYNAMIC_MITIGATIONS
> > +static void spectre_v2_reset_mitigation(void)
>
> Yeah, the reset should definitely be a single function - this ifdeffery around
> every single one is going to become yucky.
>

Do you really want it all in one big function?  Or just to relocate all the *_reset_mitigation() functions to a single place so they can all go under one ifdef?

I can do it in one big function, but it'd probably look something like:

/* Reset spectre_v1 */
setup_clear_cpu_cap(X86_FEATURE_FENCE_SWAPGS_USER);
setup_clear_cpu_cap(X86_FEATURE_FENCE_SWAPGS_KERNEL);
spectre_v1_mitigation = SPECTRE_V1_MITIGATION_AUTO;
/* Reset mds */
setup_clear_cpu_cap(X86_FEATURE_CLEAR_CPU_BUF);
static_branch_disable(&cpu_buf_idle_clear);
mds_mitigation = IS_ENABLED(CONFIG_MITIGATION_MDS) ?
                MDS_MITIGATION_AUTO : MDS_MITIGATION_OFF;
/* Reset spectre_v2 */
Etc.

--David Kaplan

Re: [RFC PATCH 05/56] x86/bugs: Reset spectre_v2 mitigations

[Borislav Petkov]

On Mon, Nov 03, 2025 at 08:10:39PM +0000, Kaplan, David wrote:
> Do you really want it all in one big function?  Or just to relocate all the
> *_reset_mitigation() functions to a single place so they can all go under
> one ifdef?
> 
> I can do it in one big function, but it'd probably look something like:
> 
> /* Reset spectre_v1 */
> setup_clear_cpu_cap(X86_FEATURE_FENCE_SWAPGS_USER);
> setup_clear_cpu_cap(X86_FEATURE_FENCE_SWAPGS_KERNEL);
> spectre_v1_mitigation = SPECTRE_V1_MITIGATION_AUTO;
> /* Reset mds */
> setup_clear_cpu_cap(X86_FEATURE_CLEAR_CPU_BUF);
> static_branch_disable(&cpu_buf_idle_clear);
> mds_mitigation = IS_ENABLED(CONFIG_MITIGATION_MDS) ?
>                 MDS_MITIGATION_AUTO : MDS_MITIGATION_OFF;
> /* Reset spectre_v2 */
> Etc.

Yap, that's what I thought too.

Since there's no point to have separate functions, the comment separation is
perfectly sufficient, I'd say.

Thx.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

RE: [RFC PATCH 50/56] x86/alternative: Add re-patch support

[Kaplan, David]

[AMD Official Use Only - AMD Internal Distribution Only]

> -----Original Message-----
> From: Nikolay Borisov <nik.borisov@suse.com>
> Sent: Friday, October 31, 2025 5:23 AM
> To: Kaplan, David <David.Kaplan@amd.com>; Thomas Gleixner
> <tglx@linutronix.de>; Borislav Petkov <bp@alien8.de>; Peter Zijlstra
> <peterz@infradead.org>; Josh Poimboeuf <jpoimboe@kernel.org>; Pawan
> Gupta <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar
> <mingo@redhat.com>; Dave Hansen <dave.hansen@linux.intel.com>;
> x86@kernel.org; H . Peter Anvin <hpa@zytor.com>
> Cc: Alexander Graf <graf@amazon.com>; Boris Ostrovsky
> <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
> Subject: Re: [RFC PATCH 50/56] x86/alternative: Add re-patch support
>
> Caution: This message originated from an External Source. Use proper caution
> when opening attachments, clicking links, or responding.
>
>
> On 10/13/25 17:34, David Kaplan wrote:
> > Updating alternatives is done under the biggest hammers possible.  The
> > freezer is used to freeze all processes and kernel threads at safe
> > points to ensure they are not in the middle of a sequence we're about to
> > patch.  Then stop_machine_nmi() synchronizes all CPUs and puts them into
> > a tight spin loop while re-patching occurs.  The actual patching is done
> > using simple memcpy, just like during boot.
> >
> > Signed-off-by: David Kaplan <david.kaplan@amd.com>
> > ---
> >   arch/x86/include/asm/alternative.h |   6 ++
> >   arch/x86/kernel/alternative.c      | 131
> +++++++++++++++++++++++++++++
> >   2 files changed, 137 insertions(+)
> >
> > diff --git a/arch/x86/include/asm/alternative.h
> b/arch/x86/include/asm/alternative.h
> > index 61ce8a4b1aa6..f0b863292c3c 100644
> > --- a/arch/x86/include/asm/alternative.h
> > +++ b/arch/x86/include/asm/alternative.h
> > @@ -19,6 +19,7 @@
> >   #ifndef __ASSEMBLER__
> >
> >   #include <linux/stddef.h>
> > +#include <linux/static_call_types.h>
> >
> >   /*
> >    * Alternative inline assembly for SMP.
> > @@ -89,6 +90,9 @@ extern s32 __cfi_sites[],   __cfi_sites_end[];
> >   extern s32 __ibt_endbr_seal[],      __ibt_endbr_seal_end[];
> >   extern s32 __smp_locks[],   __smp_locks_end[];
> >
> > +extern struct static_call_site __start_static_call_sites[],
> > +                            __stop_static_call_sites[];
> > +
> >   /*
> >    * Debug flag that can be tested to see whether alternative
> >    * instructions were patched in already:
> > @@ -98,6 +102,8 @@ extern int alternatives_patched;
> >   struct module;
> >
> >   #ifdef CONFIG_DYNAMIC_MITIGATIONS
> > +extern void cpu_update_alternatives(void);
> > +extern void cpu_prepare_repatch_alternatives(void);
> >   extern void reset_retpolines(s32 *start, s32 *end, struct module *mod);
> >   extern void reset_returns(s32 *start, s32 *end, struct module *mod);
> >   extern void reset_alternatives(struct alt_instr *start, struct alt_instr *end,
> > diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c
> > index 23bb3386ec5e..613cb645bd9f 100644
> > --- a/arch/x86/kernel/alternative.c
> > +++ b/arch/x86/kernel/alternative.c
> > @@ -6,12 +6,15 @@
> >   #include <linux/vmalloc.h>
> >   #include <linux/memory.h>
> >   #include <linux/execmem.h>
> > +#include <linux/stop_machine.h>
> > +#include <linux/freezer.h>
> >
> >   #include <asm/text-patching.h>
> >   #include <asm/insn.h>
> >   #include <asm/ibt.h>
> >   #include <asm/set_memory.h>
> >   #include <asm/nmi.h>
> > +#include <asm/bugs.h>
> >
> >   int __read_mostly alternatives_patched;
> >
> > @@ -3468,4 +3471,132 @@ void its_free_all(struct module *mod)
> >       its_page = NULL;
> >   }
> >   #endif
> > +static atomic_t thread_ack;
> > +
> > +/*
> > + * This function is called by ALL online CPUs but only CPU0 will do the
> > + * re-patching.  It is important that all other cores spin in the tight loop
> > + * below (and not in multi_cpu_stop) because they cannot safely do return
> > + * instructions while returns are being patched.  Therefore, spin them here
> > + * (with interrupts disabled) until CPU0 has finished its work.
> > + */
> > +static int __cpu_update_alternatives(void *__unused)
> > +{
> > +     if (smp_processor_id()) {
> > +             atomic_dec(&thread_ack);
> > +             while (!READ_ONCE(alternatives_patched))
> > +                     cpu_relax();
> > +
> > +             cpu_bugs_update_speculation_msrs();
> > +     } else {
> > +             repatch_in_progress = true;
> > +
> > +             /* Wait for all cores to enter this function. */
> > +             while (atomic_read(&thread_ack))
> > +                     cpu_relax();
> > +
> > +             /* These must be un-done in the opposite order in which they were
> applied. */
> > +             reset_alternatives(__alt_instructions, __alt_instructions_end, NULL);
> > +             reset_builtin_callthunks();
> > +             reset_returns(__return_sites, __return_sites_end, NULL);
> > +             reset_retpolines(__retpoline_sites, __retpoline_sites_end, NULL);
> > +
> > +             apply_retpolines(__retpoline_sites, __retpoline_sites_end, NULL);
> > +             apply_returns(__return_sites, __return_sites_end, NULL);
>
> This triggers the following splat:
>
> [  363.467469] BUG: sleeping function called from invalid context at
> kernel/locking/mutex.c:575
> [  363.467472] in_atomic(): 1, irqs_disabled(): 1, non_block: 0, pid: 18, name:
> migration/0
>
<snip>

>
> The reason being apply_returns->__static_call_fixup acquires text_mutex from
> NMI context.
>

Thank you for testing the code and reporting this!  I am looking into how best to resolve this.

--David Kaplan

Re: [RFC PATCH 05/56] x86/bugs: Reset spectre_v2 mitigations

[Josh Poimboeuf]

On Mon, Nov 03, 2025 at 09:28:11PM +0100, Borislav Petkov wrote:
> On Mon, Nov 03, 2025 at 08:10:39PM +0000, Kaplan, David wrote:
> > Do you really want it all in one big function?  Or just to relocate all the
> > *_reset_mitigation() functions to a single place so they can all go under
> > one ifdef?
> > 
> > I can do it in one big function, but it'd probably look something like:
> > 
> > /* Reset spectre_v1 */
> > setup_clear_cpu_cap(X86_FEATURE_FENCE_SWAPGS_USER);
> > setup_clear_cpu_cap(X86_FEATURE_FENCE_SWAPGS_KERNEL);
> > spectre_v1_mitigation = SPECTRE_V1_MITIGATION_AUTO;
> > /* Reset mds */
> > setup_clear_cpu_cap(X86_FEATURE_CLEAR_CPU_BUF);
> > static_branch_disable(&cpu_buf_idle_clear);
> > mds_mitigation = IS_ENABLED(CONFIG_MITIGATION_MDS) ?
> >                 MDS_MITIGATION_AUTO : MDS_MITIGATION_OFF;
> > /* Reset spectre_v2 */
> > Etc.
> 
> Yap, that's what I thought too.
> 
> Since there's no point to have separate functions, the comment separation is
> perfectly sufficient, I'd say.

Separate functions allows each reset function to stay close to its
select/update/apply counterparts.  That makes it easier to tell that
it's undoing all the right things.  Plus it preserves the existing
logical code layout/separation between mitigations.

-- 
Josh

Re: [RFC PATCH 05/56] x86/bugs: Reset spectre_v2 mitigations

[Borislav Petkov]

On Tue, Nov 04, 2025 at 06:29:20PM -0800, Josh Poimboeuf wrote:
> Separate functions allows each reset function to stay close to its
> select/update/apply counterparts.  That makes it easier to tell that
> it's undoing all the right things.  Plus it preserves the existing
> logical code layout/separation between mitigations.

... with a forward declaration for each one? Because we don't have enough
functions in this file already? And even if the code structure is begging for
us to turn it a OOO design, we're not doing it?

Sorry, but one big function with comments is at least keeping it saner. And
I'm sure your favourite editor can help you jump back'n'forth easily. :-P

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: [RFC PATCH 05/56] x86/bugs: Reset spectre_v2 mitigations

[Josh Poimboeuf]

On Wed, Nov 05, 2025 at 12:03:18PM +0100, Borislav Petkov wrote:
> On Tue, Nov 04, 2025 at 06:29:20PM -0800, Josh Poimboeuf wrote:
> > Separate functions allows each reset function to stay close to its
> > select/update/apply counterparts.  That makes it easier to tell that
> > it's undoing all the right things.  Plus it preserves the existing
> > logical code layout/separation between mitigations.
> 
> ... with a forward declaration for each one?

Nope, these patches don't add any forward declarations because they
sanely put the caller below the callees.

We should put cpu_select_mitigations() at the bottom too, then all those
existing forward declarations can go away.

> Because we don't have enough functions in this file already?

I don't see how the solution to "too many functions" is to start
squirreling away some arbitrary parts of (otherwise logically separated)
code in a hidden uber-function away from the rest?

> And even if the code structure is begging for
> us to turn it a OOO design, we're not doing it?

If "functions bad" then why not make cpu_select_mitigations() one big
happy function with a ton of comments?  Just think of all the function
savings ;-)

-- 
Josh

Re: [RFC PATCH 05/56] x86/bugs: Reset spectre_v2 mitigations

[Borislav Petkov]

On Wed, Nov 05, 2025 at 09:06:36AM -0800, Josh Poimboeuf wrote:
> Nope, these patches don't add any forward declarations because they
> sanely put the caller below the callees.

Not happy about the added ifdeffery tho. I guess we can move it inside the
functions themselves or mark them __maybe_unused - whatever makes the
compilers not complain.

> We should put cpu_select_mitigations() at the bottom too, then all those
> existing forward declarations can go away.

That's a good idea. We should, if it doesn't get too hairy.

> I don't see how the solution to "too many functions" is to start
> squirreling away some arbitrary parts of (otherwise logically separated)
> code in a hidden uber-function away from the rest?

I aim for this file to "keep it as simple as possible and leave enough
breadcrumbs as possible for later."

But your argument about keeping all the mitigations and their functions
together has some merit too.

Maybe we should do

arch/x86/kernel/cpu/bugs/mtg_<bla>.c
arch/x86/kernel/cpu/bugs/mtg_<foo>.c

:-P
 
> If "functions bad" then why not make cpu_select_mitigations() one big
> happy function with a ton of comments?  Just think of all the function
> savings ;-)

If it makes it more readable, always. But I see your point.

Thx.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

RE: [RFC PATCH 05/56] x86/bugs: Reset spectre_v2 mitigations

[Kaplan, David]

[AMD Official Use Only - AMD Internal Distribution Only]

> -----Original Message-----
> From: Borislav Petkov <bp@alien8.de>
> Sent: Wednesday, November 5, 2025 2:05 PM
> To: Josh Poimboeuf <jpoimboe@kernel.org>
> Cc: Kaplan, David <David.Kaplan@amd.com>; Thomas Gleixner
> <tglx@linutronix.de>; Peter Zijlstra <peterz@infradead.org>; Pawan Gupta
> <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar <mingo@redhat.com>;
> Dave Hansen <dave.hansen@linux.intel.com>; x86@kernel.org; H . Peter
> Anvin <hpa@zytor.com>; Alexander Graf <graf@amazon.com>; Boris
> Ostrovsky <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
> Subject: Re: [RFC PATCH 05/56] x86/bugs: Reset spectre_v2 mitigations
>
> Caution: This message originated from an External Source. Use proper
> caution when opening attachments, clicking links, or responding.
>
>
> On Wed, Nov 05, 2025 at 09:06:36AM -0800, Josh Poimboeuf wrote:
> > Nope, these patches don't add any forward declarations because they
> > sanely put the caller below the callees.
>
> Not happy about the added ifdeffery tho. I guess we can move it inside the
> functions themselves or mark them __maybe_unused - whatever makes the
> compilers not complain.
>
> > We should put cpu_select_mitigations() at the bottom too, then all those
> > existing forward declarations can go away.
>
> That's a good idea. We should, if it doesn't get too hairy.
>
> > I don't see how the solution to "too many functions" is to start
> > squirreling away some arbitrary parts of (otherwise logically separated)
> > code in a hidden uber-function away from the rest?
>
> I aim for this file to "keep it as simple as possible and leave enough
> breadcrumbs as possible for later."
>
> But your argument about keeping all the mitigations and their functions
> together has some merit too.
>
> Maybe we should do
>
> arch/x86/kernel/cpu/bugs/mtg_<bla>.c
> arch/x86/kernel/cpu/bugs/mtg_<foo>.c
>
> :-P
>
> > If "functions bad" then why not make cpu_select_mitigations() one big
> > happy function with a ton of comments?  Just think of all the function
> > savings ;-)
>
> If it makes it more readable, always. But I see your point.
>

Josh's thinking was aligned with my original thinking.  And if the #ifdefs are the only problem, I think I can just make them __maybe_unused instead.

That said, using a single function also allows for some de-duplication of code.  There are several mitigations that all use the same feature flags or other things (like x86_return_thunk) and those only need to be reset once.  Having them all in a single function makes that easier to optimize if desired.  Here's the single function version if you want to check that out and see if this is better or not:

void arch_cpu_reset_mitigations(void)
{
        /* Re-enable SMT in case it was disabled before. */
        if (cpu_smt_control == CPU_SMT_DISABLED)
                bugs_smt_disable(true);

        /* Spectre v1 */
        setup_clear_cpu_cap(X86_FEATURE_FENCE_SWAPGS_USER);
        setup_clear_cpu_cap(X86_FEATURE_FENCE_SWAPGS_KERNEL);
        spectre_v1_mitigation = SPECTRE_V1_MITIGATION_AUTO;

        /* Spectre v2 */
        x86_spec_ctrl_base &= ~SPEC_CTRL_IBRS;
        x86_spec_ctrl_base &= ~SPEC_CTRL_RRSBA_DIS_S;
        rrsba_disabled = false;
        setup_clear_cpu_cap(X86_FEATURE_KERNEL_IBRS);
        setup_clear_cpu_cap(X86_FEATURE_RETPOLINE_LFENCE);
        setup_clear_cpu_cap(X86_FEATURE_RETPOLINE);
        setup_clear_cpu_cap(X86_FEATURE_RSB_CTXSW);
        setup_clear_cpu_cap(X86_FEATURE_USE_IBPB_FW);
        spectre_v2_enabled = SPECTRE_V2_NONE;
        nospectre_v2 = false;
        spectre_v2_cmd = IS_ENABLED(CONFIG_MITIGATION_SPECTRE_V2) ?
                SPECTRE_V2_CMD_AUTO : SPECTRE_V2_CMD_NONE;

        /* Retbleed */
        setup_clear_cpu_cap(X86_FEATURE_RETHUNK);
        setup_clear_cpu_cap(X86_FEATURE_UNRET);
        setup_clear_cpu_cap(X86_FEATURE_ENTRY_IBPB);
        setup_clear_cpu_cap(X86_FEATURE_IBPB_ON_VMEXIT);
        setup_clear_cpu_cap(X86_FEATURE_CALL_DEPTH);
        x86_return_thunk = __x86_return_thunk;
        retbleed_mitigation = IS_ENABLED(CONFIG_MITIGATION_RETBLEED) ?
                RETBLEED_MITIGATION_AUTO : RETBLEED_MITIGATION_NONE;

        /* Spectre v2 user */
        static_branch_disable(&switch_vcpu_ibpb);
        static_branch_disable(&switch_mm_always_ibpb);
        static_branch_disable(&switch_mm_cond_ibpb);
        spectre_v2_user_stibp = SPECTRE_V2_USER_NONE;
        spectre_v2_user_ibpb = SPECTRE_V2_USER_NONE;
        spectre_v2_user_cmd = SPECTRE_V2_USER_CMD_AUTO;

        /* SSB */
        setup_clear_cpu_cap(X86_FEATURE_SPEC_STORE_BYPASS_DISABLE);
        x86_spec_ctrl_base &= ~SPEC_CTRL_SSBD;
        nossb = false;
        ssb_mode = IS_ENABLED(CONFIG_MITIGATION_SSB) ?
                SPEC_STORE_BYPASS_AUTO : SPEC_STORE_BYPASS_NONE;

        /* L1TF */
        setup_clear_cpu_cap(X86_FEATURE_L1TF_PTEINV);
        l1tf_mitigation = IS_ENABLED(CONFIG_MITIGATION_L1TF) ?
                L1TF_MITIGATION_AUTO : L1TF_MITIGATION_OFF;

        /* MDS */
        setup_clear_cpu_cap(X86_FEATURE_CLEAR_CPU_BUF);
        static_branch_disable(&cpu_buf_idle_clear);
        mds_mitigation = IS_ENABLED(CONFIG_MITIGATION_MDS) ?
                MDS_MITIGATION_AUTO : MDS_MITIGATION_OFF;

        /* MMIO */
        static_branch_disable(&cpu_buf_vm_clear);
        mmio_mitigation = IS_ENABLED(CONFIG_MITIGATION_MMIO_STALE_DATA) ?
                MMIO_MITIGATION_AUTO : MMIO_MITIGATION_OFF;

        /* SRBDS */
        srbds_mitigation = IS_ENABLED(CONFIG_MITIGATION_SRBDS) ?
                SRBDS_MITIGATION_AUTO : SRBDS_MITIGATION_OFF;

       /* SRSO */
        setup_clear_cpu_cap(X86_FEATURE_SRSO_ALIAS);
        setup_clear_cpu_cap(X86_FEATURE_SRSO);
        x86_pred_cmd = PRED_CMD_IBPB;
        srso_mitigation = SRSO_MITIGATION_AUTO;

        /* GDS */
        gds_mitigation =  IS_ENABLED(CONFIG_MITIGATION_GDS) ?
                GDS_MITIGATION_AUTO : GDS_MITIGATION_OFF;

        /* BHI */
        setup_clear_cpu_cap(X86_FEATURE_CLEAR_BHB_VMEXIT);
        setup_clear_cpu_cap(X86_FEATURE_CLEAR_BHB_LOOP);
        setup_clear_cpu_cap(X86_FEATURE_CLEAR_BHB_HW);
        x86_spec_ctrl_base &= ~SPEC_CTRL_BHI_DIS_S;
        bhi_mitigation = IS_ENABLED(CONFIG_MITIGATION_SPECTRE_BHI) ?
                BHI_MITIGATION_AUTO : BHI_MITIGATION_OFF;

        /* ITS */
        setup_clear_cpu_cap(X86_FEATURE_INDIRECT_THUNK_ITS);
        its_mitigation = IS_ENABLED(CONFIG_MITIGATION_ITS) ?
                ITS_MITIGATION_AUTO : ITS_MITIGATION_OFF;

        /* TSA */
        setup_clear_cpu_cap(X86_FEATURE_CLEAR_CPU_BUF_VM);
        tsa_mitigation =
                IS_ENABLED(CONFIG_MITIGATION_TSA) ? TSA_MITIGATION_AUTO : TSA_MITIGATION_NONE;

        /* VMSCAPE */
        setup_clear_cpu_cap(X86_FEATURE_IBPB_EXIT_TO_USER);
        vmscape_mitigation = IS_ENABLED(CONFIG_MITIGATION_VMSCAPE) ?
                VMSCAPE_MITIGATION_AUTO : VMSCAPE_MITIGATION_NONE;
}

--David Kaplan

Re: [RFC PATCH 05/56] x86/bugs: Reset spectre_v2 mitigations

[Josh Poimboeuf]

On Wed, Nov 05, 2025 at 08:21:08PM +0000, Kaplan, David wrote:
> > > If "functions bad" then why not make cpu_select_mitigations() one big
> > > happy function with a ton of comments?  Just think of all the function
> > > savings ;-)
> >
> > If it makes it more readable, always. But I see your point.
> >
> 
> Josh's thinking was aligned with my original thinking.  And if the
> #ifdefs are the only problem, I think I can just make them
> __maybe_unused instead.
> 
> That said, using a single function also allows for some de-duplication
> of code.  There are several mitigations that all use the same feature
> flags or other things (like x86_return_thunk) and those only need to
> be reset once.  Having them all in a single function makes that easier
> to optimize if desired.  Here's the single function version if you
> want to check that out and see if this is better or not:

I'm thinking it's probably best to keep things separate and contained
rather than intermixing / deduplicating, otherwise we might have to
disentangle them again later on.

Someday in the distant future we might want to start phasing out some of
the mitigations due to abandoned HW.

Or we might end up deciding to have a more modular per-mitigation build.

A little bit of duplicate effort is no big deal.  Running in
stop_machine_nmi(), it's not exactly performance-sensitive code :-)

-- 
Josh

[PATCH] x86/bugs: Get rid of the forward declarations

[Borislav Petkov]

On Wed, Nov 05, 2025 at 09:04:47PM +0100, Borislav Petkov wrote:
> > We should put cpu_select_mitigations() at the bottom too, then all those
> > existing forward declarations can go away.
> 
> That's a good idea. We should, if it doesn't get too hairy.

That was easier than I expected. Only build-tested.

---
From: "Borislav Petkov (AMD)" <bp@alien8.de>
Date: Fri, 14 Nov 2025 18:10:04 +0100

Get rid of the forward declarations of the mitigation functions by
moving their single caller below them.

No functional changes.

Suggested-by: Josh Poimboeuf <jpoimboe@kernel.org>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
---
 arch/x86/kernel/cpu/bugs.c | 233 +++++++++++++++----------------------
 1 file changed, 93 insertions(+), 140 deletions(-)

diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
index d7fa03bf51b4..4fa8284c53a1 100644
--- a/arch/x86/kernel/cpu/bugs.c
+++ b/arch/x86/kernel/cpu/bugs.c
@@ -53,53 +53,6 @@
  * mitigation option.
  */
 
-static void __init spectre_v1_select_mitigation(void);
-static void __init spectre_v1_apply_mitigation(void);
-static void __init spectre_v2_select_mitigation(void);
-static void __init spectre_v2_update_mitigation(void);
-static void __init spectre_v2_apply_mitigation(void);
-static void __init retbleed_select_mitigation(void);
-static void __init retbleed_update_mitigation(void);
-static void __init retbleed_apply_mitigation(void);
-static void __init spectre_v2_user_select_mitigation(void);
-static void __init spectre_v2_user_update_mitigation(void);
-static void __init spectre_v2_user_apply_mitigation(void);
-static void __init ssb_select_mitigation(void);
-static void __init ssb_apply_mitigation(void);
-static void __init l1tf_select_mitigation(void);
-static void __init l1tf_apply_mitigation(void);
-static void __init mds_select_mitigation(void);
-static void __init mds_update_mitigation(void);
-static void __init mds_apply_mitigation(void);
-static void __init taa_select_mitigation(void);
-static void __init taa_update_mitigation(void);
-static void __init taa_apply_mitigation(void);
-static void __init mmio_select_mitigation(void);
-static void __init mmio_update_mitigation(void);
-static void __init mmio_apply_mitigation(void);
-static void __init rfds_select_mitigation(void);
-static void __init rfds_update_mitigation(void);
-static void __init rfds_apply_mitigation(void);
-static void __init srbds_select_mitigation(void);
-static void __init srbds_apply_mitigation(void);
-static void __init l1d_flush_select_mitigation(void);
-static void __init srso_select_mitigation(void);
-static void __init srso_update_mitigation(void);
-static void __init srso_apply_mitigation(void);
-static void __init gds_select_mitigation(void);
-static void __init gds_apply_mitigation(void);
-static void __init bhi_select_mitigation(void);
-static void __init bhi_update_mitigation(void);
-static void __init bhi_apply_mitigation(void);
-static void __init its_select_mitigation(void);
-static void __init its_update_mitigation(void);
-static void __init its_apply_mitigation(void);
-static void __init tsa_select_mitigation(void);
-static void __init tsa_apply_mitigation(void);
-static void __init vmscape_select_mitigation(void);
-static void __init vmscape_update_mitigation(void);
-static void __init vmscape_apply_mitigation(void);
-
 /* The base value of the SPEC_CTRL MSR without task-specific bits set */
 u64 x86_spec_ctrl_base;
 EXPORT_SYMBOL_GPL(x86_spec_ctrl_base);
@@ -233,99 +186,6 @@ static void __init cpu_print_attack_vectors(void)
 	}
 }
 
-void __init cpu_select_mitigations(void)
-{
-	/*
-	 * Read the SPEC_CTRL MSR to account for reserved bits which may
-	 * have unknown values. AMD64_LS_CFG MSR is cached in the early AMD
-	 * init code as it is not enumerated and depends on the family.
-	 */
-	if (cpu_feature_enabled(X86_FEATURE_MSR_SPEC_CTRL)) {
-		rdmsrq(MSR_IA32_SPEC_CTRL, x86_spec_ctrl_base);
-
-		/*
-		 * Previously running kernel (kexec), may have some controls
-		 * turned ON. Clear them and let the mitigations setup below
-		 * rediscover them based on configuration.
-		 */
-		x86_spec_ctrl_base &= ~SPEC_CTRL_MITIGATIONS_MASK;
-	}
-
-	x86_arch_cap_msr = x86_read_arch_cap_msr();
-
-	cpu_print_attack_vectors();
-
-	/* Select the proper CPU mitigations before patching alternatives: */
-	spectre_v1_select_mitigation();
-	spectre_v2_select_mitigation();
-	retbleed_select_mitigation();
-	spectre_v2_user_select_mitigation();
-	ssb_select_mitigation();
-	l1tf_select_mitigation();
-	mds_select_mitigation();
-	taa_select_mitigation();
-	mmio_select_mitigation();
-	rfds_select_mitigation();
-	srbds_select_mitigation();
-	l1d_flush_select_mitigation();
-	srso_select_mitigation();
-	gds_select_mitigation();
-	its_select_mitigation();
-	bhi_select_mitigation();
-	tsa_select_mitigation();
-	vmscape_select_mitigation();
-
-	/*
-	 * After mitigations are selected, some may need to update their
-	 * choices.
-	 */
-	spectre_v2_update_mitigation();
-	/*
-	 * retbleed_update_mitigation() relies on the state set by
-	 * spectre_v2_update_mitigation(); specifically it wants to know about
-	 * spectre_v2=ibrs.
-	 */
-	retbleed_update_mitigation();
-	/*
-	 * its_update_mitigation() depends on spectre_v2_update_mitigation()
-	 * and retbleed_update_mitigation().
-	 */
-	its_update_mitigation();
-
-	/*
-	 * spectre_v2_user_update_mitigation() depends on
-	 * retbleed_update_mitigation(), specifically the STIBP
-	 * selection is forced for UNRET or IBPB.
-	 */
-	spectre_v2_user_update_mitigation();
-	mds_update_mitigation();
-	taa_update_mitigation();
-	mmio_update_mitigation();
-	rfds_update_mitigation();
-	bhi_update_mitigation();
-	/* srso_update_mitigation() depends on retbleed_update_mitigation(). */
-	srso_update_mitigation();
-	vmscape_update_mitigation();
-
-	spectre_v1_apply_mitigation();
-	spectre_v2_apply_mitigation();
-	retbleed_apply_mitigation();
-	spectre_v2_user_apply_mitigation();
-	ssb_apply_mitigation();
-	l1tf_apply_mitigation();
-	mds_apply_mitigation();
-	taa_apply_mitigation();
-	mmio_apply_mitigation();
-	rfds_apply_mitigation();
-	srbds_apply_mitigation();
-	srso_apply_mitigation();
-	gds_apply_mitigation();
-	its_apply_mitigation();
-	bhi_apply_mitigation();
-	tsa_apply_mitigation();
-	vmscape_apply_mitigation();
-}
-
 /*
  * NOTE: This function is *only* called for SVM, since Intel uses
  * MSR_IA32_SPEC_CTRL for SSBD.
@@ -3371,6 +3231,99 @@ void cpu_bugs_smt_update(void)
 	mutex_unlock(&spec_ctrl_mutex);
 }
 
+void __init cpu_select_mitigations(void)
+{
+	/*
+	 * Read the SPEC_CTRL MSR to account for reserved bits which may
+	 * have unknown values. AMD64_LS_CFG MSR is cached in the early AMD
+	 * init code as it is not enumerated and depends on the family.
+	 */
+	if (cpu_feature_enabled(X86_FEATURE_MSR_SPEC_CTRL)) {
+		rdmsrq(MSR_IA32_SPEC_CTRL, x86_spec_ctrl_base);
+
+		/*
+		 * Previously running kernel (kexec), may have some controls
+		 * turned ON. Clear them and let the mitigations setup below
+		 * rediscover them based on configuration.
+		 */
+		x86_spec_ctrl_base &= ~SPEC_CTRL_MITIGATIONS_MASK;
+	}
+
+	x86_arch_cap_msr = x86_read_arch_cap_msr();
+
+	cpu_print_attack_vectors();
+
+	/* Select the proper CPU mitigations before patching alternatives: */
+	spectre_v1_select_mitigation();
+	spectre_v2_select_mitigation();
+	retbleed_select_mitigation();
+	spectre_v2_user_select_mitigation();
+	ssb_select_mitigation();
+	l1tf_select_mitigation();
+	mds_select_mitigation();
+	taa_select_mitigation();
+	mmio_select_mitigation();
+	rfds_select_mitigation();
+	srbds_select_mitigation();
+	l1d_flush_select_mitigation();
+	srso_select_mitigation();
+	gds_select_mitigation();
+	its_select_mitigation();
+	bhi_select_mitigation();
+	tsa_select_mitigation();
+	vmscape_select_mitigation();
+
+	/*
+	 * After mitigations are selected, some may need to update their
+	 * choices.
+	 */
+	spectre_v2_update_mitigation();
+	/*
+	 * retbleed_update_mitigation() relies on the state set by
+	 * spectre_v2_update_mitigation(); specifically it wants to know about
+	 * spectre_v2=ibrs.
+	 */
+	retbleed_update_mitigation();
+	/*
+	 * its_update_mitigation() depends on spectre_v2_update_mitigation()
+	 * and retbleed_update_mitigation().
+	 */
+	its_update_mitigation();
+
+	/*
+	 * spectre_v2_user_update_mitigation() depends on
+	 * retbleed_update_mitigation(), specifically the STIBP
+	 * selection is forced for UNRET or IBPB.
+	 */
+	spectre_v2_user_update_mitigation();
+	mds_update_mitigation();
+	taa_update_mitigation();
+	mmio_update_mitigation();
+	rfds_update_mitigation();
+	bhi_update_mitigation();
+	/* srso_update_mitigation() depends on retbleed_update_mitigation(). */
+	srso_update_mitigation();
+	vmscape_update_mitigation();
+
+	spectre_v1_apply_mitigation();
+	spectre_v2_apply_mitigation();
+	retbleed_apply_mitigation();
+	spectre_v2_user_apply_mitigation();
+	ssb_apply_mitigation();
+	l1tf_apply_mitigation();
+	mds_apply_mitigation();
+	taa_apply_mitigation();
+	mmio_apply_mitigation();
+	rfds_apply_mitigation();
+	srbds_apply_mitigation();
+	srso_apply_mitigation();
+	gds_apply_mitigation();
+	its_apply_mitigation();
+	bhi_apply_mitigation();
+	tsa_apply_mitigation();
+	vmscape_apply_mitigation();
+}
+
 #ifdef CONFIG_SYSFS
 
 #define L1TF_DEFAULT_MSG "Mitigation: PTE Inversion"
-- 
2.51.0

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: [PATCH] x86/bugs: Get rid of the forward declarations

[Josh Poimboeuf]

On Fri, Nov 14, 2025 at 06:14:14PM +0100, Borislav Petkov wrote:
> On Wed, Nov 05, 2025 at 09:04:47PM +0100, Borislav Petkov wrote:
> > > We should put cpu_select_mitigations() at the bottom too, then all those
> > > existing forward declarations can go away.
> > 
> > That's a good idea. We should, if it doesn't get too hairy.
> 
> That was easier than I expected. Only build-tested.
> 
> ---
> From: "Borislav Petkov (AMD)" <bp@alien8.de>
> Date: Fri, 14 Nov 2025 18:10:04 +0100
> 
> Get rid of the forward declarations of the mitigation functions by
> moving their single caller below them.
> 
> No functional changes.
> 
> Suggested-by: Josh Poimboeuf <jpoimboe@kernel.org>
> Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>

Nice!

Acked-by: Josh Poimboeuf <jpoimboe@kernel.org>

-- 
Josh

Re: [PATCH] x86/bugs: Get rid of the forward declarations

[Borislav Petkov]

On Fri, Nov 14, 2025 at 11:19:52AM -0800, Josh Poimboeuf wrote:
> Nice!
> 
> Acked-by: Josh Poimboeuf <jpoimboe@kernel.org>

Thanks.

Lemme put it through the motions and queue it - that's a nice cleanup.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: [PATCH] x86/bugs: Get rid of the forward declarations

[Pawan Gupta]

On Fri, Nov 14, 2025 at 06:14:14PM +0100, Borislav Petkov wrote:
> On Wed, Nov 05, 2025 at 09:04:47PM +0100, Borislav Petkov wrote:
> > > We should put cpu_select_mitigations() at the bottom too, then all those
> > > existing forward declarations can go away.
> > 
> > That's a good idea. We should, if it doesn't get too hairy.
> 
> That was easier than I expected. Only build-tested.
> 
> ---
> From: "Borislav Petkov (AMD)" <bp@alien8.de>
> Date: Fri, 14 Nov 2025 18:10:04 +0100
> 
> Get rid of the forward declarations of the mitigation functions by
> moving their single caller below them.
> 
> No functional changes.
> 
> Suggested-by: Josh Poimboeuf <jpoimboe@kernel.org>
> Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>

Indeed a nice cleanup.

Reviewed-by: Pawan Gupta <pawan.kumar.gupta@linux.intel.com>

Re: [RFC PATCH 07/56] x86/bugs: Reset spectre_v2_user mitigations

[Borislav Petkov]

On Thu, Oct 16, 2025 at 04:13:25PM +0000, Brendan Jackman wrote:
> This seems a bit over-complicated though, in practice it seems pretty
> likely that just hitting people with the unexpected errors is sort of
> OK? At least for the usecase I'm seeing everything through the lens of,
> I think it would be. 

Yah, this topic does get my brain in a twist...

But I think I know what the right thing to do here should be:

If the admin goes and dynamically changes mitigation settings, no matter what
processes have selected previously, their settings will be changed once they
go through cond_mitigation().

If what userspace has selected previously and if that selection is not
compatible with what the dynamic, system-wide mitigation setting got changed
to, then the per-process setting will get changed.

Because the admin wanted it so and the admin can do anything. It is basically
the same as any other OS resource change - userspace needs to handle it and
adjust to it.

If the admin decides that the system wants to force mitigations on, then all
processes will have to comply. It is what it is.

It'll need a proper audit of all the TIF_SPEC_ and other flags and we would
have to even override them and issue warnings why we do so.

But I'm thinking: if the admin changes system-wide settings, the only right
thing for tasks is to comply.

I'd say.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

RE: [RFC PATCH 07/56] x86/bugs: Reset spectre_v2_user mitigations

[Kaplan, David]

[AMD Official Use Only - AMD Internal Distribution Only]

> -----Original Message-----
> From: Borislav Petkov <bp@alien8.de>
> Sent: Wednesday, November 26, 2025 5:23 AM
> To: Brendan Jackman <jackmanb@google.com>
> Cc: Kaplan, David <David.Kaplan@amd.com>; Thomas Gleixner
> <tglx@linutronix.de>; Peter Zijlstra <peterz@infradead.org>; Josh Poimboeuf
> <jpoimboe@kernel.org>; Pawan Gupta
> <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar <mingo@redhat.com>;
> Dave Hansen <dave.hansen@linux.intel.com>; x86@kernel.org; H . Peter
> Anvin <hpa@zytor.com>; Alexander Graf <graf@amazon.com>; Boris
> Ostrovsky <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
> Subject: Re: [RFC PATCH 07/56] x86/bugs: Reset spectre_v2_user mitigations
>
> Caution: This message originated from an External Source. Use proper
> caution when opening attachments, clicking links, or responding.
>
>
> On Thu, Oct 16, 2025 at 04:13:25PM +0000, Brendan Jackman wrote:
> > This seems a bit over-complicated though, in practice it seems pretty
> > likely that just hitting people with the unexpected errors is sort of
> > OK? At least for the usecase I'm seeing everything through the lens of,
> > I think it would be.
>
> Yah, this topic does get my brain in a twist...
>
> But I think I know what the right thing to do here should be:
>
> If the admin goes and dynamically changes mitigation settings, no matter
> what
> processes have selected previously, their settings will be changed once they
> go through cond_mitigation().
>
> If what userspace has selected previously and if that selection is not
> compatible with what the dynamic, system-wide mitigation setting got
> changed
> to, then the per-process setting will get changed.
>
> Because the admin wanted it so and the admin can do anything. It is basically
> the same as any other OS resource change - userspace needs to handle it and
> adjust to it.
>
> If the admin decides that the system wants to force mitigations on, then all
> processes will have to comply. It is what it is.
>
> It'll need a proper audit of all the TIF_SPEC_ and other flags and we would
> have to even override them and issue warnings why we do so.
>
> But I'm thinking: if the admin changes system-wide settings, the only right
> thing for tasks is to comply.
>
> I'd say.
>

What does it mean for a 'task to comply'?

I think it's one thing for the task to ask for a mitigation and then the admin goes and overrides it with a system-wide setting.  That doesn't worry me too much...the admin should be able to do that.  If it's just a mater of enabling/disabling some mitigation feature, that doesn't affect the functionality of the process and it can be rather transparently done.

But this particular interface is tricky because it returns EPERM in some cases, like when the task asks for something but the admin doesn't allow it.  In theory you could have processes that don't expect to see such an error in the middle of execution...that is, they might try to configure settings initially, find that succeeds, and then later modify them and not expect them to fail.  Brendan is saying maybe that's ok, are you aligned with that?

If so, I don't think we really have to do anything special then...the behavior of this prctl() may change due to dynamic mitigation changes and userspace might see unexpected EPERM errors if that happens.

--David Kaplan

Re: [RFC PATCH 07/56] x86/bugs: Reset spectre_v2_user mitigations

[Borislav Petkov]

On Mon, Dec 01, 2025 at 04:53:58PM +0000, Kaplan, David wrote:
> What does it mean for a 'task to comply'?

If the task has selected a mitigation which is not compatible with the setting
the admin selected later, that task's mitigation settings will be changed when
it goes through __switch_to.

> I think it's one thing for the task to ask for a mitigation and then the
> admin goes and overrides it with a system-wide setting.  That doesn't worry
> me too much...the admin should be able to do that. 

Ack.

> But this particular interface is tricky because it returns EPERM in some
> cases, like when the task asks for something but the admin doesn't allow it.
> In theory you could have processes that don't expect to see such an error in
> the middle of execution...that is, they might try to configure settings
> initially, find that succeeds, and then later modify them and not expect
> them to fail.  Brendan is saying maybe that's ok, are you aligned with that?

No, not in the middle of execution - their mitigation setting will get changed
when they get scheduled. Example:

1. task sets PR_SPEC_DISABLE and enables indirect branches for it
2. admin comes in and disables indirect branches system-wide - i.e.,
   SPECTRE_V2_USER_STRICT.
3. task comes through schedule() and it gets PR_SPEC_ENABLE forced on it

Makes sense?

> If so, I don't think we really have to do anything special then...the
> behavior of this prctl() may change due to dynamic mitigation changes and
> userspace might see unexpected EPERM errors if that happens.

I'm thinking the task will run the prctl() when it starts. Then, after the
system-wide setting has been changed, the above will happen.

A *new* task will then see EPERM and should be able to handle it.

Or am I missing a case...?

Thx.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

RE: [RFC PATCH 07/56] x86/bugs: Reset spectre_v2_user mitigations

[Kaplan, David]

[AMD Official Use Only - AMD Internal Distribution Only]

> -----Original Message-----
> From: Borislav Petkov <bp@alien8.de>
> Sent: Wednesday, December 3, 2025 6:31 AM
> To: Kaplan, David <David.Kaplan@amd.com>
> Cc: Brendan Jackman <jackmanb@google.com>; Thomas Gleixner
> <tglx@linutronix.de>; Peter Zijlstra <peterz@infradead.org>; Josh Poimboeuf
> <jpoimboe@kernel.org>; Pawan Gupta
> <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar <mingo@redhat.com>;
> Dave Hansen <dave.hansen@linux.intel.com>; x86@kernel.org; H . Peter
> Anvin <hpa@zytor.com>; Alexander Graf <graf@amazon.com>; Boris
> Ostrovsky <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
> Subject: Re: [RFC PATCH 07/56] x86/bugs: Reset spectre_v2_user mitigations
>
> Caution: This message originated from an External Source. Use proper
> caution when opening attachments, clicking links, or responding.
>
>
> On Mon, Dec 01, 2025 at 04:53:58PM +0000, Kaplan, David wrote:
> > What does it mean for a 'task to comply'?
>
> If the task has selected a mitigation which is not compatible with the setting
> the admin selected later, that task's mitigation settings will be changed when
> it goes through __switch_to.
>
> > I think it's one thing for the task to ask for a mitigation and then the
> > admin goes and overrides it with a system-wide setting.  That doesn't worry
> > me too much...the admin should be able to do that.
>
> Ack.
>
> > But this particular interface is tricky because it returns EPERM in some
> > cases, like when the task asks for something but the admin doesn't allow it.
> > In theory you could have processes that don't expect to see such an error in
> > the middle of execution...that is, they might try to configure settings
> > initially, find that succeeds, and then later modify them and not expect
> > them to fail.  Brendan is saying maybe that's ok, are you aligned with that?
>
> No, not in the middle of execution - their mitigation setting will get changed
> when they get scheduled. Example:
>
> 1. task sets PR_SPEC_DISABLE and enables indirect branches for it
> 2. admin comes in and disables indirect branches system-wide - i.e.,
>    SPECTRE_V2_USER_STRICT.
> 3. task comes through schedule() and it gets PR_SPEC_ENABLE forced on it
>
> Makes sense?
>
> > If so, I don't think we really have to do anything special then...the
> > behavior of this prctl() may change due to dynamic mitigation changes and
> > userspace might see unexpected EPERM errors if that happens.
>
> I'm thinking the task will run the prctl() when it starts. Then, after the
> system-wide setting has been changed, the above will happen.
>
> A *new* task will then see EPERM and should be able to handle it.
>
> Or am I missing a case...?
>

We don't know how tasks are using this prctl().  Maybe the task only sets PR_SPEC_DISABLE around one specific function call.

What if a program starts up and queries the kernel and gets PR_SPEC_PRCTL so it thinks it can control things.  And then it calls PR_SPEC_DISABLE/PR_SPEC_ENABLE around one particular sensitive function.

And then at some point, it starts getting -EPERM...

It would be cleaner if userspace never saw errors, but the mitigation were just silently applied/not-applied.

--David Kaplan

Re: [RFC PATCH 07/56] x86/bugs: Reset spectre_v2_user mitigations

[Borislav Petkov]

On Wed, Dec 03, 2025 at 05:02:24PM +0000, Kaplan, David wrote:
> We don't know how tasks are using this prctl().  Maybe the task only sets
> PR_SPEC_DISABLE around one specific function call.
> 
> What if a program starts up and queries the kernel and gets PR_SPEC_PRCTL so
> it thinks it can control things.  And then it calls
> PR_SPEC_DISABLE/PR_SPEC_ENABLE around one particular sensitive function.
> 
> And then at some point, it starts getting -EPERM...

Well, we can't have the cake and eat it too - at some point the admin will
override the setting the task did. There's no other way.

> It would be cleaner if userspace never saw errors, but the mitigation were
> just silently applied/not-applied.

As in: if dynamic mitigations are enabled, kernel stops returning EPERM but
simply overrides the mitigation setting and issues a pr_warn_once() that
PR_SPEC_PRCTL doesn't take effect anymore due to system-wide override?

Works for me...

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

RE: [RFC PATCH 07/56] x86/bugs: Reset spectre_v2_user mitigations

[Kaplan, David]

[AMD Official Use Only - AMD Internal Distribution Only]

> -----Original Message-----
> From: Borislav Petkov <bp@alien8.de>
> Sent: Wednesday, December 3, 2025 11:36 AM
> To: Kaplan, David <David.Kaplan@amd.com>
> Cc: Brendan Jackman <jackmanb@google.com>; Thomas Gleixner
> <tglx@linutronix.de>; Peter Zijlstra <peterz@infradead.org>; Josh Poimboeuf
> <jpoimboe@kernel.org>; Pawan Gupta
> <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar <mingo@redhat.com>;
> Dave Hansen <dave.hansen@linux.intel.com>; x86@kernel.org; H . Peter
> Anvin <hpa@zytor.com>; Alexander Graf <graf@amazon.com>; Boris
> Ostrovsky <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
> Subject: Re: [RFC PATCH 07/56] x86/bugs: Reset spectre_v2_user mitigations
>
> Caution: This message originated from an External Source. Use proper
> caution when opening attachments, clicking links, or responding.
>
>
> On Wed, Dec 03, 2025 at 05:02:24PM +0000, Kaplan, David wrote:
> > We don't know how tasks are using this prctl().  Maybe the task only sets
> > PR_SPEC_DISABLE around one specific function call.
> >
> > What if a program starts up and queries the kernel and gets PR_SPEC_PRCTL
> so
> > it thinks it can control things.  And then it calls
> > PR_SPEC_DISABLE/PR_SPEC_ENABLE around one particular sensitive
> function.
> >
> > And then at some point, it starts getting -EPERM...
>
> Well, we can't have the cake and eat it too - at some point the admin will
> override the setting the task did. There's no other way.
>
> > It would be cleaner if userspace never saw errors, but the mitigation were
> > just silently applied/not-applied.
>
> As in: if dynamic mitigations are enabled, kernel stops returning EPERM but
> simply overrides the mitigation setting and issues a pr_warn_once() that
> PR_SPEC_PRCTL doesn't take effect anymore due to system-wide override?
>
> Works for me...
>

Yeah, I think that's worth considering.  I think for the get functions (e.g. ib_prctl_get()) they can return whatever the current mitigation status is.  But for the set functions (e.g. ib_prctl_set()) would stop returning EPERM due to system-wide mitigation settings.

In other words, maybe something like this?  (And similar for the other ones like ssb_prctl_seg)

diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
index 0f0e688c1fec..8b83068d0372 100644
--- a/arch/x86/kernel/cpu/bugs.c
+++ b/arch/x86/kernel/cpu/bugs.c
@@ -2859,6 +2859,7 @@ static bool is_spec_ib_user_controlled(void)
                spectre_v2_user_stibp == SPECTRE_V2_USER_SECCOMP;
 }

+#define PR_SPEC_MSG "PR_SET_SPECULATION_CTRL ineffective due to system-wide mitigation settings.\n"
 static int ib_prctl_set(struct task_struct *task, unsigned long ctrl)
 {
        switch (ctrl) {
@@ -2882,10 +2883,22 @@ static int ib_prctl_set(struct task_struct *task, unsigned long ctrl)
                 * spectre_v2_user_ibpb == SPECTRE_V2_USER_SECCOMP and
                 * spectre_v2_user_stibp == SPECTRE_V2_USER_STRICT_PREFERRED.
                 */
-               if (!is_spec_ib_user_controlled() ||
-                   task_spec_ib_force_disable(task))
+               if (task_spec_ib_force_disable(task))
                        return -EPERM;

+               /*
+                * A system-wide mitigation setting change could render a task
+                * unable to change their mitigation options.  Don't return
+                * EPERM to avoid confusion in this case.
+                */
+               if (!is_spec_ib_user_controlled()) {
+                       if (!IS_ENABLED(CONFIG_DYNAMIC_MITIGATIONS))
+                               return -EPERM;
+
+                       pr_warn_once(PR_SPEC_MSG);
+                       return 0;
+               }
+
                task_clear_spec_ib_disable(task);
                task_update_spec_tif(task);
                break;
@@ -2896,11 +2909,16 @@ static int ib_prctl_set(struct task_struct *task, unsigned long ctrl)
                 * mitigation is force disabled.
                 */
                if (spectre_v2_user_ibpb == SPECTRE_V2_USER_NONE &&
-                   spectre_v2_user_stibp == SPECTRE_V2_USER_NONE)
+                   spectre_v2_user_stibp == SPECTRE_V2_USER_NONE &&
+                   !IS_ENABLED(CONFIG_DYNAMIC_MITIGATIONS))
                        return -EPERM;

-               if (!is_spec_ib_user_controlled())
+               if (!is_spec_ib_user_controlled()) {
+                       if (IS_ENABLED(CONFIG_DYNAMIC_MITIGATIONS))
+                               pr_warn_once(PR_SPEC_MSG);
+
                        return 0;
+               }

                task_set_spec_ib_disable(task);
                if (ctrl == PR_SPEC_FORCE_DISABLE)

Re: [RFC PATCH 07/56] x86/bugs: Reset spectre_v2_user mitigations

[Borislav Petkov]

On Wed, Dec 03, 2025 at 08:14:08PM +0000, Kaplan, David wrote:
> Yeah, I think that's worth considering.  I think for the get functions (e.g.
> ib_prctl_get()) they can return whatever the current mitigation status is.
> But for the set functions (e.g. ib_prctl_set()) would stop returning EPERM
> due to system-wide mitigation settings.
> 
> In other words, maybe something like this?  (And similar for the other ones
> like ssb_prctl_seg)

Looks about right to me.

Thx.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: [RFC PATCH 28/56] stop_machine: Add stop_machine_nmi()

[Chang S. Bae]

On 10/13/2025 7:34 AM, David Kaplan wrote:
>   
> +/**
> + * stop_machine_nmi: freeze the machine and run this function in NMI context
> + * @fn: the function to run
> + * @data: the data ptr for the @fn()
> + * @cpus: the cpus to run the @fn() on (NULL = any online cpu)
> + *
> + * Like stop_machine() but runs the function in NMI context to avoid any risk of
> + * interruption due to NMIs.
> + *
> + * Protects against CPU hotplug.
> + */
> +int stop_machine_nmi(cpu_stop_fn_t fn, void *data, const struct cpumask *cpus);
> +
> +/**
> + * stop_machine_cpuslocked_nmi: freeze and run this function in NMI context
> + * @fn: the function to run
> + * @data: the data ptr for the @fn()
> + * @cpus: the cpus to run the @fn() on (NULL = any online cpu)
> + *
> + * Same as above. Must be called from within a cpus_read_lock() protected
> + * region. Avoids nested calls to cpus_read_lock().
> + */
> +int stop_machine_cpuslocked_nmi(cpu_stop_fn_t fn, void *data, const struct cpumask *cpus);

<snip>

> +int stop_machine_cpuslocked_nmi(cpu_stop_fn_t fn, void *data,
> +				const struct cpumask *cpus)
> +{
> +	return __stop_machine_cpuslocked(fn, data, cpus, true);
> +}
> +

It looks like this is readily missing the static key switching which is 
handled below. I think the body could be something like:
   ...
   static_branch_enable_cpuslocked(&stop_machine_nmi_handler_enable);
   ret = __stop_machine_cpuslocked(fn, data, cpus, true);
   static_branch_disable_cpuslocked(&stop_machine_nmi_handler_enable);
   ...

> +int stop_machine_nmi(cpu_stop_fn_t fn, void *data, const struct cpumask *cpus)
> +{
> +	int ret;
> +
> +	cpus_read_lock();
> +	static_branch_enable_cpuslocked(&stop_machine_nmi_handler_enable);
> +	ret = stop_machine_cpuslocked_nmi(fn, data, cpus);
> +	static_branch_disable_cpuslocked(&stop_machine_nmi_handler_enable);
> +	cpus_read_unlock();
> +	return ret;
> +}

With that, here __stop_machine_cpuslocked() can be invoked instead.

Thanks,
Chang

RE: [RFC PATCH 28/56] stop_machine: Add stop_machine_nmi()

[Kaplan, David]

[AMD Official Use Only - AMD Internal Distribution Only]

> -----Original Message-----
> From: Chang S. Bae <chang.seok.bae@intel.com>
> Sent: Friday, January 9, 2026 4:16 PM
> To: Kaplan, David <David.Kaplan@amd.com>; Thomas Gleixner
> <tglx@linutronix.de>; Borislav Petkov <bp@alien8.de>; Peter Zijlstra
> <peterz@infradead.org>; Josh Poimboeuf <jpoimboe@kernel.org>; Pawan
> Gupta <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar
> <mingo@redhat.com>; Dave Hansen <dave.hansen@linux.intel.com>;
> x86@kernel.org; H . Peter Anvin <hpa@zytor.com>
> Cc: Alexander Graf <graf@amazon.com>; Boris Ostrovsky
> <boris.ostrovsky@oracle.com>; linux-kernel@vger.kernel.org
> Subject: Re: [RFC PATCH 28/56] stop_machine: Add stop_machine_nmi()
>
> Caution: This message originated from an External Source. Use proper
> caution when opening attachments, clicking links, or responding.
>
>
> On 10/13/2025 7:34 AM, David Kaplan wrote:
> >
> > +/**
> > + * stop_machine_nmi: freeze the machine and run this function in NMI
> context
> > + * @fn: the function to run
> > + * @data: the data ptr for the @fn()
> > + * @cpus: the cpus to run the @fn() on (NULL = any online cpu)
> > + *
> > + * Like stop_machine() but runs the function in NMI context to avoid any
> risk of
> > + * interruption due to NMIs.
> > + *
> > + * Protects against CPU hotplug.
> > + */
> > +int stop_machine_nmi(cpu_stop_fn_t fn, void *data, const struct cpumask
> *cpus);
> > +
> > +/**
> > + * stop_machine_cpuslocked_nmi: freeze and run this function in NMI
> context
> > + * @fn: the function to run
> > + * @data: the data ptr for the @fn()
> > + * @cpus: the cpus to run the @fn() on (NULL = any online cpu)
> > + *
> > + * Same as above. Must be called from within a cpus_read_lock()
> protected
> > + * region. Avoids nested calls to cpus_read_lock().
> > + */
> > +int stop_machine_cpuslocked_nmi(cpu_stop_fn_t fn, void *data, const
> struct cpumask *cpus);
>
> <snip>
>
> > +int stop_machine_cpuslocked_nmi(cpu_stop_fn_t fn, void *data,
> > +                             const struct cpumask *cpus)
> > +{
> > +     return __stop_machine_cpuslocked(fn, data, cpus, true);
> > +}
> > +
>
> It looks like this is readily missing the static key switching which is
> handled below. I think the body could be something like:
>    ...
>    static_branch_enable_cpuslocked(&stop_machine_nmi_handler_enable);
>    ret = __stop_machine_cpuslocked(fn, data, cpus, true);
>    static_branch_disable_cpuslocked(&stop_machine_nmi_handler_enable);
>    ...
>
> > +int stop_machine_nmi(cpu_stop_fn_t fn, void *data, const struct cpumask
> *cpus)
> > +{
> > +     int ret;
> > +
> > +     cpus_read_lock();
> > +
> static_branch_enable_cpuslocked(&stop_machine_nmi_handler_enable);
> > +     ret = stop_machine_cpuslocked_nmi(fn, data, cpus);
> > +
> static_branch_disable_cpuslocked(&stop_machine_nmi_handler_enable);
> > +     cpus_read_unlock();
> > +     return ret;
> > +}
>
> With that, here __stop_machine_cpuslocked() can be invoked instead.
>

Ah, yeah I agree that doing the static key switching is better done in stop_machine_cpuslocked_nmi instead.  That's a good point.

Thanks --David Kaplan

Re: [RFC PATCH 08/56] x86/bugs: Reset SSB mitigations

[Borislav Petkov]

On Mon, Oct 13, 2025 at 09:33:56AM -0500, David Kaplan wrote:
> @@ -2916,6 +2937,8 @@ void x86_spec_ctrl_setup_ap(void)
>  
>  	if (ssb_mode == SPEC_STORE_BYPASS_DISABLE)
>  		x86_amd_ssb_disable();
> +	else
> +		x86_amd_ssb_enable();

I'm assuming we need this for the case when we do alternatives-patch and then
some CPUs are coming online later so they have to get SSBD properly set
there...

In any case, lemme suggest a simplification (I hope I've gotten the booleans
right):

---

diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
index 6b25192560f0..e78e010b4752 100644
--- a/arch/x86/kernel/cpu/bugs.c
+++ b/arch/x86/kernel/cpu/bugs.c
@@ -221,24 +221,20 @@ x86_virt_spec_ctrl(u64 guest_virt_spec_ctrl, bool setguest)
 }
 EXPORT_SYMBOL_FOR_KVM(x86_virt_spec_ctrl);
 
-static void x86_amd_ssb_disable(void)
+static void x86_amd_ssb_toggle(bool disable)
 {
-	u64 msrval = x86_amd_ls_cfg_base | x86_amd_ls_cfg_ssbd_mask;
+	u64 msrval  = x86_amd_ls_cfg_base;
+	u64 msrvirt = 0;
 
-	if (boot_cpu_has(X86_FEATURE_VIRT_SSBD))
-		wrmsrq(MSR_AMD64_VIRT_SPEC_CTRL, SPEC_CTRL_SSBD);
-	else if (boot_cpu_has(X86_FEATURE_LS_CFG_SSBD))
-		wrmsrq(MSR_AMD64_LS_CFG, msrval);
-}
-
-static void x86_amd_ssb_enable(void)
-{
-	u64 msrval = x86_amd_ls_cfg_base;
+	if (disable) {
+		msrval  |= x86_amd_ls_cfg_ssbd_mask;
+		msrvirt  = SPEC_CTRL_SSBD;
+	}
 
 	if (boot_cpu_has(X86_FEATURE_VIRT_SSBD))
-		wrmsrl(MSR_AMD64_VIRT_SPEC_CTRL, 0);
+		wrmsrq(MSR_AMD64_VIRT_SPEC_CTRL, msrvirt);
 	else if (boot_cpu_has(X86_FEATURE_LS_CFG_SSBD))
-		wrmsrl(MSR_AMD64_LS_CFG, msrval);
+		wrmsrq(MSR_AMD64_LS_CFG, msrval);
 }
 
 #undef pr_fmt
@@ -2524,7 +2520,7 @@ static void __init ssb_apply_mitigation(void)
 		 */
 		if (!static_cpu_has(X86_FEATURE_SPEC_CTRL_SSBD) &&
 		    !static_cpu_has(X86_FEATURE_AMD_SSBD)) {
-			x86_amd_ssb_disable();
+			x86_amd_ssb_toggle(true);
 		} else {
 			x86_spec_ctrl_base |= SPEC_CTRL_SSBD;
 			update_spec_ctrl(x86_spec_ctrl_base);
@@ -2785,10 +2781,7 @@ void x86_spec_ctrl_setup_ap(void)
 	if (boot_cpu_has(X86_FEATURE_MSR_SPEC_CTRL))
 		update_spec_ctrl(x86_spec_ctrl_base);
 
-	if (ssb_mode == SPEC_STORE_BYPASS_DISABLE)
-		x86_amd_ssb_disable();
-	else
-		x86_amd_ssb_enable();
+	x86_amd_ssb_toggle(ssb_mode == SPEC_STORE_BYPASS_DISABLE);
 }
 
 bool itlb_multihit_kvm_mitigation;

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: [RFC PATCH 11/56] x86/bugs: Reset MMIO mitigations

[Borislav Petkov]

On Mon, Oct 13, 2025 at 09:33:59AM -0500, David Kaplan wrote:
> Add function to reset MMIO mitigations back to their boot-time defaults.
> 
> Signed-off-by: David Kaplan <david.kaplan@amd.com>
> ---
>  arch/x86/kernel/cpu/bugs.c | 11 +++++++++++
>  1 file changed, 11 insertions(+)
> 
> diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
> index 5668a8b8821b..9139c8187913 100644
> --- a/arch/x86/kernel/cpu/bugs.c
> +++ b/arch/x86/kernel/cpu/bugs.c
> @@ -787,6 +787,16 @@ static void __init mmio_apply_mitigation(void)
>  		cpu_smt_disable(false);
>  }
>  
> +#ifdef CONFIG_DYNAMIC_MITIGATIONS
> +static void mmio_reset_mitigation(void)
> +{
> +	static_branch_disable(&cpu_buf_vm_clear);

This needs to be

	setup_clear_cpu_cap(X86_FEATURE_CLEAR_CPU_BUF_VM_MMIO);

I believe, after:

f6106d41ec84 ("x86/bugs: Use an x86 feature to track the MMIO Stale Data mitigation")

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

RE: [RFC PATCH 11/56] x86/bugs: Reset MMIO mitigations

[Kaplan, David]

[AMD Official Use Only - AMD Internal Distribution Only]

> -----Original Message-----
> From: Borislav Petkov <bp@alien8.de>
> Sent: Monday, January 26, 2026 7:06 AM
> To: Kaplan, David <David.Kaplan@amd.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>; Peter Zijlstra
> <peterz@infradead.org>; Josh Poimboeuf <jpoimboe@kernel.org>; Pawan
> Gupta <pawan.kumar.gupta@linux.intel.com>; Ingo Molnar
> <mingo@redhat.com>; Dave Hansen <dave.hansen@linux.intel.com>;
> x86@kernel.org; H . Peter Anvin <hpa@zytor.com>; Alexander Graf
> <graf@amazon.com>; Boris Ostrovsky <boris.ostrovsky@oracle.com>; linux-
> kernel@vger.kernel.org
> Subject: Re: [RFC PATCH 11/56] x86/bugs: Reset MMIO mitigations
>
> Caution: This message originated from an External Source. Use proper
> caution when opening attachments, clicking links, or responding.
>
>
> On Mon, Oct 13, 2025 at 09:33:59AM -0500, David Kaplan wrote:
> > Add function to reset MMIO mitigations back to their boot-time defaults.
> >
> > Signed-off-by: David Kaplan <david.kaplan@amd.com>
> > ---
> >  arch/x86/kernel/cpu/bugs.c | 11 +++++++++++
> >  1 file changed, 11 insertions(+)
> >
> > diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
> > index 5668a8b8821b..9139c8187913 100644
> > --- a/arch/x86/kernel/cpu/bugs.c
> > +++ b/arch/x86/kernel/cpu/bugs.c
> > @@ -787,6 +787,16 @@ static void __init mmio_apply_mitigation(void)
> >               cpu_smt_disable(false);
> >  }
> >
> > +#ifdef CONFIG_DYNAMIC_MITIGATIONS
> > +static void mmio_reset_mitigation(void)
> > +{
> > +     static_branch_disable(&cpu_buf_vm_clear);
>
> This needs to be
>
>         setup_clear_cpu_cap(X86_FEATURE_CLEAR_CPU_BUF_VM_MMIO);
>
> I believe, after:
>
> f6106d41ec84 ("x86/bugs: Use an x86 feature to track the MMIO Stale Data
> mitigation")
>

Ack.  Actually It looks like I need to clear X86_FEATURE_CLEAR_CPU_BUF, X86_FEATURE_CLEAR_CPU_VM, and the one you noted.

(Although technically those first two do get cleared by other reset functions).

Thanks
--David Kaplan