[PATCH v14 0/5] Add RMPOPT support.

[PATCH v14 0/5] Add RMPOPT support.

[Ashish Kalra]

From: Ashish Kalra <ashish.kalra@amd.com>

In the SEV-SNP architecture, hypervisor and non-SNP guests are subject
to RMP checks on writes to provide integrity of SEV-SNP guest memory.

The RMPOPT architecture enables optimizations whereby the RMP checks
can be skipped if 1GB regions of memory are known to not contain any
SNP guest memory.

RMPOPT is a new instruction designed to minimize the performance
overhead of RMP checks for the hypervisor and non-SNP guests.

RMPOPT instruction currently supports two functions. In case of the
verify and report status function the CPU will read the RMP contents,
verify the entire 1GB region starting at the provided SPA is HV-owned.
For the entire 1GB region it checks that all RMP entries in this region
are HV-owned (i.e, not in assigned state) and then accordingly updates
the RMPOPT table to indicate if optimization has been enabled and
provide indication to software if the optimization was successful.

In case of report status function, the CPU returns the optimization
status for the 1GB region.

The RMPOPT table is managed by a combination of software and hardware.
Software uses the RMPOPT instruction to set bits in the table,
indicating that regions of memory are entirely HV-owned.  Hardware
automatically clears bits in the RMPOPT table when RMP contents are
changed during RMPUPDATE instruction.

For more information on the RMPOPT instruction, see the AMD64 RMPOPT
technical documentation.

As SNP is enabled by default the hypervisor and non-SNP guests are
subject to RMP write checks to provide integrity of SNP guest memory.

This patch-series adds support to enable RMP optimizations for up to
2TB of system RAM across the system and allow RMPUPDATE to disable
those optimizations as SNP guests are launched.

Support for RAM larger than 2 TB will be added in follow-on series.

This series also adds support to disable CPU hotplug while SNP is
active, as the SEV firmware enumerates CPUs at SNP initialization and is
not aware of the OS bringing CPUs online or offline afterwards.  This
also keeps the set of CPUs stable for the asynchronous RMPOPT scan, so
the per-core RMPOPT_BASE MSRs programmed during setup remain valid.

This series also introduces support to re-enable RMP optimizations
during SNP guest termination, after guest pages have been converted
back to shared.

RMP optimizations are performed asynchronously by queuing work on a
dedicated workqueue after a 10 second delay.

Delaying work allows batching of multiple SNP guest terminations.

Once 1GB hugetlb guest_memfd support is merged, support for
re-enabling RMPOPT optimizations during 1GB page cleanup will be added
in follow-on series.

v14:
- 3/5 (Initialize RMPOPT MSRs): remove rmpopt_disable() and the RMPOPT_BASE
  teardown from this patch.  RMPOPT is now set up once and not torn down, so the
  RMPOPT_BASE MSRs are left in place on shutdown; the (now minimal) disable path
  moves to 4/5.
- 4/5 (async RMPOPT): rework the setup/teardown -- initialize once and do not
  tear it down.  rmpopt_disable() now only cancels the pending re-optimization
  pass.  Use a dedicated per-CPU workqueue (WQ_PERCPU) and drop the
  migrate_disable()/migrate_enable() around the local warm-up scan.  Add a
  binutils-version comment above the RMPOPT .byte encoding and trim redundant
  comments and the 2 TB pr_info().  Reword the subject to "Perform RMP
  optimizations asynchronously".
- 5/5 (Re-enable on guest shutdown): move the call-site comment onto the
  snp_rmpopt_all_physmem() definition and simplify the RMPOPT_WORK_TIMEOUT
  comment.
- Correct the Suggested-by/Reviewed-by tags across the series.

  Review feedback from Borislav Petkov.

v13:
- 3/5 (Initialize RMPOPT MSRs): use cpu_primary_thread_mask directly instead of
  building a local rmpopt_cpumask -- all primary threads are online while SNP is
  active, so the two are equivalent.  Rename snp_cleanup_rmpopt() to
  rmpopt_disable().
- 4/5 (async RMPOPT): drop the leader/follower bookkeeping.  Warm the RMP scan
  cache once, then fan the RMPOPT out to cpu_primary_thread_mask; re-running
  RMPOPT on the warm-up CPU is a cache hit, so the separate follower cpumask and
  the cpumask_andnot() are no longer needed.  Rename rmpopt_work_handler() to
  do_rmpopt_work() and drop the now-unused this_cpu.
- Add Reviewed-by: Tom Lendacky <thomas.lendacky@amd.com> to the series.

  Review feedback from Borislav Petkov.

v12:
- Merged the former "Add interface to re-enable RMP optimizations" and
  "KVM: SEV: Perform RMP optimizations on SNP guest shutdown" patches into a
  single patch (now 5/5), since the interface has no user without the KVM
  caller.  The series is now 5 patches.
- 2/5 (Disable CPU hotplug): drop a stray reference to later patches from the
  commit message and trim the snp_prepare() comment.  Move cpu_hotplug_enable()
  to the end of snp_shutdown(), after clear_rmp() and the mfd_reconfigure() IPI,
  so no all-CPU operation runs after hotplug is re-enabled.
- 3/5 (Initialize RMPOPT MSRs): restructure snp_probe_rmptable_info() so the
  contiguous probe (which clears X86_FEATURE_RMPOPT) is the common fall-through,
  also covering X86_FEATURE_SEGMENTED_RMP advertised but disabled by firmware.
  Move snp_setup_rmpopt() to the end of __sev_snp_init_locked().  Give
  snp_cleanup_rmpopt() its own short comment in snp_shutdown().
- 4/5 (async RMPOPT): document the optimization trigger points in the commit
  message.  Rename enum rmpopt_function to rmpopt_op_type.  Merge __rmpopt() into
  rmpopt() and replace rmpopt_smp() with a new rmpopt_scan_range() that loops as
  the on_each_cpu() callback, so the follower scan issues one IPI per core rather
  than one per 1GB.  Use u64 for physical addresses.  Allocate the follower
  cpumask once at setup.  Document rmpopt_wq as the setup sentinel.  Move the
  RMPOPT_WORK_TIMEOUT define to the guest-shutdown patch (its only user) as
  (10 * MSEC_PER_SEC).  Remove a spammy pr_info() and reword comments.
- 5/5 (Re-enable RMP optimizations on SNP guest shutdown): document why
  re-optimization is driven by guest teardown (the only event that returns
  guest memory to hypervisor ownership); reword the commit message
  (s/clear/disable/ for optimizations, drop "Conversely").

  Review feedback from Borislav Petkov.

v11:
- Reordered so "Disable CPU hotplug while SNP is active" (2/6) precedes
  "Initialize RMPOPT configuration MSRs" (3/6): the RMPOPT setup/cleanup code is
  then introduced with CPU hotplug already disabled and never takes
  cpus_read_lock().
- 1/6 (cpufeatures): drop the tools/arch/x86/include/asm/cpufeatures.h change and
  adopt the commit message as applied by Boris.
- 2/6 (Disable CPU hotplug): reword the commit message.  Drop the redundant
  cpus_read_lock()/cpus_read_unlock() in snp_prepare() in this same patch -- with
  hotplug disabled the online CPU mask is stable, so the read lock is not needed
  and hotplug handling has no hole when bisecting.
- 3/6 (Initialize RMPOPT MSRs): replace the rmpopt_capable bool with a small
  helper local to arch/x86/virt/svm/sev.c --
  cpu_feature_enabled(X86_FEATURE_RMPOPT) && cc_platform_has(CC_ATTR_HOST_SEV_SNP)
  -- clearing X86_FEATURE_RMPOPT for a contiguous (non-segmented) RMP in
  snp_probe_rmptable_info() (runs at BSP init, before alternatives).  Rename
  rmpopt_cleanup() to snp_cleanup_rmpopt() to match snp_setup_rmpopt().  Both are
  introduced without cpus_read_lock() (hotplug is already disabled by 2/6).
  Simplify the RMPOPT_BASE comments.  Drop the now-unused <asm/sev.h> include from
  core.c.
- 4/6 (async RMPOPT): the follower scan is likewise introduced without
  cpus_read_lock().  Drop the cond_resched() calls (nops on x86).  On
  re-initialization after a legacy SNP shutdown, re-queue the optimization pass
  instead of skipping it.  pr_warn() on cpumask allocation failure.

  Review feedback from Borislav Petkov and K Prateek Nayak.

v10:
- Rework the CPU-hotplug patch (3/6): disable CPU hotplug in
  snp_prepare(), before SnpEn is set, instead of late in
  __sev_snp_init_locked(), so no CPU can come online without SnpEn during
  SNP initialization (per upstream review).  Tie hotplug to SnpEn: it
  stays disabled while SnpEn is set -- including across a failed SNP_INIT
  and across the legacy SNP_SHUTDOWN_EX path -- and is re-enabled only
  once the firmware clears SnpEn on the x86_snp_shutdown path.  Drop the
  separate idempotent flag: snp_prepare() re-enables hotplug on its own
  early failure, and a kexec target that boots with SnpEn already set
  disables hotplug once in snp_rmptable_init().  Reword the commit log and
  comments accordingly.
- Emit a pr_warn() in rmpopt_work_handler() (4/6) when the follower
  cpumask allocation fails, instead of silently skipping the optimization
  pass.

  Sashiko AI upstream review identified several of the above issues.

v9:
- Rename rmpopt_configured to rmpopt_capable.
- Make rmpopt_cpumask a cpumask_var_t (allocated/freed at setup/cleanup)
  instead of a static cpumask_t.
- Drop the v8 WARN_ON_ONCE() on the RMPOPT_BASE writes; use a plain
  wrmsrq_on_cpu(), matching the SNP MSR-write convention in this file.
- Disable CPU hotplug with cpu_hotplug_disable()/cpu_hotplug_enable()
  (per tglx); re-enable only on the full x86_snp_shutdown path.
- Simplify rmpopt_work_handler() to a single leader-then-followers path:
  with CPU hotplug disabled while SNP is active and snp_prepare()
  requiring all CPUs online when RMPOPT_BASE is programmed, every core is
  always programmed, so the explicit-leader fallback is now unreachable.
  Drop it along with the v8 work_on_cpu()/rmpopt_leader_fn() helper.
- Drop the debugfs interface (was patch 7/7) and its report-only
  plumbing; observability will be revisited after this series is merged.
- Restrict snp_rmpopt_all_physmem()'s export to the kvm-amd module.
- Use scoped_guard(cpus_read_lock) for the per-CPU MSR and follower
  loops.

  Sashiko AI upstream review identified several of the above issues.

v8:
- Add a new patch to disable CPU hotplug while SNP is active, keeping
  the CPU set stable for the RMPOPT work handler.
- Drop the setup_clear_cpu_cap(X86_FEATURE_RMPOPT) calls; the
  rmpopt_configured bool is the runtime guard.
- WARN_ON_ONCE() on the RMPOPT_BASE MSR writes that previously ignored
  their return value.
- Simplify rmpopt_work_handler() by removing the explicit-leader
  fallback: with CPU hotplug disabled while SNP is active and
  snp_prepare() requiring all CPUs online when RMPOPT_BASE is programmed,
  every core is always programmed, so the running CPU can always be the
  leader.  This drops the smp_call_function_single() fallback (and with
  it the AB-BA deadlock and IRQ-latency concerns) and collapses the
  leader selection into a single leader-then-followers path.
- Use mod_delayed_work() in snp_rmpopt_all_physmem() so the batching
  delay tracks the last SNP guest termination.

  Sashiko AI code review identified several of the above issues.

v7:
- Sync tools/arch/x86/include/asm/cpufeatures.h to mirror the kernel
  header for X86_FEATURE_RMPOPT.
- Fix commit title to use X86_FEATURE_RMPOPT to match the code
  (was X86_FEATURE_AMD_RMPOPT).
- Add static bool rmpopt_configured, set only when segmented RMP setup
  succeeds in setup_rmptable().  Check rmpopt_configured alongside
  cpu_feature_enabled(X86_FEATURE_RMPOPT) in snp_setup_rmpopt() and
  snp_rmpopt_all_physmem(), because setup_clear_cpu_cap() is unreliable
  after alternatives are patched.  Add snp_clear_rmpopt_configured()
  called from amd_cc_platform_clear() when CC_ATTR_HOST_SEV_SNP is
  cleared.  Do not use __ro_after_init on rmpopt_configured since the
  writer snp_clear_rmpopt_configured() is not __init.
- Add cond_resched() to all three leader loops in rmpopt_work_handler()
  to prevent soft lockups on systems with up to 2TB of RAM.
- Add comment above __rmpopt() documenting the RMPOPT instruction
  encoding (F2 0F 01 FC) and register interface (RAX = system physical
  address input, RCX = operation type input, RFLAGS.CF = output).
  Note: RMPOPT does not modify RAX unlike PVALIDATE/RMPUPDATE, so
  the existing "a" (input-only) constraint is correct.

  Sashiko AI code review identified several of the above issues.

v6:
- Drop wrmsrq_on_cpus() helper; use for_each_cpu() with wrmsrq_on_cpu()
  instead, as RMPOPT_BASE MSR programming is not performance-critical.
- Rewrite rmpopt_work_handler() leader selection to use a local
  follower_mask copy instead of modifying the global rmpopt_cpumask.
  This eliminates the current_cpu_cleared tracking and the restore at
  the end, and removes the need for synchronization comments about
  transient cpumask inconsistency.
- Add three-way leader selection in rmpopt_work_handler():
  1. Current CPU is a primary thread in cpumask: run leader locally.
  2. Current CPU is a sibling thread whose primary is in cpumask:
     run leader locally (RMPOPT_BASE MSR is per-core), remove the
     primary from followers via cpumask_andnot(topology_sibling_cpumask).
  3. Current CPU's core has no RMPOPT_BASE MSR programmed: pick an
     explicit leader via cpumask_first() + smp_call_function_single()
     to avoid #UD, with cpus_read_lock() around the IPI loop.
- Add WARN_ON_ONCE guard for empty cpumask in the explicit leader
  fallback path, with migrate_enable() before goto out.
- Add .llseek = seq_lseek to rmpopt_table_fops for consistency with
  other seq_file-based debugfs files and to support tools like "less".
- Change debugfs file permissions from 0444 to 0400 to restrict access
  to root only.
- Add comment in rmpopt_table_seq_show() explaining why cpu_online_mask
  is safe: RMPOPT_BASE MSR is per-core and snp_prepare() ensures all
  CPUs are online when the MSR is programmed.

  Sashiko AI code review identified several of the above issues.

v5:
- Introduce rmpopt_cleanup() to tear down workqueue, debugfs, cpumask,
  and MSR state, called from snp_shutdown().
- Introduce rmpopt_wq_mutex to serialize snp_setup_rmpopt(),
  snp_rmpopt_all_physmem(), and rmpopt_cleanup().
- Introduce rmpopt_show_mutex to serialize debugfs reporting of
  rmpopt_report_cpumask.
- Move snp_rmpopt_all_physmem() call after SNP DECOMMISSION during
  guest shutdown.
- Use migrate_disable()/migrate_enable() for CPU pinning in the
  rmpopt_work_handler() leader loop to maintain CPU affinity without
  disabling preemption for the entire RMPOPT scan.
- Add cpus_read_lock()/cpus_read_unlock() around the follower
  on_each_cpu_mask() loop in rmpopt_work_handler().
- Guard snp_setup_rmpopt() against re-initialization when
  SNP_SHUTDOWN_EX with x86_snp_shutdown=0 skips rmpopt_cleanup()
  but clears snp_initialized, preventing workqueue and resource
  leaks on repeated init/shutdown cycles.
- Replace setup_clear_cpu_cap() with pr_err() on alloc_workqueue()
  failure in snp_setup_rmpopt(), as setup_clear_cpu_cap() cannot be
  used after alternatives are patched; callers check rmpopt_wq != NULL
  as the runtime guard instead.
- Add pr_info() when RMPOPT coverage is capped at 2TB.
- Add comments noting CPU hotplug is not supported with SNP enabled
  and only online primary threads are covered by rmpopt_cpumask.
- Add comment in setup_rmptable() noting Segmented RMP must be
  enabled to enable RMPOPT.
- Simplify cpumask setup loop to set if primary thread rather than
  skip if not primary.
- Improve grammar and clarity in snp_setup_rmpopt() comments.
- Added Reviewed-by's.

  Sashiko AI code review identified several of the above issues.

v4:
- Add new wrmsrq_on_cpus() helper to write same u64 value to a
  per-CPU MSR across a cpumask without per-cpu struct allocation
  overhead.
- Rename configure_and_enable_rmpopt() to snp_setup_rmpopt().
- Use wrmsrq_on_cpus() instead of wrmsrq_on_cpu() loop for
  programming RMPOPT_BASE MSRs.
- Add setup_clear_cpu_cap(X86_FEATURE_RMPOPT) if segmented RMP
  setup fails or workqueue allocation fails.
- Add X86_FEATURE_RMPOPT feature clear logic in amd_cc_platform_clear()
  for CC_ATTR_HOST_SEV_SNP.
- All of the above allow checking for only X86_FEATURE_RMPOPT for both
  RMPOPT setup/enable and RMP re-optimizations.
- Rename snp_perform_rmp_optimization() to snp_rmpopt_all_physmem().
- Split rmpopt() into rmpopt() and rmpopt_smp() for SMP callback use.
- Introduce separate rmpopt_report_cpumask for debugfs reporting,
  distinct from rmpopt_cpumask used for primary thread tracking.
- Remove snp_perform_rmp_optimization() call from __sev_snp_init_locked()
  and instead setup and enable RMPOPT after SNP is enabled and
  initialized.

v3:
- Drop all RMPOPT kthread support and introduce adding custom and
  dedicated workqueue to schedule delayed and asynchronous RMPOPT work.
- Drop the guest_memfd inode cleanup interface and add support to
  re-enable RMP optimizations during guest shutdown using the
  asynchronous and delayed workqueue interface.
- Introduce new __rmpopt() helper and rmpopt() and
  rmpopt_report_status() wrappers on top which use rax and rcx
  parameters to closely match RMPOPT specs.
- Use new optimized RMPOPT loop to issue RMPOPT instructions on all
  system RAM upto 2TB and all CPUs, by optimizing each range on one CPU
  first, then let other CPUs execute RMPOPT in parallel so they can skip
  most work as the range has already been optimized.
- Also add support for running the optimized RMPOPT loop only on
  one thread per core.
- Replace all PUD_SIZE references with SZ_1G to conform to 1GB regions
  as specified by RMPOPT specifications and not be dependent on PUD_SIZE
  which makes the RMPOPT patch-set independent of x86 page table sizes.
- Use wrmsrq_on_cpu() to program the RMPOPT_BASE MSR registers on
  all CPUs that removes all ugly casting to use on_each_cpu_mask().
- Fix inline commits and patch commit messages


v2:
- Drop all NUMA and Socket configuration and enablement support and
  enable RMPOPT support for up to 2TB of system RAM.
- Drop get_cpumask_of_primary_threads() and enable per-core RMPOPT
  base MSRs and issue RMPOPT instruction on all CPUs.
- Drop the configfs interface to manually re-enable RMP optimizations.
- Add new guest_memfd cleanup interface to automatically re-enable
  RMP optimizations during guest shutdown.
- Include references to the public RMPOPT documentation.
- Move debugfs directory for RMPOPT under architecuture specific
  parent directory.

Ashish Kalra (5):
  x86/cpufeatures: Add X86_FEATURE_RMPOPT feature flag
  x86/sev: Disable CPU hotplug while SNP is active
  x86/sev: Initialize RMPOPT configuration MSRs
  x86/sev: Perform RMP optimizations asynchronously
  x86/sev: Re-enable RMP optimizations on SNP guest shutdown

 arch/x86/include/asm/cpufeatures.h |   2 +-
 arch/x86/include/asm/msr-index.h   |   3 +
 arch/x86/include/asm/sev.h         |   4 +
 arch/x86/kernel/cpu/scattered.c    |   1 +
 arch/x86/kvm/svm/sev.c             |   2 +
 arch/x86/virt/svm/sev.c            | 201 +++++++++++++++++++++++++++--
 drivers/crypto/ccp/sev-dev.c       |   2 +
 7 files changed, 200 insertions(+), 15 deletions(-)

-- 
2.43.0

[PATCH v14 1/5] x86/cpufeatures: Add X86_FEATURE_RMPOPT feature flag

[Ashish Kalra]

From: Ashish Kalra <ashish.kalra@amd.com>

Add a flag indicating whether RMPOPT instruction is supported.

RMPOPT is a new instruction that reduces the performance overhead of RMP
checks for the hypervisor and non-SNP guests by allowing those checks to be
skipped when 1-GB memory regions are known to contain no SEV-SNP guest memory.

For more information on the RMPOPT instruction, see the AMD64 RMPOPT
technical documentation.

  [ bp: Zap respective tools/ change. ]

Suggested-by: Borislav Petkov (AMD) <bp@alien8.de>
Reviewed-by: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Ashish Kalra <ashish.kalra@amd.com>
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Reviewed-by: Dave Hansen <dave.hansen@linux.intel.com>
Reviewed-by: Ackerley Tng <ackerleytng@google.com>
Link: https://patch.msgid.link/39e9ee269a572c516a3f4e937bfe12d00697d5e6.1782841284.git.ashish.kalra@amd.com
---
 arch/x86/include/asm/cpufeatures.h | 2 +-
 arch/x86/kernel/cpu/scattered.c    | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h
index f70ee74b5f92..3b5b32d3391b 100644
--- a/arch/x86/include/asm/cpufeatures.h
+++ b/arch/x86/include/asm/cpufeatures.h
@@ -76,7 +76,7 @@
 #define X86_FEATURE_K8			( 3*32+ 4) /* Opteron, Athlon64 */
 #define X86_FEATURE_ZEN5		( 3*32+ 5) /* CPU based on Zen5 microarchitecture */
 #define X86_FEATURE_ZEN6		( 3*32+ 6) /* CPU based on Zen6 microarchitecture */
-/* Free                                 ( 3*32+ 7) */
+#define X86_FEATURE_RMPOPT		( 3*32+ 7) /* Support for AMD RMPOPT instruction */
 #define X86_FEATURE_CONSTANT_TSC	( 3*32+ 8) /* "constant_tsc" TSC ticks at a constant rate */
 /* free: was #define X86_FEATURE_UP	( 3*32+ 9) * "up" SMP kernel running on UP */
 #define X86_FEATURE_ART			( 3*32+10) /* "art" Always running timer (ART) */
diff --git a/arch/x86/kernel/cpu/scattered.c b/arch/x86/kernel/cpu/scattered.c
index 8665a6474806..d1795ce219da 100644
--- a/arch/x86/kernel/cpu/scattered.c
+++ b/arch/x86/kernel/cpu/scattered.c
@@ -67,6 +67,7 @@ static const struct cpuid_bit cpuid_bits[] = {
 	{ X86_FEATURE_PERFMON_V2,		CPUID_EAX,  0, 0x80000022, 0 },
 	{ X86_FEATURE_AMD_LBR_V2,		CPUID_EAX,  1, 0x80000022, 0 },
 	{ X86_FEATURE_AMD_LBR_PMC_FREEZE,	CPUID_EAX,  2, 0x80000022, 0 },
+	{ X86_FEATURE_RMPOPT,			CPUID_EDX,  0, 0x80000025, 0 },
 	{ X86_FEATURE_AMD_HTR_CORES,		CPUID_EAX, 30, 0x80000026, 0 },
 	{ 0, 0, 0, 0, 0 }
 };
-- 
2.43.0

[PATCH v14 2/5] x86/sev: Disable CPU hotplug while SNP is active

[Ashish Kalra]

From: Ashish Kalra <ashish.kalra@amd.com>

While SNP is active, every memory write is checked against the RMP to
protect SNP guest memory.  A core performs these RMP checks only once
SNP has been initialized via SNP_INIT and the SNP-enable bit in SYSCFG is
set on that core; the firmware requires the SNP-enable bit to be set on
every present CPU before SNP initialization.

A core that is not SNP-enabled and not SNP-initialized performs no RMP
checks at all, so there is no valid configuration with SNP active and any
CPU exempt from RMP checks.

The firmware determines which CPUs are present from the processor and the
BIOS/UEFI configuration (e.g. SMT disabled in the BIOS) and enumerates
them at SNP init; it is not aware of the OS bringing CPUs online or
offline afterwards.

SNP_INIT fails unless SnpEn is set on all CPUs, so a CPU that is offline
when SNP_INIT is issued, does not have SnpEn set, SNP_INIT fails, and
there can be no SNP guest memory.  OS CPU hotplug can thus diverge from
the firmware's expectations and break SNP.

Tie CPU hotplug to the SNP-enable bit: disable it in snp_prepare() before
SNP is enabled, and re-enable it in snp_shutdown() once the firmware has
disabled SNP.

If snp_prepare() fails before enabling SNP it re-enables hotplug itself;
once SNP is enabled hotplug stays disabled, including across a failed
SNP_INIT and across the legacy SNP_SHUTDOWN_EX path, both of which leave
SNP enabled.

A kexec target that boots with SNP already enabled, disables hotplug once
in snp_rmptable_init(), since snp_prepare() bails when SNP is already
enabled.

With CPU hotplug now disabled while SNP is active, the online CPU mask is
stable, so the cpus_read_lock() previously taken in snp_prepare() to
iterate it is redundant.  Drop cpus_read_lock()/cpus_read_unlock() here.

Suggested-by: Thomas Lendacky <thomas.lendacky@amd.com>
Reviewed-by: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Ashish Kalra <ashish.kalra@amd.com>
---
 arch/x86/virt/svm/sev.c | 36 ++++++++++++++++++++++++++----------
 1 file changed, 26 insertions(+), 10 deletions(-)

diff --git a/arch/x86/virt/svm/sev.c b/arch/x86/virt/svm/sev.c
index cff285d8ad8e..558f7924a3f8 100644
--- a/arch/x86/virt/svm/sev.c
+++ b/arch/x86/virt/svm/sev.c
@@ -513,7 +513,6 @@ static void clear_hsave_pa(void *arg)
 
 int snp_prepare(void)
 {
-	int ret;
 	u64 val;
 
 	/*
@@ -526,14 +525,18 @@ int snp_prepare(void)
 
 	clear_rmp();
 
-	cpus_read_lock();
+	/*
+	 * No CPU may come online without SnpEn while SNP is active; disable
+	 * hotplug here and re-enable it in snp_shutdown().
+	 */
+	cpu_hotplug_disable();
 
 	if (!cpumask_equal(cpu_online_mask, cpu_present_mask)) {
-		ret = -EOPNOTSUPP;
+		cpu_hotplug_enable();
 		pr_warn("SNP init failed: not all CPUs online. (%*pbl online <-> %*pbl present masks).\n",
 			cpumask_pr_args(cpu_online_mask),
 			cpumask_pr_args(cpu_present_mask));
-		goto unlock;
+		return -EOPNOTSUPP;
 	}
 
 	wbinvd_on_all_cpus();
@@ -548,12 +551,7 @@ int snp_prepare(void)
 	/* SNP_INIT requires MSR_VM_HSAVE_PA to be cleared on all CPUs. */
 	on_each_cpu(clear_hsave_pa, NULL, 1);
 
-	ret = 0;
-
-unlock:
-	cpus_read_unlock();
-
-	return ret;
+	return 0;
 }
 EXPORT_SYMBOL_FOR_MODULES(snp_prepare, "ccp");
 
@@ -567,6 +565,13 @@ void snp_shutdown(void)
 
 	clear_rmp();
 	on_each_cpu(mfd_reconfigure, NULL, 1);
+
+	/*
+	 * The firmware has disabled SNP (SnpEn is clear), so re-enable CPU
+	 * hotplug.  A legacy SNP shutdown returns above with SnpEn still set and
+	 * leaves hotplug disabled.
+	 */
+	cpu_hotplug_enable();
 }
 EXPORT_SYMBOL_FOR_MODULES(snp_shutdown, "ccp");
 
@@ -577,6 +582,8 @@ EXPORT_SYMBOL_FOR_MODULES(snp_shutdown, "ccp");
  */
 int __init snp_rmptable_init(void)
 {
+	u64 val;
+
 	if (WARN_ON_ONCE(!cc_platform_has(CC_ATTR_HOST_SEV_SNP)))
 		return -ENOSYS;
 
@@ -586,6 +593,15 @@ int __init snp_rmptable_init(void)
 	if (!setup_rmptable())
 		return -ENOSYS;
 
+	/*
+	 * On a kexec boot SNP may already be enabled (legacy firmware leaves
+	 * SnpEn set across shutdown), in which case snp_prepare() bails without
+	 * disabling CPU hotplug, so disable it here.
+	 */
+	rdmsrq(MSR_AMD64_SYSCFG, val);
+	if (val & MSR_AMD64_SYSCFG_SNP_EN)
+		cpu_hotplug_disable();
+
 	/*
 	 * Setting crash_kexec_post_notifiers to 'true' to ensure that SNP panic
 	 * notifier is invoked to do SNP IOMMU shutdown before kdump.
-- 
2.43.0

[PATCH v14 3/5] x86/sev: Initialize RMPOPT configuration MSRs

[Ashish Kalra]

From: Ashish Kalra <ashish.kalra@amd.com>

The new RMPOPT instruction helps manage per-CPU RMP optimization
structures inside the CPU. It takes a 1GB-aligned physical address
and either returns the status of the optimizations or tries to enable
the optimizations.

Per-CPU RMPOPT tables support at most 2 TB of addressable memory for
RMP optimizations.

Initialize the per-CPU RMPOPT table base to the starting physical
address. This enables RMP optimization for up to 2 TB of system RAM on
all CPUs.

Additionally, add support to setup and enable RMPOPT once SNP is
enabled and initialized.

Suggested-by: Thomas Lendacky <thomas.lendacky@amd.com>
Suggested-by: Dave Hansen <dave.hansen@linux.intel.com>
Signed-off-by: Ashish Kalra <ashish.kalra@amd.com>
---
v14: rmpopt_disable() and the RMPOPT_BASE teardown were removed from this
     patch (the disable path is reworked in the next patch, and the MSRs are
     now left in place on shutdown).  Dropped Reviewed-by: Dave Hansen and
     Tom Lendacky due to this rework.

 arch/x86/include/asm/msr-index.h |  3 +++
 arch/x86/include/asm/sev.h       |  2 ++
 arch/x86/virt/svm/sev.c          | 46 ++++++++++++++++++++++++++++----
 drivers/crypto/ccp/sev-dev.c     |  2 ++
 4 files changed, 48 insertions(+), 5 deletions(-)

diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-index.h
index 3a8e51a0c9e8..1635e2e1c576 100644
--- a/arch/x86/include/asm/msr-index.h
+++ b/arch/x86/include/asm/msr-index.h
@@ -761,6 +761,9 @@
 #define MSR_AMD64_SEG_RMP_ENABLED_BIT	0
 #define MSR_AMD64_SEG_RMP_ENABLED	BIT_ULL(MSR_AMD64_SEG_RMP_ENABLED_BIT)
 #define MSR_AMD64_RMP_SEGMENT_SHIFT(x)	(((x) & GENMASK_ULL(13, 8)) >> 8)
+#define MSR_AMD64_RMPOPT_BASE		0xc0010139
+#define MSR_AMD64_RMPOPT_ENABLE_BIT	0
+#define MSR_AMD64_RMPOPT_ENABLE		BIT_ULL(MSR_AMD64_RMPOPT_ENABLE_BIT)
 
 #define MSR_SVSM_CAA			0xc001f000
 
diff --git a/arch/x86/include/asm/sev.h b/arch/x86/include/asm/sev.h
index 9e7a077c445d..5638d09b5132 100644
--- a/arch/x86/include/asm/sev.h
+++ b/arch/x86/include/asm/sev.h
@@ -662,6 +662,7 @@ static inline void snp_leak_pages(u64 pfn, unsigned int pages)
 	__snp_leak_pages(pfn, pages, true);
 }
 int snp_prepare(void);
+void snp_setup_rmpopt(void);
 void snp_shutdown(void);
 #else
 static inline bool snp_probe_rmptable_info(void) { return false; }
@@ -680,6 +681,7 @@ static inline void snp_leak_pages(u64 pfn, unsigned int npages) {}
 static inline void kdump_sev_callback(void) { }
 static inline void snp_fixup_e820_tables(void) {}
 static inline int snp_prepare(void) { return -ENODEV; }
+static inline void snp_setup_rmpopt(void) {}
 static inline void snp_shutdown(void) {}
 #endif
 
diff --git a/arch/x86/virt/svm/sev.c b/arch/x86/virt/svm/sev.c
index 558f7924a3f8..a059327dc107 100644
--- a/arch/x86/virt/svm/sev.c
+++ b/arch/x86/virt/svm/sev.c
@@ -124,6 +124,8 @@ static void *rmp_bookkeeping __ro_after_init;
 
 static u64 probed_rmp_base, probed_rmp_size;
 
+static phys_addr_t rmpopt_pa_start;
+
 static LIST_HEAD(snp_leaked_pages_list);
 static DEFINE_SPINLOCK(snp_leaked_pages_list_lock);
 
@@ -575,6 +577,32 @@ void snp_shutdown(void)
 }
 EXPORT_SYMBOL_FOR_MODULES(snp_shutdown, "ccp");
 
+static bool rmpopt_capable(void)
+{
+	return cpu_feature_enabled(X86_FEATURE_RMPOPT) &&
+	       cc_platform_has(CC_ATTR_HOST_SEV_SNP);
+}
+
+void snp_setup_rmpopt(void)
+{
+	u64 rmpopt_base;
+	int cpu;
+
+	if (!rmpopt_capable())
+		return;
+
+	rmpopt_pa_start = ALIGN_DOWN(PFN_PHYS(min_low_pfn), SZ_1G);
+	rmpopt_base = rmpopt_pa_start | MSR_AMD64_RMPOPT_ENABLE;
+
+	/*
+	 * Per-CPU RMPOPT tables cover at most 2 TB.  Program each core's
+	 * RMPOPT_BASE with the start of RAM to optimize up to 2 TB.
+	 */
+	for_each_cpu(cpu, cpu_primary_thread_mask)
+		wrmsrq_on_cpu(cpu, MSR_AMD64_RMPOPT_BASE, rmpopt_base);
+}
+EXPORT_SYMBOL_FOR_MODULES(snp_setup_rmpopt, "ccp");
+
 /*
  * Do the necessary preparations which are verified by the firmware as
  * described in the SNP_INIT_EX firmware command description in the SNP
@@ -699,13 +727,21 @@ static bool probe_segmented_rmptable_info(void)
 
 bool snp_probe_rmptable_info(void)
 {
-	if (cpu_feature_enabled(X86_FEATURE_SEGMENTED_RMP))
+	if (cpu_feature_enabled(X86_FEATURE_SEGMENTED_RMP)) {
 		rdmsrq(MSR_AMD64_RMP_CFG, rmp_cfg);
 
-	if (rmp_cfg & MSR_AMD64_SEG_RMP_ENABLED)
-		return probe_segmented_rmptable_info();
-	else
-		return probe_contiguous_rmptable_info();
+		if (rmp_cfg & MSR_AMD64_SEG_RMP_ENABLED)
+			return probe_segmented_rmptable_info();
+	}
+
+	/*
+	 * Segmented RMP is either not supported on the platform or is
+	 * disabled by the firmware.  RMPOPT is not supported without
+	 * segmented RMP.
+	 */
+	setup_clear_cpu_cap(X86_FEATURE_RMPOPT);
+
+	return probe_contiguous_rmptable_info();
 }
 
 /*
diff --git a/drivers/crypto/ccp/sev-dev.c b/drivers/crypto/ccp/sev-dev.c
index f833cb7e4da3..287a8345854b 100644
--- a/drivers/crypto/ccp/sev-dev.c
+++ b/drivers/crypto/ccp/sev-dev.c
@@ -1663,6 +1663,8 @@ static int __sev_snp_init_locked(int *error, unsigned int max_snp_asid)
 
 	sev_es_tmr_size = SNP_TMR_SIZE;
 
+	snp_setup_rmpopt();
+
 	return 0;
 }
 
-- 
2.43.0

[PATCH v14 4/5] x86/sev: Perform RMP optimizations asynchronously

[Ashish Kalra]

From: Ashish Kalra <ashish.kalra@amd.com>

When SNP is enabled, all writes to memory are checked to ensure memory
integrity. This imposes performance overhead on the whole system.

RMPOPT is a new instruction that minimizes the performance overhead of
RMP checks on the hypervisor and on non-SNP guests by allowing RMP
checks to be skipped for 1GB regions of memory that are known not to
contain any SNP guest memory.

Add support for performing RMP optimizations asynchronously using a
dedicated workqueue.

At RMP initialization time, run an optimization pass over all physical
memory (up to 2TB of system RAM, starting from the lowest physical
memory address aligned down to a 1GB boundary), skipping RMP checks for
1GB regions that do not contain SNP guest memory (excluding preassigned
pages such as the RMP table and firmware pages).

As SNP guests are launched, RMPUPDATE assigns their private pages to
guest-owned state; when such a page falls within an optimized 1GB
region, the hardware clears that region's RMPOPT optimization and RMP
checks resume there to protect the guest memory.

Since launching SNP guests clears these optimizations, perform them
again asynchronously using the dedicated workqueue.

Suggested-by: Thomas Lendacky <thomas.lendacky@amd.com>
Suggested-by: Dave Hansen <dave.hansen@linux.intel.com>
Signed-off-by: Ashish Kalra <ashish.kalra@amd.com>
---
v14: Reworked the setup/teardown - initialize once and do not tear it down;
     rmpopt_disable() now only cancels the pending pass.  Use a dedicated
     per-CPU workqueue (WQ_PERCPU) and drop the migrate_disable()/
     migrate_enable() around the local scan.  Added a binutils-version
     comment above the RMPOPT .byte and trimmed redundant comments and the
     2 TB pr_info().  Subject reworded from "Add support to perform RMP
     optimizations asynchronously".  Dropped Reviewed-by: Ackerley Tng and
     Tom Lendacky due to the rework (kept Suggested-by).

 arch/x86/virt/svm/sev.c | 92 ++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 91 insertions(+), 1 deletion(-)

diff --git a/arch/x86/virt/svm/sev.c b/arch/x86/virt/svm/sev.c
index a059327dc107..35678b1f535d 100644
--- a/arch/x86/virt/svm/sev.c
+++ b/arch/x86/virt/svm/sev.c
@@ -19,6 +19,7 @@
 #include <linux/iommu.h>
 #include <linux/amd-iommu.h>
 #include <linux/nospec.h>
+#include <linux/workqueue.h>
 
 #include <asm/sev.h>
 #include <asm/processor.h>
@@ -124,7 +125,16 @@ static void *rmp_bookkeeping __ro_after_init;
 
 static u64 probed_rmp_base, probed_rmp_size;
 
-static phys_addr_t rmpopt_pa_start;
+static u64 rmpopt_pa_start, rmpopt_pa_end;
+
+enum rmpopt_op_type {
+	RMPOPT_OP_VERIFY_AND_REPORT_STATUS,
+	RMPOPT_OP_REPORT_STATUS
+};
+
+static struct workqueue_struct *rmpopt_wq;
+static struct delayed_work rmpopt_delayed_work;
+static DEFINE_MUTEX(rmpopt_wq_mutex);
 
 static LIST_HEAD(snp_leaked_pages_list);
 static DEFINE_SPINLOCK(snp_leaked_pages_list_lock);
@@ -557,6 +567,14 @@ int snp_prepare(void)
 }
 EXPORT_SYMBOL_FOR_MODULES(snp_prepare, "ccp");
 
+static void rmpopt_disable(void)
+{
+	guard(mutex)(&rmpopt_wq_mutex);
+
+	if (rmpopt_wq)
+		cancel_delayed_work_sync(&rmpopt_delayed_work);
+}
+
 void snp_shutdown(void)
 {
 	u64 syscfg;
@@ -565,6 +583,8 @@ void snp_shutdown(void)
 	if (syscfg & MSR_AMD64_SYSCFG_SNP_EN)
 		return;
 
+	rmpopt_disable();
+
 	clear_rmp();
 	on_each_cpu(mfd_reconfigure, NULL, 1);
 
@@ -583,6 +603,43 @@ static bool rmpopt_capable(void)
 	       cc_platform_has(CC_ATTR_HOST_SEV_SNP);
 }
 
+/*
+ * RMPOPT optimizations skip RMP checks at 1GB granularity if this range of
+ * memory does not contain any SNP guest memory.
+ *
+ * @pa is a system physical address; RMPOPT operates on the containing 1GB.
+ */
+static void rmpopt(u64 pa)
+{
+	enum rmpopt_op_type op = RMPOPT_OP_VERIFY_AND_REPORT_STATUS;
+	u64 pa_start = ALIGN_DOWN(pa, SZ_1G);
+
+	/* Supported by binutils 2.48+ */
+	asm volatile(".byte 0xf2, 0x0f, 0x01, 0xfc"
+		     :: "a" (pa_start), "c" (op)
+		     : "memory", "cc");
+}
+
+/* on_each_cpu() callback: optimize the whole RMPOPT range on this CPU. */
+static void rmpopt_scan_range(void *arg)
+{
+	u64 pa;
+
+	for (pa = rmpopt_pa_start; pa < rmpopt_pa_end; pa += SZ_1G)
+		rmpopt(pa);
+}
+
+static void do_rmpopt_work(struct work_struct *work)
+{
+	/*
+	 * Warm up the RMPOPT cache on this pinned per-CPU worker with interrupts
+	 * on, so the IRQ-disabled fan-out below only issues cache-hit RMPOPTs.
+	 */
+	rmpopt_scan_range(NULL);
+
+	on_each_cpu_mask(cpu_primary_thread_mask, rmpopt_scan_range, NULL, true);
+}
+
 void snp_setup_rmpopt(void)
 {
 	u64 rmpopt_base;
@@ -591,6 +648,30 @@ void snp_setup_rmpopt(void)
 	if (!rmpopt_capable())
 		return;
 
+	guard(mutex)(&rmpopt_wq_mutex);
+
+	/*
+	 * Set up once: the workqueue and RMPOPT_BASE MSRs are left in place on
+	 * shutdown, so a later re-initialization just re-queues the optimization
+	 * pass rather than redoing the setup.
+	 */
+	if (rmpopt_wq) {
+		queue_delayed_work(rmpopt_wq, &rmpopt_delayed_work, 0);
+		return;
+	}
+
+	/*
+	 * Use a dedicated per-CPU workqueue so the potentially lengthy warm-up
+	 * scan does not tie up a shared workqueue worker.
+	 */
+	rmpopt_wq = alloc_workqueue("rmpopt_wq", WQ_PERCPU, 1);
+	if (!rmpopt_wq) {
+		pr_err("Failed to allocate RMPOPT workqueue\n");
+		return;
+	}
+
+	INIT_DELAYED_WORK(&rmpopt_delayed_work, do_rmpopt_work);
+
 	rmpopt_pa_start = ALIGN_DOWN(PFN_PHYS(min_low_pfn), SZ_1G);
 	rmpopt_base = rmpopt_pa_start | MSR_AMD64_RMPOPT_ENABLE;
 
@@ -600,6 +681,15 @@ void snp_setup_rmpopt(void)
 	 */
 	for_each_cpu(cpu, cpu_primary_thread_mask)
 		wrmsrq_on_cpu(cpu, MSR_AMD64_RMPOPT_BASE, rmpopt_base);
+
+	rmpopt_pa_end = ALIGN(PFN_PHYS(max_pfn), SZ_1G);
+
+	if ((rmpopt_pa_end - rmpopt_pa_start) > SZ_2T)
+		rmpopt_pa_end = rmpopt_pa_start + SZ_2T;
+
+	queue_delayed_work(rmpopt_wq, &rmpopt_delayed_work, 0);
+
+	pr_info("RMPOPT optimizations enabled\n");
 }
 EXPORT_SYMBOL_FOR_MODULES(snp_setup_rmpopt, "ccp");
 
-- 
2.43.0

[PATCH v14 5/5] x86/sev: Re-enable RMP optimizations on SNP guest shutdown

[Ashish Kalra]

From: Ashish Kalra <ashish.kalra@amd.com>

The RMPOPT table is a per-CPU table which indicates whether 1GB regions
of physical memory are entirely hypervisor-owned.

When performing host memory accesses in hypervisor mode as well as
non-SNP guest mode, the processor may consult the RMPOPT table to
potentially skip an RMP access and improve performance.

Normal guest events disable RMP optimizations: pages are converted from
shared to private as SNP guests are launched, and large pages are split
and collapsed during guest operation -- both disable the RMPOPT
optimizations for the affected 1GB regions.

When guests are torn down, their pages are converted back to shared, so
those regions may become eligible for RMPOPT optimization again. Without
some intervention, all RMP optimizations would eventually be lost, so
re-optimize all of physical memory on SNP guest teardown.

Perform the re-optimization after a delay, using mod_delayed_work() so
that the delay timer is reset on each call. This batches multiple guest
terminations into a single pass: the re-optimization runs 10 seconds
after the *last* termination rather than after the first.
mod_delayed_work() also re-queues work that is already in-flight, so a
re-scan request during an active scan is not silently dropped.

Guest teardown is currently the only event that returns guest memory to
hypervisor ownership: SNP guests do not support ballooning or memory
hotplug, so pages freed during a guest's lifetime remain guest-owned.
It is therefore the only point at which memory becomes eligible for RMP
re-optimization, which is why re-optimization is driven by guest
teardown rather than by a periodic scan.

Reviewed-by: Ackerley Tng <ackerleytng@google.com>
Reviewed-by: Tom Lendacky <thomas.lendacky@amd.com>
Reviewed-by: Dave Hansen <dave.hansen@linux.intel.com>
Signed-off-by: Ashish Kalra <ashish.kalra@amd.com>
---
 arch/x86/include/asm/sev.h |  2 ++
 arch/x86/kvm/svm/sev.c     |  2 ++
 arch/x86/virt/svm/sev.c    | 31 +++++++++++++++++++++++++++++++
 3 files changed, 35 insertions(+)

diff --git a/arch/x86/include/asm/sev.h b/arch/x86/include/asm/sev.h
index 5638d09b5132..3235e171647d 100644
--- a/arch/x86/include/asm/sev.h
+++ b/arch/x86/include/asm/sev.h
@@ -662,6 +662,7 @@ static inline void snp_leak_pages(u64 pfn, unsigned int pages)
 	__snp_leak_pages(pfn, pages, true);
 }
 int snp_prepare(void);
+void snp_rmpopt_all_physmem(void);
 void snp_setup_rmpopt(void);
 void snp_shutdown(void);
 #else
@@ -681,6 +682,7 @@ static inline void snp_leak_pages(u64 pfn, unsigned int npages) {}
 static inline void kdump_sev_callback(void) { }
 static inline void snp_fixup_e820_tables(void) {}
 static inline int snp_prepare(void) { return -ENODEV; }
+static inline void snp_rmpopt_all_physmem(void) {}
 static inline void snp_setup_rmpopt(void) {}
 static inline void snp_shutdown(void) {}
 #endif
diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
index 5705723f1f41..d8e6b8a08b79 100644
--- a/arch/x86/kvm/svm/sev.c
+++ b/arch/x86/kvm/svm/sev.c
@@ -3032,6 +3032,8 @@ void sev_vm_destroy(struct kvm *kvm)
 		 */
 		if (snp_decommission_context(kvm))
 			return;
+
+		snp_rmpopt_all_physmem();
 	} else {
 		sev_unbind_asid(kvm, sev->handle);
 	}
diff --git a/arch/x86/virt/svm/sev.c b/arch/x86/virt/svm/sev.c
index 35678b1f535d..c16f82642390 100644
--- a/arch/x86/virt/svm/sev.c
+++ b/arch/x86/virt/svm/sev.c
@@ -640,6 +640,37 @@ static void do_rmpopt_work(struct work_struct *work)
 	on_each_cpu_mask(cpu_primary_thread_mask, rmpopt_scan_range, NULL, true);
 }
 
+/*
+ * Delay, in milliseconds, before the RMP re-optimization pass runs after an SNP
+ * guest is torn down, passed as the delay to mod_delayed_work().  This coalesces
+ * a burst of teardowns into a single scan and gives each guest's pages time to
+ * be converted back to the shared, hypervisor-owned state.  The 10 second value
+ * is a heuristic trading re-optimization latency against scanning too eagerly.
+ */
+#define RMPOPT_WORK_TIMEOUT	(10 * MSEC_PER_SEC)
+
+/*
+ * Perform RMP optimizations on memory freed by terminating guests.  The scan
+ * is deferred, so it normally runs after sev_gmem_invalidate() has converted
+ * this guest's pages back to shared, and picks them up then.  A very large
+ * guest whose conversion has not finished by then is picked up by a later
+ * teardown's scan.
+ */
+void snp_rmpopt_all_physmem(void)
+{
+	if (!rmpopt_capable())
+		return;
+
+	guard(mutex)(&rmpopt_wq_mutex);
+
+	if (!rmpopt_wq)
+		return;
+
+	mod_delayed_work(rmpopt_wq, &rmpopt_delayed_work,
+			 msecs_to_jiffies(RMPOPT_WORK_TIMEOUT));
+}
+EXPORT_SYMBOL_FOR_MODULES(snp_rmpopt_all_physmem, "kvm-amd");
+
 void snp_setup_rmpopt(void)
 {
 	u64 rmpopt_base;
-- 
2.43.0

Re: [PATCH v14 4/5] x86/sev: Perform RMP optimizations asynchronously

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] Re-initializing SNP after a CPU hotplug cycle skips programming the RMPOPT_BASE MSR on the newly onlined CPU, leading to an Invalid Opcode (#UD) exception when executing RMPOPT.
- [High] The inline assembly for the RMPOPT instruction fails to declare `%rax` as an output constraint or clobber, leading to silent register corruption and unpredictable loop behavior.
- [High] The RMPOPT 'warm-up' executes on a single CPU, causing the subsequent cross-CPU IPI to execute millions of slow RMPOPT instructions with IRQs disabled on all other CPUs with cold caches, triggering hard lockups.
--

commit e2c3ac4ab3639e4c47a6c85a3acca0d88b6fb5a3
Author: Ashish Kalra <ashish.kalra@amd.com>

x86/sev: Perform RMP optimizations asynchronously

This patch introduces asynchronous RMP optimizations by scanning and
optimizing up to 2TB of system memory at 1GB boundaries using a per-CPU
workqueue. When SNP guests are launched and regions are allocated, hardware
clears the optimizations, so the workqueue asynchronously reapplies them.

> diff --git a/arch/x86/virt/svm/sev.c b/arch/x86/virt/svm/sev.c
> index a059327dc1071..35678b1f535d4 100644
> --- a/arch/x86/virt/svm/sev.c
> +++ b/arch/x86/virt/svm/sev.c
[ ... ]
> +static void rmpopt(u64 pa)
> +{
> +	enum rmpopt_op_type op = RMPOPT_OP_VERIFY_AND_REPORT_STATUS;
> +	u64 pa_start = ALIGN_DOWN(pa, SZ_1G);
> +
> +	/* Supported by binutils 2.48+ */
> +	asm volatile(".byte 0xf2, 0x0f, 0x01, 0xfc"
> +		     :: "a" (pa_start), "c" (op)
> +		     : "memory", "cc");

[Severity: High]
Does this inline assembly need to declare %rax as an output or clobbered
register?

Since RMPOPT_OP_VERIFY_AND_REPORT_STATUS instructs the hardware to return a
status code in %rax, the compiler might allocate the loop counter pa to %rax
in rmpopt_scan_range(). If hardware overwrites %rax, the loop counter would
be reset, potentially causing an infinite loop.

> +}
> +
> +/* on_each_cpu() callback: optimize the whole RMPOPT range on this CPU. */
> +static void rmpopt_scan_range(void *arg)
> +{
> +	u64 pa;
> +
> +	for (pa = rmpopt_pa_start; pa < rmpopt_pa_end; pa += SZ_1G)
> +		rmpopt(pa);
> +}
> +
> +static void do_rmpopt_work(struct work_struct *work)
> +{
> +	/*
> +	 * Warm up the RMPOPT cache on this pinned per-CPU worker with interrupts
> +	 * on, so the IRQ-disabled fan-out below only issues cache-hit RMPOPTs.
> +	 */
> +	rmpopt_scan_range(NULL);
> +
> +	on_each_cpu_mask(cpu_primary_thread_mask, rmpopt_scan_range, NULL, true);

[Severity: High]
Could this fan-out cause significant stalls on other CPUs?

The RMPOPT cache is warmed up locally on the current CPU, but because the
caches are strictly per-CPU, all other CPUs will still have a cold cache.
When they receive the IPI, they will perform the full up to 2TB scan in
hardirq context with a cold cache, which seems likely to trigger NMI
watchdog timeouts across the system.

> +}
> +
> +void snp_setup_rmpopt(void)
> +{
> +	u64 rmpopt_base;
> +	int cpu;
> +
> +	if (!rmpopt_capable())
> +		return;
> +
> +	guard(mutex)(&rmpopt_wq_mutex);
> +
> +	/*
> +	 * Set up once: the workqueue and RMPOPT_BASE MSRs are left in place on
> +	 * shutdown, so a later re-initialization just re-queues the optimization
> +	 * pass rather than redoing the setup.
> +	 */
> +	if (rmpopt_wq) {
> +		queue_delayed_work(rmpopt_wq, &rmpopt_delayed_work, 0);
> +		return;

[Severity: High]
Is it safe to skip programming MSR_AMD64_RMPOPT_BASE on newly onlined CPUs?

If a CPU is hot-unplugged and hot-plugged back, its MSRs are reset to their
defaults. If SNP is disabled and re-enabled during this cycle,
snp_setup_rmpopt() will see that rmpopt_wq is already allocated and return
early. This skips the MSR programming loop for the hotplugged CPU, which
would lead to an Invalid Opcode (#UD) exception when executing RMPOPT on it.

> +	}

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1788984053.git.ashish.kalra@amd.com?part=4

Re: [PATCH v14 5/5] x86/sev: Re-enable RMP optimizations on SNP guest shutdown

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] Race condition between KVM VM destruction and CCP unbind allows RMPOPT work to be queued after SNP shutdown, leading to a potential kernel panic.
- [High] System lockup due to broadcasting a cold-cache memory scan to all CPUs with interrupts disabled.
--

commit 5610134a51dd4a088684e2eb5968c3e869f5727c
Author: Ashish Kalra <ashish.kalra@amd.com>

x86/sev: Re-enable RMP optimizations on SNP guest shutdown

This commit re-enables RMP optimizations on SNP guest shutdown by scheduling
a delayed work pass. It introduces snp_rmpopt_all_physmem() to queue the scan
10 seconds after a guest termination. The delayed work batches multiple guest
teardowns into a single optimization pass.

> diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
> index 5705723f1f412..d8e6b8a08b79f 100644
> --- a/arch/x86/kvm/svm/sev.c
> +++ b/arch/x86/kvm/svm/sev.c
[ ... ]
> @@ -3032,6 +3032,8 @@ void sev_vm_destroy(struct kvm *kvm)
>  		 */
>  		if (snp_decommission_context(kvm))
>  			return;
> +
> +		snp_rmpopt_all_physmem();

[Severity: High]
Does this create a race condition with CCP unbind that can lead to a host panic?

If the CCP device is unbound or shut down, snp_shutdown() is invoked, which
disables RMPOPT by cancelling pending work via rmpopt_disable(). However,
rmpopt_disable() does not set rmpopt_wq = NULL:

arch/x86/virt/svm/sev.c:rmpopt_disable() {
    guard(mutex)(&rmpopt_wq_mutex);

    if (rmpopt_wq)
        cancel_delayed_work_sync(&rmpopt_delayed_work);
}

If KVM concurrently destroys a VM, it calls snp_rmpopt_all_physmem() here.
Because rmpopt_wq is still valid, the background scan is blindly re-queued.

When the delayed work executes, it will issue the RMPOPT instruction while
SNP is disabled in hardware (SYSCFG_SNP_EN == 0), causing an Invalid Opcode
(#UD) exception and panicking the host system.

>  	} else {
>  		sev_unbind_asid(kvm, sev->handle);
>  	}
> diff --git a/arch/x86/virt/svm/sev.c b/arch/x86/virt/svm/sev.c
> index 35678b1f535d4..c16f826423903 100644
> --- a/arch/x86/virt/svm/sev.c
> +++ b/arch/x86/virt/svm/sev.c
> @@ -640,6 +640,37 @@ static void do_rmpopt_work(struct work_struct *work)
>  	on_each_cpu_mask(cpu_primary_thread_mask, rmpopt_scan_range, NULL, true);

[Severity: High]
Can this cause a system lockup by broadcasting a cold-cache memory scan to all
CPUs with interrupts disabled?

When an SNP guest shuts down, it queues rmpopt_delayed_work. The worker
function do_rmpopt_work() attempts to warm up the cache first:

arch/x86/virt/svm/sev.c:do_rmpopt_work() {
    rmpopt_scan_range(NULL);
    on_each_cpu_mask(cpu_primary_thread_mask, rmpopt_scan_range, NULL, true);
}

Because the RMPOPT hardware cache is per-CPU, running rmpopt_scan_range()
locally only warms up the cache for the worker's CPU. The subsequent
on_each_cpu_mask() forces all other CPUs to run the scan with interrupts
disabled.

Since all other CPUs have a cold cache, they perform a massive memory scan
(up to 2TB) while IRQs are disabled, which can cause immense latency and
lead to host lockup (exceeding NMI watchdog or RCU stall timeouts).

>  }
>  
> +/*
> + * Delay, in milliseconds, before the RMP re-optimization pass runs after an SNP
[ ... ]

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1788984053.git.ashish.kalra@amd.com?part=5

Re: [PATCH v14 2/5] x86/sev: Disable CPU hotplug while SNP is active

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] System suspend/resume bypasses cpu_hotplug_disable() and brings CPUs online without SnpEn set, breaking the SNP invariant.
--

commit 0243c46e8f299e3de42fbcddbc4c7bd93fb68c65
Author: Ashish Kalra <ashish.kalra@amd.com>

x86/sev: Disable CPU hotplug while SNP is active

This commit disables CPU hotplug while SEV-SNP is active. This ensures no
CPU can be brought online without the SnpEn bit set in MSR_AMD64_SYSCFG,
which is required for hardware RMP checks to function correctly.

> diff --git a/arch/x86/virt/svm/sev.c b/arch/x86/virt/svm/sev.c
> index cff285d8ad8e3..558f7924a3f85 100644
> --- a/arch/x86/virt/svm/sev.c
> +++ b/arch/x86/virt/svm/sev.c
> @@ -526,14 +525,18 @@ int snp_prepare(void)
>  
>  	clear_rmp();
>  
> -	cpus_read_lock();
> +	/*
> +	 * No CPU may come online without SnpEn while SNP is active; disable
> +	 * hotplug here and re-enable it in snp_shutdown().
> +	 */
> +	cpu_hotplug_disable();

[Severity: High]
Does system suspend/resume bypass cpu_hotplug_disable() and bring CPUs online
without SnpEn set, breaking the SNP invariant?

If the host initiates an ACPI system suspend (e.g., S3), freeze_secondary_cpus()
offlines all secondary CPUs via _cpu_down(), which bypasses the
cpu_hotplug_disabled counter incremented by cpu_hotplug_disable().

Upon resume, thaw_secondary_cpus() brings the CPUs back online via _cpu_up().
Because CPU MSR state is lost during power-off in suspend and snp_enable() is
not registered as a CPU hotplug callback, the secondary CPUs wake up with SnpEn
cleared. This would allow them to bypass hardware RMP checks.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1788984053.git.ashish.kalra@amd.com?part=2

Re: [PATCH v14 4/5] x86/sev: Perform RMP optimizations asynchronously

[Kalra, Ashish]

On 9/10/2026 5:11 PM, sashiko-bot@kernel.org wrote:
> Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
> - [High] Re-initializing SNP after a CPU hotplug cycle skips programming the RMPOPT_BASE MSR on the newly onlined CPU, leading to an Invalid Opcode (#UD) exception when executing RMPOPT.
> - [High] The inline assembly for the RMPOPT instruction fails to declare `%rax` as an output constraint or clobber, leading to silent register corruption and unpredictable loop behavior.
> - [High] The RMPOPT 'warm-up' executes on a single CPU, causing the subsequent cross-CPU IPI to execute millions of slow RMPOPT instructions with IRQs disabled on all other CPUs with cold caches, triggering hard lockups.
> --
> 
> commit e2c3ac4ab3639e4c47a6c85a3acca0d88b6fb5a3
> Author: Ashish Kalra <ashish.kalra@amd.com>
> 
> x86/sev: Perform RMP optimizations asynchronously
> 
> This patch introduces asynchronous RMP optimizations by scanning and
> optimizing up to 2TB of system memory at 1GB boundaries using a per-CPU
> workqueue. When SNP guests are launched and regions are allocated, hardware
> clears the optimizations, so the workqueue asynchronously reapplies them.
> 
>> diff --git a/arch/x86/virt/svm/sev.c b/arch/x86/virt/svm/sev.c
>> index a059327dc1071..35678b1f535d4 100644
>> --- a/arch/x86/virt/svm/sev.c
>> +++ b/arch/x86/virt/svm/sev.c
> [ ... ]
>> +static void rmpopt(u64 pa)
>> +{
>> +	enum rmpopt_op_type op = RMPOPT_OP_VERIFY_AND_REPORT_STATUS;
>> +	u64 pa_start = ALIGN_DOWN(pa, SZ_1G);
>> +
>> +	/* Supported by binutils 2.48+ */
>> +	asm volatile(".byte 0xf2, 0x0f, 0x01, 0xfc"
>> +		     :: "a" (pa_start), "c" (op)
>> +		     : "memory", "cc");
> 
> [Severity: High]
> Does this inline assembly need to declare %rax as an output or clobbered
> register?
> 
> Since RMPOPT_OP_VERIFY_AND_REPORT_STATUS instructs the hardware to return a
> status code in %rax, the compiler might allocate the loop counter pa to %rax
> in rmpopt_scan_range(). If hardware overwrites %rax, the loop counter would
> be reset, potentially causing an infinite loop.
> 

RMPOPT's only output is the carry flag (RFLAGS.CF); it does not modify %rax. The VERIFY_AND_REPORT_STATUS result is returned in CF, not %rax — unlike RMPREAD/PSMASH/RMPUPDATE, which return data in registers. Since RMPOPT leaves %rax unchanged, the input‑only "a" (pa_start) constraint is correct and no output/clobber for %rax is required; the loop counter cannot be corrupted.

>> +}
>> +
>> +/* on_each_cpu() callback: optimize the whole RMPOPT range on this CPU. */
>> +static void rmpopt_scan_range(void *arg)
>> +{
>> +	u64 pa;
>> +
>> +	for (pa = rmpopt_pa_start; pa < rmpopt_pa_end; pa += SZ_1G)
>> +		rmpopt(pa);
>> +}
>> +
>> +static void do_rmpopt_work(struct work_struct *work)
>> +{
>> +	/*
>> +	 * Warm up the RMPOPT cache on this pinned per-CPU worker with interrupts
>> +	 * on, so the IRQ-disabled fan-out below only issues cache-hit RMPOPTs.
>> +	 */
>> +	rmpopt_scan_range(NULL);
>> +
>> +	on_each_cpu_mask(cpu_primary_thread_mask, rmpopt_scan_range, NULL, true);
> 
> [Severity: High]
> Could this fan-out cause significant stalls on other CPUs?
> 
> The RMPOPT cache is warmed up locally on the current CPU, but because the
> caches are strictly per-CPU, all other CPUs will still have a cold cache.
> When they receive the IPI, they will perform the full up to 2TB scan in
> hardirq context with a cold cache, which seems likely to trigger NMI
> watchdog timeouts across the system.
> 

The premise that the RMPOPT caches are strictly per‑CPU is not correct. The RMPOPT optimization results are held in shared memory, not in per‑CPU caches. Once the warm‑up pass on the local CPU — run with interrupts enabled and preemptible — has verified and recorded the 1GB regions, the follower CPUs read those already‑computed results from shared memory, so their RMPOPTs short‑circuit instead of re‑scanning the RMP. This is why the fan‑out is cheap: measured via the debug follower_timings node, the worst‑case follower IPI callback completes in ≈1.3 ms (1,341,505 ns) — roughly four orders of
magnitude below the soft‑lockup/NMI‑watchdog thresholds— confirming the followers do not each perform a cold up‑to‑2TB scan in hardirq.

>> +}
>> +
>> +void snp_setup_rmpopt(void)
>> +{
>> +	u64 rmpopt_base;
>> +	int cpu;
>> +
>> +	if (!rmpopt_capable())
>> +		return;
>> +
>> +	guard(mutex)(&rmpopt_wq_mutex);
>> +
>> +	/*
>> +	 * Set up once: the workqueue and RMPOPT_BASE MSRs are left in place on
>> +	 * shutdown, so a later re-initialization just re-queues the optimization
>> +	 * pass rather than redoing the setup.
>> +	 */
>> +	if (rmpopt_wq) {
>> +		queue_delayed_work(rmpopt_wq, &rmpopt_delayed_work, 0);
>> +		return;
> 
> [Severity: High]
> Is it safe to skip programming MSR_AMD64_RMPOPT_BASE on newly onlined CPUs?
> 
> If a CPU is hot-unplugged and hot-plugged back, its MSRs are reset to their
> defaults. If SNP is disabled and re-enabled during this cycle,
> snp_setup_rmpopt() will see that rmpopt_wq is already allocated and return
> early. This skips the MSR programming loop for the hotplugged CPU, which
> would lead to an Invalid Opcode (#UD) exception when executing RMPOPT on it.
> 

CPU hotplug is disabled the entire time SNP is active (patch 2/5), so a CPU's RMPOPT_BASE can only be reset by a hotplug cycle that occurs after a full SNP shutdown has cleared SnpEn and re‑enabled hotplug.
RMPOPT is set up once and left in place; re‑initialization simply re‑queues the optimization pass rather than reprogramming the MSRs. Guarding against a full‑shutdown + CPU‑hotplug + SNP‑re‑init sequence is outside the supported flow, so the setup is intentionally not re‑done on re‑initialization.

Thanks,
Ashish

>> +	}
> 

Re: [PATCH v14 2/5] x86/sev: Disable CPU hotplug while SNP is active

[Kalra, Ashish]

On 9/10/2026 5:25 PM, sashiko-bot@kernel.org wrote:
> Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
> - [High] System suspend/resume bypasses cpu_hotplug_disable() and brings CPUs online without SnpEn set, breaking the SNP invariant.
> --
> 
> commit 0243c46e8f299e3de42fbcddbc4c7bd93fb68c65
> Author: Ashish Kalra <ashish.kalra@amd.com>
> 
> x86/sev: Disable CPU hotplug while SNP is active
> 
> This commit disables CPU hotplug while SEV-SNP is active. This ensures no
> CPU can be brought online without the SnpEn bit set in MSR_AMD64_SYSCFG,
> which is required for hardware RMP checks to function correctly.
> 
>> diff --git a/arch/x86/virt/svm/sev.c b/arch/x86/virt/svm/sev.c
>> index cff285d8ad8e3..558f7924a3f85 100644
>> --- a/arch/x86/virt/svm/sev.c
>> +++ b/arch/x86/virt/svm/sev.c
>> @@ -526,14 +525,18 @@ int snp_prepare(void)
>>  
>>  	clear_rmp();
>>  
>> -	cpus_read_lock();
>> +	/*
>> +	 * No CPU may come online without SnpEn while SNP is active; disable
>> +	 * hotplug here and re-enable it in snp_shutdown().
>> +	 */
>> +	cpu_hotplug_disable();
> 
> [Severity: High]
> Does system suspend/resume bypass cpu_hotplug_disable() and bring CPUs online
> without SnpEn set, breaking the SNP invariant?
> 
> If the host initiates an ACPI system suspend (e.g., S3), freeze_secondary_cpus()
> offlines all secondary CPUs via _cpu_down(), which bypasses the
> cpu_hotplug_disabled counter incremented by cpu_hotplug_disable().
> 
> Upon resume, thaw_secondary_cpus() brings the CPUs back online via _cpu_up().
> Because CPU MSR state is lost during power-off in suspend and snp_enable() is
> not registered as a CPU hotplug callback, the secondary CPUs wake up with SnpEn
> cleared. This would allow them to bypass hardware RMP checks.
> 

You're right that the suspend path bypasses cpu_hotplug_disable() — freeze_secondary_cpus() uses _cpu_down() directly and ignores the cpu_hotplug_disabled counter. But this is neither introduced nor affected by this patch, and does not apply to SNP hosts:

1. This patch does not touch the suspend/resume path. cpu_hotplug_disable() holds the online CPU set stable against the regular (sysfs) hotplug path while SNP is active. The cpus_read_lock() this patch replaces did not gate the freeze/thaw path either, so suspend/resume behaviour is unchanged from base SNP.
2. SNP hosts do not support system suspend/resume. It is not a supported configuration for an SNP‑enabled host, and there is no path today to re‑establish the per‑CPU SNP state (SnpEn) on resume — snp_enable() has never been registered as a CPU‑hotplug/resume callback. That is a pre‑existing property of base SNP, independent of this series.

Thanks,
Ashish

Re: [PATCH v14 5/5] x86/sev: Re-enable RMP optimizations on SNP guest shutdown

[Kalra, Ashish]

On 9/10/2026 5:12 PM, sashiko-bot@kernel.org wrote:
> Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
> - [High] Race condition between KVM VM destruction and CCP unbind allows RMPOPT work to be queued after SNP shutdown, leading to a potential kernel panic.
> - [High] System lockup due to broadcasting a cold-cache memory scan to all CPUs with interrupts disabled.
> --
> 
> commit 5610134a51dd4a088684e2eb5968c3e869f5727c
> Author: Ashish Kalra <ashish.kalra@amd.com>
> 
> x86/sev: Re-enable RMP optimizations on SNP guest shutdown
> 
> This commit re-enables RMP optimizations on SNP guest shutdown by scheduling
> a delayed work pass. It introduces snp_rmpopt_all_physmem() to queue the scan
> 10 seconds after a guest termination. The delayed work batches multiple guest
> teardowns into a single optimization pass.
> 
>> diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
>> index 5705723f1f412..d8e6b8a08b79f 100644
>> --- a/arch/x86/kvm/svm/sev.c
>> +++ b/arch/x86/kvm/svm/sev.c
> [ ... ]
>> @@ -3032,6 +3032,8 @@ void sev_vm_destroy(struct kvm *kvm)
>>  		 */
>>  		if (snp_decommission_context(kvm))
>>  			return;
>> +
>> +		snp_rmpopt_all_physmem();
> 
> [Severity: High]
> Does this create a race condition with CCP unbind that can lead to a host panic?
> 
> If the CCP device is unbound or shut down, snp_shutdown() is invoked, which
> disables RMPOPT by cancelling pending work via rmpopt_disable(). However,
> rmpopt_disable() does not set rmpopt_wq = NULL:
> 
> arch/x86/virt/svm/sev.c:rmpopt_disable() {
>     guard(mutex)(&rmpopt_wq_mutex);
> 
>     if (rmpopt_wq)
>         cancel_delayed_work_sync(&rmpopt_delayed_work);
> }
> 
> If KVM concurrently destroys a VM, it calls snp_rmpopt_all_physmem() here.
> Because rmpopt_wq is still valid, the background scan is blindly re-queued.
> 
> When the delayed work executes, it will issue the RMPOPT instruction while
> SNP is disabled in hardware (SYSCFG_SNP_EN == 0), causing an Invalid Opcode
> (#UD) exception and panicking the host system.

The concern is valid in principle. In practice, though, it requires an unsupported configuration:

- A full SNP shutdown (SNP_SHUTDOWN, which clears SnpEn) only succeeds when there are no active SNP guests. The only way snp_rmpopt_all_physmem() can re‑queue the pass after SnpEn has been cleared is if the CCP/SEV platform is being torn down (driver unbind/removal or system shutdown) concurrently with an in‑flight SNP guest teardown.
- Tearing down the SEV platform while SNP guests are still being destroyed is not a supported operation.

Given that, we keep the RMPOPT setup/teardown minimal — set up once, and re‑queue the pass only on re‑initialization — rather than add persistent "enabled" state solely to guard an unsupported teardown ordering.

Thanks,
Ashish

> 
>>  	} else {
>>  		sev_unbind_asid(kvm, sev->handle);
>>  	}
>> diff --git a/arch/x86/virt/svm/sev.c b/arch/x86/virt/svm/sev.c
>> index 35678b1f535d4..c16f826423903 100644
>> --- a/arch/x86/virt/svm/sev.c
>> +++ b/arch/x86/virt/svm/sev.c
>> @@ -640,6 +640,37 @@ static void do_rmpopt_work(struct work_struct *work)
>>  	on_each_cpu_mask(cpu_primary_thread_mask, rmpopt_scan_range, NULL, true);
> 
> [Severity: High]
> Can this cause a system lockup by broadcasting a cold-cache memory scan to all
> CPUs with interrupts disabled?
> 
> When an SNP guest shuts down, it queues rmpopt_delayed_work. The worker
> function do_rmpopt_work() attempts to warm up the cache first:
> 
> arch/x86/virt/svm/sev.c:do_rmpopt_work() {
>     rmpopt_scan_range(NULL);
>     on_each_cpu_mask(cpu_primary_thread_mask, rmpopt_scan_range, NULL, true);
> }
> 
> Because the RMPOPT hardware cache is per-CPU, running rmpopt_scan_range()
> locally only warms up the cache for the worker's CPU. The subsequent
> on_each_cpu_mask() forces all other CPUs to run the scan with interrupts
> disabled.
> 
> Since all other CPUs have a cold cache, they perform a massive memory scan
> (up to 2TB) while IRQs are disabled, which can cause immense latency and
> lead to host lockup (exceeding NMI watchdog or RCU stall timeouts).
> 

The RMPOPT optimization results are held in shared memory, not per‑CPU caches, so after the local warm‑up the follower CPUs read the already‑computed results from
shared memory and short‑circuit. Measured via follower_timings, the worst‑case follower IPI callback is ≈1.3 ms (1,341,505 ns) — ~4 orders of magnitude under the soft‑lockup/NMI/RCU‑stall thresholds. 

Thanks,
Ashish

>>  }
>>  
>> +/*
>> + * Delay, in milliseconds, before the RMP re-optimization pass runs after an SNP
> [ ... ]
> 

Re: [PATCH v14 4/5] x86/sev: Perform RMP optimizations asynchronously

[Borislav Petkov]

On Thu, Sep 10, 2026 at 10:00:08PM +0000, Ashish Kalra wrote:
>  void snp_setup_rmpopt(void)
>  {
>  	u64 rmpopt_base;
> @@ -591,6 +648,30 @@ void snp_setup_rmpopt(void)
>  	if (!rmpopt_capable())
>  		return;
>  
> +	guard(mutex)(&rmpopt_wq_mutex);
> +
> +	/*
> +	 * Set up once: the workqueue and RMPOPT_BASE MSRs are left in place on
> +	 * shutdown, so a later re-initialization just re-queues the optimization
> +	 * pass rather than redoing the setup.
> +	 */
> +	if (rmpopt_wq) {
> +		queue_delayed_work(rmpopt_wq, &rmpopt_delayed_work, 0);
> +		return;
> +	}

No, this is not how this is done. This is a *setup* function but you also use
it to start the workqueue if it has been allocated already. So it should
either setup or start but not both.

So what you do is, you try to allocate the workqueue. If it fails, you clear
X86_FEATURE_RMPOPT so that rmpopt_capable() is false and that can be your
start_workqueue function.

This way you get rid of all that

	if (rmpopt_wq) 

sprinkles everywhere.

> +
> +	/*
> +	 * Use a dedicated per-CPU workqueue so the potentially lengthy warm-up
> +	 * scan does not tie up a shared workqueue worker.
> +	 */
> +	rmpopt_wq = alloc_workqueue("rmpopt_wq", WQ_PERCPU, 1);
> +	if (!rmpopt_wq) {
> +		pr_err("Failed to allocate RMPOPT workqueue\n");
> +		return;
> +	}
> +
> +	INIT_DELAYED_WORK(&rmpopt_delayed_work, do_rmpopt_work);
> +
>  	rmpopt_pa_start = ALIGN_DOWN(PFN_PHYS(min_low_pfn), SZ_1G);
>  	rmpopt_base = rmpopt_pa_start | MSR_AMD64_RMPOPT_ENABLE;
>  
> @@ -600,6 +681,15 @@ void snp_setup_rmpopt(void)
>  	 */
>  	for_each_cpu(cpu, cpu_primary_thread_mask)
>  		wrmsrq_on_cpu(cpu, MSR_AMD64_RMPOPT_BASE, rmpopt_base);
> +
> +	rmpopt_pa_end = ALIGN(PFN_PHYS(max_pfn), SZ_1G);
> +
> +	if ((rmpopt_pa_end - rmpopt_pa_start) > SZ_2T)
> +		rmpopt_pa_end = rmpopt_pa_start + SZ_2T;
> +
> +	queue_delayed_work(rmpopt_wq, &rmpopt_delayed_work, 0);
> +
> +	pr_info("RMPOPT optimizations enabled\n");
>  }
>  EXPORT_SYMBOL_FOR_MODULES(snp_setup_rmpopt, "ccp");

There is no ccp driver patch calling this so this export needs to happen when
you're actually adding the ccp code.

Same thing for the snp_rmpopt_all_physmem() export to kvm-amd.

Looking at this more, I would like to get rid of the snp_setup_rmpopt() export
and have this function do the necessary setup stuff from an initcall in this
file. This way you set up the stuff at kernel init time and have everything
ready to go.

Then the ccp will *only* call a function which is called snp_enable_rmpopt()
after it has enabled SNP. That function simply enables the workqueue.

And then kvm-amd can call that function too so we end up with one export.

Oh, and you can zap those comments while at it:

diff --git a/arch/x86/virt/svm/sev.c b/arch/x86/virt/svm/sev.c
index ca99617142be..c8ba71431a5e 100644
--- a/arch/x86/virt/svm/sev.c
+++ b/arch/x86/virt/svm/sev.c
@@ -619,7 +619,6 @@ static void rmpopt(u64 pa)
 		     : "memory", "cc");
 }
 
-/* on_each_cpu() callback: optimize the whole RMPOPT range on this CPU. */
 static void rmpopt_scan_range(void *arg)
 {
 	u64 pa;
@@ -632,7 +631,7 @@ static void do_rmpopt_work(struct work_struct *work)
 {
 	/*
 	 * Warm up the RMPOPT cache on this pinned per-CPU worker with interrupts
-	 * on, so the IRQ-disabled fan-out below only issues cache-hit RMPOPTs.
+	 * enabled, so the IRQ-disabled fan-out below only issues cache-hit RMPOPTs.
 	 */
 	rmpopt_scan_range(NULL);
 
@@ -649,11 +648,6 @@ void snp_setup_rmpopt(void)
 
 	guard(mutex)(&rmpopt_wq_mutex);
 
-	/*
-	 * Set up once: the workqueue and RMPOPT_BASE MSRs are left in place on
-	 * shutdown, so a later re-initialization just re-queues the optimization
-	 * pass rather than redoing the setup.
-	 */
 	if (rmpopt_wq) {
 		queue_delayed_work(rmpopt_wq, &rmpopt_delayed_work, 0);
 		return;

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette