* [PATCH v9 5/6] x86/sev: Add interface to re-enable RMP optimizations.
From: Ashish Kalra @ 2026-06-24 21:58 UTC (permalink / raw)
To: tglx, mingo, bp, dave.hansen, x86, hpa, seanjc, peterz,
thomas.lendacky, herbert, davem, ardb
Cc: pbonzini, aik, Michael.Roth, KPrateek.Nayak, Tycho.Andersen,
Nathan.Fontenot, ackerleytng, jackyli, pgonda, rientjes, jacobhxu,
xin, pawan.kumar.gupta, babu.moger, dyoung, nikunj, john.allen,
darwi, linux-kernel, linux-crypto, kvm, linux-coco
In-Reply-To: <cover.1782336473.git.ashish.kalra@amd.com>
From: Ashish Kalra <ashish.kalra@amd.com>
RMPOPT table is a per-CPU table which indicates if 1GB regions of
physical memory are entirely hypervisor-owned or not.
When performing host memory accesses in hypervisor mode as well as
non-SNP guest mode, the processor may consult the RMPOPT table to
potentially skip an RMP access and improve performance.
Normal guest events clear RMP optimizations: pages are converted from
shared to private as SNP guests are launched, and large pages are split
and collapsed during guest operation -- both clear the RMPOPT
optimizations for the affected 1GB regions. Conversely, guest pages are
converted back to shared during SNP guest termination, so those regions
may become eligible for RMPOPT optimization again.
Without some intervention, all RMP optimizations would eventually be
lost. Add an interface to re-optimize all of physical memory.
The interface uses mod_delayed_work() instead of queue_delayed_work()
so that the delay timer is reset on each call. This provides proper
batching semantics: re-optimization runs 10 seconds after the *last*
VM termination rather than after the first. mod_delayed_work() also
re-queues work that is already in-flight, so a re-scan request
during an active scan is not silently dropped.
Reviewed-by: Ackerley Tng <ackerleytng@google.com>
Signed-off-by: Ashish Kalra <ashish.kalra@amd.com>
---
arch/x86/include/asm/sev.h | 2 ++
arch/x86/virt/svm/sev.c | 15 +++++++++++++++
2 files changed, 17 insertions(+)
diff --git a/arch/x86/include/asm/sev.h b/arch/x86/include/asm/sev.h
index 440c813fedde..d40beafbebb6 100644
--- a/arch/x86/include/asm/sev.h
+++ b/arch/x86/include/asm/sev.h
@@ -662,6 +662,7 @@ static inline void snp_leak_pages(u64 pfn, unsigned int pages)
__snp_leak_pages(pfn, pages, true);
}
int snp_prepare(void);
+void snp_rmpopt_all_physmem(void);
void snp_setup_rmpopt(void);
void snp_clear_rmpopt_capable(void);
void snp_disable_cpu_hotplug(void);
@@ -683,6 +684,7 @@ static inline void snp_leak_pages(u64 pfn, unsigned int npages) {}
static inline void kdump_sev_callback(void) { }
static inline void snp_fixup_e820_tables(void) {}
static inline int snp_prepare(void) { return -ENODEV; }
+static inline void snp_rmpopt_all_physmem(void) {}
static inline void snp_setup_rmpopt(void) {}
static inline void snp_clear_rmpopt_capable(void) {}
static inline void snp_disable_cpu_hotplug(void) {}
diff --git a/arch/x86/virt/svm/sev.c b/arch/x86/virt/svm/sev.c
index 5f99cbbc6cbd..4661e5271a2d 100644
--- a/arch/x86/virt/svm/sev.c
+++ b/arch/x86/virt/svm/sev.c
@@ -743,6 +743,21 @@ static void rmpopt_work_handler(struct work_struct *work)
free_cpumask_var(follower_mask);
}
+void snp_rmpopt_all_physmem(void)
+{
+ if (!cpu_feature_enabled(X86_FEATURE_RMPOPT) || !rmpopt_capable)
+ return;
+
+ guard(mutex)(&rmpopt_wq_mutex);
+
+ if (!rmpopt_wq)
+ return;
+
+ mod_delayed_work(rmpopt_wq, &rmpopt_delayed_work,
+ msecs_to_jiffies(RMPOPT_WORK_TIMEOUT));
+}
+EXPORT_SYMBOL_FOR_MODULES(snp_rmpopt_all_physmem, "kvm-amd");
+
void snp_setup_rmpopt(void)
{
u64 rmpopt_base;
--
2.43.0
^ permalink raw reply related
* [PATCH v9 4/6] x86/sev: Add support to perform RMP optimizations asynchronously
From: Ashish Kalra @ 2026-06-24 21:57 UTC (permalink / raw)
To: tglx, mingo, bp, dave.hansen, x86, hpa, seanjc, peterz,
thomas.lendacky, herbert, davem, ardb
Cc: pbonzini, aik, Michael.Roth, KPrateek.Nayak, Tycho.Andersen,
Nathan.Fontenot, ackerleytng, jackyli, pgonda, rientjes, jacobhxu,
xin, pawan.kumar.gupta, babu.moger, dyoung, nikunj, john.allen,
darwi, linux-kernel, linux-crypto, kvm, linux-coco
In-Reply-To: <cover.1782336473.git.ashish.kalra@amd.com>
From: Ashish Kalra <ashish.kalra@amd.com>
When SEV-SNP is enabled, all writes to memory are checked to ensure
integrity of SNP guest memory. This imposes performance overhead on the
whole system.
RMPOPT is a new instruction that minimizes the performance overhead of
RMP checks on the hypervisor and on non-SNP guests by allowing RMP
checks to be skipped for 1GB regions of memory that are known not to
contain any SEV-SNP guest memory.
Add support for performing RMP optimizations asynchronously using a
dedicated workqueue.
Enable RMPOPT optimizations for up to 2TB of system RAM starting from
the lowest physical memory address aligned down to a 1GB boundary at
RMP initialization time. RMP checks can initially be skipped for 1GB
memory ranges that do not contain SEV-SNP guest memory (excluding
preassigned pages such as the RMP table and firmware pages). As SNP
guests are launched, RMPUPDATE will disable the corresponding RMPOPT
optimizations.
Suggested-by: Thomas Lendacky <thomas.lendacky@amd.com>
Suggested-by: Dave Hansen <dave.hansen@linux.intel.com>
Reviewed-by: Ackerley Tng <ackerleytng@google.com>
Signed-off-by: Ashish Kalra <ashish.kalra@amd.com>
---
arch/x86/virt/svm/sev.c | 165 +++++++++++++++++++++++++++++++++++++++-
1 file changed, 162 insertions(+), 3 deletions(-)
diff --git a/arch/x86/virt/svm/sev.c b/arch/x86/virt/svm/sev.c
index 60984f76b4e9..5f99cbbc6cbd 100644
--- a/arch/x86/virt/svm/sev.c
+++ b/arch/x86/virt/svm/sev.c
@@ -19,6 +19,7 @@
#include <linux/iommu.h>
#include <linux/amd-iommu.h>
#include <linux/nospec.h>
+#include <linux/workqueue.h>
#include <asm/sev.h>
#include <asm/processor.h>
@@ -125,9 +126,20 @@ static void *rmp_bookkeeping __ro_after_init;
static u64 probed_rmp_base, probed_rmp_size;
static cpumask_var_t rmpopt_cpumask;
-static phys_addr_t rmpopt_pa_start;
+static phys_addr_t rmpopt_pa_start, rmpopt_pa_end;
static bool rmpopt_capable;
+enum rmpopt_function {
+ RMPOPT_FUNC_VERIFY_AND_REPORT_STATUS,
+ RMPOPT_FUNC_REPORT_STATUS
+};
+
+#define RMPOPT_WORK_TIMEOUT 10000
+
+static struct workqueue_struct *rmpopt_wq;
+static struct delayed_work rmpopt_delayed_work;
+static DEFINE_MUTEX(rmpopt_wq_mutex);
+
static LIST_HEAD(snp_leaked_pages_list);
static DEFINE_SPINLOCK(snp_leaked_pages_list_lock);
@@ -571,13 +583,22 @@ static void rmpopt_cleanup(void)
{
int cpu;
+ guard(mutex)(&rmpopt_wq_mutex);
+
+ if (!rmpopt_wq)
+ return;
+
+ cancel_delayed_work_sync(&rmpopt_delayed_work);
+ destroy_workqueue(rmpopt_wq);
+
scoped_guard(cpus_read_lock) {
for_each_cpu(cpu, rmpopt_cpumask)
wrmsrq_on_cpu(cpu, MSR_AMD64_RMPOPT_BASE, 0);
}
free_cpumask_var(rmpopt_cpumask);
- rmpopt_pa_start = 0;
+ rmpopt_pa_start = rmpopt_pa_end = 0;
+ rmpopt_wq = NULL;
}
/*
@@ -627,6 +648,101 @@ void snp_clear_rmpopt_capable(void)
rmpopt_capable = false;
}
+/*
+ * RMPOPT: F2 0F 01 FC
+ * Input: RAX = system physical address (1GB aligned)
+ * RCX = operation type
+ * Output: CF set if the range was optimized
+ */
+static inline bool __rmpopt(u64 pa_start, u64 op_type)
+{
+ bool optimized;
+
+ asm volatile(".byte 0xf2, 0x0f, 0x01, 0xfc"
+ : "=@ccc" (optimized)
+ : "a" (pa_start), "c" (op_type)
+ : "memory", "cc");
+
+ return optimized;
+}
+
+static void rmpopt(u64 pa)
+{
+ u64 pa_start = ALIGN_DOWN(pa, SZ_1G);
+ u64 op_type = RMPOPT_FUNC_VERIFY_AND_REPORT_STATUS;
+
+ __rmpopt(pa_start, op_type);
+}
+
+/*
+ * 'val' is a system physical address.
+ */
+static void rmpopt_smp(void *val)
+{
+ rmpopt((u64)val);
+}
+
+/*
+ * RMPOPT optimizations skip RMP checks at 1GB granularity if this
+ * range of memory does not contain any SNP guest memory.
+ */
+static void rmpopt_work_handler(struct work_struct *work)
+{
+ cpumask_var_t follower_mask;
+ phys_addr_t pa;
+ int this_cpu;
+
+ pr_info("Attempt RMP optimizations on physical address range @1GB alignment [0x%016llx - 0x%016llx]\n",
+ rmpopt_pa_start, rmpopt_pa_end);
+
+ if (!alloc_cpumask_var(&follower_mask, GFP_KERNEL))
+ return;
+
+ /*
+ * RMPOPT scans the RMP table, stores the result of the scan in the
+ * reserved processor memory. The RMP scan is the most expensive
+ * part. If a second RMPOPT occurs, it can skip the expensive scan
+ * if they can see a cached result in the reserved processor memory.
+ *
+ * Do RMPOPT on one CPU alone. Then, follow that up with RMPOPT
+ * on every other primary thread. Followers are "designed to"
+ * skip the scan if they see the "cached" scan results.
+ *
+ * Pin the worker to the current CPU for the leader loop so that
+ * this_cpu remains valid and the RMPOPT instruction executes on
+ * the correct CPU. Use migrate_disable() rather than get_cpu() to
+ * prevent migration while still allowing preemption.
+ */
+ migrate_disable();
+ this_cpu = smp_processor_id();
+
+ cpumask_andnot(follower_mask, rmpopt_cpumask,
+ topology_sibling_cpumask(this_cpu));
+
+ for (pa = rmpopt_pa_start; pa < rmpopt_pa_end; pa += SZ_1G) {
+ rmpopt(pa);
+ cond_resched();
+ }
+ migrate_enable();
+
+ /*
+ * Followers: run RMPOPT on remaining cores. CPUs cannot go offline
+ * while SNP is active, so the follower set stays valid across the
+ * scan and cpus_read_lock() is uncontended.
+ */
+ scoped_guard(cpus_read_lock) {
+ for (pa = rmpopt_pa_start; pa < rmpopt_pa_end; pa += SZ_1G) {
+ on_each_cpu_mask(follower_mask, rmpopt_smp,
+ (void *)pa, true);
+
+ /* Give a chance for other threads to run */
+ cond_resched();
+ }
+ }
+
+ free_cpumask_var(follower_mask);
+}
+
void snp_setup_rmpopt(void)
{
u64 rmpopt_base;
@@ -635,14 +751,42 @@ void snp_setup_rmpopt(void)
if (!cpu_feature_enabled(X86_FEATURE_RMPOPT) || !rmpopt_capable)
return;
+ guard(mutex)(&rmpopt_wq_mutex);
+
+ /*
+ * Guard against re-initialization. When SNP_SHUTDOWN_EX is issued
+ * with x86_snp_shutdown=0, snp_shutdown() is not called and
+ * rmpopt_cleanup() is skipped, but snp_initialized is still cleared.
+ * A subsequent __sev_snp_init_locked() would call snp_setup_rmpopt()
+ * again, leaking the existing workqueue, delayed work, and cpumask
+ * state.
+ */
+ if (rmpopt_wq)
+ return;
+
+ /*
+ * Create an RMPOPT-specific workqueue to avoid scheduling
+ * RMPOPT workitem on the global system workqueue.
+ */
+ rmpopt_wq = alloc_workqueue("rmpopt_wq", WQ_UNBOUND, 1);
+ if (!rmpopt_wq) {
+ pr_err("Failed to allocate RMPOPT workqueue\n");
+ return;
+ }
+
+ INIT_DELAYED_WORK(&rmpopt_delayed_work, rmpopt_work_handler);
+
if (!zalloc_cpumask_var(&rmpopt_cpumask, GFP_KERNEL)) {
pr_err("Failed to allocate RMPOPT cpumask\n");
+ destroy_workqueue(rmpopt_wq);
+ rmpopt_wq = NULL;
return;
}
/*
* The RMPOPT_BASE MSR is per-core, so only one thread per core needs
- * to set up the RMPOPT_BASE MSR.
+ * to set up the RMPOPT_BASE MSR. Likewise, only one thread per core
+ * needs to issue the RMPOPT instruction.
*
* Note: only online primary threads are included. If a core's
* primary thread is offline, that core is not covered. CPU hotplug
@@ -665,6 +809,21 @@ void snp_setup_rmpopt(void)
for_each_cpu(cpu, rmpopt_cpumask)
wrmsrq_on_cpu(cpu, MSR_AMD64_RMPOPT_BASE, rmpopt_base);
}
+
+ rmpopt_pa_end = ALIGN(PFN_PHYS(max_pfn), SZ_1G);
+
+ /* Limit memory scanning to 2TB of RAM */
+ if ((rmpopt_pa_end - rmpopt_pa_start) > SZ_2T) {
+ pr_info("RMPOPT coverage limited to 2TB; memory above 0x%llx not optimized\n",
+ rmpopt_pa_start + SZ_2T);
+ rmpopt_pa_end = rmpopt_pa_start + SZ_2T;
+ }
+
+ /*
+ * Once all per-CPU RMPOPT tables have been configured, enable RMPOPT
+ * optimizations on all physical memory.
+ */
+ queue_delayed_work(rmpopt_wq, &rmpopt_delayed_work, 0);
}
EXPORT_SYMBOL_FOR_MODULES(snp_setup_rmpopt, "ccp");
--
2.43.0
^ permalink raw reply related
* [PATCH v9 3/6] x86/sev: Disable CPU hotplug while SNP is active
From: Ashish Kalra @ 2026-06-24 21:56 UTC (permalink / raw)
To: tglx, mingo, bp, dave.hansen, x86, hpa, seanjc, peterz,
thomas.lendacky, herbert, davem, ardb
Cc: pbonzini, aik, Michael.Roth, KPrateek.Nayak, Tycho.Andersen,
Nathan.Fontenot, ackerleytng, jackyli, pgonda, rientjes, jacobhxu,
xin, pawan.kumar.gupta, babu.moger, dyoung, nikunj, john.allen,
darwi, linux-kernel, linux-crypto, kvm, linux-coco
In-Reply-To: <cover.1782336473.git.ashish.kalra@amd.com>
From: Ashish Kalra <ashish.kalra@amd.com>
While SNP is active, every memory write is checked against the RMP to
protect the integrity of SEV-SNP guest memory. By the SNP architecture
these checks cannot be disabled on a subset of CPUs: they are gated
per-core by SYSCFG[SNP_EN], which the SEV firmware requires to be set on
every present CPU before SNP initialization. A CPU that does not have
SNP_EN set and was not initialized via SNP_INIT performs no RMP checks at
all, so there is no valid configuration with SNP active and any CPU exempt
from RMP checks.
The firmware determines which CPUs are present from the processor and the
BIOS/UEFI configuration (e.g. SMT disabled in the BIOS) and enumerates
them at SNP init; it is not aware of the OS bringing CPUs online or
offline afterwards. A CPU brought online after SNP init was not
enumerated at SNP_INIT and does not have SNP_EN set, so writes from it are
not RMP-checked and could corrupt SEV-SNP guest memory, and there is no
way to keep work off such a CPU once it is online. OS CPU hotplug can thus
diverge from the firmware's expectations and break SNP. Disable CPU
hotplug while SNP is active.
Use cpu_hotplug_disable() at SNP init and cpu_hotplug_enable() only on the
full x86_snp_shutdown path; the legacy SNP_SHUTDOWN_EX path leaves SNP
active and must keep hotplug disabled. A flag in built-in SNP code keeps
the disable balanced across the teardown paths, re-init and kexec, and
survives a ccp module reload.
This also keeps the CPU set stable for the asynchronous RMPOPT scan added
later in this series, and ensures cpus_read_lock() in the scan is
uncontended.
Suggested-by: Thomas Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Ashish Kalra <ashish.kalra@amd.com>
---
arch/x86/include/asm/sev.h | 2 ++
arch/x86/virt/svm/sev.c | 30 ++++++++++++++++++++++++++++++
drivers/crypto/ccp/sev-dev.c | 3 +++
3 files changed, 35 insertions(+)
diff --git a/arch/x86/include/asm/sev.h b/arch/x86/include/asm/sev.h
index 0243989f229b..440c813fedde 100644
--- a/arch/x86/include/asm/sev.h
+++ b/arch/x86/include/asm/sev.h
@@ -664,6 +664,7 @@ static inline void snp_leak_pages(u64 pfn, unsigned int pages)
int snp_prepare(void);
void snp_setup_rmpopt(void);
void snp_clear_rmpopt_capable(void);
+void snp_disable_cpu_hotplug(void);
void snp_shutdown(void);
#else
static inline bool snp_probe_rmptable_info(void) { return false; }
@@ -684,6 +685,7 @@ static inline void snp_fixup_e820_tables(void) {}
static inline int snp_prepare(void) { return -ENODEV; }
static inline void snp_setup_rmpopt(void) {}
static inline void snp_clear_rmpopt_capable(void) {}
+static inline void snp_disable_cpu_hotplug(void) {}
static inline void snp_shutdown(void) {}
#endif
diff --git a/arch/x86/virt/svm/sev.c b/arch/x86/virt/svm/sev.c
index dab6e1c290bc..60984f76b4e9 100644
--- a/arch/x86/virt/svm/sev.c
+++ b/arch/x86/virt/svm/sev.c
@@ -133,6 +133,9 @@ static DEFINE_SPINLOCK(snp_leaked_pages_list_lock);
static unsigned long snp_nr_leaked_pages;
+/* Set while SNP has CPU hotplug disabled (kernel-lifetime; survives ccp reload). */
+static bool snp_cpu_hotplug_disabled;
+
#undef pr_fmt
#define pr_fmt(fmt) "SEV-SNP: " fmt
@@ -577,6 +580,22 @@ static void rmpopt_cleanup(void)
rmpopt_pa_start = 0;
}
+/*
+ * Disable CPU hotplug while SNP is active. Applied once per SNP-active
+ * window and balanced by cpu_hotplug_enable() in snp_shutdown().
+ * The legacy SNP_SHUTDOWN_EX path leaves SNP enabled without re-enabling
+ * hotplug, so a re-init while SNP is still active must not stack the
+ * disable count.
+ */
+void snp_disable_cpu_hotplug(void)
+{
+ if (!snp_cpu_hotplug_disabled) {
+ cpu_hotplug_disable();
+ snp_cpu_hotplug_disabled = true;
+ }
+}
+EXPORT_SYMBOL_FOR_MODULES(snp_disable_cpu_hotplug, "ccp");
+
void snp_shutdown(void)
{
u64 syscfg;
@@ -587,6 +606,17 @@ void snp_shutdown(void)
rmpopt_cleanup();
+ /*
+ * Re-enable CPU hotplug now that SNP is fully shut down. Done here
+ * (x86_snp_shutdown path) only -- the legacy path leaves SNP active
+ * and must keep hotplug disabled. After rmpopt_cleanup() so the
+ * per-core RMPOPT_BASE MSRs are cleared with hotplug still disabled.
+ */
+ if (snp_cpu_hotplug_disabled) {
+ cpu_hotplug_enable();
+ snp_cpu_hotplug_disabled = false;
+ }
+
clear_rmp();
on_each_cpu(mfd_reconfigure, NULL, 1);
}
diff --git a/drivers/crypto/ccp/sev-dev.c b/drivers/crypto/ccp/sev-dev.c
index 217b6b19802e..66475145b3fa 100644
--- a/drivers/crypto/ccp/sev-dev.c
+++ b/drivers/crypto/ccp/sev-dev.c
@@ -1479,6 +1479,9 @@ static int __sev_snp_init_locked(int *error, unsigned int max_snp_asid)
snp_hv_fixed_pages_state_update(sev, HV_FIXED);
+ /* Disable CPU hotplug while SNP is active (see snp_disable_cpu_hotplug). */
+ snp_disable_cpu_hotplug();
+
snp_setup_rmpopt();
sev->snp_initialized = true;
--
2.43.0
^ permalink raw reply related
* [PATCH v9 2/6] x86/sev: Initialize RMPOPT configuration MSRs
From: Ashish Kalra @ 2026-06-24 21:56 UTC (permalink / raw)
To: tglx, mingo, bp, dave.hansen, x86, hpa, seanjc, peterz,
thomas.lendacky, herbert, davem, ardb
Cc: pbonzini, aik, Michael.Roth, KPrateek.Nayak, Tycho.Andersen,
Nathan.Fontenot, ackerleytng, jackyli, pgonda, rientjes, jacobhxu,
xin, pawan.kumar.gupta, babu.moger, dyoung, nikunj, john.allen,
darwi, linux-kernel, linux-crypto, kvm, linux-coco
In-Reply-To: <cover.1782336473.git.ashish.kalra@amd.com>
From: Ashish Kalra <ashish.kalra@amd.com>
The new RMPOPT instruction helps manage per-CPU RMP optimization
structures inside the CPU. It takes a 1GB-aligned physical address
and either returns the status of the optimizations or tries to enable
the optimizations.
Per-CPU RMPOPT tables support at most 2 TB of addressable memory for
RMP optimizations.
Initialize the per-CPU RMPOPT table base to the starting physical
address. This enables RMP optimization for up to 2 TB of system RAM on
all CPUs.
Additionally, add support to setup and enable RMPOPT once SNP is
enabled and initialized.
Suggested-by: Thomas Lendacky <thomas.lendacky@amd.com>
Suggested-by: Dave Hansen <dave.hansen@linux.intel.com>
Reviewed-by: Dave Hansen <dave.hansen@linux.intel.com>
Signed-off-by: Ashish Kalra <ashish.kalra@amd.com>
---
arch/x86/coco/core.c | 2 +
arch/x86/include/asm/msr-index.h | 3 ++
arch/x86/include/asm/sev.h | 4 ++
arch/x86/virt/svm/sev.c | 70 ++++++++++++++++++++++++++++++++
drivers/crypto/ccp/sev-dev.c | 3 ++
5 files changed, 82 insertions(+)
diff --git a/arch/x86/coco/core.c b/arch/x86/coco/core.c
index 989ca9f72ba3..f0ed6c62d86c 100644
--- a/arch/x86/coco/core.c
+++ b/arch/x86/coco/core.c
@@ -16,6 +16,7 @@
#include <asm/archrandom.h>
#include <asm/coco.h>
#include <asm/processor.h>
+#include <asm/sev.h>
enum cc_vendor cc_vendor __ro_after_init = CC_VENDOR_NONE;
SYM_PIC_ALIAS(cc_vendor);
@@ -172,6 +173,7 @@ static void amd_cc_platform_clear(enum cc_attr attr)
switch (attr) {
case CC_ATTR_HOST_SEV_SNP:
cc_flags.host_sev_snp = 0;
+ snp_clear_rmpopt_capable();
break;
default:
break;
diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-index.h
index 86554de9a3f5..28540744f1eb 100644
--- a/arch/x86/include/asm/msr-index.h
+++ b/arch/x86/include/asm/msr-index.h
@@ -761,6 +761,9 @@
#define MSR_AMD64_SEG_RMP_ENABLED_BIT 0
#define MSR_AMD64_SEG_RMP_ENABLED BIT_ULL(MSR_AMD64_SEG_RMP_ENABLED_BIT)
#define MSR_AMD64_RMP_SEGMENT_SHIFT(x) (((x) & GENMASK_ULL(13, 8)) >> 8)
+#define MSR_AMD64_RMPOPT_BASE 0xc0010139
+#define MSR_AMD64_RMPOPT_ENABLE_BIT 0
+#define MSR_AMD64_RMPOPT_ENABLE BIT_ULL(MSR_AMD64_RMPOPT_ENABLE_BIT)
#define MSR_SVSM_CAA 0xc001f000
diff --git a/arch/x86/include/asm/sev.h b/arch/x86/include/asm/sev.h
index 594cfa19cbd4..0243989f229b 100644
--- a/arch/x86/include/asm/sev.h
+++ b/arch/x86/include/asm/sev.h
@@ -662,6 +662,8 @@ static inline void snp_leak_pages(u64 pfn, unsigned int pages)
__snp_leak_pages(pfn, pages, true);
}
int snp_prepare(void);
+void snp_setup_rmpopt(void);
+void snp_clear_rmpopt_capable(void);
void snp_shutdown(void);
#else
static inline bool snp_probe_rmptable_info(void) { return false; }
@@ -680,6 +682,8 @@ static inline void snp_leak_pages(u64 pfn, unsigned int npages) {}
static inline void kdump_sev_callback(void) { }
static inline void snp_fixup_e820_tables(void) {}
static inline int snp_prepare(void) { return -ENODEV; }
+static inline void snp_setup_rmpopt(void) {}
+static inline void snp_clear_rmpopt_capable(void) {}
static inline void snp_shutdown(void) {}
#endif
diff --git a/arch/x86/virt/svm/sev.c b/arch/x86/virt/svm/sev.c
index 8bcdce98f6dc..dab6e1c290bc 100644
--- a/arch/x86/virt/svm/sev.c
+++ b/arch/x86/virt/svm/sev.c
@@ -124,6 +124,10 @@ static void *rmp_bookkeeping __ro_after_init;
static u64 probed_rmp_base, probed_rmp_size;
+static cpumask_var_t rmpopt_cpumask;
+static phys_addr_t rmpopt_pa_start;
+static bool rmpopt_capable;
+
static LIST_HEAD(snp_leaked_pages_list);
static DEFINE_SPINLOCK(snp_leaked_pages_list_lock);
@@ -490,6 +494,11 @@ static bool __init setup_rmptable(void)
if (rmp_cfg & MSR_AMD64_SEG_RMP_ENABLED) {
if (!setup_segmented_rmptable())
return false;
+ /*
+ * RMPOPT requires a segmented RMP, so indicate that the
+ * system is capable of configuring and running RMPOPT.
+ */
+ rmpopt_capable = true;
} else {
if (!setup_contiguous_rmptable())
return false;
@@ -555,6 +564,19 @@ int snp_prepare(void)
}
EXPORT_SYMBOL_FOR_MODULES(snp_prepare, "ccp");
+static void rmpopt_cleanup(void)
+{
+ int cpu;
+
+ scoped_guard(cpus_read_lock) {
+ for_each_cpu(cpu, rmpopt_cpumask)
+ wrmsrq_on_cpu(cpu, MSR_AMD64_RMPOPT_BASE, 0);
+ }
+
+ free_cpumask_var(rmpopt_cpumask);
+ rmpopt_pa_start = 0;
+}
+
void snp_shutdown(void)
{
u64 syscfg;
@@ -563,11 +585,59 @@ void snp_shutdown(void)
if (syscfg & MSR_AMD64_SYSCFG_SNP_EN)
return;
+ rmpopt_cleanup();
+
clear_rmp();
on_each_cpu(mfd_reconfigure, NULL, 1);
}
EXPORT_SYMBOL_FOR_MODULES(snp_shutdown, "ccp");
+void snp_clear_rmpopt_capable(void)
+{
+ rmpopt_capable = false;
+}
+
+void snp_setup_rmpopt(void)
+{
+ u64 rmpopt_base;
+ int cpu;
+
+ if (!cpu_feature_enabled(X86_FEATURE_RMPOPT) || !rmpopt_capable)
+ return;
+
+ if (!zalloc_cpumask_var(&rmpopt_cpumask, GFP_KERNEL)) {
+ pr_err("Failed to allocate RMPOPT cpumask\n");
+ return;
+ }
+
+ /*
+ * The RMPOPT_BASE MSR is per-core, so only one thread per core needs
+ * to set up the RMPOPT_BASE MSR.
+ *
+ * Note: only online primary threads are included. If a core's
+ * primary thread is offline, that core is not covered. CPU hotplug
+ * is not currently supported with SNP enabled.
+ */
+ scoped_guard(cpus_read_lock) {
+ for_each_online_cpu(cpu)
+ if (topology_is_primary_thread(cpu))
+ cpumask_set_cpu(cpu, rmpopt_cpumask);
+
+ rmpopt_pa_start = ALIGN_DOWN(PFN_PHYS(min_low_pfn), SZ_1G);
+ rmpopt_base = rmpopt_pa_start | MSR_AMD64_RMPOPT_ENABLE;
+
+ /*
+ * Per-CPU RMPOPT tables support at most 2 TB of addressable memory
+ * for RMP optimizations. Initialize the per-CPU RMPOPT table base
+ * to the starting physical address to enable RMP optimizations for
+ * up to 2 TB of system RAM on all CPUs.
+ */
+ for_each_cpu(cpu, rmpopt_cpumask)
+ wrmsrq_on_cpu(cpu, MSR_AMD64_RMPOPT_BASE, rmpopt_base);
+ }
+}
+EXPORT_SYMBOL_FOR_MODULES(snp_setup_rmpopt, "ccp");
+
/*
* Do the necessary preparations which are verified by the firmware as
* described in the SNP_INIT_EX firmware command description in the SNP
diff --git a/drivers/crypto/ccp/sev-dev.c b/drivers/crypto/ccp/sev-dev.c
index 78f98aee7a66..217b6b19802e 100644
--- a/drivers/crypto/ccp/sev-dev.c
+++ b/drivers/crypto/ccp/sev-dev.c
@@ -1478,6 +1478,9 @@ static int __sev_snp_init_locked(int *error, unsigned int max_snp_asid)
}
snp_hv_fixed_pages_state_update(sev, HV_FIXED);
+
+ snp_setup_rmpopt();
+
sev->snp_initialized = true;
dev_dbg(sev->dev, "SEV-SNP firmware initialized, SEV-TIO is %s\n",
data.tio_en ? "enabled" : "disabled");
--
2.43.0
^ permalink raw reply related
* [PATCH v9 1/6] x86/cpufeatures: Add X86_FEATURE_RMPOPT feature flag
From: Ashish Kalra @ 2026-06-24 21:56 UTC (permalink / raw)
To: tglx, mingo, bp, dave.hansen, x86, hpa, seanjc, peterz,
thomas.lendacky, herbert, davem, ardb
Cc: pbonzini, aik, Michael.Roth, KPrateek.Nayak, Tycho.Andersen,
Nathan.Fontenot, ackerleytng, jackyli, pgonda, rientjes, jacobhxu,
xin, pawan.kumar.gupta, babu.moger, dyoung, nikunj, john.allen,
darwi, linux-kernel, linux-crypto, kvm, linux-coco
In-Reply-To: <cover.1782336473.git.ashish.kalra@amd.com>
From: Ashish Kalra <ashish.kalra@amd.com>
Add a flag indicating whether RMPOPT instruction is supported.
RMPOPT is a new instruction that reduces the performance overhead of
RMP checks for the hypervisor and non-SNP guests by allowing those
checks to be skipped when 1-GB memory regions are known to contain no
SEV-SNP guest memory.
For more information on the RMPOPT instruction, see the AMD64 RMPOPT
technical documentation.
Suggested-by: Borislav Petkov (AMD) <bp@alien8.de>
Reviewed-by: Dave Hansen <dave.hansen@linux.intel.com>
Reviewed-by: Ackerley Tng <ackerleytng@google.com>
Signed-off-by: Ashish Kalra <ashish.kalra@amd.com>
---
arch/x86/include/asm/cpufeatures.h | 2 +-
arch/x86/kernel/cpu/scattered.c | 1 +
tools/arch/x86/include/asm/cpufeatures.h | 2 +-
3 files changed, 3 insertions(+), 2 deletions(-)
diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h
index 1d506e5d6f46..794cc96b8493 100644
--- a/arch/x86/include/asm/cpufeatures.h
+++ b/arch/x86/include/asm/cpufeatures.h
@@ -76,7 +76,7 @@
#define X86_FEATURE_K8 ( 3*32+ 4) /* Opteron, Athlon64 */
#define X86_FEATURE_ZEN5 ( 3*32+ 5) /* CPU based on Zen5 microarchitecture */
#define X86_FEATURE_ZEN6 ( 3*32+ 6) /* CPU based on Zen6 microarchitecture */
-/* Free ( 3*32+ 7) */
+#define X86_FEATURE_RMPOPT ( 3*32+ 7) /* Support for AMD RMPOPT instruction */
#define X86_FEATURE_CONSTANT_TSC ( 3*32+ 8) /* "constant_tsc" TSC ticks at a constant rate */
#define X86_FEATURE_UP ( 3*32+ 9) /* "up" SMP kernel running on UP */
#define X86_FEATURE_ART ( 3*32+10) /* "art" Always running timer (ART) */
diff --git a/arch/x86/kernel/cpu/scattered.c b/arch/x86/kernel/cpu/scattered.c
index 937129ce6a96..021c0bf22de2 100644
--- a/arch/x86/kernel/cpu/scattered.c
+++ b/arch/x86/kernel/cpu/scattered.c
@@ -67,6 +67,7 @@ static const struct cpuid_bit cpuid_bits[] = {
{ X86_FEATURE_PERFMON_V2, CPUID_EAX, 0, 0x80000022, 0 },
{ X86_FEATURE_AMD_LBR_V2, CPUID_EAX, 1, 0x80000022, 0 },
{ X86_FEATURE_AMD_LBR_PMC_FREEZE, CPUID_EAX, 2, 0x80000022, 0 },
+ { X86_FEATURE_RMPOPT, CPUID_EDX, 0, 0x80000025, 0 },
{ X86_FEATURE_AMD_HTR_CORES, CPUID_EAX, 30, 0x80000026, 0 },
{ 0, 0, 0, 0, 0 }
};
diff --git a/tools/arch/x86/include/asm/cpufeatures.h b/tools/arch/x86/include/asm/cpufeatures.h
index 86d17b195e79..7ce681af1dd7 100644
--- a/tools/arch/x86/include/asm/cpufeatures.h
+++ b/tools/arch/x86/include/asm/cpufeatures.h
@@ -76,7 +76,7 @@
#define X86_FEATURE_K8 ( 3*32+ 4) /* Opteron, Athlon64 */
#define X86_FEATURE_ZEN5 ( 3*32+ 5) /* CPU based on Zen5 microarchitecture */
#define X86_FEATURE_ZEN6 ( 3*32+ 6) /* CPU based on Zen6 microarchitecture */
-/* Free ( 3*32+ 7) */
+#define X86_FEATURE_RMPOPT ( 3*32+ 7) /* Support for AMD RMPOPT instruction */
#define X86_FEATURE_CONSTANT_TSC ( 3*32+ 8) /* "constant_tsc" TSC ticks at a constant rate */
#define X86_FEATURE_UP ( 3*32+ 9) /* "up" SMP kernel running on UP */
#define X86_FEATURE_ART ( 3*32+10) /* "art" Always running timer (ART) */
--
2.43.0
^ permalink raw reply related
* [PATCH v9 0/6] Add RMPOPT support.
From: Ashish Kalra @ 2026-06-24 21:55 UTC (permalink / raw)
To: tglx, mingo, bp, dave.hansen, x86, hpa, seanjc, peterz,
thomas.lendacky, herbert, davem, ardb
Cc: pbonzini, aik, Michael.Roth, KPrateek.Nayak, Tycho.Andersen,
Nathan.Fontenot, ackerleytng, jackyli, pgonda, rientjes, jacobhxu,
xin, pawan.kumar.gupta, babu.moger, dyoung, nikunj, john.allen,
darwi, linux-kernel, linux-crypto, kvm, linux-coco
From: Ashish Kalra <ashish.kalra@amd.com>
In the SEV-SNP architecture, hypervisor and non-SNP guests are subject
to RMP checks on writes to provide integrity of SEV-SNP guest memory.
The RMPOPT architecture enables optimizations whereby the RMP checks
can be skipped if 1GB regions of memory are known to not contain any
SNP guest memory.
RMPOPT is a new instruction designed to minimize the performance
overhead of RMP checks for the hypervisor and non-SNP guests.
RMPOPT instruction currently supports two functions. In case of the
verify and report status function the CPU will read the RMP contents,
verify the entire 1GB region starting at the provided SPA is HV-owned.
For the entire 1GB region it checks that all RMP entries in this region
are HV-owned (i.e, not in assigned state) and then accordingly updates
the RMPOPT table to indicate if optimization has been enabled and
provide indication to software if the optimization was successful.
In case of report status function, the CPU returns the optimization
status for the 1GB region.
The RMPOPT table is managed by a combination of software and hardware.
Software uses the RMPOPT instruction to set bits in the table,
indicating that regions of memory are entirely HV-owned. Hardware
automatically clears bits in the RMPOPT table when RMP contents are
changed during RMPUPDATE instruction.
For more information on the RMPOPT instruction, see the AMD64 RMPOPT
technical documentation.
As SNP is enabled by default the hypervisor and non-SNP guests are
subject to RMP write checks to provide integrity of SNP guest memory.
This patch-series adds support to enable RMP optimizations for up to
2TB of system RAM across the system and allow RMPUPDATE to disable
those optimizations as SNP guests are launched.
Support for RAM larger than 2 TB will be added in follow-on series.
This series also adds support to disable CPU hotplug while SNP is
active, as the SEV firmware enumerates CPUs at SNP initialization and is
not aware of the OS bringing CPUs online or offline afterwards. This
also keeps the set of CPUs stable for the asynchronous RMPOPT scan, so
the per-core RMPOPT_BASE MSRs programmed during setup remain valid.
This series also introduces support to re-enable RMP optimizations
during SNP guest termination, after guest pages have been converted
back to shared.
RMP optimizations are performed asynchronously by queuing work on a
dedicated workqueue after a 10 second delay.
Delaying work allows batching of multiple SNP guest terminations.
Once 1GB hugetlb guest_memfd support is merged, support for
re-enabling RMPOPT optimizations during 1GB page cleanup will be added
in follow-on series.
v9:
- Rename rmpopt_configured to rmpopt_capable.
- Make rmpopt_cpumask a cpumask_var_t (allocated/freed at setup/cleanup)
instead of a static cpumask_t.
- Drop the v8 WARN_ON_ONCE() on the RMPOPT_BASE writes; use a plain
wrmsrq_on_cpu(), matching the SNP MSR-write convention in this file.
- Disable CPU hotplug with cpu_hotplug_disable()/cpu_hotplug_enable()
(per tglx); re-enable only on the full x86_snp_shutdown path.
- Simplify rmpopt_work_handler() to a single leader-then-followers path:
with CPU hotplug disabled while SNP is active and snp_prepare()
requiring all CPUs online when RMPOPT_BASE is programmed, every core is
always programmed, so the explicit-leader fallback is now unreachable.
Drop it along with the v8 work_on_cpu()/rmpopt_leader_fn() helper.
- Drop the debugfs interface (was patch 7/7) and its report-only
plumbing; observability will be revisited after this series is merged.
- Restrict snp_rmpopt_all_physmem()'s export to the kvm-amd module.
- Use scoped_guard(cpus_read_lock) for the per-CPU MSR and follower
loops.
Sashiko AI upstream review identified several of the above issues.
v8:
- Add a new patch to disable CPU hotplug while SNP is active, keeping
the CPU set stable for the RMPOPT work handler.
- Drop the setup_clear_cpu_cap(X86_FEATURE_RMPOPT) calls; the
rmpopt_configured bool is the runtime guard.
- WARN_ON_ONCE() on the RMPOPT_BASE MSR writes that previously ignored
their return value.
- Simplify rmpopt_work_handler() by removing the explicit-leader
fallback: with CPU hotplug disabled while SNP is active and
snp_prepare() requiring all CPUs online when RMPOPT_BASE is programmed,
every core is always programmed, so the running CPU can always be the
leader. This drops the smp_call_function_single() fallback (and with
it the AB-BA deadlock and IRQ-latency concerns) and collapses the
leader selection into a single leader-then-followers path.
- Use mod_delayed_work() in snp_rmpopt_all_physmem() so the batching
delay tracks the last SNP guest termination.
Sashiko AI code review identified several of the above issues.
v7:
- Sync tools/arch/x86/include/asm/cpufeatures.h to mirror the kernel
header for X86_FEATURE_RMPOPT.
- Fix commit title to use X86_FEATURE_RMPOPT to match the code
(was X86_FEATURE_AMD_RMPOPT).
- Add static bool rmpopt_configured, set only when segmented RMP setup
succeeds in setup_rmptable(). Check rmpopt_configured alongside
cpu_feature_enabled(X86_FEATURE_RMPOPT) in snp_setup_rmpopt() and
snp_rmpopt_all_physmem(), because setup_clear_cpu_cap() is unreliable
after alternatives are patched. Add snp_clear_rmpopt_configured()
called from amd_cc_platform_clear() when CC_ATTR_HOST_SEV_SNP is
cleared. Do not use __ro_after_init on rmpopt_configured since the
writer snp_clear_rmpopt_configured() is not __init.
- Add cond_resched() to all three leader loops in rmpopt_work_handler()
to prevent soft lockups on systems with up to 2TB of RAM.
- Add comment above __rmpopt() documenting the RMPOPT instruction
encoding (F2 0F 01 FC) and register interface (RAX = system physical
address input, RCX = operation type input, RFLAGS.CF = output).
Note: RMPOPT does not modify RAX unlike PVALIDATE/RMPUPDATE, so
the existing "a" (input-only) constraint is correct.
Sashiko AI code review identified several of the above issues.
v6:
- Drop wrmsrq_on_cpus() helper; use for_each_cpu() with wrmsrq_on_cpu()
instead, as RMPOPT_BASE MSR programming is not performance-critical.
- Rewrite rmpopt_work_handler() leader selection to use a local
follower_mask copy instead of modifying the global rmpopt_cpumask.
This eliminates the current_cpu_cleared tracking and the restore at
the end, and removes the need for synchronization comments about
transient cpumask inconsistency.
- Add three-way leader selection in rmpopt_work_handler():
1. Current CPU is a primary thread in cpumask: run leader locally.
2. Current CPU is a sibling thread whose primary is in cpumask:
run leader locally (RMPOPT_BASE MSR is per-core), remove the
primary from followers via cpumask_andnot(topology_sibling_cpumask).
3. Current CPU's core has no RMPOPT_BASE MSR programmed: pick an
explicit leader via cpumask_first() + smp_call_function_single()
to avoid #UD, with cpus_read_lock() around the IPI loop.
- Add WARN_ON_ONCE guard for empty cpumask in the explicit leader
fallback path, with migrate_enable() before goto out.
- Add .llseek = seq_lseek to rmpopt_table_fops for consistency with
other seq_file-based debugfs files and to support tools like "less".
- Change debugfs file permissions from 0444 to 0400 to restrict access
to root only.
- Add comment in rmpopt_table_seq_show() explaining why cpu_online_mask
is safe: RMPOPT_BASE MSR is per-core and snp_prepare() ensures all
CPUs are online when the MSR is programmed.
Sashiko AI code review identified several of the above issues.
v5:
- Introduce rmpopt_cleanup() to tear down workqueue, debugfs, cpumask,
and MSR state, called from snp_shutdown().
- Introduce rmpopt_wq_mutex to serialize snp_setup_rmpopt(),
snp_rmpopt_all_physmem(), and rmpopt_cleanup().
- Introduce rmpopt_show_mutex to serialize debugfs reporting of
rmpopt_report_cpumask.
- Move snp_rmpopt_all_physmem() call after SNP DECOMMISSION during
guest shutdown.
- Use migrate_disable()/migrate_enable() for CPU pinning in the
rmpopt_work_handler() leader loop to maintain CPU affinity without
disabling preemption for the entire RMPOPT scan.
- Add cpus_read_lock()/cpus_read_unlock() around the follower
on_each_cpu_mask() loop in rmpopt_work_handler().
- Guard snp_setup_rmpopt() against re-initialization when
SNP_SHUTDOWN_EX with x86_snp_shutdown=0 skips rmpopt_cleanup()
but clears snp_initialized, preventing workqueue and resource
leaks on repeated init/shutdown cycles.
- Replace setup_clear_cpu_cap() with pr_err() on alloc_workqueue()
failure in snp_setup_rmpopt(), as setup_clear_cpu_cap() cannot be
used after alternatives are patched; callers check rmpopt_wq != NULL
as the runtime guard instead.
- Add pr_info() when RMPOPT coverage is capped at 2TB.
- Add comments noting CPU hotplug is not supported with SNP enabled
and only online primary threads are covered by rmpopt_cpumask.
- Add comment in setup_rmptable() noting Segmented RMP must be
enabled to enable RMPOPT.
- Simplify cpumask setup loop to set if primary thread rather than
skip if not primary.
- Improve grammar and clarity in snp_setup_rmpopt() comments.
- Added Reviewed-by's.
Sashiko AI code review identified several of the above issues.
v4:
- Add new wrmsrq_on_cpus() helper to write same u64 value to a
per-CPU MSR across a cpumask without per-cpu struct allocation
overhead.
- Rename configure_and_enable_rmpopt() to snp_setup_rmpopt().
- Use wrmsrq_on_cpus() instead of wrmsrq_on_cpu() loop for
programming RMPOPT_BASE MSRs.
- Add setup_clear_cpu_cap(X86_FEATURE_RMPOPT) if segmented RMP
setup fails or workqueue allocation fails.
- Add X86_FEATURE_RMPOPT feature clear logic in amd_cc_platform_clear()
for CC_ATTR_HOST_SEV_SNP.
- All of the above allow checking for only X86_FEATURE_RMPOPT for both
RMPOPT setup/enable and RMP re-optimizations.
- Rename snp_perform_rmp_optimization() to snp_rmpopt_all_physmem().
- Split rmpopt() into rmpopt() and rmpopt_smp() for SMP callback use.
- Introduce separate rmpopt_report_cpumask for debugfs reporting,
distinct from rmpopt_cpumask used for primary thread tracking.
- Remove snp_perform_rmp_optimization() call from __sev_snp_init_locked()
and instead setup and enable RMPOPT after SNP is enabled and
initialized.
v3:
- Drop all RMPOPT kthread support and introduce adding custom and
dedicated workqueue to schedule delayed and asynchronous RMPOPT work.
- Drop the guest_memfd inode cleanup interface and add support to
re-enable RMP optimizations during guest shutdown using the
asynchronous and delayed workqueue interface.
- Introduce new __rmpopt() helper and rmpopt() and
rmpopt_report_status() wrappers on top which use rax and rcx
parameters to closely match RMPOPT specs.
- Use new optimized RMPOPT loop to issue RMPOPT instructions on all
system RAM upto 2TB and all CPUs, by optimizing each range on one CPU
first, then let other CPUs execute RMPOPT in parallel so they can skip
most work as the range has already been optimized.
- Also add support for running the optimized RMPOPT loop only on
one thread per core.
- Replace all PUD_SIZE references with SZ_1G to conform to 1GB regions
as specified by RMPOPT specifications and not be dependent on PUD_SIZE
which makes the RMPOPT patch-set independent of x86 page table sizes.
- Use wrmsrq_on_cpu() to program the RMPOPT_BASE MSR registers on
all CPUs that removes all ugly casting to use on_each_cpu_mask().
- Fix inline commits and patch commit messages
v2:
- Drop all NUMA and Socket configuration and enablement support and
enable RMPOPT support for up to 2TB of system RAM.
- Drop get_cpumask_of_primary_threads() and enable per-core RMPOPT
base MSRs and issue RMPOPT instruction on all CPUs.
- Drop the configfs interface to manually re-enable RMP optimizations.
- Add new guest_memfd cleanup interface to automatically re-enable
RMP optimizations during guest shutdown.
- Include references to the public RMPOPT documentation.
- Move debugfs directory for RMPOPT under architecuture specific
parent directory.
Ashish Kalra (6):
x86/cpufeatures: Add X86_FEATURE_RMPOPT feature flag
x86/sev: Initialize RMPOPT configuration MSRs
x86/sev: Disable CPU hotplug while SNP is active
x86/sev: Add support to perform RMP optimizations asynchronously
x86/sev: Add interface to re-enable RMP optimizations.
KVM: SEV: Perform RMP optimizations on SNP guest shutdown
arch/x86/coco/core.c | 2 +
arch/x86/include/asm/cpufeatures.h | 2 +-
arch/x86/include/asm/msr-index.h | 3 +
arch/x86/include/asm/sev.h | 8 +
arch/x86/kernel/cpu/scattered.c | 1 +
arch/x86/kvm/svm/sev.c | 10 +
arch/x86/virt/svm/sev.c | 274 +++++++++++++++++++++++
drivers/crypto/ccp/sev-dev.c | 6 +
tools/arch/x86/include/asm/cpufeatures.h | 2 +-
9 files changed, 306 insertions(+), 2 deletions(-)
--
2.43.0
^ permalink raw reply
* Re: [PATCH v4 0/3] crypto: skcipher - per-request multi-data-unit batching
From: Leonid Ravich @ 2026-06-24 19:52 UTC (permalink / raw)
To: Eric Biggers, Herbert Xu
Cc: Alasdair Kergon, Ard Biesheuvel, Jens Axboe, dm-devel,
linux-block, linux-crypto
In-Reply-To: <20260622182328.GB1250822@google.com>
On Mon, Jun 22, 2026 at 06:23:28PM +0000, Eric Biggers wrote:
> I don't think there's a path forward without an in-tree user that's
> shown to be worthwhile over just using the acceleration built directly
> into the CPU. As well as confirmation of no regression to existing
> users, including in cases where the inline sg list can't be used.
Agreed. Proposing a smaller v5 that meets the no-regression bar now and
leaves "beats the CPU" to a follow-up with a real in-tree user.
dm-crypt submits one request per contiguous bio segment (a single
bio_vec) with data_unit_size = sector_size, instead of one per sector.
E.g. default sector_size 512 with a 4 KiB bio_vec: one request of 8
data units, which the fallback splitter walks as 8 per-sector calls --
dm-crypt no longer open-codes the per-data-unit loop itself.
- Uses only the existing inline sg_in[0]/sg_out[0] entry. No per-bio
scatterlist, no kmalloc -- the "inline sg list can't be used" case
doesn't exist here, so there's nothing to regress.
- For a non-native algorithm the core auto-splits into the same
per-sector calls dm-crypt makes today: identical output and cost.
This is what Herbert predicted -- the per-unit indirect call just
moves from the caller into the API; the fallback is no slower.
So it stands on no-regression alone, with no software throughput claim.
What it adds is the interface a native one-pass driver needs. I'd land
that now and bring a native offload user + numbers as the follow-up,
rather than block the interface on the driver.
Acceptable? If so I'll respin v5 as the minimal version.
Thanks,
Leonid
^ permalink raw reply
* Re: [RFC] ML-KEM (FIPS 203) implementation with reusable decapsulation pool
From: kstzavertaylo @ 2026-06-24 14:44 UTC (permalink / raw)
To: Eric Biggers; +Cc: linux-crypto, herbert
In-Reply-To: <CAMho2Re4pc4f_TkApfwsbfguiaN_Ccw770_A+0fc9G7L_GBAnA@mail.gmail.com>
Hello,
Since our previous discussion, I have significantly updated the
userspace implementation and completed a series of benchmarks and
validation tests.
The implementation allocates all memory required for a given key
during key creation and subsequently operates entirely within that
preallocated memory until the key is destroyed.
During decapsulation, the same memory allocated at key creation is
reused across operations, eliminating runtime memory allocations in
the decapsulation path.
I compared the implementation against PQClean ML-KEM. The main observations are:
* Stack usage remains approximately 1 KB regardless of the selected
ML-KEM security level.
* No memory allocations are performed during decapsulation.
* Throughput is generally comparable to PQClean.
* For ML-KEM-1024, the implementation is currently about 5–8% slower
than PQClean.
* The trade-off is increased persistent memory consumption and a more
complex internal architecture based on reusable preallocated
resources.
I also took your earlier feedback into account and came to the
conclusion that integration through lib/crypto would be a better
direction than the KPP interface. As a result, I am now seriously
considering adapting the implementation for lib/crypto instead.
In addition, I am continuing to investigate further optimizations in
pure C. If these experiments produce meaningful results, I will
publish them and would be happy to share the findings in the future.
Benchmark results, methodology, stack usage measurements, memory usage
analysis, and PQClean comparisons are available here:
Release:
https://github.com/kstzv/ml-kem/releases/tag/v1.1.0
Benchmarks:
https://github.com/kstzv/ml-kem/tree/v1.1.0/portable/userspace/benchmarks
Thank you again for your earlier feedback.
Best regards,
K. S. Zavertailo
On Sun, Jun 14, 2026 at 10:50 AM kstzavertaylo <kstzavertaylo@gmail.com> wrote:
>
> Thank you for the detailed feedback and for outlining the historical
> context regarding pools in the crypto subsystem.
>
>
> I understand your point of view and the preference for keeping the
> core implementation simple with per-operation allocations (or
> caller-provided workspaces), especially given the lack of precedent
> for pool-based designs in lib/crypto. My approach with the reusable
> decapsulation pool was driven by a focus on constrained environments
> where minimizing stack usage and relying on reusable preallocated
> working memory during the hot path can be particularly valuable.
> However, I fully agree that concrete data is needed to properly
> evaluate the trade-offs.
>
>
> I see your point regarding preallocated workspaces and caller-managed
> caching. One of the goals of my prototype was to explore a design
> where decapsulation operates on reusable preallocated contexts rather
> than per-call working memory, primarily to reduce stack requirements
> and move memory management into an initialization phase. I need to
> analyze more carefully how much of this can already be achieved
> through a caller-provided workspace model and whether the additional
> complexity of a dedicated pool is actually justified.
>
>
> I am currently working on benchmarks that compare stack consumption,
> allocation behavior, memory footprint, and performance between the
> different approaches. Once I have solid numbers, I will share the
> results and my conclusions.
>
>
> I also appreciate the clarification regarding KPP. My original
> prototype used KPP because it appeared to be the closest existing
> interface for key establishment, but I am not specifically attached to
> that approach and will spend some time evaluating how the same ideas
> could fit into the lib/crypto model as well. In the meantime, I will
> also look into how the pre-allocated workspace support you suggested
> could be integrated.
>
>
> Best regards,
> K. Zavertailo
>
>
> On Fri, Jun 12, 2026 at 9:32 PM Eric Biggers <ebiggers@kernel.org> wrote:
> >
> > On Fri, Jun 12, 2026 at 05:14:54PM +0300, kstzavertaylo wrote:
> > > Thank you for the detailed reply and for pointing me to the existing
> > > ML-KEM/X-Wing patchset. I spent some time reviewing the implementation
> > > to better understand the design choices and how they compare to the
> > > approach I took in my own work.
> > >
> > > After reviewing the patchset, I can see several strengths in the
> > > implementation. It integrates cleanly into the existing lib/crypto
> > > infrastructure, reuses kernel cryptographic primitives, avoids large
> > > stack allocations, and includes KUnit-based validation. The
> > > implementation also appears intentionally compact and well aligned
> > > with existing kernel conventions.
> > >
> > > While reviewing the implementation, I noticed that decapsulation
> > > allocates a temporary workspace for each operation. This is one of the
> > > areas where my design diverged, which is what originally motivated the
> > > reusable pool approach.
> > >
> > > My implementation was developed with a somewhat different goal in
> > > mind. I experimented with a reusable decapsulation workspace model
> > > where memory is allocated during key initialization and then reused
> > > across subsequent decapsulation operations. The main motivation was
> > > reducing allocation frequency and minimizing both stack usage and
> > > repeated memory management during decapsulation.
> > >
> > > As a result, the implementation avoids allocations during
> > > decapsulation entirely by reusing preallocated workspaces associated
> > > with the key context. My original hypothesis was that moving memory
> > > allocation to key initialization, thereby eliminating allocations from
> > > the decapsulation path, could reduce allocation overhead during
> > > repeated decapsulation operations and be beneficial in environments
> > > where allocation activity is considered undesirable.
> >
> > In my ML-KEM code, all the decapsulation memory is consolidated into
> > struct mlkem_decap_workspace. It would be straightforward to support
> > the caller providing a pre-allocated workspace.
> >
> > In the case of X-Wing, we could also support pre-expanding the
> > decapsulation key.
> >
> > It just depends on what is actually going to be needed by the kernel
> > feature(s) that are going to use this. Which we don't really know yet.
> >
> > We do know that it hasn't been found to be useful for the crypto
> > subsystem to provide pools for any other algorithm in the kernel, for a
> > variety of reasons. Usually callers can just allocate per-operation, or
> > they have some sort of object (inode, block device, socket, etc.) that's
> > a natural place for them to cache whatever they need anyway. In the
> > rare cases where some sort of pool is needed it's implemented in the
> > caller, optimized for the particular use case. So I think there's a
> > good chance your pool idea is going off on the wrong track.
> >
> > > Another difference is the integration level. My prototype explored
> > > direct integration through the KPP interface, whereas the patchset
> > > focuses on providing a reusable cryptographic library component within
> > > lib/crypto. These approaches address somewhat different layers of the
> > > kernel crypto stack.
> >
> > We don't need crypto_kpp support, as it's much more complex and harder
> > to use than the crypto library
> > (https://docs.kernel.org/crypto/libcrypto.html). Also it seems it's not
> > really possible anyway, since crypto_kpp is an old design that works for
> > Diffie-Hellman but not KEMs.
> >
> > - Eric
^ permalink raw reply
* Re: [PATCH v3 1/7] list: Add mutable iterator variants
From: David Laight @ 2026-06-24 14:23 UTC (permalink / raw)
To: Christian König
Cc: Kaitao Cheng, Andrew Morton, David Hildenbrand, Jens Axboe,
Tejun Heo, Alexander Viro, Christian Brauner, Alexei Starovoitov,
Daniel Borkmann, Andrii Nakryiko, Johannes Weiner, Peter Zijlstra,
Ingo Molnar, Arnaldo Carvalho de Melo, Namhyung Kim,
Thomas Gleixner, Juri Lelli, Vincent Guittot, Paul Moore,
Andy Shevchenko, Paul E. McKenney, Shakeel Butt, David Howells,
Simona Vetter, Randy Dunlap, Luca Ceresoli, Philipp Stanner,
linux-block, linux-kernel, cgroups, linux-ntfs-dev, linux-fsdevel,
io-uring, audit, bpf, netdev, dri-devel, linux-perf-users,
linux-trace-kernel, kexec, live-patching, linux-modules,
linux-crypto, linux-pm, rcu, sched-ext, linux-mm, virtualization,
damon, llvm, Kaitao Cheng
In-Reply-To: <cf8467c7-b98f-44a5-9cf9-60b43b5da711@amd.com>
On Wed, 24 Jun 2026 15:23:47 +0200
Christian König <christian.koenig@amd.com> wrote:
> On 6/24/26 15:14, Kaitao Cheng wrote:
> >
> >
> > 在 2026/6/22 16:42, David Laight 写道:
> >> On Mon, 22 Jun 2026 12:05:31 +0800
> >> Kaitao Cheng <kaitao.cheng@linux.dev> wrote:
> >>
> >>> From: Kaitao Cheng <chengkaitao@kylinos.cn>
> >>>
> >>> The list_for_each*_safe() helpers are used when the loop body may
> >>> remove the current entry. Their API exposes the temporary cursor at
> >>> every call site, even though most users only need it for the iterator
> >>> implementation and never reference it in the loop body.
> >>>
> >>> Add *_mutable() variants for list and hlist iteration. The new helpers
> >>> support both forms: callers may keep passing an explicit temporary cursor
> >>> when they need to inspect or reset it, or omit it and let the helper use
> >>> a unique internal cursor.
> >>
> >> I'm not really sure 'mutable' means anything either.
> >> It is possible to make it valid for the loop body (or even other threads)
> >> to delete arbitrary list items - but that needs significant extra overheads.
> >>
> >> It might be worth doing something that doesn't need the extra variable,
> >> but there is little point doing all the churn just to rename things.
> >>
> >>>
> >>> This makes call sites that only mutate the list through the current entry
> >>> less noisy, while keeping the existing *_safe() helpers available for
> >>> compatibility.
> >>>
> >>> Signed-off-by: Kaitao Cheng <chengkaitao@kylinos.cn>
> >>> ---
> >>> include/linux/list.h | 269 +++++++++++++++++++++++++++++++++++++------
> >>> 1 file changed, 231 insertions(+), 38 deletions(-)
> >>>
> >>> diff --git a/include/linux/list.h b/include/linux/list.h
> >>> index 09d979976b3b..1081def7cea9 100644
> >>> --- a/include/linux/list.h
> >>> +++ b/include/linux/list.h
> >>> @@ -7,6 +7,7 @@
> >>> #include <linux/stddef.h>
> >>> #include <linux/poison.h>
> >>> #include <linux/const.h>
> >>> +#include <linux/args.h>
> >>>
> >>> #include <asm/barrier.h>
> >>>
> >>> @@ -763,28 +764,72 @@ static inline void list_splice_tail_init(struct list_head *list,
> >>> #define list_for_each_prev(pos, head) \
> >>> for (pos = (head)->prev; !list_is_head(pos, (head)); pos = pos->prev)
> >>>
> >>> -/**
> >>> - * list_for_each_safe - iterate over a list safe against removal of list entry
> >>> - * @pos: the &struct list_head to use as a loop cursor.
> >>> - * @n: another &struct list_head to use as temporary storage
> >>> - * @head: the head for your list.
> >>> +/*
> >>> + * list_for_each_safe is an old interface, use list_for_each_mutable instead.
> >>> */
> >>> #define list_for_each_safe(pos, n, head) \
> >>> for (pos = (head)->next, n = pos->next; \
> >>> !list_is_head(pos, (head)); \
> >>> pos = n, n = pos->next)
> >>>
> >>> +#define __list_for_each_mutable_internal(pos, tmp, head) \
> >>> + for (typeof(pos) tmp = (pos = (head)->next)->next; \
> >>
> >> Use auto
> >>
> >>> + !list_is_head(pos, (head)); \
> >>> + pos = tmp, tmp = pos->next)
> >>> +
> >>> +#define __list_for_each_mutable1(pos, head) \
> >>> + __list_for_each_mutable_internal(pos, __UNIQUE_ID(next), head)
> >>> +
> >>> +#define __list_for_each_mutable2(pos, next, head) \
> >>> + list_for_each_safe(pos, next, head)
> >>> +
> >>> /**
> >>> - * list_for_each_prev_safe - iterate over a list backwards safe against removal of list entry
> >>> + * list_for_each_mutable - iterate over a list safe against entry removal
> >>> * @pos: the &struct list_head to use as a loop cursor.
> >>> - * @n: another &struct list_head to use as temporary storage
> >>> - * @head: the head for your list.
> >>> + * @...: either (head) or (next, head)
> >>> + *
> >>> + * next: another &struct list_head to use as optional temporary storage.
> >>> + * The temporary cursor is internal unless explicitly supplied by
> >>> + * the caller.
> >>> + * head: the head for your list.
> >>> + */
> >>> +#define list_for_each_mutable(pos, ...) \
> >>> + CONCATENATE(__list_for_each_mutable, COUNT_ARGS(__VA_ARGS__)) \
> >>> + (pos, __VA_ARGS__)
> >>
> >> The variable argument count logic really just slows down compilation.
> >> Maybe there aren't enough copies of this code to make that significant.
> >> But just because you can do it doesn't mean it is a gooD idea.
> >> I'm also not sure it really adds anything to the readability.
> >>
> >> And, it you are going to make the middle argument optional there is
> >> no need to change the macro name.
> >
> > Christian König and Jani Nikula also disagree with the variadic-argument
> > implementation approach. If we abandon that method, it means we will
> > inevitably need to add some new macros. If mutable is not a good name,
> > suggestions for better alternatives would be welcome; coming up with a
> > suitable name is indeed rather tricky.
>
> I don't think you need to add a new macro for the specific use case that people want to modify the next element of the iteration.
>
> If I remember your numbers correctly that is a really corner case and keeping using the existing *_safe() macros for that sounds perfectly fine to me.
IIRC currently you have a choice of either:
define Item that can't be deleted
list_for_each() The current item.
list_for_each_safe() The next item.
There is also likely to be code that updates the variables to allow
for other scenarios.
Note that if increase a reference count and release a lock then list_for_each()
is likely safer than list_for_each_safe() :-)
list.h has 9 variants of the 'safe' loop.
The bloat of another 9 is getting excessive.
It has to be said that this is one of my least favourite type of list...
David
>
> Regards,
> Christian.
^ permalink raw reply
* Re: [PATCH v3 1/7] list: Add mutable iterator variants
From: Christian König @ 2026-06-24 13:23 UTC (permalink / raw)
To: Kaitao Cheng, David Laight
Cc: Andrew Morton, David Hildenbrand, Jens Axboe, Tejun Heo,
Alexander Viro, Christian Brauner, Alexei Starovoitov,
Daniel Borkmann, Andrii Nakryiko, Johannes Weiner, Peter Zijlstra,
Ingo Molnar, Arnaldo Carvalho de Melo, Namhyung Kim,
Thomas Gleixner, Juri Lelli, Vincent Guittot, Paul Moore,
Andy Shevchenko, Paul E. McKenney, Shakeel Butt, David Howells,
Simona Vetter, Randy Dunlap, Luca Ceresoli, Philipp Stanner,
linux-block, linux-kernel, cgroups, linux-ntfs-dev, linux-fsdevel,
io-uring, audit, bpf, netdev, dri-devel, linux-perf-users,
linux-trace-kernel, kexec, live-patching, linux-modules,
linux-crypto, linux-pm, rcu, sched-ext, linux-mm, virtualization,
damon, llvm, Kaitao Cheng
In-Reply-To: <351a6b67-b394-4c58-aee2-88b6c8089ad5@linux.dev>
On 6/24/26 15:14, Kaitao Cheng wrote:
>
>
> 在 2026/6/22 16:42, David Laight 写道:
>> On Mon, 22 Jun 2026 12:05:31 +0800
>> Kaitao Cheng <kaitao.cheng@linux.dev> wrote:
>>
>>> From: Kaitao Cheng <chengkaitao@kylinos.cn>
>>>
>>> The list_for_each*_safe() helpers are used when the loop body may
>>> remove the current entry. Their API exposes the temporary cursor at
>>> every call site, even though most users only need it for the iterator
>>> implementation and never reference it in the loop body.
>>>
>>> Add *_mutable() variants for list and hlist iteration. The new helpers
>>> support both forms: callers may keep passing an explicit temporary cursor
>>> when they need to inspect or reset it, or omit it and let the helper use
>>> a unique internal cursor.
>>
>> I'm not really sure 'mutable' means anything either.
>> It is possible to make it valid for the loop body (or even other threads)
>> to delete arbitrary list items - but that needs significant extra overheads.
>>
>> It might be worth doing something that doesn't need the extra variable,
>> but there is little point doing all the churn just to rename things.
>>
>>>
>>> This makes call sites that only mutate the list through the current entry
>>> less noisy, while keeping the existing *_safe() helpers available for
>>> compatibility.
>>>
>>> Signed-off-by: Kaitao Cheng <chengkaitao@kylinos.cn>
>>> ---
>>> include/linux/list.h | 269 +++++++++++++++++++++++++++++++++++++------
>>> 1 file changed, 231 insertions(+), 38 deletions(-)
>>>
>>> diff --git a/include/linux/list.h b/include/linux/list.h
>>> index 09d979976b3b..1081def7cea9 100644
>>> --- a/include/linux/list.h
>>> +++ b/include/linux/list.h
>>> @@ -7,6 +7,7 @@
>>> #include <linux/stddef.h>
>>> #include <linux/poison.h>
>>> #include <linux/const.h>
>>> +#include <linux/args.h>
>>>
>>> #include <asm/barrier.h>
>>>
>>> @@ -763,28 +764,72 @@ static inline void list_splice_tail_init(struct list_head *list,
>>> #define list_for_each_prev(pos, head) \
>>> for (pos = (head)->prev; !list_is_head(pos, (head)); pos = pos->prev)
>>>
>>> -/**
>>> - * list_for_each_safe - iterate over a list safe against removal of list entry
>>> - * @pos: the &struct list_head to use as a loop cursor.
>>> - * @n: another &struct list_head to use as temporary storage
>>> - * @head: the head for your list.
>>> +/*
>>> + * list_for_each_safe is an old interface, use list_for_each_mutable instead.
>>> */
>>> #define list_for_each_safe(pos, n, head) \
>>> for (pos = (head)->next, n = pos->next; \
>>> !list_is_head(pos, (head)); \
>>> pos = n, n = pos->next)
>>>
>>> +#define __list_for_each_mutable_internal(pos, tmp, head) \
>>> + for (typeof(pos) tmp = (pos = (head)->next)->next; \
>>
>> Use auto
>>
>>> + !list_is_head(pos, (head)); \
>>> + pos = tmp, tmp = pos->next)
>>> +
>>> +#define __list_for_each_mutable1(pos, head) \
>>> + __list_for_each_mutable_internal(pos, __UNIQUE_ID(next), head)
>>> +
>>> +#define __list_for_each_mutable2(pos, next, head) \
>>> + list_for_each_safe(pos, next, head)
>>> +
>>> /**
>>> - * list_for_each_prev_safe - iterate over a list backwards safe against removal of list entry
>>> + * list_for_each_mutable - iterate over a list safe against entry removal
>>> * @pos: the &struct list_head to use as a loop cursor.
>>> - * @n: another &struct list_head to use as temporary storage
>>> - * @head: the head for your list.
>>> + * @...: either (head) or (next, head)
>>> + *
>>> + * next: another &struct list_head to use as optional temporary storage.
>>> + * The temporary cursor is internal unless explicitly supplied by
>>> + * the caller.
>>> + * head: the head for your list.
>>> + */
>>> +#define list_for_each_mutable(pos, ...) \
>>> + CONCATENATE(__list_for_each_mutable, COUNT_ARGS(__VA_ARGS__)) \
>>> + (pos, __VA_ARGS__)
>>
>> The variable argument count logic really just slows down compilation.
>> Maybe there aren't enough copies of this code to make that significant.
>> But just because you can do it doesn't mean it is a gooD idea.
>> I'm also not sure it really adds anything to the readability.
>>
>> And, it you are going to make the middle argument optional there is
>> no need to change the macro name.
>
> Christian König and Jani Nikula also disagree with the variadic-argument
> implementation approach. If we abandon that method, it means we will
> inevitably need to add some new macros. If mutable is not a good name,
> suggestions for better alternatives would be welcome; coming up with a
> suitable name is indeed rather tricky.
I don't think you need to add a new macro for the specific use case that people want to modify the next element of the iteration.
If I remember your numbers correctly that is a really corner case and keeping using the existing *_safe() macros for that sounds perfectly fine to me.
Regards,
Christian.
^ permalink raw reply
* Re: [PATCH v3 1/7] list: Add mutable iterator variants
From: Kaitao Cheng @ 2026-06-24 13:14 UTC (permalink / raw)
To: David Laight
Cc: Andrew Morton, David Hildenbrand, Jens Axboe, Tejun Heo,
Alexander Viro, Christian Brauner, Alexei Starovoitov,
Daniel Borkmann, Andrii Nakryiko, Johannes Weiner, Peter Zijlstra,
Ingo Molnar, Arnaldo Carvalho de Melo, Namhyung Kim,
Thomas Gleixner, Juri Lelli, Vincent Guittot, Paul Moore,
Andy Shevchenko, Paul E. McKenney, Shakeel Butt,
Christian König, David Howells, Simona Vetter, Randy Dunlap,
Luca Ceresoli, Philipp Stanner, linux-block, linux-kernel,
cgroups, linux-ntfs-dev, linux-fsdevel, io-uring, audit, bpf,
netdev, dri-devel, linux-perf-users, linux-trace-kernel, kexec,
live-patching, linux-modules, linux-crypto, linux-pm, rcu,
sched-ext, linux-mm, virtualization, damon, llvm, Kaitao Cheng
In-Reply-To: <20260622094242.64531b9a@pumpkin>
在 2026/6/22 16:42, David Laight 写道:
> On Mon, 22 Jun 2026 12:05:31 +0800
> Kaitao Cheng <kaitao.cheng@linux.dev> wrote:
>
>> From: Kaitao Cheng <chengkaitao@kylinos.cn>
>>
>> The list_for_each*_safe() helpers are used when the loop body may
>> remove the current entry. Their API exposes the temporary cursor at
>> every call site, even though most users only need it for the iterator
>> implementation and never reference it in the loop body.
>>
>> Add *_mutable() variants for list and hlist iteration. The new helpers
>> support both forms: callers may keep passing an explicit temporary cursor
>> when they need to inspect or reset it, or omit it and let the helper use
>> a unique internal cursor.
>
> I'm not really sure 'mutable' means anything either.
> It is possible to make it valid for the loop body (or even other threads)
> to delete arbitrary list items - but that needs significant extra overheads.
>
> It might be worth doing something that doesn't need the extra variable,
> but there is little point doing all the churn just to rename things.
>
>>
>> This makes call sites that only mutate the list through the current entry
>> less noisy, while keeping the existing *_safe() helpers available for
>> compatibility.
>>
>> Signed-off-by: Kaitao Cheng <chengkaitao@kylinos.cn>
>> ---
>> include/linux/list.h | 269 +++++++++++++++++++++++++++++++++++++------
>> 1 file changed, 231 insertions(+), 38 deletions(-)
>>
>> diff --git a/include/linux/list.h b/include/linux/list.h
>> index 09d979976b3b..1081def7cea9 100644
>> --- a/include/linux/list.h
>> +++ b/include/linux/list.h
>> @@ -7,6 +7,7 @@
>> #include <linux/stddef.h>
>> #include <linux/poison.h>
>> #include <linux/const.h>
>> +#include <linux/args.h>
>>
>> #include <asm/barrier.h>
>>
>> @@ -763,28 +764,72 @@ static inline void list_splice_tail_init(struct list_head *list,
>> #define list_for_each_prev(pos, head) \
>> for (pos = (head)->prev; !list_is_head(pos, (head)); pos = pos->prev)
>>
>> -/**
>> - * list_for_each_safe - iterate over a list safe against removal of list entry
>> - * @pos: the &struct list_head to use as a loop cursor.
>> - * @n: another &struct list_head to use as temporary storage
>> - * @head: the head for your list.
>> +/*
>> + * list_for_each_safe is an old interface, use list_for_each_mutable instead.
>> */
>> #define list_for_each_safe(pos, n, head) \
>> for (pos = (head)->next, n = pos->next; \
>> !list_is_head(pos, (head)); \
>> pos = n, n = pos->next)
>>
>> +#define __list_for_each_mutable_internal(pos, tmp, head) \
>> + for (typeof(pos) tmp = (pos = (head)->next)->next; \
>
> Use auto
>
>> + !list_is_head(pos, (head)); \
>> + pos = tmp, tmp = pos->next)
>> +
>> +#define __list_for_each_mutable1(pos, head) \
>> + __list_for_each_mutable_internal(pos, __UNIQUE_ID(next), head)
>> +
>> +#define __list_for_each_mutable2(pos, next, head) \
>> + list_for_each_safe(pos, next, head)
>> +
>> /**
>> - * list_for_each_prev_safe - iterate over a list backwards safe against removal of list entry
>> + * list_for_each_mutable - iterate over a list safe against entry removal
>> * @pos: the &struct list_head to use as a loop cursor.
>> - * @n: another &struct list_head to use as temporary storage
>> - * @head: the head for your list.
>> + * @...: either (head) or (next, head)
>> + *
>> + * next: another &struct list_head to use as optional temporary storage.
>> + * The temporary cursor is internal unless explicitly supplied by
>> + * the caller.
>> + * head: the head for your list.
>> + */
>> +#define list_for_each_mutable(pos, ...) \
>> + CONCATENATE(__list_for_each_mutable, COUNT_ARGS(__VA_ARGS__)) \
>> + (pos, __VA_ARGS__)
>
> The variable argument count logic really just slows down compilation.
> Maybe there aren't enough copies of this code to make that significant.
> But just because you can do it doesn't mean it is a gooD idea.
> I'm also not sure it really adds anything to the readability.
>
> And, it you are going to make the middle argument optional there is
> no need to change the macro name.
Christian König and Jani Nikula also disagree with the variadic-argument
implementation approach. If we abandon that method, it means we will
inevitably need to add some new macros. If mutable is not a good name,
suggestions for better alternatives would be welcome; coming up with a
suitable name is indeed rather tricky.
--
Thanks
Kaitao Cheng
^ permalink raw reply
* Re: [PATCH v3 0/7] Prepare mutable list iterators to cache cursor state
From: Kaitao Cheng @ 2026-06-24 13:05 UTC (permalink / raw)
To: Jani Nikula, Andrew Morton, David Hildenbrand, Jens Axboe,
Tejun Heo, Alexander Viro, Christian Brauner, Alexei Starovoitov,
Daniel Borkmann, Andrii Nakryiko, Johannes Weiner, Peter Zijlstra,
Ingo Molnar, Arnaldo Carvalho de Melo, Namhyung Kim,
Thomas Gleixner, Juri Lelli, Vincent Guittot, Paul Moore,
Andy Shevchenko, Paul E. McKenney, Shakeel Butt,
Christian König
Cc: David Howells, Simona Vetter, Randy Dunlap, Luca Ceresoli,
Philipp Stanner, linux-block, linux-kernel, cgroups,
linux-ntfs-dev, linux-fsdevel, io-uring, audit, bpf, netdev,
dri-devel, linux-perf-users, linux-trace-kernel, kexec,
live-patching, linux-modules, linux-crypto, linux-pm, rcu,
sched-ext, linux-mm, virtualization, damon, llvm, chengkaitao
In-Reply-To: <88f34c7fa5a3d1700cc8005818751d6aa31f09df@intel.com>
在 2026/6/22 16:37, Jani Nikula 写道:
> On Mon, 22 Jun 2026, Kaitao Cheng <kaitao.cheng@linux.dev> wrote:
>> Add *_mutable() iterator variants for list, hlist and llist. The new
>> helpers are variadic and support both forms. In the common case, the
>> caller omits the temporary cursor and the macro creates a unique internal
>> cursor with typeof(pos) and __UNIQUE_ID(). If a loop really needs an
>> explicit temporary cursor, the caller can still pass it and the helper
>> keeps the existing *_safe() behaviour.
>>
>> For example, a call site may use the shorter form:
>>
>> list_for_each_entry_mutable(pos, head, member)
>>
>> or keep the explicit temporary cursor form:
>>
>> list_for_each_entry_mutable(pos, tmp, head, member)
>
> I'm unconvinced it's a good idea to allow two forms with macro trickery,
> *especially* when it's not the last argument you can omit. I think it's
> a footgun.
>
> IMO stick with the first form only, and there'll always be the _safe
> variant that can be used when the temp pointer is needed.
Could we go back to the v1 version? What do you think of that
implementation approach?
https://lore.kernel.org/all/20260529082149.76764-1-kaitao.cheng@linux.dev/
--
Thanks
Kaitao Cheng
^ permalink raw reply
* Re: [PATCH v3 0/7] Prepare mutable list iterators to cache cursor state
From: Kaitao Cheng @ 2026-06-24 12:58 UTC (permalink / raw)
To: David Hildenbrand (Arm), Alexei Starovoitov
Cc: Andrew Morton, Jens Axboe, Tejun Heo, Alexander Viro,
Christian Brauner, Alexei Starovoitov, Daniel Borkmann,
Andrii Nakryiko, Johannes Weiner, Peter Zijlstra, Ingo Molnar,
Arnaldo Carvalho de Melo, Namhyung Kim, Thomas Gleixner,
Juri Lelli, Vincent Guittot, Paul Moore, Andy Shevchenko,
Paul E. McKenney, Shakeel Butt, Christian König,
David Howells, Simona Vetter, Randy Dunlap, Luca Ceresoli,
Philipp Stanner, linux-block, LKML,
open list:CONTROL GROUP (CGROUP), linux-ntfs-dev, Linux-Fsdevel,
io-uring, audit, bpf, Network Development, dri-devel,
linux-perf-use., linux-trace-kernel, kexec, live-patching,
linux-modules, Linux Crypto Mailing List, Linux Power Management,
rcu, sched-ext, linux-mm, virtualization, damon,
clang-built-linux, chengkaitao
In-Reply-To: <8f98a3a6-f97b-4673-964f-fb09c8879e2e@kernel.org>
在 2026/6/22 19:27, David Hildenbrand (Arm) 写道:
> On 6/22/26 07:28, Alexei Starovoitov wrote:
>> On Sun, Jun 21, 2026 at 9:06 PM Kaitao Cheng <kaitao.cheng@linux.dev> wrote:
>>>
>>> From: chengkaitao <chengkaitao@kylinos.cn>
>>>
>>> The list_for_each*_safe() helpers are used when the loop body may remove
>>> the current entry. Their current interface, however, forces every caller
>>> to define a temporary cursor outside the macro and pass it in, even when
>>> the caller never uses that cursor directly. For most call sites this
>>> extra cursor is just boilerplate required by the macro implementation.
>>>
>>> This is awkward because the saved next pointer is an internal detail of
>>> the iteration. Callers that only remove or move the current entry do not
>>> need to spell it out.
>>>
>>> The _safe() suffix has also caused confusion. Christian Koenig pointed
>>> out that the name is easy to read as a thread-safe variant, especially
>>> for beginners, even though it only means that the iterator keeps enough
>>> state to tolerate removal of the current entry. He suggested _mutable()
>>> as a clearer description of what the loop permits.
>>>
>>> Add *_mutable() iterator variants for list, hlist and llist. The new
>>> helpers are variadic and support both forms. In the common case, the
>>> caller omits the temporary cursor and the macro creates a unique internal
>>> cursor with typeof(pos) and __UNIQUE_ID(). If a loop really needs an
>>> explicit temporary cursor, the caller can still pass it and the helper
>>> keeps the existing *_safe() behaviour.
>>>
>>> For example, a call site may use the shorter form:
>>>
>>> list_for_each_entry_mutable(pos, head, member)
>>>
>>> or keep the explicit temporary cursor form:
>>>
>>> list_for_each_entry_mutable(pos, tmp, head, member)
>>>
>>> The existing *_safe() helpers remain available for compatibility. This
>>> series only converts users in mm, block, kernel, init and io_uring. If
>>> this approach looks acceptable, the remaining users can be converted in
>>> follow-up series.
>>>
>>> Changes in v3 (Christian König, Andy Shevchenko):
>>> - Convert safe list walks to mutable iterators
>>>
>>> Changes in v2 (Muchun Song, Andy Shevchenko):
>>> - Drop the list_for_each_entry_mutable*() helpers from v1 and make the
>>> cursor change directly in the existing list_for_each_entry*() helpers.
>>> - Open-code special list walks that rely on updating the loop cursor in
>>> the body, preserving their existing traversal semantics.
>>>
>>> Link to v2:
>>> https://lore.kernel.org/all/20260609061347.93688-1-kaitao.cheng@linux.dev/
>>>
>>> Link to v1:
>>> https://lore.kernel.org/all/20260529082149.76764-1-kaitao.cheng@linux.dev/
>>>
>>> Kaitao Cheng (7):
>>> list: Add mutable iterator variants
>>> llist: Add mutable iterator variants
>>> mm: Use mutable list iterators
>>> block: Use mutable list iterators
>>> kernel: Use mutable list iterators
>>> initramfs: Use mutable list iterator
>>> io_uring: Use mutable list iterators
>>>
>>> block/bfq-iosched.c | 17 +-
>>> block/blk-cgroup.c | 12 +-
>>> block/blk-flush.c | 4 +-
>>> block/blk-iocost.c | 18 +-
>>> block/blk-mq.c | 8 +-
>>> block/blk-throttle.c | 4 +-
>>> block/kyber-iosched.c | 4 +-
>>> block/partitions/ldm.c | 8 +-
>>> block/sed-opal.c | 4 +-
>>> include/linux/list.h | 269 ++++++++++++++++++++++++----
>>> include/linux/llist.h | 81 +++++++--
>>> init/initramfs.c | 5 +-
>>> io_uring/cancel.c | 6 +-
>>> io_uring/poll.c | 3 +-
>>> io_uring/rw.c | 4 +-
>>> io_uring/timeout.c | 8 +-
>>> io_uring/uring_cmd.c | 3 +-
>>> kernel/audit_tree.c | 4 +-
>>> kernel/audit_watch.c | 16 +-
>>> kernel/auditfilter.c | 4 +-
>>> kernel/auditsc.c | 4 +-
>>> kernel/bpf/arena.c | 10 +-
>>> kernel/bpf/arraymap.c | 8 +-
>>> kernel/bpf/bpf_local_storage.c | 3 +-
>>> kernel/bpf/bpf_lru_list.c | 25 ++-
>>> kernel/bpf/btf.c | 18 +-
>>> kernel/bpf/cgroup.c | 7 +-
>>> kernel/bpf/cpumap.c | 4 +-
>>> kernel/bpf/devmap.c | 10 +-
>>> kernel/bpf/helpers.c | 8 +-
>>> kernel/bpf/local_storage.c | 4 +-
>>> kernel/bpf/memalloc.c | 16 +-
>>> kernel/bpf/offload.c | 8 +-
>>> kernel/bpf/states.c | 4 +-
>>> kernel/bpf/stream.c | 4 +-
>>> kernel/bpf/verifier.c | 6 +-
>>> kernel/cgroup/cgroup-v1.c | 4 +-
>>> kernel/cgroup/cgroup.c | 54 +++---
>>> kernel/cgroup/dmem.c | 12 +-
>>> kernel/cgroup/rdma.c | 8 +-
>>> kernel/events/core.c | 44 +++--
>>> kernel/events/uprobes.c | 12 +-
>>> kernel/exit.c | 8 +-
>>> kernel/fail_function.c | 4 +-
>>> kernel/gcov/clang.c | 4 +-
>>> kernel/irq_work.c | 4 +-
>>> kernel/kexec_core.c | 4 +-
>>> kernel/kprobes.c | 16 +-
>>> kernel/livepatch/core.c | 4 +-
>>> kernel/livepatch/core.h | 4 +-
>>> kernel/liveupdate/kho_block.c | 4 +-
>>> kernel/liveupdate/luo_flb.c | 4 +-
>>> kernel/locking/rwsem.c | 2 +-
>>> kernel/locking/test-ww_mutex.c | 2 +-
>>> kernel/module/main.c | 11 +-
>>> kernel/padata.c | 4 +-
>>> kernel/power/snapshot.c | 8 +-
>>> kernel/power/wakelock.c | 4 +-
>>> kernel/printk/printk.c | 11 +-
>>> kernel/ptrace.c | 4 +-
>>> kernel/rcu/rcutorture.c | 3 +-
>>> kernel/rcu/tasks.h | 9 +-
>>> kernel/rcu/tree.c | 6 +-
>>> kernel/resource.c | 4 +-
>>> kernel/sched/core.c | 4 +-
>>> kernel/sched/ext.c | 22 +--
>>> kernel/sched/fair.c | 28 +--
>>> kernel/sched/topology.c | 4 +-
>>> kernel/sched/wait.c | 4 +-
>>> kernel/seccomp.c | 4 +-
>>> kernel/signal.c | 11 +-
>>> kernel/smp.c | 4 +-
>>> kernel/taskstats.c | 8 +-
>>> kernel/time/clockevents.c | 6 +-
>>> kernel/time/clocksource.c | 4 +-
>>> kernel/time/posix-cpu-timers.c | 4 +-
>>> kernel/time/posix-timers.c | 3 +-
>>> kernel/torture.c | 3 +-
>>> kernel/trace/bpf_trace.c | 4 +-
>>> kernel/trace/ftrace.c | 49 +++--
>>> kernel/trace/ring_buffer.c | 25 ++-
>>> kernel/trace/trace.c | 12 +-
>>> kernel/trace/trace_dynevent.c | 6 +-
>>> kernel/trace/trace_dynevent.h | 5 +-
>>> kernel/trace/trace_events.c | 35 ++--
>>> kernel/trace/trace_events_filter.c | 4 +-
>>> kernel/trace/trace_events_hist.c | 8 +-
>>> kernel/trace/trace_events_trigger.c | 17 +-
>>> kernel/trace/trace_events_user.c | 16 +-
>>> kernel/trace/trace_stat.c | 4 +-
>>> kernel/user-return-notifier.c | 3 +-
>>> kernel/workqueue.c | 16 +-
>>> mm/backing-dev.c | 8 +-
>>> mm/balloon.c | 8 +-
>>> mm/cma.c | 4 +-
>>> mm/compaction.c | 4 +-
>>> mm/damon/core.c | 4 +-
>>> mm/damon/sysfs-schemes.c | 4 +-
>>> mm/dmapool.c | 4 +-
>>> mm/huge_memory.c | 8 +-
>>> mm/hugetlb.c | 56 +++---
>>> mm/hugetlb_vmemmap.c | 16 +-
>>> mm/khugepaged.c | 14 +-
>>> mm/kmemleak.c | 7 +-
>>> mm/ksm.c | 25 +--
>>> mm/list_lru.c | 4 +-
>>> mm/memcontrol-v1.c | 8 +-
>>> mm/memory-failure.c | 12 +-
>>> mm/memory-tiers.c | 4 +-
>>> mm/migrate.c | 23 ++-
>>> mm/mmu_notifier.c | 9 +-
>>> mm/page_alloc.c | 8 +-
>>> mm/page_reporting.c | 2 +-
>>> mm/percpu.c | 11 +-
>>> mm/pgtable-generic.c | 4 +-
>>> mm/rmap.c | 10 +-
>>> mm/shmem.c | 9 +-
>>> mm/slab_common.c | 14 +-
>>> mm/slub.c | 33 ++--
>>> mm/swapfile.c | 4 +-
>>> mm/userfaultfd.c | 12 +-
>>> mm/vmalloc.c | 24 +--
>>> mm/vmscan.c | 7 +-
>>> mm/zsmalloc.c | 4 +-
>>> 124 files changed, 875 insertions(+), 681 deletions(-)
>>
>> Not sure what you were thinking, but this diff stat
>> is not landable.
>
> Agreed. If we decide we want this, I guess we should target per-subsystem
> conversions.
>
> If this goes through the MM tree, I would even appreciate doing this on a per-MM
> component granularity.
>
> (unless we have some magic "Linus converts all of them" script, which I doubt we
> will have)
I strongly agree with the point above.
> Is there a way forward to replace list_for_each_*_safe entirely, possibly just
> reusing the old name but simply the parameter?
David Laight, Christian König, and Jani Nikula do not agree with using
clever macro syntax to support both calling forms at the same time,
so for now it is not possible to keep the original macro name and only
simplify the parameter. I may revert to the v1 version and ask everyone
for their opinions again.
--
Thanks
Kaitao Cheng
^ permalink raw reply
* Re: [PATCH v3 0/7] Prepare mutable list iterators to cache cursor state
From: Kaitao Cheng @ 2026-06-24 12:29 UTC (permalink / raw)
To: Andy Shevchenko
Cc: Alexei Starovoitov, Andrew Morton, David Hildenbrand, Jens Axboe,
Tejun Heo, Alexander Viro, Christian Brauner, Alexei Starovoitov,
Daniel Borkmann, Andrii Nakryiko, Johannes Weiner, Peter Zijlstra,
Ingo Molnar, Arnaldo Carvalho de Melo, Namhyung Kim,
Thomas Gleixner, Juri Lelli, Vincent Guittot, Paul Moore,
Paul E. McKenney, Shakeel Butt, Christian König,
David Howells, Simona Vetter, Randy Dunlap, Luca Ceresoli,
Philipp Stanner, linux-block, LKML,
open list:CONTROL GROUP (CGROUP), linux-ntfs-dev, Linux-Fsdevel,
io-uring, audit, bpf, Network Development, dri-devel,
linux-perf-use., linux-trace-kernel, kexec, live-patching,
linux-modules, Linux Crypto Mailing List, Linux Power Management,
rcu, sched-ext, linux-mm, virtualization, damon,
clang-built-linux, chengkaitao, Muchun Song
In-Reply-To: <ajkSftEbdGoiJXYs@ashevche-desk.local>
在 2026/6/22 18:46, Andy Shevchenko 写道:
> On Mon, Jun 22, 2026 at 02:15:01PM +0800, Kaitao Cheng wrote:
>> 在 2026/6/22 13:28, Alexei Starovoitov 写道:
>>> On Sun, Jun 21, 2026 at 9:06 PM Kaitao Cheng <kaitao.cheng@linux.dev> wrote:
>
> ...
>
>>>> block/bfq-iosched.c | 17 +-
>>>> block/blk-cgroup.c | 12 +-
>>>> block/blk-flush.c | 4 +-
>>>> block/blk-iocost.c | 18 +-
>>>> block/blk-mq.c | 8 +-
>>>> block/blk-throttle.c | 4 +-
>>>> block/kyber-iosched.c | 4 +-
>>>> block/partitions/ldm.c | 8 +-
>>>> block/sed-opal.c | 4 +-
>>>> include/linux/list.h | 269 ++++++++++++++++++++++++----
>>>> include/linux/llist.h | 81 +++++++--
>>>> init/initramfs.c | 5 +-
>>>> io_uring/cancel.c | 6 +-
>>>> io_uring/poll.c | 3 +-
>>>> io_uring/rw.c | 4 +-
>>>> io_uring/timeout.c | 8 +-
>>>> io_uring/uring_cmd.c | 3 +-
>>>> kernel/audit_tree.c | 4 +-
>>>> kernel/audit_watch.c | 16 +-
>>>> kernel/auditfilter.c | 4 +-
>>>> kernel/auditsc.c | 4 +-
>>>> kernel/bpf/arena.c | 10 +-
>>>> kernel/bpf/arraymap.c | 8 +-
>>>> kernel/bpf/bpf_local_storage.c | 3 +-
>>>> kernel/bpf/bpf_lru_list.c | 25 ++-
>>>> kernel/bpf/btf.c | 18 +-
>>>> kernel/bpf/cgroup.c | 7 +-
>>>> kernel/bpf/cpumap.c | 4 +-
>>>> kernel/bpf/devmap.c | 10 +-
>>>> kernel/bpf/helpers.c | 8 +-
>>>> kernel/bpf/local_storage.c | 4 +-
>>>> kernel/bpf/memalloc.c | 16 +-
>>>> kernel/bpf/offload.c | 8 +-
>>>> kernel/bpf/states.c | 4 +-
>>>> kernel/bpf/stream.c | 4 +-
>>>> kernel/bpf/verifier.c | 6 +-
>>>> kernel/cgroup/cgroup-v1.c | 4 +-
>>>> kernel/cgroup/cgroup.c | 54 +++---
>>>> kernel/cgroup/dmem.c | 12 +-
>>>> kernel/cgroup/rdma.c | 8 +-
>>>> kernel/events/core.c | 44 +++--
>>>> kernel/events/uprobes.c | 12 +-
>>>> kernel/exit.c | 8 +-
>>>> kernel/fail_function.c | 4 +-
>>>> kernel/gcov/clang.c | 4 +-
>>>> kernel/irq_work.c | 4 +-
>>>> kernel/kexec_core.c | 4 +-
>>>> kernel/kprobes.c | 16 +-
>>>> kernel/livepatch/core.c | 4 +-
>>>> kernel/livepatch/core.h | 4 +-
>>>> kernel/liveupdate/kho_block.c | 4 +-
>>>> kernel/liveupdate/luo_flb.c | 4 +-
>>>> kernel/locking/rwsem.c | 2 +-
>>>> kernel/locking/test-ww_mutex.c | 2 +-
>>>> kernel/module/main.c | 11 +-
>>>> kernel/padata.c | 4 +-
>>>> kernel/power/snapshot.c | 8 +-
>>>> kernel/power/wakelock.c | 4 +-
>>>> kernel/printk/printk.c | 11 +-
>>>> kernel/ptrace.c | 4 +-
>>>> kernel/rcu/rcutorture.c | 3 +-
>>>> kernel/rcu/tasks.h | 9 +-
>>>> kernel/rcu/tree.c | 6 +-
>>>> kernel/resource.c | 4 +-
>>>> kernel/sched/core.c | 4 +-
>>>> kernel/sched/ext.c | 22 +--
>>>> kernel/sched/fair.c | 28 +--
>>>> kernel/sched/topology.c | 4 +-
>>>> kernel/sched/wait.c | 4 +-
>>>> kernel/seccomp.c | 4 +-
>>>> kernel/signal.c | 11 +-
>>>> kernel/smp.c | 4 +-
>>>> kernel/taskstats.c | 8 +-
>>>> kernel/time/clockevents.c | 6 +-
>>>> kernel/time/clocksource.c | 4 +-
>>>> kernel/time/posix-cpu-timers.c | 4 +-
>>>> kernel/time/posix-timers.c | 3 +-
>>>> kernel/torture.c | 3 +-
>>>> kernel/trace/bpf_trace.c | 4 +-
>>>> kernel/trace/ftrace.c | 49 +++--
>>>> kernel/trace/ring_buffer.c | 25 ++-
>>>> kernel/trace/trace.c | 12 +-
>>>> kernel/trace/trace_dynevent.c | 6 +-
>>>> kernel/trace/trace_dynevent.h | 5 +-
>>>> kernel/trace/trace_events.c | 35 ++--
>>>> kernel/trace/trace_events_filter.c | 4 +-
>>>> kernel/trace/trace_events_hist.c | 8 +-
>>>> kernel/trace/trace_events_trigger.c | 17 +-
>>>> kernel/trace/trace_events_user.c | 16 +-
>>>> kernel/trace/trace_stat.c | 4 +-
>>>> kernel/user-return-notifier.c | 3 +-
>>>> kernel/workqueue.c | 16 +-
>>>> mm/backing-dev.c | 8 +-
>>>> mm/balloon.c | 8 +-
>>>> mm/cma.c | 4 +-
>>>> mm/compaction.c | 4 +-
>>>> mm/damon/core.c | 4 +-
>>>> mm/damon/sysfs-schemes.c | 4 +-
>>>> mm/dmapool.c | 4 +-
>>>> mm/huge_memory.c | 8 +-
>>>> mm/hugetlb.c | 56 +++---
>>>> mm/hugetlb_vmemmap.c | 16 +-
>>>> mm/khugepaged.c | 14 +-
>>>> mm/kmemleak.c | 7 +-
>>>> mm/ksm.c | 25 +--
>>>> mm/list_lru.c | 4 +-
>>>> mm/memcontrol-v1.c | 8 +-
>>>> mm/memory-failure.c | 12 +-
>>>> mm/memory-tiers.c | 4 +-
>>>> mm/migrate.c | 23 ++-
>>>> mm/mmu_notifier.c | 9 +-
>>>> mm/page_alloc.c | 8 +-
>>>> mm/page_reporting.c | 2 +-
>>>> mm/percpu.c | 11 +-
>>>> mm/pgtable-generic.c | 4 +-
>>>> mm/rmap.c | 10 +-
>>>> mm/shmem.c | 9 +-
>>>> mm/slab_common.c | 14 +-
>>>> mm/slub.c | 33 ++--
>>>> mm/swapfile.c | 4 +-
>>>> mm/userfaultfd.c | 12 +-
>>>> mm/vmalloc.c | 24 +--
>>>> mm/vmscan.c | 7 +-
>>>> mm/zsmalloc.c | 4 +-
>>>> 124 files changed, 875 insertions(+), 681 deletions(-)
>>>
>>> Not sure what you were thinking, but this diff stat
>>> is not landable.
>>
>> [PATCH v3 1/7] and [PATCH v3 2/7] contain the main logic and can
>> be merged directly. They are also compatible with the old API.
>> [PATCH v3 3/7] through [PATCH v3 7/7] are just simple interface
>> replacements and do not change any functional logic. They can be
>> left unmerged for now; individual modules can pick them up later
>> if needed.
>>
>> In v2, Andy Shevchenko mentioned: "If it's done by Linus himself
>> during the day when he prepares -rc1, it's fine."
>
> Yes, but you need to get his blessing first to go with this.
> Have you communicated with him on this?
Not yet, because the overall approach is still not mature. People
have different opinions on the implementation details and on how
to move this forward, so I think we should iterate through a few
versions first before making a final decision.
>> Even so, the
>> changes in this patch series are indeed quite large and touch
>> almost every subsystem. I have only converted part of them for
>> now, so I wanted to send this out first and see what people think.
>
> That's why it's better to provide a script to convert (e.g., coccinelle)
> instead of tons of patches.
I tried writing conversion scripts with Coccinelle, but there were
always cases that got missed. In contrast, I found that using AI
for focused replacements was actually more efficient.
As David Hildenbrand mentioned, "If we decide we want this, I guess
we should target per-subsystem conversions." I would like to provide
the new interface first; adapting each subsystem on demand later may
be easier to achieve.
--
Thanks
Kaitao Cheng
^ permalink raw reply
* [PATCH] crypto: keembay: Fix AEAD unregister count in error path
From: Myeonghun Pak @ 2026-06-24 7:15 UTC (permalink / raw)
To: Daniele Alessandrelli, Herbert Xu, David S. Miller
Cc: linux-crypto, linux-kernel, Myeonghun Pak, Ijae Kim
register_aes_algs() registers the AEAD algorithms before registering the
skcipher algorithms. If skcipher registration fails, the function unwinds
the earlier AEAD registration with crypto_engine_unregister_aeads(), but it
passes ARRAY_SIZE(algs), which is the skcipher table size.
Use ARRAY_SIZE(algs_aead) for the AEAD unwind path so the unregister helper
iterates over the same table that was registered. Also clarify the nearby
comment: the crypto registration helpers clean up algorithms registered
within the same call, while this function must still unwind earlier
successful registration steps.
Fixes: 885743324513 ("crypto: keembay - Add support for Keem Bay OCS AES/SM4")
Co-developed-by: Ijae Kim <ae878000@gmail.com>
Signed-off-by: Ijae Kim <ae878000@gmail.com>
Signed-off-by: Myeonghun Pak <mhun512@gmail.com>
---
drivers/crypto/intel/keembay/keembay-ocs-aes-core.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/drivers/crypto/intel/keembay/keembay-ocs-aes-core.c b/drivers/crypto/intel/keembay/keembay-ocs-aes-core.c
index 8a8f6c81e0..0e42402422 100644
--- a/drivers/crypto/intel/keembay/keembay-ocs-aes-core.c
+++ b/drivers/crypto/intel/keembay/keembay-ocs-aes-core.c
@@ -1541,7 +1541,7 @@ static int register_aes_algs(struct ocs_aes_dev *aes_dev)
/*
* If any algorithm fails to register, all preceding algorithms that
- * were successfully registered will be automatically unregistered.
+ * were registered in the same call are automatically unregistered.
*/
ret = crypto_engine_register_aeads(algs_aead, ARRAY_SIZE(algs_aead));
if (ret)
@@ -1549,7 +1549,7 @@ static int register_aes_algs(struct ocs_aes_dev *aes_dev)
ret = crypto_engine_register_skciphers(algs, ARRAY_SIZE(algs));
if (ret)
- crypto_engine_unregister_aeads(algs_aead, ARRAY_SIZE(algs));
+ crypto_engine_unregister_aeads(algs_aead, ARRAY_SIZE(algs_aead));
return ret;
}
--
2.47.1
^ permalink raw reply related
* Re: [PATCH] crypto: af_alg - Add af_alg_restrict sysctl, defaulting to 1
From: Demi Marie Obenour @ 2026-06-24 2:09 UTC (permalink / raw)
To: Eric Biggers, Andy Lutomirski
Cc: linux-crypto, Herbert Xu, linux-kernel, linux-doc,
linux-bluetooth, iwd, linux-hardening, Milan Broz
In-Reply-To: <20260623192715.GE1850517@google.com>
[-- Attachment #1.1.1: Type: text/plain, Size: 2658 bytes --]
On 6/23/26 15:27, Eric Biggers wrote:
> On Tue, Jun 23, 2026 at 12:12:24PM -0700, Andy Lutomirski wrote:
>> On Mon, Jun 22, 2026 at 4:49 PM Eric Biggers <ebiggers@kernel.org> wrote:
>>>
>>> AF_ALG is a frequent source of vulnerabilities and a maintenance
>>> nightmare. It exposes far more functionality to userspace than ever
>>> should have been exposed, especially to unprivileged processes. Recent
>>> exploits have targeted kernel internal implementation details like
>>> "authencesn" that have zero use case for userspace access.
>>>
>>> Fortunately, AF_ALG is rarely used in practice, as userspace crypto
>>> libraries exist. And when it is used, only some functionality is known
>>> to be used, and many users are known to hold capabilities already.
>>> iwd for example requires CAP_NET_ADMIN and has a known algorithm list
>>> (https://lore.kernel.org/linux-crypto/bcbbef00-5881-421b-8892-7be6c04b832d@gmail.com/).
>>>
>>> Thus, let's restrict the set of allowed algorithms by default, depending
>>> on the capabilities held.
>>>
>>> Add a sysctl /proc/sys/crypto/af_alg_restrict with meaning:
>>>
>>> 0: unrestricted
>>> 1: limited functionality
>>> 2: completely disabled
>>>
>>> Set the default value to 1, which enables an algorithm allowlist for
>>> unprivileged processes and a slightly longer allowlist for privileged
>>> processes.
>>
>> In our brave new world of containers, this is a bit awkward. The
>> admin is sort of asking two separate questions:
>>
>> 1. Is the actual running distro and its privileged components capable
>> of working without AF_ALG or with only the parts marked as being
>> unprivileged?
>>
>> 2. Is the system running contains that need the unprivileged parts?
>> (Which is maybe just sha1 for ip? I really don't know.)
>>
>> Should there maybe be two separate options so that all options are
>> available? Or maybe something between 2 and 3 that means "limited
>> functionality and privileged modes are completely disabled"?
>
> If we want to offer more settings we could. I could see this getting
> quite complex pretty quickly once everyone weighs in, though. There's
> quite a bit of value in keeping things simple, even if the offered
> settings won't be optimal for every case.
>
> - Eric
What about exposing both allowlists to userspace and making them
configurable?
I'm mostly concerned about systems running code (possibly
closed-source) that uses algorithms that nobody here knows about.
It would be better to allow a single algorithm than to turn off all
restrictions.
--
Sincerely,
Demi Marie Obenour (she/her/hers)
[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 7253 bytes --]
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply
* Re: [PATCH] crypto: af_alg - Add af_alg_restrict sysctl, defaulting to 1
From: Rosen Penev @ 2026-06-23 22:06 UTC (permalink / raw)
To: Eric Biggers
Cc: linux-crypto, Herbert Xu, linux-kernel, linux-doc,
linux-bluetooth, iwd, linux-hardening, Milan Broz,
Demi Marie Obenour, Andy Lutomirski
In-Reply-To: <20260623214730.GA3281861@google.com>
On Tue, Jun 23, 2026 at 2:47 PM Eric Biggers <ebiggers@kernel.org> wrote:
>
> On Tue, Jun 23, 2026 at 02:28:17PM -0700, Rosen Penev wrote:
> > > +static const struct af_alg_allowlist_entry hash_allowlist[] = {
> > > + { "cmac(aes)", true }, /* iwd, bluez */
> > > + { "hmac(md5)", true }, /* iwd */
> > > + { "hmac(sha1)", true }, /* iwd */
> > > + { "hmac(sha224)", true }, /* iwd */
> > > + { "hmac(sha256)", true }, /* iwd */
> > > + { "hmac(sha384)", true }, /* iwd */
> > > + { "hmac(sha512)", true }, /* iwd, sha512hmac */
> > > + { "md4", true }, /* iwd */
> > > + { "md5", true }, /* iwd */
> > > + { "sha1", false }, /* iwd, iproute2 < 7.0 */
> > > + { "sha224", true }, /* iwd */
> > > + { "sha256", true }, /* iwd */
> > > + { "sha384", true }, /* iwd */
> > > + { "sha512", true }, /* iwd */
> > > + {},
> > In OpenWrt, https://gitlab.com/linux-afs/kafs-client and strongswan
> > seem to be the other users of the user API. I haven't looked into what
> > they need.
>
> [Please trim your replies, thanks!]
Not sure what you mean.
>
> https://gitlab.com/linux-afs/kafs-client uses AF_ALG only for
> "hmac(md5)", which I already put on the privileged allowlist due to iwd
> also using it. So it would still work by default with the current
> patch, unless it needs to use it unprivileged.
>
> (FWIW, a use of a single obsolete algorithm like this is also a good
> candidate for just replacing with local code...)
>
> https://github.com/strongswan/strongswan already supports userspace
> crypto libraries.
Oh lovely. Looks like this needs fixing in OpenWrt.
>
> - Eric
^ permalink raw reply
* Re: [PATCH] crypto: af_alg - Add af_alg_restrict sysctl, defaulting to 1
From: Eric Biggers @ 2026-06-23 21:47 UTC (permalink / raw)
To: Rosen Penev
Cc: linux-crypto, Herbert Xu, linux-kernel, linux-doc,
linux-bluetooth, iwd, linux-hardening, Milan Broz,
Demi Marie Obenour, Andy Lutomirski
In-Reply-To: <CAKxU2N_EGTWkvtPOxQXBroxGVXDf1atPoFVyRRu0wHOtEXVWaA@mail.gmail.com>
On Tue, Jun 23, 2026 at 02:28:17PM -0700, Rosen Penev wrote:
> > +static const struct af_alg_allowlist_entry hash_allowlist[] = {
> > + { "cmac(aes)", true }, /* iwd, bluez */
> > + { "hmac(md5)", true }, /* iwd */
> > + { "hmac(sha1)", true }, /* iwd */
> > + { "hmac(sha224)", true }, /* iwd */
> > + { "hmac(sha256)", true }, /* iwd */
> > + { "hmac(sha384)", true }, /* iwd */
> > + { "hmac(sha512)", true }, /* iwd, sha512hmac */
> > + { "md4", true }, /* iwd */
> > + { "md5", true }, /* iwd */
> > + { "sha1", false }, /* iwd, iproute2 < 7.0 */
> > + { "sha224", true }, /* iwd */
> > + { "sha256", true }, /* iwd */
> > + { "sha384", true }, /* iwd */
> > + { "sha512", true }, /* iwd */
> > + {},
> In OpenWrt, https://gitlab.com/linux-afs/kafs-client and strongswan
> seem to be the other users of the user API. I haven't looked into what
> they need.
[Please trim your replies, thanks!]
https://gitlab.com/linux-afs/kafs-client uses AF_ALG only for
"hmac(md5)", which I already put on the privileged allowlist due to iwd
also using it. So it would still work by default with the current
patch, unless it needs to use it unprivileged.
(FWIW, a use of a single obsolete algorithm like this is also a good
candidate for just replacing with local code...)
https://github.com/strongswan/strongswan already supports userspace
crypto libraries.
- Eric
^ permalink raw reply
* Re: [PATCH] crypto: af_alg - Add af_alg_restrict sysctl, defaulting to 1
From: Rosen Penev @ 2026-06-23 21:28 UTC (permalink / raw)
To: Eric Biggers
Cc: linux-crypto, Herbert Xu, linux-kernel, linux-doc,
linux-bluetooth, iwd, linux-hardening, Milan Broz,
Demi Marie Obenour, Andy Lutomirski
In-Reply-To: <20260622234803.6982-1-ebiggers@kernel.org>
On Mon, Jun 22, 2026 at 4:49 PM Eric Biggers <ebiggers@kernel.org> wrote:
>
> AF_ALG is a frequent source of vulnerabilities and a maintenance
> nightmare. It exposes far more functionality to userspace than ever
> should have been exposed, especially to unprivileged processes. Recent
> exploits have targeted kernel internal implementation details like
> "authencesn" that have zero use case for userspace access.
>
> Fortunately, AF_ALG is rarely used in practice, as userspace crypto
> libraries exist. And when it is used, only some functionality is known
> to be used, and many users are known to hold capabilities already.
> iwd for example requires CAP_NET_ADMIN and has a known algorithm list
> (https://lore.kernel.org/linux-crypto/bcbbef00-5881-421b-8892-7be6c04b832d@gmail.com/).
>
> Thus, let's restrict the set of allowed algorithms by default, depending
> on the capabilities held.
>
> Add a sysctl /proc/sys/crypto/af_alg_restrict with meaning:
>
> 0: unrestricted
> 1: limited functionality
> 2: completely disabled
>
> Set the default value to 1, which enables an algorithm allowlist for
> unprivileged processes and a slightly longer allowlist for privileged
> processes.
>
> Note that the list may be tweaked in the future. However, the common
> use cases such as iwd and bluez are taken into account already. I've
> tested that iwd still works with the default value of 1.
>
> Signed-off-by: Eric Biggers <ebiggers@kernel.org>
> ---
> Documentation/admin-guide/sysctl/crypto.rst | 36 +++++++++++
> Documentation/crypto/userspace-if.rst | 13 +++-
> crypto/af_alg.c | 72 +++++++++++++++++++--
> crypto/algif_aead.c | 11 ++++
> crypto/algif_hash.c | 24 +++++++
> crypto/algif_rng.c | 9 +++
> crypto/algif_skcipher.c | 20 ++++++
> include/crypto/if_alg.h | 8 +++
> 8 files changed, 184 insertions(+), 9 deletions(-)
>
> diff --git a/Documentation/admin-guide/sysctl/crypto.rst b/Documentation/admin-guide/sysctl/crypto.rst
> index b707bd314a64..9a1bd53287f4 100644
> --- a/Documentation/admin-guide/sysctl/crypto.rst
> +++ b/Documentation/admin-guide/sysctl/crypto.rst
> @@ -5,10 +5,46 @@
> These files show up in ``/proc/sys/crypto/``, depending on the
> kernel configuration:
>
> .. contents:: :local:
>
> +.. _af_alg_restrict:
> +
> +af_alg_restrict
> +===============
> +
> +Controls the level of restriction of AF_ALG.
> +
> +AF_ALG is a deprecated and rarely-used userspace interface that is a
> +frequent source of vulnerabilities. It also unnecessarily exposes a
> +large number of kernel implementation details. For more information
> +about AF_ALG, see :ref:`Documentation/crypto/userspace-if.rst
> +<crypto_userspace_interface>`.
> +
> +Starting in Linux v7.3, AF_ALG supports only a limited set of
> +algorithms by default. This sysctl allows the system administrator to
> +remove this restriction when needed for compatibility reasons, or to
> +go further and disable AF_ALG entirely. The default value is 1.
> +
> +=== ==================================================================
> +0 AF_ALG is unrestricted.
> +
> +1 AF_ALG is supported with a limited list of algorithms. The list
> + is designed for compatibility with known users such as iwd and
> + bluez that haven't yet been fixed to use userspace crypto code.
> +
> + Specifically, there is an allowlist for unprivileged processes
> + and a somewhat longer allowlist for processes that hold
> + CAP_SYS_ADMIN or CAP_NET_ADMIN in the initial user namespace.
> +
> + Attempts to bind() an AF_ALG socket with a disallowed algorithm
> + fail with ENOENT.
> +
> +2 AF_ALG is completely disabled. Attempts to create an AF_ALG
> + socket fail with EAFNOSUPPORT.
> +=== ==================================================================
> +
> fips_enabled
> ============
>
> Read-only flag that indicates whether FIPS mode is enabled.
>
> diff --git a/Documentation/crypto/userspace-if.rst b/Documentation/crypto/userspace-if.rst
> index ab93300c8e04..d6194346e366 100644
> --- a/Documentation/crypto/userspace-if.rst
> +++ b/Documentation/crypto/userspace-if.rst
> @@ -1,5 +1,7 @@
> +.. _crypto_userspace_interface:
> +
> User Space Interface
> ====================
>
> Introduction
> ------------
> @@ -10,13 +12,18 @@ code.
>
> AF_ALG is insecure and is deprecated. Originally added to the kernel in 2010,
> most kernel developers now consider it to be a mistake. Support for hardware
> accelerators, which was the original purpose of AF_ALG, has been removed.
>
> -AF_ALG continues to be supported only for backwards compatibility. On systems
> -where no programs using AF_ALG remain, the support for it should be disabled by
> -disabling ``CONFIG_CRYPTO_USER_API_*``.
> +AF_ALG continues to be supported only for backwards compatibility.
> +
> +Starting in Linux v7.3, the set of algorithms supported by AF_ALG is limited by
> +default. See :ref:`/proc/sys/crypto/af_alg_restrict <af_alg_restrict>`.
> +
> +On systems where no programs using AF_ALG remain, the support for it should be
> +disabled entirely by setting ``/proc/sys/crypto/af_alg_restrict`` to 2 or by
> +disabling ``CONFIG_CRYPTO_USER_API_*`` in the kernel configuration.
>
> Deprecation
> -----------
>
> AF_ALG was originally intended to provide userspace programs access to crypto
> diff --git a/crypto/af_alg.c b/crypto/af_alg.c
> index cce000e8590e..34b801568fba 100644
> --- a/crypto/af_alg.c
> +++ b/crypto/af_alg.c
> @@ -6,10 +6,11 @@
> *
> * Copyright (c) 2010 Herbert Xu <herbert@gondor.apana.org.au>
> */
>
> #include <linux/atomic.h>
> +#include <linux/capability.h>
> #include <crypto/if_alg.h>
> #include <linux/crypto.h>
> #include <linux/init.h>
> #include <linux/kernel.h>
> #include <linux/key.h>
> @@ -20,14 +21,32 @@
> #include <linux/rwsem.h>
> #include <linux/sched.h>
> #include <linux/sched/signal.h>
> #include <linux/security.h>
> #include <linux/string.h>
> +#include <linux/sysctl.h>
> +#include <linux/user_namespace.h>
> #include <keys/user-type.h>
> #include <keys/trusted-type.h>
> #include <keys/encrypted-type.h>
>
> +static int af_alg_restrict = 1;
> +
> +static const struct ctl_table af_alg_table[] = {
> + {
> + .procname = "af_alg_restrict",
> + .data = &af_alg_restrict,
> + .maxlen = sizeof(int),
> + .mode = 0644,
> + .proc_handler = proc_dointvec_minmax,
> + .extra1 = SYSCTL_ZERO,
> + .extra2 = SYSCTL_TWO,
> + },
> +};
> +
> +static struct ctl_table_header *af_alg_header;
> +
> struct alg_type_list {
> const struct af_alg_type *type;
> struct list_head list;
> };
>
> @@ -108,10 +127,43 @@ int af_alg_unregister_type(const struct af_alg_type *type)
>
> return err;
> }
> EXPORT_SYMBOL_GPL(af_alg_unregister_type);
>
> +static bool af_alg_capable(void)
> +{
> + return ns_capable_noaudit(&init_user_ns, CAP_NET_ADMIN) ||
> + capable(CAP_SYS_ADMIN);
> +}
> +
> +int af_alg_check_restriction(const char *name,
> + const struct af_alg_allowlist_entry allowlist[])
> +{
> + int level = READ_ONCE(af_alg_restrict);
> +
> + if (level == 0)
> + return 0;
> + if (level == 1) {
> + for (const struct af_alg_allowlist_entry *ent = allowlist;
> + ent->name; ent++) {
> + if (strcmp(name, ent->name) == 0 &&
> + (!ent->privileged || af_alg_capable()))
> + return 0;
> + }
> + }
> + /*
> + * Use -ENOENT (the error code for "algorithm not found") instead of
> + * -EACCES or -EPERM, for the highest chance of correctly triggering
> + * fallback code paths in userspace programs.
> + *
> + * Don't log a warning, since it would be noisy. iwd tries to bind a
> + * bunch of algorithms that it never uses.
> + */
> + return -ENOENT;
> +}
> +EXPORT_SYMBOL_GPL(af_alg_check_restriction);
> +
> static void alg_do_release(const struct af_alg_type *type, void *private)
> {
> if (!type)
> return;
>
> @@ -504,10 +556,13 @@ static int alg_create(struct net *net, struct socket *sock, int protocol,
> int kern)
> {
> struct sock *sk;
> int err;
>
> + if (READ_ONCE(af_alg_restrict) == 2)
> + return -EAFNOSUPPORT;
> +
> if (sock->type != SOCK_SEQPACKET)
> return -ESOCKTNOSUPPORT;
> if (protocol != 0)
> return -EPROTONOSUPPORT;
>
> @@ -1220,31 +1275,36 @@ int af_alg_get_rsgl(struct sock *sk, struct msghdr *msg, int flags,
> }
> EXPORT_SYMBOL_GPL(af_alg_get_rsgl);
>
> static int __init af_alg_init(void)
> {
> - int err = proto_register(&alg_proto, 0);
> + int err;
> +
> + af_alg_header = register_sysctl("crypto", af_alg_table);
>
> + err = proto_register(&alg_proto, 0);
> if (err)
> - goto out;
> + goto out_unregister_sysctl;
>
> err = sock_register(&alg_family);
> - if (err != 0)
> + if (err)
> goto out_unregister_proto;
>
> -out:
> - return err;
> + return 0;
>
> out_unregister_proto:
> proto_unregister(&alg_proto);
> - goto out;
> +out_unregister_sysctl:
> + unregister_sysctl_table(af_alg_header);
> + return err;
> }
>
> static void __exit af_alg_exit(void)
> {
> sock_unregister(PF_ALG);
> proto_unregister(&alg_proto);
> + unregister_sysctl_table(af_alg_header);
> }
>
> module_init(af_alg_init);
> module_exit(af_alg_exit);
> MODULE_DESCRIPTION("Crypto userspace interface");
> diff --git a/crypto/algif_aead.c b/crypto/algif_aead.c
> index 787aac8aeb24..b9217f9086aa 100644
> --- a/crypto/algif_aead.c
> +++ b/crypto/algif_aead.c
> @@ -32,10 +32,15 @@
> #include <linux/mm.h>
> #include <linux/module.h>
> #include <linux/net.h>
> #include <net/sock.h>
>
> +static const struct af_alg_allowlist_entry aead_allowlist[] = {
> + { "ccm(aes)", true }, /* bluez */
> + {},
> +};
> +
> static inline bool aead_sufficient_data(struct sock *sk)
> {
> struct alg_sock *ask = alg_sk(sk);
> struct sock *psk = ask->parent;
> struct alg_sock *pask = alg_sk(psk);
> @@ -342,10 +347,16 @@ static struct proto_ops algif_aead_ops_nokey = {
> .poll = af_alg_poll,
> };
>
> static void *aead_bind(const char *name)
> {
> + int err;
> +
> + err = af_alg_check_restriction(name, aead_allowlist);
> + if (err)
> + return ERR_PTR(err);
> +
> return crypto_alloc_aead(name, 0, AF_ALG_CRYPTOAPI_MASK);
> }
>
> static void aead_release(void *private)
> {
> diff --git a/crypto/algif_hash.c b/crypto/algif_hash.c
> index 5452ad6c1506..a8d958d51ece 100644
> --- a/crypto/algif_hash.c
> +++ b/crypto/algif_hash.c
> @@ -14,10 +14,28 @@
> #include <linux/mm.h>
> #include <linux/module.h>
> #include <linux/net.h>
> #include <net/sock.h>
>
> +static const struct af_alg_allowlist_entry hash_allowlist[] = {
> + { "cmac(aes)", true }, /* iwd, bluez */
> + { "hmac(md5)", true }, /* iwd */
> + { "hmac(sha1)", true }, /* iwd */
> + { "hmac(sha224)", true }, /* iwd */
> + { "hmac(sha256)", true }, /* iwd */
> + { "hmac(sha384)", true }, /* iwd */
> + { "hmac(sha512)", true }, /* iwd, sha512hmac */
> + { "md4", true }, /* iwd */
> + { "md5", true }, /* iwd */
> + { "sha1", false }, /* iwd, iproute2 < 7.0 */
> + { "sha224", true }, /* iwd */
> + { "sha256", true }, /* iwd */
> + { "sha384", true }, /* iwd */
> + { "sha512", true }, /* iwd */
> + {},
In OpenWrt, https://gitlab.com/linux-afs/kafs-client and strongswan
seem to be the other users of the user API. I haven't looked into what
they need.
> +};
> +
> struct hash_ctx {
> struct af_alg_sgl sgl;
>
> u8 *result;
>
> @@ -380,10 +398,16 @@ static struct proto_ops algif_hash_ops_nokey = {
> .accept = hash_accept_nokey,
> };
>
> static void *hash_bind(const char *name)
> {
> + int err;
> +
> + err = af_alg_check_restriction(name, hash_allowlist);
> + if (err)
> + return ERR_PTR(err);
> +
> return crypto_alloc_ahash(name, 0, AF_ALG_CRYPTOAPI_MASK);
> }
>
> static void hash_release(void *private)
> {
> diff --git a/crypto/algif_rng.c b/crypto/algif_rng.c
> index 4dfe7899f8fa..bd522915d56d 100644
> --- a/crypto/algif_rng.c
> +++ b/crypto/algif_rng.c
> @@ -48,10 +48,14 @@
>
> MODULE_LICENSE("GPL");
> MODULE_AUTHOR("Stephan Mueller <smueller@chronox.de>");
> MODULE_DESCRIPTION("User-space interface for random number generators");
>
> +static const struct af_alg_allowlist_entry rng_allowlist[] = {
> + {},
> +};
> +
> struct rng_ctx {
> #define MAXSIZE 128
> unsigned int len;
> struct crypto_rng *drng;
> u8 *addtl;
> @@ -199,10 +203,15 @@ static struct proto_ops __maybe_unused algif_rng_test_ops = {
>
> static void *rng_bind(const char *name)
> {
> struct rng_parent_ctx *pctx;
> struct crypto_rng *rng;
> + int err;
> +
> + err = af_alg_check_restriction(name, rng_allowlist);
> + if (err)
> + return ERR_PTR(err);
>
> pctx = kzalloc_obj(*pctx);
> if (!pctx)
> return ERR_PTR(-ENOMEM);
>
> diff --git a/crypto/algif_skcipher.c b/crypto/algif_skcipher.c
> index df20bdfe1f1f..2b8069667974 100644
> --- a/crypto/algif_skcipher.c
> +++ b/crypto/algif_skcipher.c
> @@ -32,10 +32,24 @@
> #include <linux/mm.h>
> #include <linux/module.h>
> #include <linux/net.h>
> #include <net/sock.h>
>
> +static const struct af_alg_allowlist_entry skcipher_allowlist[] = {
> + { "adiantum(xchacha12,aes)", false }, /* cryptsetup */
> + { "adiantum(xchacha20,aes)", false }, /* cryptsetup */
> + { "cbc(aes)", true }, /* iwd */
> + { "cbc(des)", true }, /* iwd */
> + { "cbc(des3_ede)", true }, /* iwd */
> + { "ctr(aes)", true }, /* iwd */
> + { "ecb(aes)", true }, /* iwd, bluez */
> + { "ecb(des)", true }, /* iwd */
> + { "hctr2(aes)", false }, /* cryptsetup */
> + { "xts(aes)", false }, /* cryptsetup benchmark */
> + {},
> +};
> +
> static int skcipher_sendmsg(struct socket *sock, struct msghdr *msg,
> size_t size)
> {
> struct sock *sk = sock->sk;
> struct alg_sock *ask = alg_sk(sk);
> @@ -307,10 +321,16 @@ static struct proto_ops algif_skcipher_ops_nokey = {
> .poll = af_alg_poll,
> };
>
> static void *skcipher_bind(const char *name)
> {
> + int err;
> +
> + err = af_alg_check_restriction(name, skcipher_allowlist);
> + if (err)
> + return ERR_PTR(err);
> +
> return crypto_alloc_skcipher(name, 0, AF_ALG_CRYPTOAPI_MASK);
> }
>
> static void skcipher_release(void *private)
> {
> diff --git a/include/crypto/if_alg.h b/include/crypto/if_alg.h
> index 7643ba954125..4e9ed8e73403 100644
> --- a/include/crypto/if_alg.h
> +++ b/include/crypto/if_alg.h
> @@ -159,13 +159,21 @@ struct af_alg_ctx {
> unsigned int len;
>
> unsigned int inflight;
> };
>
> +struct af_alg_allowlist_entry {
> + const char *name;
> + bool privileged;
> +};
> +
> int af_alg_register_type(const struct af_alg_type *type);
> int af_alg_unregister_type(const struct af_alg_type *type);
>
> +int af_alg_check_restriction(const char *name,
> + const struct af_alg_allowlist_entry allowlist[]);
> +
> int af_alg_release(struct socket *sock);
> void af_alg_release_parent(struct sock *sk);
> int af_alg_accept(struct sock *sk, struct socket *newsock,
> struct proto_accept_arg *arg);
>
>
> base-commit: 1dc18801be29bc54709aa355b8acd80e183b03cd
> --
> 2.54.0
>
>
^ permalink raw reply
* Re: [PATCH] crypto: af_alg - Add af_alg_restrict sysctl, defaulting to 1
From: Eric Biggers @ 2026-06-23 19:49 UTC (permalink / raw)
To: Kees Cook
Cc: linux-crypto, Herbert Xu, linux-kernel, linux-doc,
linux-bluetooth, iwd, linux-hardening, Milan Broz,
Demi Marie Obenour, Andy Lutomirski
In-Reply-To: <202606231216.14A774833@keescook>
On Tue, Jun 23, 2026 at 12:24:28PM -0700, Kees Cook wrote:
> On Mon, Jun 22, 2026 at 04:48:03PM -0700, Eric Biggers wrote:
> > AF_ALG is a frequent source of vulnerabilities and a maintenance
> > nightmare. It exposes far more functionality to userspace than ever
> > should have been exposed, especially to unprivileged processes. Recent
> > exploits have targeted kernel internal implementation details like
> > "authencesn" that have zero use case for userspace access.
>
> I absolutely want to see this attack surface reduction.
>
> > Add a sysctl /proc/sys/crypto/af_alg_restrict with meaning:
> > [...]
> > Note that the list may be tweaked in the future. However, the common
> > use cases such as iwd and bluez are taken into account already. I've
> > tested that iwd still works with the default value of 1.
>
> I wince at this bit, though. This is a "security policy in the kernel"
> which we try to avoid, and it's could be done already in userspace with
> modprobe blacklist.
>
> But, as you say, AF_ALG is deprecated. I understand that to mean that
> the alg list is only ever going to *shrink* in the future.
>
> Using a sysctl means monolithic kernels are protected, but wouldn't
> those systems just compile AF_ALG out?
>
> So, I guess, I would want a more clear rationale for why we do it this
> way instead of via modprobe blacklist. I see a few reasons, but they
> don't really convince me that we should ignore the "no security policy
> in the kernel" rule to do it this way.
As we saw when distros tried to mitigate copy.fail, a lot of distros
have CONFIG_CRYPTO_USER_API_* set to 'y', so algif_aead.ko couldn't be
blacklisted. (Ironically because of FIPS 140, which is yet another
example of how FIPS 140 harms real-world security.)
But even when 'm', the module blacklist is just a binary choice for each
algorithm type: aead, skcipher, hash, and rng. Loading algif_aead.ko
allows not just "ccm(aes)" that bluez needs, but also bizarre things
like "authencesn(hmac(sha256),cbc(aes))" that are used only in exploits.
And sure, userspace could theoretically gather the complete list of
algorithm modules (e.g. authencesn.ko) and blacklist them individually.
But no one does that, and many are built-in anyway -- and this time not
just because of FIPS.
So we need an allowlist at the algorithm level, not just the algorithm
type level. Putting the allowlist in the kernel, taking into account
the real use cases like iwd and bluez, and having a simple tristate
sysctl similar to some of the existing ones, is the simplest and most
practical way to achieve this by default across Linux distros.
If we did something like delegate the algorithm allowlist to LSMs, I
think that in practice it's just going to be almost never used.
- Eric
^ permalink raw reply
* Re: [PATCH] crypto: af_alg - Add af_alg_restrict sysctl, defaulting to 1
From: Eric Biggers @ 2026-06-23 19:27 UTC (permalink / raw)
To: Andy Lutomirski
Cc: linux-crypto, Herbert Xu, linux-kernel, linux-doc,
linux-bluetooth, iwd, linux-hardening, Milan Broz,
Demi Marie Obenour
In-Reply-To: <CALCETrXPj0u=FZ=aFcZAHk3fFZa7rCuPEjx6cOMXmT3sdkC7SA@mail.gmail.com>
On Tue, Jun 23, 2026 at 12:12:24PM -0700, Andy Lutomirski wrote:
> On Mon, Jun 22, 2026 at 4:49 PM Eric Biggers <ebiggers@kernel.org> wrote:
> >
> > AF_ALG is a frequent source of vulnerabilities and a maintenance
> > nightmare. It exposes far more functionality to userspace than ever
> > should have been exposed, especially to unprivileged processes. Recent
> > exploits have targeted kernel internal implementation details like
> > "authencesn" that have zero use case for userspace access.
> >
> > Fortunately, AF_ALG is rarely used in practice, as userspace crypto
> > libraries exist. And when it is used, only some functionality is known
> > to be used, and many users are known to hold capabilities already.
> > iwd for example requires CAP_NET_ADMIN and has a known algorithm list
> > (https://lore.kernel.org/linux-crypto/bcbbef00-5881-421b-8892-7be6c04b832d@gmail.com/).
> >
> > Thus, let's restrict the set of allowed algorithms by default, depending
> > on the capabilities held.
> >
> > Add a sysctl /proc/sys/crypto/af_alg_restrict with meaning:
> >
> > 0: unrestricted
> > 1: limited functionality
> > 2: completely disabled
> >
> > Set the default value to 1, which enables an algorithm allowlist for
> > unprivileged processes and a slightly longer allowlist for privileged
> > processes.
>
> In our brave new world of containers, this is a bit awkward. The
> admin is sort of asking two separate questions:
>
> 1. Is the actual running distro and its privileged components capable
> of working without AF_ALG or with only the parts marked as being
> unprivileged?
>
> 2. Is the system running contains that need the unprivileged parts?
> (Which is maybe just sha1 for ip? I really don't know.)
>
> Should there maybe be two separate options so that all options are
> available? Or maybe something between 2 and 3 that means "limited
> functionality and privileged modes are completely disabled"?
If we want to offer more settings we could. I could see this getting
quite complex pretty quickly once everyone weighs in, though. There's
quite a bit of value in keeping things simple, even if the offered
settings won't be optimal for every case.
- Eric
^ permalink raw reply
* Re: [PATCH] crypto: af_alg - Add af_alg_restrict sysctl, defaulting to 1
From: Kees Cook @ 2026-06-23 19:24 UTC (permalink / raw)
To: Eric Biggers
Cc: linux-crypto, Herbert Xu, linux-kernel, linux-doc,
linux-bluetooth, iwd, linux-hardening, Milan Broz,
Demi Marie Obenour, Andy Lutomirski
In-Reply-To: <20260622234803.6982-1-ebiggers@kernel.org>
On Mon, Jun 22, 2026 at 04:48:03PM -0700, Eric Biggers wrote:
> AF_ALG is a frequent source of vulnerabilities and a maintenance
> nightmare. It exposes far more functionality to userspace than ever
> should have been exposed, especially to unprivileged processes. Recent
> exploits have targeted kernel internal implementation details like
> "authencesn" that have zero use case for userspace access.
I absolutely want to see this attack surface reduction.
> Add a sysctl /proc/sys/crypto/af_alg_restrict with meaning:
> [...]
> Note that the list may be tweaked in the future. However, the common
> use cases such as iwd and bluez are taken into account already. I've
> tested that iwd still works with the default value of 1.
I wince at this bit, though. This is a "security policy in the kernel"
which we try to avoid, and it's could be done already in userspace with
modprobe blacklist.
But, as you say, AF_ALG is deprecated. I understand that to mean that
the alg list is only ever going to *shrink* in the future.
Using a sysctl means monolithic kernels are protected, but wouldn't
those systems just compile AF_ALG out?
So, I guess, I would want a more clear rationale for why we do it this
way instead of via modprobe blacklist. I see a few reasons, but they
don't really convince me that we should ignore the "no security policy
in the kernel" rule to do it this way.
-Kees
--
Kees Cook
^ permalink raw reply
* Re: [PATCH] crypto: af_alg - Document the deprecation of AF_ALG
From: Eric Biggers @ 2026-06-23 19:19 UTC (permalink / raw)
To: Linus Torvalds
Cc: Bastien Nocera, linux-crypto, Herbert Xu, Marcel Holtmann,
Luiz Augusto von Dentz, linux-doc, linux-api, linux-kernel,
netdev, linux-bluetooth, ell
In-Reply-To: <CAHk-=wgNG=F3xO9PjL0RcKy3UWvq0Np9uZu+nFUQBAA8So9xdA@mail.gmail.com>
On Tue, Jun 23, 2026 at 11:56:10AM -0700, Linus Torvalds wrote:
> On Tue, 23 Jun 2026 at 09:51, Eric Biggers <ebiggers@kernel.org> wrote:
> >
> > We're aware of that and are taking it into account in the allowlist:
>
> Note that if we can just unconditionally make it depend on
> CAP_NET_ADMIN, that would be good - independently of any allowlist.
>
> Because if iwd and abluetoothd are the main two users, and both of
> those already require CAP_NET_ADMIN anyway...
There's also cryptsetup, including unprivileged benchmarking and also
(in theory) formatting support, and pre-7.0 versions of iproute2 which
used it for computing SHA-1 hashes of BPF programs.
If we broke unprivileged 'cryptsetup benchmark', some people would
definitely notice. However, since it's just a manually-run benchmark
anyway, users could just run it with sudo.
I don't know about the iproute2 case.
It depends how aggressive we want to be. My current proposal
(https://lore.kernel.org/linux-crypto/20260622234803.6982-1-ebiggers@kernel.org/)
has the entries in the allowlist marked as either privileged or
unprivileged. There are just a few unprivileged ones, for cryptsetup
and iproute2 as mentioned. But we could try doing away with the
unprivileged ones entirely and see who complains.
- Eric
^ permalink raw reply
* Re: [PATCH v8 3/7] crypto/ccp: Disable CPU hotplug while SNP is active
From: Kalra, Ashish @ 2026-06-23 19:15 UTC (permalink / raw)
To: Ackerley Tng, Jethro Beekman, tglx, mingo, bp, dave.hansen, x86,
hpa, seanjc, peterz, thomas.lendacky, herbert, davem, ardb
Cc: pbonzini, aik, Michael.Roth, KPrateek.Nayak, Tycho.Andersen,
Nathan.Fontenot, jackyli, pgonda, rientjes, jacobhxu, xin,
pawan.kumar.gupta, babu.moger, dyoung, nikunj, john.allen, darwi,
linux-kernel, linux-crypto, kvm, linux-coco
In-Reply-To: <CAEvNRgHDNGCETxLsy0v-_cBO1=1U+tXtOXWEFrXLU7pYz7U9ow@mail.gmail.com>
Hello Ackerley,
On 6/23/2026 12:48 PM, Ackerley Tng wrote:
> Jethro Beekman <jethro@fortanix.com> writes:
>
>> On 2026-06-15 21:49, Ashish Kalra wrote:
>>> From: Ashish Kalra <ashish.kalra@amd.com>
>>>
>>> The SEV firmware enumerates the CPUs at SNP initialization and is not
>>> aware of the OS bringing CPUs online or offline afterwards, so OS CPU
>>> hotplug can diverge from the firmware's expectations and break SNP.
>>> Disable CPU hotplug while SNP is active.
>>
>> I think this is too broad. If I have a hypervisor that supports SNP virtualization, a (non-confidential) L1 guest running Linux should still support CPU hotplug while also running confidential L2 guests.
>>
>> --
>> Jethro Beekman | CTO | Fortanix
>>
>
> Were any other solutions considered other than disabling CPU hotplug?
>
> Is this temporary until something else is implemented?
>
> I'm not sure how commonly CPU hotplug is used, and if people are okay
> with trading in CPU hotplug to get SNP.
>
> Is it that fundamentally the SEV firmware can't support hotplug, so
> there's no point in keeping it enabled anyway?
Yes, essentially. The SEV firmware knows nothing about when the OS takes CPUs online or offline. At SNP_INIT it accounts for all
the CPUs enabled via the BIOS/UEFI and establishes the per-core SNP state for them; it has no notion of the OS bringing CPUs up or
down afterwards. So OS hotplug actions can diverge from the firmware's expectations and break SNP. Disabling hotplug just makes
that constraint explicit — there's nothing useful to keep it enabled for: a hot-removed core still "exists" as far as
the firmware and the per-core RMP/RMPOPT state are concerned, and a core brought online later was never set up for SNP.
>
> Is there some way of supporting hotplug for CPUs that won't be used with
> SNP, for serving non-SNP VMs on the same host as SNP VMs, or is that too
> complicated?
>
Not really. SNP's memory-integrity guarantee rests on a single invariant: every memory write is subject to RMP checks to protect
against corruption of SEV-SNP guest memory. The moment any CPU can issue writes that aren't RMP-checked, that protection is
broken for the whole system — it's not something that can be confined to "that one core."
That's because SNP isn't per-core in that sense — it's a system-wide mode. SYSCFG[SNP_EN] is set on every core, the RMP covers all
of physical memory, and once SNP is enabled every memory write is subject to RMP checks on every core. A non-SNP guest sharing the
host still runs on cores that are part of the SNP-enabled system.
By the SNP architecture there simply can't be a CPU that isn't doing RMP checks while SNP is active, so SNP_EN has to be enforced
on every core. RMP enforcement is gated per-core by SYSCFG[SNP_EN] and it must be set on every core before SNP_INIT; a core with
SNP_EN clear performs no RMP checks at all, which the architecture doesn't allow once SNP is up. A newly hotplugged CPU comes up
without SNP_EN (SNP not enabled on it), and since it wasn't present when SNP_INIT ran it isn't part of the initialized SNP
configuration either — so it does no RMP checking. And because an SNP guest's vCPUs (or any guest for that matter) can be scheduled
on any online CPU, the guest could end up running on that core, accessing memory with no RMP enforcement and breaking SNP's memory integrity.
There's no way to prevent that: KVM doesn't fence SNP guests (or any guests for that matter) off particular online cores.
And carving out cores for non-SNP-only use isn't possible by the architecture: SNP requires RMP checks on every CPU, so there's no valid
configuration with SNP active and a subset of cores exempt.
Thanks,
Ashish
>>>
>>> [...snip...]
>>>
^ permalink raw reply
* Re: [PATCH] crypto: af_alg - Add af_alg_restrict sysctl, defaulting to 1
From: Andy Lutomirski @ 2026-06-23 19:12 UTC (permalink / raw)
To: Eric Biggers
Cc: linux-crypto, Herbert Xu, linux-kernel, linux-doc,
linux-bluetooth, iwd, linux-hardening, Milan Broz,
Demi Marie Obenour
In-Reply-To: <20260622234803.6982-1-ebiggers@kernel.org>
On Mon, Jun 22, 2026 at 4:49 PM Eric Biggers <ebiggers@kernel.org> wrote:
>
> AF_ALG is a frequent source of vulnerabilities and a maintenance
> nightmare. It exposes far more functionality to userspace than ever
> should have been exposed, especially to unprivileged processes. Recent
> exploits have targeted kernel internal implementation details like
> "authencesn" that have zero use case for userspace access.
>
> Fortunately, AF_ALG is rarely used in practice, as userspace crypto
> libraries exist. And when it is used, only some functionality is known
> to be used, and many users are known to hold capabilities already.
> iwd for example requires CAP_NET_ADMIN and has a known algorithm list
> (https://lore.kernel.org/linux-crypto/bcbbef00-5881-421b-8892-7be6c04b832d@gmail.com/).
>
> Thus, let's restrict the set of allowed algorithms by default, depending
> on the capabilities held.
>
> Add a sysctl /proc/sys/crypto/af_alg_restrict with meaning:
>
> 0: unrestricted
> 1: limited functionality
> 2: completely disabled
>
> Set the default value to 1, which enables an algorithm allowlist for
> unprivileged processes and a slightly longer allowlist for privileged
> processes.
In our brave new world of containers, this is a bit awkward. The
admin is sort of asking two separate questions:
1. Is the actual running distro and its privileged components capable
of working without AF_ALG or with only the parts marked as being
unprivileged?
2. Is the system running contains that need the unprivileged parts?
(Which is maybe just sha1 for ip? I really don't know.)
Should there maybe be two separate options so that all options are
available? Or maybe something between 2 and 3 that means "limited
functionality and privileged modes are completely disabled"?
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox