[PATCH v9 00/14] Allow preemption during IPI completion waiting to improve real-time performance

[PATCH v9 00/14] Allow preemption during IPI completion waiting to improve real-time performance

[Chuyi Zhou]

Changes in v9:
 - Rewrite changelogs in [PATCH v9 1/14], [PATCH v9 2/14],
   [PATCH v9 4/14], [PATCH v9 5/14], [PATCH v9 6/14],
   [PATCH v9 7/14], and [PATCH v9 11/14].
 - Change smp_call_function_single()'s wait argument to bool and fix
   function/kernel-doc argument alignment in [PATCH v9 3/14].
 - Clean up task_struct IPI mask indentation, declaration ordering, and
   related initialization placement in [PATCH v9 4/14].
 - Update the CSD allocation comment in [PATCH v9 5/14].
 - In [PATCH v9 6/14], make put_cpu() unconditional after IPI
   queue/dispatch and clarify that the final csd_lock_wait() only waits
   for completion.
 - Add a comment for FLUSH_TLB_INFO_ALIGN, reference commit 780e0106d468
   directly, and add Sebastian's Reviewed-by tag in [PATCH v9 11/14].

Changes in v8:
 - Use MIN(SMP_CACHE_BYTES, 64) for FLUSH_TLB_INFO_ALIGN in
   [PATCH v8 11/14].
 - In flush_tlb_mm_range(), name the stack object "info" directly instead
   of keeping a temporary "_info" plus pointer in [PATCH v8 12/14].

Changes in v7:
 - Split the x86/mm flush_tlb_info stack-storage change into
   [PATCH v7 10/14], [PATCH v7 11/14], and [PATCH v7 12/14].
 - Rewrite the flush_tlb_info changelog with the historical context, test
   data, Link tag, and Acked-by tag.
 - Add a preparatory KVM patch for kvm_flush_tlb_multi() in
   [PATCH v7 13/14].
 - Refactor flush_tlb_mm_range() and arch_tlbbatch_flush() to avoid gotos
   in [PATCH v7 14/14].
 - Drop the flush_tlb_kernel_range() preemption optimization for now.
 - Tighten the smp_task_ipi_mask_alloc()/smp_task_ipi_mask_free()
   declarations in [PATCH v7 4/14].
 - Update the smp_call_function_many_cond() comment wording.
 - Add Paul E. McKenney's Tested-by tags and Sebastian's Reviewed-by tag.

Changes in v6:
 - Make the task-local cpumask selection explicit and drop preemptible()
   check in smp_call_function_many_cond(). The early put_cpu() decision now
   depends only on whether a task-local cpumask is available.
 - Keep smp_task_ipi_mask() private to kernel/smp.c in [PATCH v6 4/12].
 - Add #include <linux/slab.h> to kernel/smp.c in [PATCH v6 4/12] for
   kmalloc()/kfree(), fixing the kernel test robot build failure reported
   at:
   https://lore.kernel.org/oe-kbuild-all/202605241101.w6T2LApw-lkp@intel.com/
 - Update the csd_lock_wait() comment in [PATCH v6 6/12].
 - Add Sebastian's Reviewed-by tags to the reviewed patches.

Changes in v5:
 - Replace "smp: Remove get_cpu from smp_call_function_any" with a new
   approach that extracts a common __smp_call_function_single() to safely
   keep the remote CPU selection and IPI dispatch process within a single
   preemption-disabled region in [PATCH v5 3/12].
 - Fix a typo in comments (s/cpumask_stack/task_mask/) and remove the
   obsolete "Preemption must be disabled" constraint from the kernel-doc
   in [PATCH v5 6/12].
 - Adjust the WARN_ON_ONCE() validation condition to avoid a false positive
   warning caused by CPU hotplug races when use_cpus_read_lock is false in
   [PATCH v5 9/12].
 - Move the preemptible() check in smp_call_function_many_cond() from
   [PATCH v5 4/12] to [PATCH v5 6/12].
 - Rebase to commit 4ac4d6549a65 ("sched: Use trace_call__<tp>() to save a
   static branch").

Changes in v4:
 - Use task-local IPI cpumask rather than on-stack cpumask in
   [PATCH v4 4/12] (suggested by sebastian).
 - Skip to free csd memory in smpcfd_dead_cpu() to guarantee csd memory
   access safety, instead of using RCU mechanism in [PATCH v4 5/12]
   (suggested by sebastian).
 - Align flush_tlb_info with SMP_CACHE_BYTES to avoid performance
   degradation caused by unnecessary cache line movements in [PATCH v4
   10/12](suggested by sebastian and Nadav).
 - Collect Acked-bys and Reviewed-bys.

Changes in v3:
 - Add benchmarks to measure the performance impact of changing
   flush_tlb_info to stack variable in [PATCH v3 10/12] (suggested by
   peter)
 - Adjust the rcu_read_unlock() location in [PATCH v3 5/12] (suggested
   by muchun)
 - Use raw_smp_processor_id() to prevent warning[1] from
   check_preemption_disabled() in [PATCH v3 12/12].
 - Collect Acked-bys and Reviewed-by.

[1]: https://lore.kernel.org/lkml/20260302075216.2170675-1-zhouchuyi@bytedance.com/T/#mc39999cbeb3f50be176f0903d0fa4075688b073d

Changes in v2:
 - Simplify the code comments in [PATCH v2 2/12] (pointed by peter and
   muchun)
 - Adjust the preemption disabling logic in smp_call_function_any() in
   [PATCH v2 3/12] (suggested by peter).
 - Use on-stack cpumask only when !CONFIG_CPUMASK_OFFSTACK in [PATCH V2
   4/12] (pointed by peter)
 - Add [PATCH v2 5/12] to replace migrate_disable with the rcu mechanism
 - Adjust the preemption disabling logic to allow flush_tlb_multi() to be
   preemptible and migratable in [PATCH v2 11/12]
 - Collect Acked-bys and Reviewed-bys


Introduction
============

The vast majority of smp_call_function*() callers block until remote CPUs
complete the IPI function execution. As smp_call_function*() runs with
preemption disabled throughout, scheduling latency increases dramatically
with the number of remote CPUs and other factors (such as interrupts being
disabled).

On x86-64 architectures, TLB flushes are performed via IPIs; thus, during
process exit or when process-mapped pages are reclaimed, numerous IPI
operations must be awaited, leading to increased scheduling latency for
other threads on the current CPU. In our production environment, we
observed IPI wait-induced scheduling latency reaching up to 16ms on a
16-core machine. Our goal is to allow preemption during IPI completion
waiting to improve real-time performance.

Background
==========

In our production environments, latency-sensitive workloads (DPDK) are
configured with the highest priority to preempt lower-priority tasks at any
time. We discovered that DPDK's wake-up latency is primarily caused by the
current CPU having preemption disabled. Therefore, we collected the maximum
preemption disabled events within every 30-second interval and then
calculated the P50/P99 of these max preemption disabled events:


                        p50(ns)               p99(ns)
cpu0                   254956                 5465050
cpu1                   115801                 120782
cpu2                   43324                  72957
cpu3                   256637                 16723307
cpu4                   58979                  87237
cpu5                   47464                  79815
cpu6                   48881                  81371
cpu7                   52263                  82294
cpu8                   263555                 4657713
cpu9                   44935                  73962
cpu10                  37659                  65026
cpu11                  257008                 2706878
cpu12                  49669                  90006
cpu13                  45186                  74666
cpu14                  60705                  83866
cpu15                  51311                  86885

Meanwhile, we have collected the distribution of preemption disabling
events exceeding 1ms across different CPUs over several hours(I omitted
CPU data that were all zeros):

CPU        1~10ms   10~50ms   50~100ms
cpu0        29       5       0
cpu3        38       13      0
cpu8        34       6       0
cpu11       24       10      0

The preemption disabled for several milliseconds or even 10ms+ mostly
originates from TLB flush:

@stack[
    trace_preempt_on+143
    trace_preempt_on+143
    preempt_count_sub+67
    arch_tlbbatch_flush/flush_tlb_mm_range
    task_exit/page_reclaim/...
]

Further analysis confirms that the majority of the time is consumed in
csd_lock_wait().

Now smp_call*() always needs to disable preemption, mainly to protect its
internal per-CPU data structures and synchronize with CPU offline
operations. This patchset attempts to make csd_lock_wait() preemptible,
thereby reducing the preemption-disabled critical section and improving
kernel real-time performance.

Effect
======

After applying this patchset, we no longer observe preemption disabled for
more than 1ms on the arch_tlbbatch_flush/flush_tlb_mm_range path. The
overall P99 of max preemption disabled events in every 30-second interval
is reduced to around 1.5ms. The remaining latency is primarily due to lock
contention.

                     before patch    after patch    reduced by
                     -----------    --------------  ------------
p99(ns)                16723307        1556034        ~90.70%

Chuyi Zhou (14):
  smp: Disable preemption explicitly in __csd_lock_wait()
  smp: Enable preemption early in smp_call_function_single()
  smp: Refactor remote CPU selection in smp_call_function_any()
  smp: Use task-local IPI cpumask in smp_call_function_many_cond()
  smp: Alloc percpu csd data in smpcfd_prepare_cpu() only once
  smp: Enable preemption early in smp_call_function_many_cond()
  smp: Remove preempt_disable() from smp_call_function()
  smp: Remove preempt_disable() from on_each_cpu_cond_mask()
  scftorture: Remove preempt_disable() in scftorture_invoke_one()
  x86/mm: Factor out flush_tlb_info initialization
  x86/mm: Cap flush_tlb_info alignment at 64 bytes
  x86/mm: Move flush_tlb_info back to the stack
  x86/kvm: Disable preemption in kvm_flush_tlb_multi()
  x86/mm: Re-enable preemption before flush_tlb_multi()

 arch/x86/include/asm/tlbflush.h |   9 +-
 arch/x86/kernel/kvm.c           |   4 +-
 arch/x86/mm/tlb.c               |  95 +++++++---------
 include/linux/sched.h           |   6 ++
 include/linux/smp.h             |  15 ++-
 kernel/fork.c                   |   9 +-
 kernel/scftorture.c             |  13 +--
 kernel/smp.c                    | 186 ++++++++++++++++++++++----------
 kernel/up.c                     |   3 +-
 9 files changed, 210 insertions(+), 130 deletions(-)

-- 
2.20.1

[PATCH v9 01/14] smp: Disable preemption explicitly in __csd_lock_wait()

[Chuyi Zhou]

The CSD lock wait debugging code in __csd_lock_wait() must run with
preemption disabled. The smp function call mechanisms which invoke it
currently keep preemption disabled across the wait, so the debugging code
inherits that guarantee from its callers.

Keeping preemption disabled across the whole smp function call operation
can induce large scheduling latencies. Shortening the caller-side
preemption-disabled region would invoke __csd_lock_wait() with preemption
enabled.

Prepare for that by disabling preemption explicitly around the CSD lock
wait debugging code in __csd_lock_wait().

Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
Tested-by: Paul E. McKenney <paulmck@kernel.org>
Reviewed-by: Steven Rostedt (Google) <rostedt@goodmis.org>
Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Acked-by: Muchun Song <muchun.song@linux.dev>
---
 kernel/smp.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/kernel/smp.c b/kernel/smp.c
index a0bb56bd8dda..b58975480e11 100644
--- a/kernel/smp.c
+++ b/kernel/smp.c
@@ -323,6 +323,8 @@ static void __csd_lock_wait(call_single_data_t *csd)
 	int bug_id = 0;
 	u64 ts0, ts1;
 
+	guard(preempt)();
+
 	ts1 = ts0 = ktime_get_mono_fast_ns();
 	for (;;) {
 		if (csd_lock_wait_toolong(csd, ts0, &ts1, &bug_id, &nmessages))
-- 
2.20.1

[PATCH v9 02/14] smp: Enable preemption early in smp_call_function_single()

[Chuyi Zhou]

smp_call_function_single() disables preemption while it validates the
target CPU, prepares the call single data, queues the callback and sends
the IPI.

For the !wait case, preemption protects the per-CPU csd_data from
concurrent modification by another task on the same CPU. For the wait
case, the CSD is stack allocated and no other task can reuse it. CPU
pinning is still required until the callback has been queued and the IPI
has been sent, to ensure that the target CPU cannot be offlined after the
online check but before dispatch.

After generic_exec_single() has queued the callback, the synchronous
csd_lock_wait() invocation at the end of the execution does not require
the caller to remain pinned to the current CPU.

Enable preemption before csd_lock_wait() to shorten the
preemption-disabled section.

Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
Tested-by: Paul E. McKenney <paulmck@kernel.org>
Reviewed-by: Muchun Song <muchun.song@linux.dev>
Reviewed-by: Steven Rostedt (Google) <rostedt@goodmis.org>
Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
---
 kernel/smp.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/kernel/smp.c b/kernel/smp.c
index b58975480e11..292eefadddbc 100644
--- a/kernel/smp.c
+++ b/kernel/smp.c
@@ -700,11 +700,16 @@ int smp_call_function_single(int cpu, smp_call_func_t func, void *info,
 
 	err = generic_exec_single(cpu, csd);
 
+	/*
+	 * @csd is stack-allocated when @wait is true. No concurrent access
+	 * except from the IPI completion path, so we can re-enable preemption
+	 * early to reduce latency.
+	 */
+	put_cpu();
+
 	if (wait)
 		csd_lock_wait(csd);
 
-	put_cpu();
-
 	return err;
 }
 EXPORT_SYMBOL(smp_call_function_single);
-- 
2.20.1

[PATCH v9 03/14] smp: Refactor remote CPU selection in smp_call_function_any()

[Chuyi Zhou]

smp_call_function_any() disables preemption across the entire operation:
selecting a target CPU, enqueueing the IPI, and synchronously waiting for
the remote CPU. smp_call_function_single() already re-enables preemption
before the synchronous csd_lock_wait(), so callers of
smp_call_function_any() should benefit from the same shorter
preemption-disabled section.

Simply removing get_cpu() and put_cpu() from smp_call_function_any()
would leave the preemption disablement entirely to
smp_call_function_single(). That opens a preemption window between
selecting the remote CPU, for example via sched_numa_find_nth_cpu(), and
dispatching the IPI in smp_call_function_single(). If the selected CPU is
fully offlined in that window, smp_call_function_single() fails its
cpu_online() check and returns -ENXIO to the caller, violating the
guarantee that smp_call_function_any() executes on any online CPU in the
mask.

Move the remote CPU selection into a common
__smp_call_function_single() helper. Keep the target CPU selection and
IPI dispatch within the same preemption-disabled region, while still
allowing the wait path to use the shorter preemption-disabled section
provided by smp_call_function_single().

Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
Tested-by: Paul E. McKenney <paulmck@kernel.org>
Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
---
 include/linux/smp.h |  3 +--
 kernel/smp.c        | 56 ++++++++++++++++++++++++---------------------
 kernel/up.c         |  3 +--
 3 files changed, 32 insertions(+), 30 deletions(-)

diff --git a/include/linux/smp.h b/include/linux/smp.h
index 6925d15ccaa7..11e36c7bc4d6 100644
--- a/include/linux/smp.h
+++ b/include/linux/smp.h
@@ -47,8 +47,7 @@ extern void __smp_call_single_queue(int cpu, struct llist_node *node);
 /* total number of cpus in this system (may exceed NR_CPUS) */
 extern unsigned int total_cpus;
 
-int smp_call_function_single(int cpuid, smp_call_func_t func, void *info,
-			     int wait);
+int smp_call_function_single(int cpuid, smp_call_func_t func, void *info, bool wait);
 
 void on_each_cpu_cond_mask(smp_cond_func_t cond_func, smp_call_func_t func,
 			   void *info, bool wait, const struct cpumask *mask);
diff --git a/kernel/smp.c b/kernel/smp.c
index 292eefadddbc..92e1dffe4589 100644
--- a/kernel/smp.c
+++ b/kernel/smp.c
@@ -641,17 +641,9 @@ void flush_smp_call_function_queue(void)
 	local_irq_restore(flags);
 }
 
-/**
- * smp_call_function_single - Run a function on a specific CPU
- * @cpu: Specific target CPU for this function.
- * @func: The function to run. This must be fast and non-blocking.
- * @info: An arbitrary pointer to pass to the function.
- * @wait: If true, wait until function has completed on other CPUs.
- *
- * Returns: %0 on success, else a negative status code.
- */
-int smp_call_function_single(int cpu, smp_call_func_t func, void *info,
-			     int wait)
+static int __smp_call_function_single(int cpu, smp_call_func_t func,
+				      void *info, const struct cpumask *mask,
+				      bool wait)
 {
 	call_single_data_t *csd;
 	call_single_data_t csd_stack = {
@@ -668,6 +660,14 @@ int smp_call_function_single(int cpu, smp_call_func_t func, void *info,
 	 */
 	this_cpu = get_cpu();
 
+	if (mask) {
+		/* Try for same CPU (cheapest) */
+		if (!cpumask_test_cpu(this_cpu, mask))
+			cpu = sched_numa_find_nth_cpu(mask, 0, cpu_to_node(this_cpu));
+		else
+			cpu = this_cpu;
+	}
+
 	/*
 	 * Can deadlock when called with interrupts disabled.
 	 * We allow cpu's that are not yet online though, as no one else can
@@ -712,6 +712,20 @@ int smp_call_function_single(int cpu, smp_call_func_t func, void *info,
 
 	return err;
 }
+
+/**
+ * smp_call_function_single - Run a function on a specific CPU
+ * @cpu:	Specific target CPU for this function.
+ * @func:	The function to run. This must be fast and non-blocking.
+ * @info:	An arbitrary pointer to pass to the function.
+ * @wait:	If true, wait until function has completed on other CPUs.
+ *
+ * Returns: %0 on success, else a negative status code.
+ */
+int smp_call_function_single(int cpu, smp_call_func_t func, void *info, bool wait)
+{
+	return __smp_call_function_single(cpu, func, info, NULL, wait);
+}
 EXPORT_SYMBOL(smp_call_function_single);
 
 /**
@@ -762,10 +776,10 @@ EXPORT_SYMBOL_GPL(smp_call_function_single_async);
 
 /**
  * smp_call_function_any - Run a function on any of the given cpus
- * @mask: The mask of cpus it can run on.
- * @func: The function to run. This must be fast and non-blocking.
- * @info: An arbitrary pointer to pass to the function.
- * @wait: If true, wait until function has completed.
+ * @mask:	The mask of cpus it can run on.
+ * @func:	The function to run. This must be fast and non-blocking.
+ * @info:	An arbitrary pointer to pass to the function.
+ * @wait:	If true, wait until function has completed.
  *
  * Selection preference:
  *	1) current cpu if in @mask
@@ -776,17 +790,7 @@ EXPORT_SYMBOL_GPL(smp_call_function_single_async);
 int smp_call_function_any(const struct cpumask *mask,
 			  smp_call_func_t func, void *info, int wait)
 {
-	unsigned int cpu;
-	int ret;
-
-	/* Try for same CPU (cheapest) */
-	cpu = get_cpu();
-	if (!cpumask_test_cpu(cpu, mask))
-		cpu = sched_numa_find_nth_cpu(mask, 0, cpu_to_node(cpu));
-
-	ret = smp_call_function_single(cpu, func, info, wait);
-	put_cpu();
-	return ret;
+	return __smp_call_function_single(-1, func, info, mask, wait);
 }
 EXPORT_SYMBOL_GPL(smp_call_function_any);
 
diff --git a/kernel/up.c b/kernel/up.c
index df50828cc2f0..6d4ac9502e8b 100644
--- a/kernel/up.c
+++ b/kernel/up.c
@@ -9,8 +9,7 @@
 #include <linux/smp.h>
 #include <linux/hypervisor.h>
 
-int smp_call_function_single(int cpu, void (*func) (void *info), void *info,
-				int wait)
+int smp_call_function_single(int cpu, void (*func)(void *info), void *info, bool wait)
 {
 	unsigned long flags;
 
-- 
2.20.1

[PATCH v9 04/14] smp: Use task-local IPI cpumask in smp_call_function_many_cond()

[Chuyi Zhou]

smp_call_function_many_cond() uses the per-CPU cfd->cpumask as the list
of remote CPUs to wait for. That is safe while the caller remains pinned
to the current CPU for the whole operation, because another task cannot
run on the same CPU and reuse the per-CPU mask.

The synchronous wait is the long-latency part of the operation. To make
that wait preemptible, the mask iterated by csd_lock_wait() must remain
stable even if the task is preempted or migrates. If the wait used the
per-CPU cfd->cpumask after dropping CPU pinning, another task scheduled
on the original CPU could enter smp_call_function_many_cond() and
overwrite the mask while the first task is still iterating it.

Give each task private IPI cpumask storage and use it as the wait mask in
smp_call_function_many_cond(). Other cpumask storage choices do not fit
this use case:

 - Per-CPU storage is the state that becomes unsafe once the wait is
   made preemptible. After the caller drops CPU pinning, another task
   scheduled on the original CPU can enter smp_call_function_many_cond()
   and reuse the same per-CPU mask.

 - Stack storage is not suitable for large NR_CPUS or
   CONFIG_CPUMASK_OFFSTACK=y configurations. The wait mask needs to
   scale with cpumask_size(), and putting that storage on the stack is
   not acceptable on large systems.

 - Allocating the mask inside smp_call_function_many_cond() would put an
   allocation and a failure path in the generic IPI path. A sleeping
   allocation is not suitable because callers have historically only
   provided a preempt-disabled context, not a sleepable one. GFP_ATOMIC
   would avoid sleeping, but a failure fallback would make the latency
   improvement opportunistic instead of guaranteed.

The users are not limited to a small, pre-identifiable class of tasks. On
x86, ordinary tasks can reach this path through TLB flushes during exit,
unmap and reclaim, so allocating the mask only for a known subset of
tasks is not straightforward.

The memory cost is explicit: one word is added to task_struct. When
cpumask_size() fits in that word, the mask is stored inline and no
separate allocation is needed. Larger systems allocate cpumask_size() per
task; on x86-64 NR_CPUS=8192 this is about 1 KiB per task. For context,
x86 already carries several KiB of per-task architecture and FPU state,
depending on the enabled features and configuration. That does not make
the extra cpumask free, but it puts the large-NR_CPUS case in
perspective.

Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
Tested-by: Paul E. McKenney <paulmck@kernel.org>
Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
---
 include/linux/sched.h |  6 ++++
 include/linux/smp.h   | 12 ++++++++
 kernel/fork.c         |  9 +++++-
 kernel/smp.c          | 70 ++++++++++++++++++++++++++++++++++++++-----
 4 files changed, 88 insertions(+), 9 deletions(-)

diff --git a/include/linux/sched.h b/include/linux/sched.h
index 35e6183ef615..695a2d21a374 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -1364,6 +1364,12 @@ struct task_struct {
 	struct list_head		perf_event_list;
 	struct perf_ctx_data __rcu	*perf_ctx_data;
 #endif
+#if defined(CONFIG_SMP) && defined(CONFIG_PREEMPTION)
+	union {
+		cpumask_t			*ipi_mask_ptr;
+		unsigned long			ipi_mask_val;
+	};
+#endif
 #ifdef CONFIG_DEBUG_PREEMPT
 	unsigned long			preempt_disable_ip;
 #endif
diff --git a/include/linux/smp.h b/include/linux/smp.h
index 11e36c7bc4d6..2dfa7390717a 100644
--- a/include/linux/smp.h
+++ b/include/linux/smp.h
@@ -238,6 +238,18 @@ static inline int get_boot_cpu_id(void)
 
 #endif /* !SMP */
 
+#if defined(CONFIG_PREEMPTION) && defined(CONFIG_SMP)
+int smp_task_ipi_mask_alloc(struct task_struct *task);
+void smp_task_ipi_mask_free(struct task_struct *task);
+#else
+static inline int smp_task_ipi_mask_alloc(struct task_struct *task)
+{
+	return 0;
+}
+
+static inline void smp_task_ipi_mask_free(struct task_struct *task) { }
+#endif
+
 /*
  * raw_smp_processor_id() - get the current (unstable) CPU id
  *
diff --git a/kernel/fork.c b/kernel/fork.c
index 6fcca1db0af3..37f8343a3b74 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -535,6 +535,7 @@ void free_task(struct task_struct *tsk)
 #endif
 	release_user_cpus_ptr(tsk);
 	scs_release(tsk);
+	smp_task_ipi_mask_free(tsk);
 
 #ifndef CONFIG_THREAD_INFO_IN_TASK
 	/*
@@ -933,10 +934,14 @@ static struct task_struct *dup_task_struct(struct task_struct *orig, int node)
 #endif
 	account_kernel_stack(tsk, 1);
 
-	err = scs_prepare(tsk, node);
+	err = smp_task_ipi_mask_alloc(tsk);
 	if (err)
 		goto free_stack;
 
+	err = scs_prepare(tsk, node);
+	if (err)
+		goto free_ipi_mask;
+
 #ifdef CONFIG_SECCOMP
 	/*
 	 * We must handle setting up seccomp filters once we're under
@@ -1007,6 +1012,8 @@ static struct task_struct *dup_task_struct(struct task_struct *orig, int node)
 #endif
 	return tsk;
 
+free_ipi_mask:
+	smp_task_ipi_mask_free(tsk);
 free_stack:
 	exit_task_stack_account(tsk);
 	free_thread_stack(tsk);
diff --git a/kernel/smp.c b/kernel/smp.c
index 92e1dffe4589..e9d647385df1 100644
--- a/kernel/smp.c
+++ b/kernel/smp.c
@@ -16,6 +16,7 @@
 #include <linux/init.h>
 #include <linux/interrupt.h>
 #include <linux/gfp.h>
+#include <linux/slab.h>
 #include <linux/smp.h>
 #include <linux/cpu.h>
 #include <linux/sched.h>
@@ -794,6 +795,49 @@ int smp_call_function_any(const struct cpumask *mask,
 }
 EXPORT_SYMBOL_GPL(smp_call_function_any);
 
+static DEFINE_STATIC_KEY_FALSE(ipi_mask_inlined);
+
+#ifdef CONFIG_PREEMPTION
+
+int smp_task_ipi_mask_alloc(struct task_struct *task)
+{
+	if (static_branch_unlikely(&ipi_mask_inlined))
+		return 0;
+
+	task->ipi_mask_ptr = kmalloc(cpumask_size(), GFP_KERNEL);
+	if (!task->ipi_mask_ptr)
+		return -ENOMEM;
+
+	return 0;
+}
+
+void smp_task_ipi_mask_free(struct task_struct *task)
+{
+	if (static_branch_unlikely(&ipi_mask_inlined))
+		return;
+
+	kfree(task->ipi_mask_ptr);
+}
+
+static cpumask_t *smp_task_ipi_mask(struct task_struct *cur)
+{
+	/*
+	 * If cpumask_size() is smaller than or equal to the pointer
+	 * size, it stashes the cpumask in the pointer itself to
+	 * avoid extra memory allocations.
+	 */
+	if (static_branch_unlikely(&ipi_mask_inlined))
+		return (cpumask_t *)&cur->ipi_mask_val;
+
+	return cur->ipi_mask_ptr;
+}
+#else
+static cpumask_t *smp_task_ipi_mask(struct task_struct *cur)
+{
+	return NULL;
+}
+#endif
+
 /*
  * Flags to be used as scf_flags argument of smp_call_function_many_cond().
  *
@@ -809,13 +853,21 @@ static void smp_call_function_many_cond(const struct cpumask *mask,
 					smp_cond_func_t cond_func)
 {
 	int cpu, last_cpu, this_cpu = smp_processor_id();
-	struct call_function_data *cfd;
+	struct cpumask *cpumask, *task_mask;
 	bool wait = scf_flags & SCF_WAIT;
-	int nr_cpus = 0;
+	struct call_function_data *cfd;
 	bool run_remote = false;
+	int nr_cpus = 0;
 
 	lockdep_assert_preemption_disabled();
 
+	cfd = this_cpu_ptr(&cfd_data);
+	task_mask = smp_task_ipi_mask(current);
+	if (task_mask)
+		cpumask = task_mask;
+	else
+		cpumask = cfd->cpumask;
+
 	/*
 	 * Can deadlock when called with interrupts disabled.
 	 * We allow cpu's that are not yet online though, as no one else can
@@ -836,16 +888,15 @@ static void smp_call_function_many_cond(const struct cpumask *mask,
 
 	/* Check if we need remote execution, i.e., any CPU excluding this one. */
 	if (cpumask_any_and_but(mask, cpu_online_mask, this_cpu) < nr_cpu_ids) {
-		cfd = this_cpu_ptr(&cfd_data);
-		cpumask_and(cfd->cpumask, mask, cpu_online_mask);
-		__cpumask_clear_cpu(this_cpu, cfd->cpumask);
+		cpumask_and(cpumask, mask, cpu_online_mask);
+		__cpumask_clear_cpu(this_cpu, cpumask);
 
 		cpumask_clear(cfd->cpumask_ipi);
-		for_each_cpu(cpu, cfd->cpumask) {
+		for_each_cpu(cpu, cpumask) {
 			call_single_data_t *csd = per_cpu_ptr(cfd->csd, cpu);
 
 			if (cond_func && !cond_func(cpu, info)) {
-				__cpumask_clear_cpu(cpu, cfd->cpumask);
+				__cpumask_clear_cpu(cpu, cpumask);
 				continue;
 			}
 
@@ -896,7 +947,7 @@ static void smp_call_function_many_cond(const struct cpumask *mask,
 	}
 
 	if (run_remote && wait) {
-		for_each_cpu(cpu, cfd->cpumask) {
+		for_each_cpu(cpu, cpumask) {
 			call_single_data_t *csd;
 
 			csd = per_cpu_ptr(cfd->csd, cpu);
@@ -1010,6 +1061,9 @@ EXPORT_SYMBOL(nr_cpu_ids);
 void __init setup_nr_cpu_ids(void)
 {
 	set_nr_cpu_ids(find_last_bit(cpumask_bits(cpu_possible_mask), NR_CPUS) + 1);
+
+	if (IS_ENABLED(CONFIG_PREEMPTION) && cpumask_size() <= sizeof(unsigned long))
+		static_branch_enable(&ipi_mask_inlined);
 }
 
 /* Called by boot processor to activate the rest. */
-- 
2.20.1

[PATCH v9 05/14] smp: Alloc percpu csd data in smpcfd_prepare_cpu() only once

[Chuyi Zhou]

smp_call_function_many_cond() uses per-CPU CSD objects when queueing
callbacks to remote CPUs, and the wait path later dereferences those CSDs
from csd_lock_wait().

Making the wait path preemptible allows the initiating task to be
preempted or migrated before it waits for completion. A target CPU can be
offlined in that window. If smpcfd_dead_cpu() frees the target CPU's
per-CPU CSD storage, csd_lock_wait() can later dereference freed memory.

One way to protect the CSD storage is to free it via RCU or after a
synchronization step in the CPU offline path, but that would add
unnecessary complexity and can delay CPU shutdown.

Allocate the per-CPU CSD storage the first time a CPU comes up and keep
it allocated when the CPU is offlined. This allows csd_lock_wait() to
access the CSD even when the target CPU is offlined after preemption is
re-enabled and before the wait is invoked.

Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
Tested-by: Paul E. McKenney <paulmck@kernel.org>
Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Acked-by: Muchun Song <muchun.song@linux.dev>
---
 kernel/smp.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/kernel/smp.c b/kernel/smp.c
index e9d647385df1..e76de3010b30 100644
--- a/kernel/smp.c
+++ b/kernel/smp.c
@@ -64,7 +64,14 @@ int smpcfd_prepare_cpu(unsigned int cpu)
 		free_cpumask_var(cfd->cpumask);
 		return -ENOMEM;
 	}
-	cfd->csd = alloc_percpu(call_single_data_t);
+
+	/*
+	 * Allocate the per-CPU CSD the first time a CPU comes up. It is
+	 * not freed when the CPU is offlined, so csd_lock_wait() can access
+	 * it even when the CPU was offlined after preemption was re-enabled.
+	 */
+	if (!cfd->csd)
+		cfd->csd = alloc_percpu(call_single_data_t);
 	if (!cfd->csd) {
 		free_cpumask_var(cfd->cpumask);
 		free_cpumask_var(cfd->cpumask_ipi);
@@ -80,7 +87,6 @@ int smpcfd_dead_cpu(unsigned int cpu)
 
 	free_cpumask_var(cfd->cpumask);
 	free_cpumask_var(cfd->cpumask_ipi);
-	free_percpu(cfd->csd);
 	return 0;
 }
 
-- 
2.20.1

[PATCH v9 06/14] smp: Enable preemption early in smp_call_function_many_cond()

[Chuyi Zhou]

smp_call_function_many_cond() still has to keep the caller pinned to the
current CPU while the remote IPI request is built and dispatched. This
protects the queueing state and CPU-hotplug boundary that are required
before the synchronous wait starts:

 - It protects the current CPU's per-CPU scratch cpumask,
   cfd->cpumask_ipi. Another task running on the same CPU could otherwise
   enter smp_call_function_many_cond() and reuse that scratch cpumask
   before the current caller has finished building and sending the IPI
   request.

 - It provides the CPU-hotplug exclusion required by the CSD queueing
   side. New CSDs must not be queued after smpcfd_dying_cpu() has flushed
   the outgoing CPU's callback queue. Keeping preemption disabled until
   all required CSDs have been queued and the corresponding IPIs have
   been sent prevents CPU offline from crossing that boundary in the
   middle of the queueing operation.

The CSD acquisition side also relies on that caller-side CPU pinning.
csd_lock() waits for CSD_FLAG_LOCK to clear and then marks the CSD busy
with a regular store, so another task on the same CPU must not be
allowed to acquire and reinitialize the same per-CPU CSD concurrently.

After the callbacks have been queued and the IPIs have been sent, the
caller only performs the final csd_lock_wait() completion wait. If it is
preempted there, another task running on the original CPU may enter
smp_call_function_many_cond(), but any attempt to reuse the same per-CPU
CSD will block in csd_lock() until the previous callback clears
CSD_FLAG_LOCK. The final csd_lock_wait() does not acquire or reinitialize
the CSD, so it does not need the same caller-side preemption-disabled
protection.

The wait mask is task-local, so it cannot be overwritten by another task
on the original CPU. The per-CPU CSD storage also remains allocated
across CPU offline, so csd_lock_wait() can safely dereference it even if
the target CPU is offlined after the caller is unpinned.

With those requirements satisfied, enable preemption before the
synchronous csd_lock_wait() loop. This makes the potentially long wait
preemptible and migratable while keeping the CPU-pinned section around
the remote CPU selection and IPI dispatch.

Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
Tested-by: Paul E. McKenney <paulmck@kernel.org>
Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
---
 kernel/smp.c | 28 ++++++++++++++++++----------
 1 file changed, 18 insertions(+), 10 deletions(-)

diff --git a/kernel/smp.c b/kernel/smp.c
index e76de3010b30..92f984754139 100644
--- a/kernel/smp.c
+++ b/kernel/smp.c
@@ -858,15 +858,14 @@ static void smp_call_function_many_cond(const struct cpumask *mask,
 					unsigned int scf_flags,
 					smp_cond_func_t cond_func)
 {
-	int cpu, last_cpu, this_cpu = smp_processor_id();
 	struct cpumask *cpumask, *task_mask;
 	bool wait = scf_flags & SCF_WAIT;
 	struct call_function_data *cfd;
+	int cpu, last_cpu, this_cpu;
 	bool run_remote = false;
 	int nr_cpus = 0;
 
-	lockdep_assert_preemption_disabled();
-
+	this_cpu = get_cpu();
 	cfd = this_cpu_ptr(&cfd_data);
 	task_mask = smp_task_ipi_mask(current);
 	if (task_mask)
@@ -952,6 +951,16 @@ static void smp_call_function_many_cond(const struct cpumask *mask,
 		local_irq_restore(flags);
 	}
 
+	/*
+	 * The IPI work has been queued and dispatched. On PREEMPT kernels,
+	 * tasks created through dup_task_struct() have task-local wait masks.
+	 * The boot init_task can fall back to cfd->cpumask when the mask is
+	 * not inlined, but other tasks still use task-local masks and cannot
+	 * overwrite it. On !PREEMPT kernels, preempt_enable() cannot schedule
+	 * another task, so the per-CPU mask remains protected.
+	 */
+	put_cpu();
+
 	if (run_remote && wait) {
 		for_each_cpu(cpu, cpumask) {
 			call_single_data_t *csd;
@@ -964,15 +973,14 @@ static void smp_call_function_many_cond(const struct cpumask *mask,
 
 /**
  * smp_call_function_many() - Run a function on a set of CPUs.
- * @mask: The set of cpus to run on (only runs on online subset).
- * @func: The function to run. This must be fast and non-blocking.
- * @info: An arbitrary pointer to pass to the function.
- * @wait: If true, wait (atomically) until function has completed
- *        on other CPUs.
+ * @mask:	The set of cpus to run on (only runs on online subset).
+ * @func:	The function to run. This must be fast and non-blocking.
+ * @info:	An arbitrary pointer to pass to the function.
+ * @wait:	If true, wait (atomically) until function has completed
+ *		on other CPUs.
  *
  * You must not call this function with disabled interrupts or from a
- * hardware interrupt handler or from a bottom half handler. Preemption
- * must be disabled when calling this function.
+ * hardware interrupt handler or from a bottom half handler.
  *
  * @func is not called on the local CPU even if @mask contains it.  Consider
  * using on_each_cpu_cond_mask() instead if this is not desirable.
-- 
2.20.1

[PATCH v9 07/14] smp: Remove preempt_disable() from smp_call_function()

[Chuyi Zhou]

smp_call_function_many_cond() handles the preemption and CPU pinning
requirements internally. smp_call_function() only forwards the request to
that helper for cpu_online_mask and does not access CPU-local state on
its own.

Remove the outer preempt_disable() and preempt_enable() pair from
smp_call_function().

Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
Tested-by: Paul E. McKenney <paulmck@kernel.org>
Reviewed-by: Muchun Song <muchun.song@linux.dev>
Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
---
 kernel/smp.c | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)

diff --git a/kernel/smp.c b/kernel/smp.c
index 92f984754139..933a14dbb8f8 100644
--- a/kernel/smp.c
+++ b/kernel/smp.c
@@ -994,10 +994,10 @@ EXPORT_SYMBOL(smp_call_function_many);
 
 /**
  * smp_call_function() - Run a function on all other CPUs.
- * @func: The function to run. This must be fast and non-blocking.
- * @info: An arbitrary pointer to pass to the function.
- * @wait: If true, wait (atomically) until function has completed
- *        on other CPUs.
+ * @func:	The function to run. This must be fast and non-blocking.
+ * @info:	An arbitrary pointer to pass to the function.
+ * @wait:	If true, wait (atomically) until function has completed
+ *		on other CPUs.
  *
  * If @wait is true, then returns once @func has returned; otherwise
  * it returns just before the target cpu calls @func.
@@ -1007,9 +1007,8 @@ EXPORT_SYMBOL(smp_call_function_many);
  */
 void smp_call_function(smp_call_func_t func, void *info, int wait)
 {
-	preempt_disable();
-	smp_call_function_many(cpu_online_mask, func, info, wait);
-	preempt_enable();
+	smp_call_function_many_cond(cpu_online_mask, func, info,
+				    wait ? SCF_WAIT : 0, NULL);
 }
 EXPORT_SYMBOL(smp_call_function);
 
-- 
2.20.1

[PATCH v9 08/14] smp: Remove preempt_disable() from on_each_cpu_cond_mask()

[Chuyi Zhou]

smp_call_function_many_cond() handles the preemption and CPU pinning
requirements internally. on_each_cpu_cond_mask() only builds the call
flags and forwards the request to that helper.

Remove the outer preempt_disable() and preempt_enable() pair from
on_each_cpu_cond_mask().

Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
Tested-by: Paul E. McKenney <paulmck@kernel.org>
Reviewed-by: Muchun Song <muchun.song@linux.dev>
Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
---
 kernel/smp.c | 2 --
 1 file changed, 2 deletions(-)

diff --git a/kernel/smp.c b/kernel/smp.c
index 933a14dbb8f8..db0a123911d1 100644
--- a/kernel/smp.c
+++ b/kernel/smp.c
@@ -1131,9 +1131,7 @@ void on_each_cpu_cond_mask(smp_cond_func_t cond_func, smp_call_func_t func,
 	if (wait)
 		scf_flags |= SCF_WAIT;
 
-	preempt_disable();
 	smp_call_function_many_cond(mask, func, info, scf_flags, cond_func);
-	preempt_enable();
 }
 EXPORT_SYMBOL(on_each_cpu_cond_mask);
 
-- 
2.20.1

[PATCH v9 09/14] scftorture: Remove preempt_disable() in scftorture_invoke_one()

[Chuyi Zhou]

The smp_call*() functions handle their required preemption and CPU
pinning internally. The explicit preempt_disable() in
scftorture_invoke_one() is therefore no longer required for correctness.
Keeping the outer preempt_disable() would also prevent scftorture from
exercising the narrowed internal preemption-disabled regions during IPI
dispatch.

Removing the outer preemption protection can expose a CPU hotplug race in
the test validation when use_cpus_read_lock is false. For multicast
operations, SCF_PRIM_MANY or SCF_PRIM_ALL, if only one CPU is online,
smp_call_function_many() correctly skips sending IPIs and leaves scfc_out
false. Without preemption disabled, a CPU hotplug thread can preempt the
test thread, bring a second CPU online and increment num_online_cpus().
When the test thread resumes, the validation check can observe
num_online_cpus() > 1 and falsely trigger the memory-ordering warning,
leaking the scfcp structure.

Remove the preempt_disable() and preempt_enable() pairs around the
smp_call*() invocations in scftorture_invoke_one(). Restrict the
num_online_cpus() > 1 validation to the use_cpus_read_lock=true case,
where the CPU count is stable during the evaluation.

Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
Tested-by: Paul E. McKenney <paulmck@kernel.org>
Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
---
 kernel/scftorture.c | 13 ++++---------
 1 file changed, 4 insertions(+), 9 deletions(-)

diff --git a/kernel/scftorture.c b/kernel/scftorture.c
index 327c315f411c..2082f9b44370 100644
--- a/kernel/scftorture.c
+++ b/kernel/scftorture.c
@@ -348,6 +348,8 @@ static void scftorture_invoke_one(struct scf_statistics *scfp, struct torture_ra
 	int ret = 0;
 	struct scf_check *scfcp = NULL;
 	struct scf_selector *scfsp = scf_sel_rand(trsp);
+	bool is_single = (scfsp->scfs_prim == SCF_PRIM_SINGLE ||
+			  scfsp->scfs_prim == SCF_PRIM_SINGLE_RPC);
 
 	if (scfsp->scfs_prim == SCF_PRIM_SINGLE || scfsp->scfs_wait) {
 		scfcp = kmalloc_obj(*scfcp, GFP_ATOMIC);
@@ -364,8 +366,6 @@ static void scftorture_invoke_one(struct scf_statistics *scfp, struct torture_ra
 	}
 	if (use_cpus_read_lock)
 		cpus_read_lock();
-	else
-		preempt_disable();
 	switch (scfsp->scfs_prim) {
 	case SCF_PRIM_RESCHED:
 		if (IS_BUILTIN(CONFIG_SCF_TORTURE_TEST)) {
@@ -411,13 +411,10 @@ static void scftorture_invoke_one(struct scf_statistics *scfp, struct torture_ra
 		if (!ret) {
 			if (use_cpus_read_lock)
 				cpus_read_unlock();
-			else
-				preempt_enable();
+
 			wait_for_completion(&scfcp->scfc_completion);
 			if (use_cpus_read_lock)
 				cpus_read_lock();
-			else
-				preempt_disable();
 		} else {
 			scfp->n_single_rpc_ofl++;
 			scf_add_to_free_list(scfcp);
@@ -452,7 +449,7 @@ static void scftorture_invoke_one(struct scf_statistics *scfp, struct torture_ra
 			scfcp->scfc_out = true;
 	}
 	if (scfcp && scfsp->scfs_wait) {
-		if (WARN_ON_ONCE((num_online_cpus() > 1 || scfsp->scfs_prim == SCF_PRIM_SINGLE) &&
+		if (WARN_ON_ONCE(((use_cpus_read_lock && num_online_cpus() > 1) || is_single) &&
 				 !scfcp->scfc_out)) {
 			pr_warn("%s: Memory-ordering failure, scfs_prim: %d.\n", __func__, scfsp->scfs_prim);
 			atomic_inc(&n_mb_out_errs); // Leak rather than trash!
@@ -463,8 +460,6 @@ static void scftorture_invoke_one(struct scf_statistics *scfp, struct torture_ra
 	}
 	if (use_cpus_read_lock)
 		cpus_read_unlock();
-	else
-		preempt_enable();
 	if (allocfail)
 		schedule_timeout_idle((1 + longwait) * HZ);  // Let no-wait handlers complete.
 	else if (!(torture_random(trsp) & 0xfff))
-- 
2.20.1

[PATCH v9 10/14] x86/mm: Factor out flush_tlb_info initialization

[Chuyi Zhou]

get_flush_tlb_info() has two responsibilities: it reserves the per-CPU
flush_tlb_info storage and it initializes the fields that describe the
flush operation. The per-CPU storage also carries the DEBUG_VM
reentrancy check and the matching put_flush_tlb_info() lifetime rules.

Moving flush_tlb_info back to caller-provided storage requires the same
field initialization without tying the caller to the per-CPU object.
Leaving the field setup embedded in get_flush_tlb_info() would either
keep those callers tied to the per-CPU object or duplicate the
initialization logic.

Split the field setup into init_flush_tlb_info(). Keep the per-CPU
storage selection, DEBUG_VM reentrancy check and put_flush_tlb_info()
lifetime rules in get_flush_tlb_info().

No functional change intended.

Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
---
 arch/x86/mm/tlb.c | 42 +++++++++++++++++++++++++++---------------
 1 file changed, 27 insertions(+), 15 deletions(-)

diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c
index af43d177087e..f76f576d3899 100644
--- a/arch/x86/mm/tlb.c
+++ b/arch/x86/mm/tlb.c
@@ -1379,22 +1379,12 @@ static DEFINE_PER_CPU_SHARED_ALIGNED(struct flush_tlb_info, flush_tlb_info);
 static DEFINE_PER_CPU(unsigned int, flush_tlb_info_idx);
 #endif
 
-static struct flush_tlb_info *get_flush_tlb_info(struct mm_struct *mm,
-			unsigned long start, unsigned long end,
-			unsigned int stride_shift, bool freed_tables,
-			u64 new_tlb_gen)
+static void init_flush_tlb_info(struct flush_tlb_info *info,
+				struct mm_struct *mm,
+				unsigned long start, unsigned long end,
+				unsigned int stride_shift, bool freed_tables,
+				u64 new_tlb_gen)
 {
-	struct flush_tlb_info *info = this_cpu_ptr(&flush_tlb_info);
-
-#ifdef CONFIG_DEBUG_VM
-	/*
-	 * Ensure that the following code is non-reentrant and flush_tlb_info
-	 * is not overwritten. This means no TLB flushing is initiated by
-	 * interrupt handlers and machine-check exception handlers.
-	 */
-	BUG_ON(this_cpu_inc_return(flush_tlb_info_idx) != 1);
-#endif
-
 	/*
 	 * If the number of flushes is so large that a full flush
 	 * would be faster, do a full flush.
@@ -1412,6 +1402,28 @@ static struct flush_tlb_info *get_flush_tlb_info(struct mm_struct *mm,
 	info->new_tlb_gen	= new_tlb_gen;
 	info->initiating_cpu	= smp_processor_id();
 	info->trim_cpumask	= 0;
+}
+
+static struct flush_tlb_info *get_flush_tlb_info(struct mm_struct *mm,
+						 unsigned long start,
+						 unsigned long end,
+						 unsigned int stride_shift,
+						 bool freed_tables,
+						 u64 new_tlb_gen)
+{
+	struct flush_tlb_info *info = this_cpu_ptr(&flush_tlb_info);
+
+#ifdef CONFIG_DEBUG_VM
+	/*
+	 * Ensure that the following code is non-reentrant and flush_tlb_info
+	 * is not overwritten. This means no TLB flushing is initiated by
+	 * interrupt handlers and machine-check exception handlers.
+	 */
+	BUG_ON(this_cpu_inc_return(flush_tlb_info_idx) != 1);
+#endif
+
+	init_flush_tlb_info(info, mm, start, end, stride_shift, freed_tables,
+			    new_tlb_gen);
 
 	return info;
 }
-- 
2.20.1

[PATCH v9 11/14] x86/mm: Cap flush_tlb_info alignment at 64 bytes

[Chuyi Zhou]

A stack allocated flush_tlb_info should keep cacheline alignment to avoid
the regression that motivated the per-CPU storage, but using
SMP_CACHE_BYTES directly can make the stack frame grow excessively on
configurations with large cache lines. This was addressed by
commit 780e0106d468 ("x86/mm/tlb: Revert "x86/mm: Align TLB
invalidation info""), where the stack consumption reached 320 bytes.

Add FLUSH_TLB_INFO_ALIGN and cap the type alignment at 64 bytes. The
existing per-CPU flush_tlb_info instance remains
DEFINE_PER_CPU_SHARED_ALIGNED(), so its per-CPU shared-cacheline
alignment is unchanged.

This prepares for moving flush_tlb_info back to stack storage without
reintroducing the old large-cacheline stack usage problem.

Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
---
 arch/x86/include/asm/tlbflush.h | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h
index 0545fe75c3fa..70098d448e99 100644
--- a/arch/x86/include/asm/tlbflush.h
+++ b/arch/x86/include/asm/tlbflush.h
@@ -4,6 +4,7 @@
 
 #include <linux/mm_types.h>
 #include <linux/mmu_notifier.h>
+#include <linux/minmax.h>
 #include <linux/sched.h>
 
 #include <asm/barrier.h>
@@ -211,6 +212,12 @@ extern u16 invlpgb_count_max;
 
 extern void initialize_tlbstate_and_flush(void);
 
+/*
+ * Keep stack-allocated flush_tlb_info cacheline aligned, but cap the
+ * alignment to avoid excessive stack usage on large-cacheline systems.
+ */
+#define FLUSH_TLB_INFO_ALIGN MIN(SMP_CACHE_BYTES, 64)
+
 /*
  * TLB flushing:
  *
@@ -249,7 +256,7 @@ struct flush_tlb_info {
 	u8			stride_shift;
 	u8			freed_tables;
 	u8			trim_cpumask;
-};
+} __aligned(FLUSH_TLB_INFO_ALIGN);
 
 void flush_tlb_local(void);
 void flush_tlb_one_user(unsigned long addr);
-- 
2.20.1

[PATCH v9 12/14] x86/mm: Move flush_tlb_info back to the stack

[Chuyi Zhou]

flush_tlb_info benefits from cacheline alignment, but using
cacheline-aligned stack storage directly can grow stack usage too much on
configurations with large SMP_CACHE_BYTES values. Commit 515ab7c41306
("x86/mm: Align TLB invalidation info") attempted to align stack storage,
and commit 780e0106d468 ("x86/mm/tlb: Revert "x86/mm: Align TLB
invalidation info"") reverted it because using SMP_CACHE_BYTES led to 320
bytes of stack consumption. Commit 3db6d5a5ecaf ("x86/mm/tlb: Remove
'struct flush_tlb_info' from the stack") moved flush_tlb_info to per-CPU
storage, which avoided the stack growth problem while preserving
cacheline alignment. That was a good fit while the callers kept
preemption disabled for the whole flush operation.

However, a single per-CPU flush_tlb_info also requires all flush_tlb*()
operations to keep preemption disabled while the object is in use, so
that it cannot be overwritten by another flush on the same CPU.
flush_tlb*() may send IPIs to remote CPUs and synchronously wait for all
remote CPUs to complete their local TLB flushes. That wait can take tens
of milliseconds when interrupts are disabled on a remote CPU or when a
large number of remote CPUs are involved.

To shorten the CPU-pinned and preemption-disabled section around those
remote TLB flush waits, move flush_tlb_info back to caller-private stack
storage. The caller then does not have to stay on the same CPU until the
remote flush completes.

The type alignment is capped at 64 bytes. This keeps the alignment
benefit for stack objects without reintroducing the old large-cacheline
stack usage problem.

To evaluate the performance impact, use the following script to reproduce
the microbenchmark mentioned in commit 3db6d5a5ecaf ("x86/mm/tlb: Remove
'struct flush_tlb_info' from the stack"). The test environment is an Ice
Lake system (Intel(R) Xeon(R) Platinum 8336C) with 128 CPUs and 2 NUMA
nodes. During the test, the threads were bound to specific CPUs, and both
pti and mitigations were disabled:

    #include <stdio.h>
    #include <stdlib.h>
    #include <pthread.h>
    #include <sys/mman.h>
    #include <sys/time.h>
    #include <unistd.h>

    #define NUM_OPS 1000000
    #define NUM_THREADS 3
    #define NUM_RUNS 5
    #define PAGE_SIZE 4096

    volatile int stop_threads = 0;

    void *busy_wait_thread(void *arg) {
        while (!stop_threads) {
            __asm__ volatile ("nop");
        }
        return NULL;
    }

    long long get_usec() {
        struct timeval tv;
        gettimeofday(&tv, NULL);
        return tv.tv_sec * 1000000LL + tv.tv_usec;
    }

    int main() {
        pthread_t threads[NUM_THREADS];
        char *addr;
        int i, r;
        addr = mmap(NULL, PAGE_SIZE, PROT_READ | PROT_WRITE, MAP_PRIVATE
                | MAP_ANONYMOUS, -1, 0);

        if (addr == MAP_FAILED) {
            perror("mmap");
            exit(1);
        }

        for (i = 0; i < NUM_THREADS; i++) {
            if (pthread_create(&threads[i], NULL, busy_wait_thread, NULL))
                exit(1);
        }

        printf("Running benchmark: %d runs, %d ops each, %d background\n"
               "threads\n", NUM_RUNS, NUM_OPS, NUM_THREADS);

        for (r = 0; r < NUM_RUNS; r++) {
            long long start, end;
            start = get_usec();
            for (i = 0; i < NUM_OPS; i++) {
                addr[0] = 1;
                if (madvise(addr, PAGE_SIZE, MADV_DONTNEED)) {
                    perror("madvise");
                    exit(1);
                }
            }
            end = get_usec();
            double duration = (double)(end - start);
            double avg_lat = duration / NUM_OPS;
            printf("Run %d: Total time %.2f us, Avg latency %.4f us/op\n",
                   r + 1, duration, avg_lat);
        }
        stop_threads = 1;
        for (i = 0; i < NUM_THREADS; i++)
            pthread_join(threads[i], NULL);
        munmap(addr, PAGE_SIZE);
        return 0;
    }

                   base   on-stack-aligned  on-stack-not-aligned
                   ----       ---------      -----------
avg (usec/op)     2.5278       2.5261         2.5508
stddev            0.0007       0.0027         0.0023

The benchmark results show that the average latency difference between
the baseline (base) and the properly aligned stack variable
(on-stack-aligned) is within the standard deviation (stddev). This
indicates that the variations are caused by testing noise, and reverting
to a stack variable with proper alignment causes no performance
regression compared to the per-CPU implementation. The unaligned version
(on-stack-not-aligned) shows a minor performance drop. The
CPU-pinned/preemption-disabled section can therefore be shortened without
sacrificing performance.

With caller-private storage there is no shared per-CPU object to protect,
so remove the DEBUG_VM reentrancy counter as well.

Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
Tested-by: Paul E. McKenney <paulmck@kernel.org>
Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Acked-by: Nadav Amit <nadav.amit@gmail.com>
---
 arch/x86/mm/tlb.c | 80 +++++++++++------------------------------------
 1 file changed, 18 insertions(+), 62 deletions(-)

diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c
index f76f576d3899..0620c001981f 100644
--- a/arch/x86/mm/tlb.c
+++ b/arch/x86/mm/tlb.c
@@ -1373,12 +1373,6 @@ void flush_tlb_multi(const struct cpumask *cpumask,
  */
 unsigned long tlb_single_page_flush_ceiling __read_mostly = 33;
 
-static DEFINE_PER_CPU_SHARED_ALIGNED(struct flush_tlb_info, flush_tlb_info);
-
-#ifdef CONFIG_DEBUG_VM
-static DEFINE_PER_CPU(unsigned int, flush_tlb_info_idx);
-#endif
-
 static void init_flush_tlb_info(struct flush_tlb_info *info,
 				struct mm_struct *mm,
 				unsigned long start, unsigned long end,
@@ -1404,52 +1398,19 @@ static void init_flush_tlb_info(struct flush_tlb_info *info,
 	info->trim_cpumask	= 0;
 }
 
-static struct flush_tlb_info *get_flush_tlb_info(struct mm_struct *mm,
-						 unsigned long start,
-						 unsigned long end,
-						 unsigned int stride_shift,
-						 bool freed_tables,
-						 u64 new_tlb_gen)
-{
-	struct flush_tlb_info *info = this_cpu_ptr(&flush_tlb_info);
-
-#ifdef CONFIG_DEBUG_VM
-	/*
-	 * Ensure that the following code is non-reentrant and flush_tlb_info
-	 * is not overwritten. This means no TLB flushing is initiated by
-	 * interrupt handlers and machine-check exception handlers.
-	 */
-	BUG_ON(this_cpu_inc_return(flush_tlb_info_idx) != 1);
-#endif
-
-	init_flush_tlb_info(info, mm, start, end, stride_shift, freed_tables,
-			    new_tlb_gen);
-
-	return info;
-}
-
-static void put_flush_tlb_info(void)
-{
-#ifdef CONFIG_DEBUG_VM
-	/* Complete reentrancy prevention checks */
-	barrier();
-	this_cpu_dec(flush_tlb_info_idx);
-#endif
-}
-
 void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start,
 				unsigned long end, unsigned int stride_shift,
 				bool freed_tables)
 {
-	struct flush_tlb_info *info;
+	struct flush_tlb_info info;
 	int cpu = get_cpu();
 	u64 new_tlb_gen;
 
 	/* This is also a barrier that synchronizes with switch_mm(). */
 	new_tlb_gen = inc_mm_tlb_gen(mm);
 
-	info = get_flush_tlb_info(mm, start, end, stride_shift, freed_tables,
-				  new_tlb_gen);
+	init_flush_tlb_info(&info, mm, start, end, stride_shift, freed_tables,
+			    new_tlb_gen);
 
 	/*
 	 * flush_tlb_multi() is not optimized for the common case in which only
@@ -1457,19 +1418,18 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start,
 	 * flush_tlb_func_local() directly in this case.
 	 */
 	if (mm_global_asid(mm)) {
-		broadcast_tlb_flush(info);
+		broadcast_tlb_flush(&info);
 	} else if (cpumask_any_but(mm_cpumask(mm), cpu) < nr_cpu_ids) {
-		info->trim_cpumask = should_trim_cpumask(mm);
-		flush_tlb_multi(mm_cpumask(mm), info);
+		info.trim_cpumask = should_trim_cpumask(mm);
+		flush_tlb_multi(mm_cpumask(mm), &info);
 		consider_global_asid(mm);
 	} else if (mm == this_cpu_read(cpu_tlbstate.loaded_mm)) {
 		lockdep_assert_irqs_enabled();
 		local_irq_disable();
-		flush_tlb_func(info);
+		flush_tlb_func(&info);
 		local_irq_enable();
 	}
 
-	put_flush_tlb_info();
 	put_cpu();
 	mmu_notifier_arch_invalidate_secondary_tlbs(mm, start, end);
 }
@@ -1539,19 +1499,16 @@ static void kernel_tlb_flush_range(struct flush_tlb_info *info)
 
 void flush_tlb_kernel_range(unsigned long start, unsigned long end)
 {
-	struct flush_tlb_info *info;
+	struct flush_tlb_info info;
 
 	guard(preempt)();
+	init_flush_tlb_info(&info, NULL, start, end, PAGE_SHIFT, false,
+			    TLB_GENERATION_INVALID);
 
-	info = get_flush_tlb_info(NULL, start, end, PAGE_SHIFT, false,
-				  TLB_GENERATION_INVALID);
-
-	if (info->end == TLB_FLUSH_ALL)
-		kernel_tlb_flush_all(info);
+	if (info.end == TLB_FLUSH_ALL)
+		kernel_tlb_flush_all(&info);
 	else
-		kernel_tlb_flush_range(info);
-
-	put_flush_tlb_info();
+		kernel_tlb_flush_range(&info);
 }
 
 /*
@@ -1719,12 +1676,12 @@ EXPORT_SYMBOL_FOR_KVM(__flush_tlb_all);
 
 void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)
 {
-	struct flush_tlb_info *info;
+	struct flush_tlb_info info;
 
 	int cpu = get_cpu();
 
-	info = get_flush_tlb_info(NULL, 0, TLB_FLUSH_ALL, 0, false,
-				  TLB_GENERATION_INVALID);
+	init_flush_tlb_info(&info, NULL, 0, TLB_FLUSH_ALL, 0, false,
+			    TLB_GENERATION_INVALID);
 	/*
 	 * flush_tlb_multi() is not optimized for the common case in which only
 	 * a local TLB flush is needed. Optimize this use-case by calling
@@ -1734,17 +1691,16 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)
 		invlpgb_flush_all_nonglobals();
 		batch->unmapped_pages = false;
 	} else if (cpumask_any_but(&batch->cpumask, cpu) < nr_cpu_ids) {
-		flush_tlb_multi(&batch->cpumask, info);
+		flush_tlb_multi(&batch->cpumask, &info);
 	} else if (cpumask_test_cpu(cpu, &batch->cpumask)) {
 		lockdep_assert_irqs_enabled();
 		local_irq_disable();
-		flush_tlb_func(info);
+		flush_tlb_func(&info);
 		local_irq_enable();
 	}
 
 	cpumask_clear(&batch->cpumask);
 
-	put_flush_tlb_info();
 	put_cpu();
 }
 
-- 
2.20.1

[PATCH v9 13/14] x86/kvm: Disable preemption in kvm_flush_tlb_multi()

[Chuyi Zhou]

kvm_flush_tlb_multi() is installed as an x86 PV TLB flush backend, so
flush_tlb_multi() can reach it through pv_ops when running as a KVM
guest.

kvm_flush_tlb_multi() uses the per-CPU scratch cpumask __pv_cpu_mask.
That buffer must remain tied to the current CPU until the mask has been
copied, filtered, and consumed by native_flush_tlb_multi().

The x86/mm callers currently enter flush_tlb_multi() while pinned to a
CPU. To let those callers drop CPU pinning before issuing the remote TLB
flush, each PV backend must protect its own CPU-local scratch state.

Make the KVM backend protect its per-CPU scratch cpumask by disabling
preemption locally. This is harmless with the current callers, where the
preemption disable is nested, and makes the KVM pv_ops dependency
explicit before changing the x86/mm call sites.

Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
---
 arch/x86/kernel/kvm.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c
index 29226d112029..d540f54f4d16 100644
--- a/arch/x86/kernel/kvm.c
+++ b/arch/x86/kernel/kvm.c
@@ -662,8 +662,10 @@ static void kvm_flush_tlb_multi(const struct cpumask *cpumask,
 	u8 state;
 	int cpu;
 	struct kvm_steal_time *src;
-	struct cpumask *flushmask = this_cpu_cpumask_var_ptr(__pv_cpu_mask);
+	struct cpumask *flushmask;
 
+	guard(preempt)();
+	flushmask = this_cpu_cpumask_var_ptr(__pv_cpu_mask);
 	cpumask_copy(flushmask, cpumask);
 	/*
 	 * We have to call flush only on online vCPUs. And
-- 
2.20.1

[PATCH v9 14/14] x86/mm: Re-enable preemption before flush_tlb_multi()

[Chuyi Zhou]

flush_tlb_mm_range() and arch_tlbbatch_flush() pin the current CPU while
they decide whether the flush can be handled locally or must be sent to
remote CPUs. The CPU pinning is needed for the current CPU number and for
the local TLB flush path, which reads per-CPU TLB state.

The caller does not need to remain pinned while waiting for a remote TLB
flush to complete. After the remote-flush path has been selected,
flush_tlb_info is caller-private stack storage, so the caller no longer
has to stay on the same CPU to protect a shared per-CPU flush_tlb_info
object.

flush_tlb_multi() may also route through x86 PV backends. Those backends
must protect their own CPU-local scratch state instead of relying on the
caller to stay pinned. Hyper-V already does this by disabling interrupts
while using hyperv_pcpu_input_arg, and Xen's multicall path brackets its
per-CPU multicall buffer with xen_mc_batch() and xen_mc_issue().
kvm_flush_tlb_multi() also disables preemption while using __pv_cpu_mask.

Remote TLB flushes may synchronously wait for many CPUs, and the wait can
take tens of milliseconds when remote CPUs have interrupts disabled or
when many CPUs are involved. Keeping preemption disabled for that whole
wait unnecessarily increases scheduling latency on the initiating CPU.

Drop the CPU pinning before calling flush_tlb_multi() in the remote paths
of flush_tlb_mm_range() and arch_tlbbatch_flush(). Keep the local paths
inside the pinned section because they still access this CPU's TLB state.

Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
Tested-by: Paul E. McKenney <paulmck@kernel.org>
Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
---
 arch/x86/mm/tlb.c | 23 ++++++++++++++++-------
 1 file changed, 16 insertions(+), 7 deletions(-)

diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c
index 0620c001981f..3b021930cc69 100644
--- a/arch/x86/mm/tlb.c
+++ b/arch/x86/mm/tlb.c
@@ -1403,6 +1403,7 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start,
 				bool freed_tables)
 {
 	struct flush_tlb_info info;
+	bool remote_flush = false;
 	int cpu = get_cpu();
 	u64 new_tlb_gen;
 
@@ -1420,9 +1421,7 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start,
 	if (mm_global_asid(mm)) {
 		broadcast_tlb_flush(&info);
 	} else if (cpumask_any_but(mm_cpumask(mm), cpu) < nr_cpu_ids) {
-		info.trim_cpumask = should_trim_cpumask(mm);
-		flush_tlb_multi(mm_cpumask(mm), &info);
-		consider_global_asid(mm);
+		remote_flush = true;
 	} else if (mm == this_cpu_read(cpu_tlbstate.loaded_mm)) {
 		lockdep_assert_irqs_enabled();
 		local_irq_disable();
@@ -1431,6 +1430,13 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start,
 	}
 
 	put_cpu();
+
+	if (remote_flush) {
+		info.trim_cpumask = should_trim_cpumask(mm);
+		flush_tlb_multi(mm_cpumask(mm), &info);
+		consider_global_asid(mm);
+	}
+
 	mmu_notifier_arch_invalidate_secondary_tlbs(mm, start, end);
 }
 
@@ -1677,7 +1683,7 @@ EXPORT_SYMBOL_FOR_KVM(__flush_tlb_all);
 void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)
 {
 	struct flush_tlb_info info;
-
+	bool remote_flush = false;
 	int cpu = get_cpu();
 
 	init_flush_tlb_info(&info, NULL, 0, TLB_FLUSH_ALL, 0, false,
@@ -1691,7 +1697,7 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)
 		invlpgb_flush_all_nonglobals();
 		batch->unmapped_pages = false;
 	} else if (cpumask_any_but(&batch->cpumask, cpu) < nr_cpu_ids) {
-		flush_tlb_multi(&batch->cpumask, &info);
+		remote_flush = true;
 	} else if (cpumask_test_cpu(cpu, &batch->cpumask)) {
 		lockdep_assert_irqs_enabled();
 		local_irq_disable();
@@ -1699,9 +1705,12 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)
 		local_irq_enable();
 	}
 
-	cpumask_clear(&batch->cpumask);
-
 	put_cpu();
+
+	if (remote_flush)
+		flush_tlb_multi(&batch->cpumask, &info);
+
+	cpumask_clear(&batch->cpumask);
 }
 
 /*
-- 
2.20.1

Re: [PATCH v9 00/14] Allow preemption during IPI completion waiting to improve real-time performance

[Chuyi Zhou]

friendly ping...

On 2026-06-30 7:09 p.m., Chuyi Zhou wrote:
> Changes in v9:
>   - Rewrite changelogs in [PATCH v9 1/14], [PATCH v9 2/14],
>     [PATCH v9 4/14], [PATCH v9 5/14], [PATCH v9 6/14],
>     [PATCH v9 7/14], and [PATCH v9 11/14].
>   - Change smp_call_function_single()'s wait argument to bool and fix
>     function/kernel-doc argument alignment in [PATCH v9 3/14].
>   - Clean up task_struct IPI mask indentation, declaration ordering, and
>     related initialization placement in [PATCH v9 4/14].
>   - Update the CSD allocation comment in [PATCH v9 5/14].
>   - In [PATCH v9 6/14], make put_cpu() unconditional after IPI
>     queue/dispatch and clarify that the final csd_lock_wait() only waits
>     for completion.
>   - Add a comment for FLUSH_TLB_INFO_ALIGN, reference commit 780e0106d468
>     directly, and add Sebastian's Reviewed-by tag in [PATCH v9 11/14].
> 
> Changes in v8:
>   - Use MIN(SMP_CACHE_BYTES, 64) for FLUSH_TLB_INFO_ALIGN in
>     [PATCH v8 11/14].
>   - In flush_tlb_mm_range(), name the stack object "info" directly instead
>     of keeping a temporary "_info" plus pointer in [PATCH v8 12/14].
> 
> Changes in v7:
>   - Split the x86/mm flush_tlb_info stack-storage change into
>     [PATCH v7 10/14], [PATCH v7 11/14], and [PATCH v7 12/14].
>   - Rewrite the flush_tlb_info changelog with the historical context, test
>     data, Link tag, and Acked-by tag.
>   - Add a preparatory KVM patch for kvm_flush_tlb_multi() in
>     [PATCH v7 13/14].
>   - Refactor flush_tlb_mm_range() and arch_tlbbatch_flush() to avoid gotos
>     in [PATCH v7 14/14].
>   - Drop the flush_tlb_kernel_range() preemption optimization for now.
>   - Tighten the smp_task_ipi_mask_alloc()/smp_task_ipi_mask_free()
>     declarations in [PATCH v7 4/14].
>   - Update the smp_call_function_many_cond() comment wording.
>   - Add Paul E. McKenney's Tested-by tags and Sebastian's Reviewed-by tag.
> 
> Changes in v6:
>   - Make the task-local cpumask selection explicit and drop preemptible()
>     check in smp_call_function_many_cond(). The early put_cpu() decision now
>     depends only on whether a task-local cpumask is available.
>   - Keep smp_task_ipi_mask() private to kernel/smp.c in [PATCH v6 4/12].
>   - Add #include <linux/slab.h> to kernel/smp.c in [PATCH v6 4/12] for
>     kmalloc()/kfree(), fixing the kernel test robot build failure reported
>     at:
>     https://lore.kernel.org/oe-kbuild-all/202605241101.w6T2LApw-lkp@intel.com/
>   - Update the csd_lock_wait() comment in [PATCH v6 6/12].
>   - Add Sebastian's Reviewed-by tags to the reviewed patches.
> 
> Changes in v5:
>   - Replace "smp: Remove get_cpu from smp_call_function_any" with a new
>     approach that extracts a common __smp_call_function_single() to safely
>     keep the remote CPU selection and IPI dispatch process within a single
>     preemption-disabled region in [PATCH v5 3/12].
>   - Fix a typo in comments (s/cpumask_stack/task_mask/) and remove the
>     obsolete "Preemption must be disabled" constraint from the kernel-doc
>     in [PATCH v5 6/12].
>   - Adjust the WARN_ON_ONCE() validation condition to avoid a false positive
>     warning caused by CPU hotplug races when use_cpus_read_lock is false in
>     [PATCH v5 9/12].
>   - Move the preemptible() check in smp_call_function_many_cond() from
>     [PATCH v5 4/12] to [PATCH v5 6/12].
>   - Rebase to commit 4ac4d6549a65 ("sched: Use trace_call__<tp>() to save a
>     static branch").
> 
> Changes in v4:
>   - Use task-local IPI cpumask rather than on-stack cpumask in
>     [PATCH v4 4/12] (suggested by sebastian).
>   - Skip to free csd memory in smpcfd_dead_cpu() to guarantee csd memory
>     access safety, instead of using RCU mechanism in [PATCH v4 5/12]
>     (suggested by sebastian).
>   - Align flush_tlb_info with SMP_CACHE_BYTES to avoid performance
>     degradation caused by unnecessary cache line movements in [PATCH v4
>     10/12](suggested by sebastian and Nadav).
>   - Collect Acked-bys and Reviewed-bys.
> 
> Changes in v3:
>   - Add benchmarks to measure the performance impact of changing
>     flush_tlb_info to stack variable in [PATCH v3 10/12] (suggested by
>     peter)
>   - Adjust the rcu_read_unlock() location in [PATCH v3 5/12] (suggested
>     by muchun)
>   - Use raw_smp_processor_id() to prevent warning[1] from
>     check_preemption_disabled() in [PATCH v3 12/12].
>   - Collect Acked-bys and Reviewed-by.
> 
> [1]: https://lore.kernel.org/lkml/20260302075216.2170675-1-zhouchuyi@bytedance.com/T/#mc39999cbeb3f50be176f0903d0fa4075688b073d
> 
> Changes in v2:
>   - Simplify the code comments in [PATCH v2 2/12] (pointed by peter and
>     muchun)
>   - Adjust the preemption disabling logic in smp_call_function_any() in
>     [PATCH v2 3/12] (suggested by peter).
>   - Use on-stack cpumask only when !CONFIG_CPUMASK_OFFSTACK in [PATCH V2
>     4/12] (pointed by peter)
>   - Add [PATCH v2 5/12] to replace migrate_disable with the rcu mechanism
>   - Adjust the preemption disabling logic to allow flush_tlb_multi() to be
>     preemptible and migratable in [PATCH v2 11/12]
>   - Collect Acked-bys and Reviewed-bys
> 
> 
> Introduction
> ============
> 
> The vast majority of smp_call_function*() callers block until remote CPUs
> complete the IPI function execution. As smp_call_function*() runs with
> preemption disabled throughout, scheduling latency increases dramatically
> with the number of remote CPUs and other factors (such as interrupts being
> disabled).
> 
> On x86-64 architectures, TLB flushes are performed via IPIs; thus, during
> process exit or when process-mapped pages are reclaimed, numerous IPI
> operations must be awaited, leading to increased scheduling latency for
> other threads on the current CPU. In our production environment, we
> observed IPI wait-induced scheduling latency reaching up to 16ms on a
> 16-core machine. Our goal is to allow preemption during IPI completion
> waiting to improve real-time performance.
> 
> Background
> ==========
> 
> In our production environments, latency-sensitive workloads (DPDK) are
> configured with the highest priority to preempt lower-priority tasks at any
> time. We discovered that DPDK's wake-up latency is primarily caused by the
> current CPU having preemption disabled. Therefore, we collected the maximum
> preemption disabled events within every 30-second interval and then
> calculated the P50/P99 of these max preemption disabled events:
> 
> 
>                          p50(ns)               p99(ns)
> cpu0                   254956                 5465050
> cpu1                   115801                 120782
> cpu2                   43324                  72957
> cpu3                   256637                 16723307
> cpu4                   58979                  87237
> cpu5                   47464                  79815
> cpu6                   48881                  81371
> cpu7                   52263                  82294
> cpu8                   263555                 4657713
> cpu9                   44935                  73962
> cpu10                  37659                  65026
> cpu11                  257008                 2706878
> cpu12                  49669                  90006
> cpu13                  45186                  74666
> cpu14                  60705                  83866
> cpu15                  51311                  86885
> 
> Meanwhile, we have collected the distribution of preemption disabling
> events exceeding 1ms across different CPUs over several hours(I omitted
> CPU data that were all zeros):
> 
> CPU        1~10ms   10~50ms   50~100ms
> cpu0        29       5       0
> cpu3        38       13      0
> cpu8        34       6       0
> cpu11       24       10      0
> 
> The preemption disabled for several milliseconds or even 10ms+ mostly
> originates from TLB flush:
> 
> @stack[
>      trace_preempt_on+143
>      trace_preempt_on+143
>      preempt_count_sub+67
>      arch_tlbbatch_flush/flush_tlb_mm_range
>      task_exit/page_reclaim/...
> ]
> 
> Further analysis confirms that the majority of the time is consumed in
> csd_lock_wait().
> 
> Now smp_call*() always needs to disable preemption, mainly to protect its
> internal per-CPU data structures and synchronize with CPU offline
> operations. This patchset attempts to make csd_lock_wait() preemptible,
> thereby reducing the preemption-disabled critical section and improving
> kernel real-time performance.
> 
> Effect
> ======
> 
> After applying this patchset, we no longer observe preemption disabled for
> more than 1ms on the arch_tlbbatch_flush/flush_tlb_mm_range path. The
> overall P99 of max preemption disabled events in every 30-second interval
> is reduced to around 1.5ms. The remaining latency is primarily due to lock
> contention.
> 
>                       before patch    after patch    reduced by
>                       -----------    --------------  ------------
> p99(ns)                16723307        1556034        ~90.70%
> 
> Chuyi Zhou (14):
>    smp: Disable preemption explicitly in __csd_lock_wait()
>    smp: Enable preemption early in smp_call_function_single()
>    smp: Refactor remote CPU selection in smp_call_function_any()
>    smp: Use task-local IPI cpumask in smp_call_function_many_cond()
>    smp: Alloc percpu csd data in smpcfd_prepare_cpu() only once
>    smp: Enable preemption early in smp_call_function_many_cond()
>    smp: Remove preempt_disable() from smp_call_function()
>    smp: Remove preempt_disable() from on_each_cpu_cond_mask()
>    scftorture: Remove preempt_disable() in scftorture_invoke_one()
>    x86/mm: Factor out flush_tlb_info initialization
>    x86/mm: Cap flush_tlb_info alignment at 64 bytes
>    x86/mm: Move flush_tlb_info back to the stack
>    x86/kvm: Disable preemption in kvm_flush_tlb_multi()
>    x86/mm: Re-enable preemption before flush_tlb_multi()
> 
>   arch/x86/include/asm/tlbflush.h |   9 +-
>   arch/x86/kernel/kvm.c           |   4 +-
>   arch/x86/mm/tlb.c               |  95 +++++++---------
>   include/linux/sched.h           |   6 ++
>   include/linux/smp.h             |  15 ++-
>   kernel/fork.c                   |   9 +-
>   kernel/scftorture.c             |  13 +--
>   kernel/smp.c                    | 186 ++++++++++++++++++++++----------
>   kernel/up.c                     |   3 +-
>   9 files changed, 210 insertions(+), 130 deletions(-)

Re: [PATCH v9 04/14] smp: Use task-local IPI cpumask in smp_call_function_many_cond()

[Thomas Gleixner]

On Tue, Jun 30 2026 at 19:09, Chuyi Zhou wrote:
> The memory cost is explicit: one word is added to task_struct. When
> cpumask_size() fits in that word, the mask is stored inline and no
> separate allocation is needed. Larger systems allocate cpumask_size() per
> task; on x86-64 NR_CPUS=8192 this is about 1 KiB per task. For context,

It's not about. It's one kilobyte, no?

> @@ -1364,6 +1364,12 @@ struct task_struct {
>  	struct list_head		perf_event_list;
>  	struct perf_ctx_data __rcu	*perf_ctx_data;
>  #endif
> +#if defined(CONFIG_SMP) && defined(CONFIG_PREEMPTION)
> +	union {
> +		cpumask_t			*ipi_mask_ptr;
> +		unsigned long			ipi_mask_val;
> +	};
> +#endif

Aside of this horrible ifdeffery, this construct makes me more than
nervous as it's too trivial to add code which deferences the pointer
directly. The proper thing to do is:

#if defined(CONFIG_SMP) && defined(CONFIG_PREEMPTION)
struct task_ipi_mask {
	union {
		cpumask_t			*ipi_mask_ptr;
		unsigned long			ipi_mask_val;
	};
};
#else
struct task_ipi_mask { };
#endif

struct task_struct {
	...
        struct task_ipi_mask			__private ipi_mask;

and have the magic accessors use

         ACCESS_PRIVATE(t->ipi_mask.member);

That allows static analysis to catch any incidential direct references.

Thanks,

        tglx

Re: [PATCH v9 04/14] smp: Use task-local IPI cpumask in smp_call_function_many_cond()

[Chuyi Zhou]

On 2026-07-08 7:17 a.m., Thomas Gleixner wrote:
> On Tue, Jun 30 2026 at 19:09, Chuyi Zhou wrote:
>> The memory cost is explicit: one word is added to task_struct. When
>> cpumask_size() fits in that word, the mask is stored inline and no
>> separate allocation is needed. Larger systems allocate cpumask_size() per
>> task; on x86-64 NR_CPUS=8192 this is about 1 KiB per task. For context,
> 
> It's not about. It's one kilobyte, no?
> 
>> @@ -1364,6 +1364,12 @@ struct task_struct {
>>   	struct list_head		perf_event_list;
>>   	struct perf_ctx_data __rcu	*perf_ctx_data;
>>   #endif
>> +#if defined(CONFIG_SMP) && defined(CONFIG_PREEMPTION)
>> +	union {
>> +		cpumask_t			*ipi_mask_ptr;
>> +		unsigned long			ipi_mask_val;
>> +	};
>> +#endif
> 
> Aside of this horrible ifdeffery, this construct makes me more than
> nervous as it's too trivial to add code which deferences the pointer
> directly. The proper thing to do is:
> 
> #if defined(CONFIG_SMP) && defined(CONFIG_PREEMPTION)
> struct task_ipi_mask {
> 	union {
> 		cpumask_t			*ipi_mask_ptr;
> 		unsigned long			ipi_mask_val;
> 	};
> };
> #else
> struct task_ipi_mask { };
> #endif
> 
> struct task_struct {
> 	...
>          struct task_ipi_mask			__private ipi_mask;
> 
> and have the magic accessors use
> 
>           ACCESS_PRIVATE(t->ipi_mask.member);
> 
> That allows static analysis to catch any incidential direct references.
> 

Agreed. Hiding the storage behind a private wrapper makes the intended
access pattern much clearer and lets sparse catch accidental direct
references.

I'll change this to use a struct task_ipi_mask __private field in
task_struct, just as you suggested, and access the union members only
through ACCESS_PRIVATE() in the helper functions.

I'll also fix the wording in changelog.

Thanks,
Chuyi


> Thanks,
> 
>          tglx