[PATCH] sched/debug: Introduce per-CPU debugfs files

[PATCH] sched/debug: Introduce per-CPU debugfs files

[Aaron Tomlin]

Currently, accessing scheduler debugging details for a specific CPU
requires reading /sys/kernel/debug/sched/debug, which outputs
information for all online CPUs. This results in redundant output and
parsing overhead on large SMP systems when targeting an individual CPU.

Add support for per-CPU debug files under:
/sys/kernel/debug/sched/cpu/cpu<N>/debug. Reading
/sys/kernel/debug/sched/cpu/cpu<N>/debug calls print_cpu() specifically
for CPU <N>, exposing CPU-specific runqueue details on demand.

Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
---
 kernel/sched/debug.c | 40 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)

diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c
index 40584b27ea0c..15785a414134 100644
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -356,6 +356,7 @@ static const struct file_operations sched_verbose_fops = {
 };
 
 static const struct seq_operations sched_debug_sops;
+static void print_cpu(struct seq_file *m, int cpu);
 
 static int sched_debug_open(struct inode *inode, struct file *filp)
 {
@@ -633,6 +634,44 @@ static void debugfs_fair_server_init(void)
 	}
 }
 
+static int sched_debug_cpu_show(struct seq_file *m, void *v)
+{
+	unsigned long cpu = (unsigned long) m->private;
+
+	print_cpu(m, cpu);
+	return 0;
+}
+
+static int sched_debug_cpu_open(struct inode *inode, struct file *filp)
+{
+	return single_open(filp, sched_debug_cpu_show, inode->i_private);
+}
+
+static const struct file_operations sched_debug_cpu_fops = {
+	.open		= sched_debug_cpu_open,
+	.read		= seq_read,
+	.llseek		= seq_lseek,
+	.release	= single_release,
+};
+
+static __init void debugfs_cpu_init(void)
+{
+	struct dentry *d_cpu_dir;
+	unsigned long cpu;
+	char buf[16];
+
+	d_cpu_dir = debugfs_create_dir("cpu", debugfs_sched);
+
+	for_each_possible_cpu(cpu) {
+		struct dentry *d_cpu;
+
+		snprintf(buf, sizeof(buf), "cpu%lu", cpu);
+		d_cpu = debugfs_create_dir(buf, d_cpu_dir);
+
+		debugfs_create_file("debug", 0444, d_cpu, (void *) cpu, &sched_debug_cpu_fops);
+	}
+}
+
 static __init int sched_init_debug(void)
 {
 	struct dentry __maybe_unused *numa, *llc;
@@ -690,6 +729,7 @@ static __init int sched_init_debug(void)
 #ifdef CONFIG_SCHED_CLASS_EXT
 	debugfs_ext_server_init();
 #endif
+	debugfs_cpu_init();
 
 	return 0;
 }
-- 
2.55.0

Re: [PATCH] sched/debug: Introduce per-CPU debugfs files

[Zhan Xusheng]

From: Zhan Xusheng <zhanxusheng1024@gmail.com>

On Mon, Jul 27, 2026 at 10:03:09PM -0400, Aaron Tomlin wrote:
> Currently, accessing scheduler debugging details for a specific CPU
> requires reading /sys/kernel/debug/sched/debug, which outputs
> information for all online CPUs. This results in redundant output and
> parsing overhead on large SMP systems when targeting an individual CPU.

This looks like a handy addition. One thing that might strengthen the
changelog: the bigger win of a targeted per-CPU file isn't really the
userspace parsing overhead (a grep over the existing file is cheap), but
that inspecting a single CPU no longer requires walking every runqueue and
taking each rq lock to produce the full dump - which is what actually
perturbs a large, busy system. Framing it that way makes the case more
compelling.

> + for_each_possible_cpu(cpu) {
> +   struct dentry *d_cpu;
> +
> +   snprintf(buf, sizeof(buf), "cpu%lu", cpu);
> +   d_cpu = debugfs_create_dir(buf, d_cpu_dir);
> +
> +   debugfs_create_file("debug", 0444, d_cpu, (void *) cpu, &sched_debug_cpu_fops);
> + }

One design question: the existing /sys/kernel/debug/sched/debug only
iterates cpu_online_mask, whereas this creates cpu<N>/debug
for_each_possible_cpu. Reading an offline (or never-onlined) CPU's file
then calls print_cpu() on a quiescent rq. As far as I can tell that is
harmless - rq->curr is that CPU's idle task from boot, so it just prints
idle/zeroed values - but it does differ from the existing interface, and
where possible CPUs greatly exceed online ones it adds many directories
for CPUs that will never report anything interesting. Was exposing all
possible CPUs intentional, or would gating the files (or their contents)
on online CPUs be preferable?

Thanks,
Zhan Xusheng

Re: [PATCH] sched/debug: Introduce per-CPU debugfs files

[Aaron Tomlin]

On Tue, Jul 28, 2026 at 02:34:36PM +0800, Zhan Xusheng wrote:
> From: Zhan Xusheng <zhanxusheng1024@gmail.com>
> 
> On Mon, Jul 27, 2026 at 10:03:09PM -0400, Aaron Tomlin wrote:
> > Currently, accessing scheduler debugging details for a specific CPU
> > requires reading /sys/kernel/debug/sched/debug, which outputs
> > information for all online CPUs. This results in redundant output and
> > parsing overhead on large SMP systems when targeting an individual CPU.
> 
> This looks like a handy addition. One thing that might strengthen the
> changelog: the bigger win of a targeted per-CPU file isn't really the
> userspace parsing overhead (a grep over the existing file is cheap), but
> that inspecting a single CPU no longer requires walking every runqueue and
> taking each rq lock to produce the full dump - which is what actually
> perturbs a large, busy system. Framing it that way makes the case more
> compelling.

Hi Zhan,

Thank you for taking the time to review this patch and for the constructive feedback.

I agree. Reframing the rationale around usability and system perturbation,
provides a much stronger justification. I shall refine the commit log in v2
to emphasise this benefit.

> 
> > + for_each_possible_cpu(cpu) {
> > +   struct dentry *d_cpu;
> > +
> > +   snprintf(buf, sizeof(buf), "cpu%lu", cpu);
> > +   d_cpu = debugfs_create_dir(buf, d_cpu_dir);
> > +
> > +   debugfs_create_file("debug", 0444, d_cpu, (void *) cpu, &sched_debug_cpu_fops);
> > + }
> 
> One design question: the existing /sys/kernel/debug/sched/debug only
> iterates cpu_online_mask, whereas this creates cpu<N>/debug
> for_each_possible_cpu. Reading an offline (or never-onlined) CPU's file
> then calls print_cpu() on a quiescent rq. As far as I can tell that is
> harmless - rq->curr is that CPU's idle task from boot, so it just prints
> idle/zeroed values - but it does differ from the existing interface, and
> where possible CPUs greatly exceed online ones it adds many directories
> for CPUs that will never report anything interesting. Was exposing all
> possible CPUs intentional, or would gating the files (or their contents)
> on online CPUs be preferable?

Creating entries for for_each_possible_cpu() was intentional to keep the
debugfs directory structure static, thereby avoiding the complexity of
registering CPU hotplug notifiers to dynamically create and destroy
dentries during CPU online and offline events (i.e., matching fair_server
and ext_server initialisation).

However, to better align with /sys/kernel/debug/sched/debug and avoid
printing uninteresting data for offline CPUs, we could gate the read output
in sched_debug_cpu_show():

        if (!cpu_online(cpu))
            return -ENODEV;


Kind regards,
-- 
Aaron Tomlin

Re: [PATCH] sched/debug: Introduce per-CPU debugfs files

[Peter Zijlstra]

On Mon, Jul 27, 2026 at 10:03:09PM -0400, Aaron Tomlin wrote:
> Currently, accessing scheduler debugging details for a specific CPU
> requires reading /sys/kernel/debug/sched/debug, which outputs
> information for all online CPUs. This results in redundant output and
> parsing overhead on large SMP systems when targeting an individual CPU.
> 
> Add support for per-CPU debug files under:
> /sys/kernel/debug/sched/cpu/cpu<N>/debug. Reading
> /sys/kernel/debug/sched/cpu/cpu<N>/debug calls print_cpu() specifically
> for CPU <N>, exposing CPU-specific runqueue details on demand.

You're failing to explain why though. In what situation does the
overhead from sched/debug matter one whit? If you're dumping this at
frequencies high enough for this to matter, you're doing something
terribly wrong.

Re: [PATCH] sched/debug: Introduce per-CPU debugfs files

[Daniel Vacek]

On Tue, 28 Jul 2026 at 12:48, Peter Zijlstra <peterz@infradead.org> wrote:
> On Mon, Jul 27, 2026 at 10:03:09PM -0400, Aaron Tomlin wrote:
> > Currently, accessing scheduler debugging details for a specific CPU
> > requires reading /sys/kernel/debug/sched/debug, which outputs
> > information for all online CPUs. This results in redundant output and
> > parsing overhead on large SMP systems when targeting an individual CPU.
> >
> > Add support for per-CPU debug files under:
> > /sys/kernel/debug/sched/cpu/cpu<N>/debug. Reading
> > /sys/kernel/debug/sched/cpu/cpu<N>/debug calls print_cpu() specifically
> > for CPU <N>, exposing CPU-specific runqueue details on demand.
>
> You're failing to explain why though. In what situation does the
> overhead from sched/debug matter one whit? If you're dumping this at
> frequencies high enough for this to matter, you're doing something
> terribly wrong.

I can imagine running a realtime app pinned on an isolated CPU (for
example any DPDK app like FlexRAN on top of Kubernetes/OpenShift,
etc.) and debugging system noise/starved victims on this CPU. Why dump
sched details from other CPUs and look for a needle in the haystack?
This actually makes sense to me.

--nX

Re: [PATCH] sched/debug: Introduce per-CPU debugfs files

[Aaron Tomlin]

On Tue, Jul 28, 2026 at 12:48:49PM +0200, Peter Zijlstra wrote:
> On Mon, Jul 27, 2026 at 10:03:09PM -0400, Aaron Tomlin wrote:
> > Currently, accessing scheduler debugging details for a specific CPU
> > requires reading /sys/kernel/debug/sched/debug, which outputs
> > information for all online CPUs. This results in redundant output and
> > parsing overhead on large SMP systems when targeting an individual CPU.
> > 
> > Add support for per-CPU debug files under:
> > /sys/kernel/debug/sched/cpu/cpu<N>/debug. Reading
> > /sys/kernel/debug/sched/cpu/cpu<N>/debug calls print_cpu() specifically
> > for CPU <N>, exposing CPU-specific runqueue details on demand.
> 
> You're failing to explain why though. In what situation does the
> overhead from sched/debug matter one whit? If you're dumping this at
> frequencies high enough for this to matter, you're doing something
> terribly wrong.

Hi Peter,

Thank you for your feedback.

Fair point. To be clear, this is certainly not intended for high-frequency
polling or automated monitoring in production hot loops.

When investigating a latency anomaly or scheduling issue isolated to a
specific CPU, I find that accessing cpu<N>/debug provides an immediate,
targeted view of that runqueue. As mentioned to Zhan [1], I shall rewrite
the commit message in v2 to properly articulate this rationale.

[1]: https://lore.kernel.org/lkml/uz622xhp6nejphseptcwjg6mnuuvrb3tu6gpzc55oqbqtlfmmh@4nltaa6ucce2/


Kind regards,
-- 
Aaron Tomlin