[PATCH v3 00/11] sched_ext: Implement core event counters

[PATCH v3 00/11] sched_ext: Implement core event counters

[Changwoo Min]

The sched_ext core often has to override the BPF scheduler decisions,
and some events could be interesting but not easily visible.

This patchset aims to address such a problem in the following manner:
  - Introduce an infrastructure to collect such events in a scalable and
    extensible way and to expose the collected events to the BPF scheduler
    in a compatible way.
  - Define seven events to be collected.
  - Modify an scx scheduler to demonstrate the usage of the new BPF APIs.

ChangeLog: v2 -> v3
  - Rename scx_bpf_event_stats() to scx_bpf_events().
  - Add a prefix SCX_EV_ to all event names.
  - Change the implementation SCX_EV_SELECT_CPU_FALLBACK such that
    record the selected cpu (p->scx.selected_cpu) in
    select_task_rq_scx() and compare it in enqueue_task_scx() to
    reliably count events.
  - Change the implementation of SCX_EV_BYPASS_DISPATCH such that log
    the event where the condition is initially detected.
  - Modify scx_qmap to print the core event counter every second.
  - Remove unnecessary white space in patch 6.

ChangeLog: v1 -> v2
  - Rename scx_event_stat and scx_bpf_event_stat() to scx_event_stats and
    scx_bpf_event_stats().
  - Rename event names following the convention of $COMPONENT_$EVENT.
  - Rename event_stats to event_stats_cpu.
  - Drop the enum scx_event_kind and related macros.
  - Revise scx_add_event() to use this_cpu_add().
  - Add __scx_add_event() to use __this_cpu_add().
  - Move the event counter resetting code to the loading of a BPF scheduler.
  - The bypass-related event is further categorized into three events:
    BYPASS_ACTIVATE, BYPASS_DISPATCH, and BYPASS_DURATION.
  - Revise SELECT_CPU_FALLBACK to capture the case of the chosen CPU is not
    allowed.
  - Move is_cpu_allowed() from core.c to sched.h to use in the
    SELECT_CPU_FALLBACK code.

Changwoo Min (11):
  sched_ext: Implement event counter infrastructure
  sched_ext: Add an event, SCX_EV_SELECT_CPU_FALLBACK
  sched_ext: Add an event, SCX_EV_DISPATCH_LOCAL_DSQ_OFFLINE
  sched_ext: Add an event, SCX_EV_DISPATCH_KEEP_LAST
  sched_ext: Add an event, SCX_EV_ENQ_SKIP_EXITING
  sched_ext: Add an event, SCX_EV_BYPASS_ACTIVATE
  sched_ext: Add an event, SCX_EV_BYPASS_DISPATCH
  sched_ext: Add an event, SCX_EV_BYPASS_DURATION
  sched_ext: Add scx_bpf_events() and scx_read_event() for BPF
    schedulers
  sched_ext: Print core event count in scx_central scheduler
  sched_ext: Print core event count in scx_qmap scheduler

 include/linux/sched/ext.h                |   1 +
 kernel/sched/ext.c                       | 197 ++++++++++++++++++++++-
 tools/sched_ext/include/scx/common.bpf.h |   4 +
 tools/sched_ext/scx_central.bpf.c        |  21 +++
 tools/sched_ext/scx_qmap.bpf.c           |  19 +++
 5 files changed, 237 insertions(+), 5 deletions(-)

-- 
2.48.1

[PATCH v3 01/11] sched_ext: Implement event counter infrastructure

[Changwoo Min]

Collect the statistics of specific types of behavior in the sched_ext core,
which are not easily visible but still interesting to an scx scheduler.

An event type is defined in 'struct scx_event_stats.' When an event occurs,
its counter is accumulated using 'scx_add_event()' and '__scx_add_event()'
to per-CPU 'struct scx_event_stats' for efficiency. 'scx_bpf_events()'
aggregates all the per-CPU counters and exposes a system-wide counters.

For convenience and readability of the code, 'scx_agg_event()' and
'scx_dump_event()' are provided.

The collected events can be observed after a BPF scheduler is unloaded
beforea new BPF scheduler is loaded so the per-CPU 'struct scx_event_stats'
are reset.

Signed-off-by: Changwoo Min <changwoo@igalia.com>
---
 kernel/sched/ext.c | 103 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 103 insertions(+)

diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c
index 5f6a425d4ffe..4e28e88e88d4 100644
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -1440,6 +1440,64 @@ static struct task_struct *scx_task_iter_next_locked(struct scx_task_iter *iter)
 	return p;
 }
 
+/*
+ * Collection of event counters. Event types are placed in descending order.
+ */
+struct scx_event_stats {
+};
+
+/*
+ * The event counter is organized by a per-CPU variable to minimize the
+ * accounting overhead without synchronization. A system-wide view on the
+ * event counter is constructed when requested by scx_bpf_get_event_stat().
+ */
+static DEFINE_PER_CPU(struct scx_event_stats, event_stats_cpu);
+
+/**
+ * scx_add_event - Increase an event counter for 'name' by 'cnt'
+ * @name: an event name defined in struct scx_event_stats
+ * @cnt: the number of the event occured
+ *
+ * This can be used when preemption is not disabled.
+ */
+#define scx_add_event(name, cnt) do {						\
+	this_cpu_add(event_stats_cpu.name, cnt);				\
+} while(0)
+
+/**
+ * __scx_add_event - Increase an event counter for 'name' by 'cnt'
+ * @name: an event name defined in struct scx_event_stats
+ * @cnt: the number of the event occured
+ *
+ * This should be used only when preemption is disabled.
+ */
+#define __scx_add_event(name, cnt) do {						\
+	__this_cpu_add(event_stats_cpu.name, cnt);				\
+} while(0)
+
+/**
+ * scx_agg_event - Aggregate an event counter 'kind' from 'src_e' to 'dst_e'
+ * @dst_e: destination event stats
+ * @src_e: source event stats
+ * @kind: a kind of event to be aggregated
+ */
+#define scx_agg_event(dst_e, src_e, kind) do {					\
+	(dst_e)->kind += READ_ONCE((src_e)->kind);				\
+} while(0)
+
+/**
+ * scx_dump_event - Dump an event 'kind' in 'events' to 's'
+ * @s: output seq_buf
+ * @events: event stats
+ * @kind: a kind of event to dump
+ */
+#define scx_dump_event(s, events, kind) do {					\
+	dump_line(&(s), "%30s: %16llu", #kind, (events)->kind);			\
+} while (0)
+
+
+static void scx_bpf_events(struct scx_event_stats *events, size_t events__sz);
+
 static enum scx_ops_enable_state scx_ops_enable_state(void)
 {
 	return atomic_read(&scx_ops_enable_state_var);
@@ -4785,6 +4843,7 @@ static void scx_dump_state(struct scx_exit_info *ei, size_t dump_len)
 		.at_jiffies = jiffies,
 	};
 	struct seq_buf s;
+	struct scx_event_stats events;
 	unsigned long flags;
 	char *buf;
 	int cpu;
@@ -4893,6 +4952,12 @@ static void scx_dump_state(struct scx_exit_info *ei, size_t dump_len)
 		rq_unlock(rq, &rf);
 	}
 
+	dump_newline(&s);
+	dump_line(&s, "Event counters");
+	dump_line(&s, "--------------");
+
+	scx_bpf_events(&events, sizeof(events));
+
 	if (seq_buf_has_overflowed(&s) && dump_len >= sizeof(trunc_marker))
 		memcpy(ei->dump + dump_len - sizeof(trunc_marker),
 		       trunc_marker, sizeof(trunc_marker));
@@ -5000,6 +5065,15 @@ static int scx_ops_enable(struct sched_ext_ops *ops, struct bpf_link *link)
 
 	mutex_lock(&scx_ops_enable_mutex);
 
+	/*
+	 * Clear event counters so a new scx scheduler gets
+	 * fresh event counter values.
+	 */
+	for_each_possible_cpu(cpu) {
+		struct scx_event_stats *e = per_cpu_ptr(&event_stats_cpu, cpu);
+		memset(e, 0, sizeof(*e));
+	}
+
 	if (!scx_ops_helper) {
 		WRITE_ONCE(scx_ops_helper,
 			   scx_create_rt_helper("sched_ext_ops_helper"));
@@ -7001,6 +7075,34 @@ __bpf_kfunc u64 scx_bpf_now(void)
 	return clock;
 }
 
+/*
+ * scx_bpf_events - Get a system-wide event counter to
+ * @events: output buffer from a BPF program
+ * @events__sz: @events len, must end in '__sz'' for the verifier
+ */
+__bpf_kfunc void scx_bpf_events(struct scx_event_stats *events,
+				size_t events__sz)
+{
+	struct scx_event_stats e_sys, *e_cpu;
+	int cpu;
+
+	/* Aggregate per-CPU event counters into the system-wide counters. */
+	memset(&e_sys, 0, sizeof(e_sys));
+	for_each_possible_cpu(cpu) {
+		e_cpu = per_cpu_ptr(&event_stats_cpu, cpu);
+	}
+
+	/*
+	 * We cannot entirely trust a BPF-provided size since a BPF program
+	 * might be compiled against a different vmlinux.h, of which
+	 * scx_event_stats would be larger (a newer vmlinux.h) or smaller
+	 * (an older vmlinux.h). Hence, we use the smaller size to avoid
+	 * memory corruption.
+	 */
+	events__sz = min(events__sz, sizeof(*events));
+	memcpy(events, &e_sys, events__sz);
+}
+
 __bpf_kfunc_end_defs();
 
 BTF_KFUNCS_START(scx_kfunc_ids_any)
@@ -7033,6 +7135,7 @@ BTF_ID_FLAGS(func, scx_bpf_cpu_rq)
 BTF_ID_FLAGS(func, scx_bpf_task_cgroup, KF_RCU | KF_ACQUIRE)
 #endif
 BTF_ID_FLAGS(func, scx_bpf_now)
+BTF_ID_FLAGS(func, scx_bpf_events, KF_TRUSTED_ARGS)
 BTF_KFUNCS_END(scx_kfunc_ids_any)
 
 static const struct btf_kfunc_id_set scx_kfunc_set_any = {
-- 
2.48.1

[PATCH v3 02/11] sched_ext: Add an event, SCX_EV_SELECT_CPU_FALLBACK

[Changwoo Min]

Add a core event, SCX_EV_SELECT_CPU_FALLBACK, which represents how many times
ops.select_cpu() returns a CPU that the task can't use.

__scx_add_event() is used since the caller holds an rq lock,
so the preemption has already been disabled.

Signed-off-by: Changwoo Min <changwoo@igalia.com>
---
 include/linux/sched/ext.h |  1 +
 kernel/sched/ext.c        | 14 ++++++++++++++
 2 files changed, 15 insertions(+)

diff --git a/include/linux/sched/ext.h b/include/linux/sched/ext.h
index 1d70a9867fb1..f7545430a548 100644
--- a/include/linux/sched/ext.h
+++ b/include/linux/sched/ext.h
@@ -146,6 +146,7 @@ struct sched_ext_entity {
 	u32			weight;
 	s32			sticky_cpu;
 	s32			holding_cpu;
+	s32			selected_cpu;
 	u32			kf_mask;	/* see scx_kf_mask above */
 	struct task_struct	*kf_tasks[2];	/* see SCX_CALL_OP_TASK() */
 	atomic_long_t		ops_state;
diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c
index 4e28e88e88d4..4d3b32aca48d 100644
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -1444,6 +1444,11 @@ static struct task_struct *scx_task_iter_next_locked(struct scx_task_iter *iter)
  * Collection of event counters. Event types are placed in descending order.
  */
 struct scx_event_stats {
+	/*
+	 * If ops.select_cpu() returns a CPU which can't be used by the task,
+	 * the core scheduler code silently picks a fallback CPU.
+	 */
+	u64		SCX_EV_SELECT_CPU_FALLBACK;
 };
 
 /*
@@ -2170,6 +2175,11 @@ static void enqueue_task_scx(struct rq *rq, struct task_struct *p, int enq_flags
 	do_enqueue_task(rq, p, enq_flags, sticky_cpu);
 out:
 	rq->scx.flags &= ~SCX_RQ_IN_WAKEUP;
+
+	if ((enq_flags & SCX_ENQ_CPU_SELECTED) &&
+	    (cpu_of(rq) != p->scx.selected_cpu)) {
+		__scx_add_event(SCX_EV_SELECT_CPU_FALLBACK, 1);
+	}
 }
 
 static void ops_dequeue(struct task_struct *p, u64 deq_flags)
@@ -3240,6 +3250,7 @@ static int select_task_rq_scx(struct task_struct *p, int prev_cpu, int wake_flag
 
 		cpu = SCX_CALL_OP_TASK_RET(SCX_KF_ENQUEUE | SCX_KF_SELECT_CPU,
 					   select_cpu, p, prev_cpu, wake_flags);
+		p->scx.selected_cpu = cpu;
 		*ddsp_taskp = NULL;
 		if (ops_cpu_valid(cpu, "from ops.select_cpu()"))
 			return cpu;
@@ -3250,6 +3261,7 @@ static int select_task_rq_scx(struct task_struct *p, int prev_cpu, int wake_flag
 		s32 cpu;
 
 		cpu = scx_select_cpu_dfl(p, prev_cpu, wake_flags, &found);
+		p->scx.selected_cpu = cpu;
 		if (found) {
 			p->scx.slice = SCX_SLICE_DFL;
 			p->scx.ddsp_dsq_id = SCX_DSQ_LOCAL;
@@ -4957,6 +4969,7 @@ static void scx_dump_state(struct scx_exit_info *ei, size_t dump_len)
 	dump_line(&s, "--------------");
 
 	scx_bpf_events(&events, sizeof(events));
+	scx_dump_event(s, &events, SCX_EV_SELECT_CPU_FALLBACK);
 
 	if (seq_buf_has_overflowed(&s) && dump_len >= sizeof(trunc_marker))
 		memcpy(ei->dump + dump_len - sizeof(trunc_marker),
@@ -7090,6 +7103,7 @@ __bpf_kfunc void scx_bpf_events(struct scx_event_stats *events,
 	memset(&e_sys, 0, sizeof(e_sys));
 	for_each_possible_cpu(cpu) {
 		e_cpu = per_cpu_ptr(&event_stats_cpu, cpu);
+		scx_agg_event(&e_sys, e_cpu, SCX_EV_SELECT_CPU_FALLBACK);
 	}
 
 	/*
-- 
2.48.1

[PATCH v3 03/11] sched_ext: Add an event, SCX_EV_DISPATCH_LOCAL_DSQ_OFFLINE

[Changwoo Min]

Add a core event, SCX_EV_DISPATCH_LOCAL_DSQ_OFFLINE, which represents how
many times a BPF scheduler tries to dispatch to an offlined local DSQ.

__scx_add_event() is used since the caller holds an rq lock,
so the preemption has already been disabled.

Signed-off-by: Changwoo Min <changwoo@igalia.com>
---
 kernel/sched/ext.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c
index 4d3b32aca48d..041b0af3551a 100644
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -1449,6 +1449,12 @@ struct scx_event_stats {
 	 * the core scheduler code silently picks a fallback CPU.
 	 */
 	u64		SCX_EV_SELECT_CPU_FALLBACK;
+
+	/*
+	 * When dispatching to a local DSQ, the CPU may have gone offline in
+	 * the meantime. In this case, the task is bounced to the global DSQ.
+	 */
+	u64		SCX_EV_DISPATCH_LOCAL_DSQ_OFFLINE;
 };
 
 /*
@@ -2644,6 +2650,7 @@ static void dispatch_to_local_dsq(struct rq *rq, struct scx_dispatch_q *dst_dsq,
 	if (unlikely(!task_can_run_on_remote_rq(p, dst_rq, true))) {
 		dispatch_enqueue(find_global_dsq(p), p,
 				 enq_flags | SCX_ENQ_CLEAR_OPSS);
+		__scx_add_event(SCX_EV_DISPATCH_LOCAL_DSQ_OFFLINE, 1);
 		return;
 	}
 
@@ -4970,6 +4977,7 @@ static void scx_dump_state(struct scx_exit_info *ei, size_t dump_len)
 
 	scx_bpf_events(&events, sizeof(events));
 	scx_dump_event(s, &events, SCX_EV_SELECT_CPU_FALLBACK);
+	scx_dump_event(s, &events, SCX_EV_DISPATCH_LOCAL_DSQ_OFFLINE);
 
 	if (seq_buf_has_overflowed(&s) && dump_len >= sizeof(trunc_marker))
 		memcpy(ei->dump + dump_len - sizeof(trunc_marker),
@@ -7104,6 +7112,7 @@ __bpf_kfunc void scx_bpf_events(struct scx_event_stats *events,
 	for_each_possible_cpu(cpu) {
 		e_cpu = per_cpu_ptr(&event_stats_cpu, cpu);
 		scx_agg_event(&e_sys, e_cpu, SCX_EV_SELECT_CPU_FALLBACK);
+		scx_agg_event(&e_sys, e_cpu, SCX_EV_DISPATCH_LOCAL_DSQ_OFFLINE);
 	}
 
 	/*
-- 
2.48.1

[PATCH v3 04/11] sched_ext: Add an event, SCX_EV_DISPATCH_KEEP_LAST

[Changwoo Min]

Add a core event, SCX_EV_DISPATCH_KEEP_LAST, which represents how many
times a task is continued to run without ops.enqueue() when
SCX_OPS_ENQ_LAST is not set.

__scx_add_event() is used since the caller holds an rq lock,
so the preemption has already been disabled.

Signed-off-by: Changwoo Min <changwoo@igalia.com>
---
 kernel/sched/ext.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c
index 041b0af3551a..7147f730850b 100644
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -1455,6 +1455,12 @@ struct scx_event_stats {
 	 * the meantime. In this case, the task is bounced to the global DSQ.
 	 */
 	u64		SCX_EV_DISPATCH_LOCAL_DSQ_OFFLINE;
+
+	/*
+	 * If SCX_OPS_ENQ_LAST is not set, the number of times that a task
+	 * continued to run because there were no other tasks on the CPU.
+	 */
+	u64		SCX_EV_DISPATCH_KEEP_LAST;
 };
 
 /*
@@ -2908,6 +2914,7 @@ static int balance_one(struct rq *rq, struct task_struct *prev)
 	if (prev_on_rq && (!static_branch_unlikely(&scx_ops_enq_last) ||
 	     scx_rq_bypassing(rq))) {
 		rq->scx.flags |= SCX_RQ_BAL_KEEP;
+		__scx_add_event(SCX_EV_DISPATCH_KEEP_LAST, 1);
 		goto has_tasks;
 	}
 	rq->scx.flags &= ~SCX_RQ_IN_BALANCE;
@@ -4978,6 +4985,7 @@ static void scx_dump_state(struct scx_exit_info *ei, size_t dump_len)
 	scx_bpf_events(&events, sizeof(events));
 	scx_dump_event(s, &events, SCX_EV_SELECT_CPU_FALLBACK);
 	scx_dump_event(s, &events, SCX_EV_DISPATCH_LOCAL_DSQ_OFFLINE);
+	scx_dump_event(s, &events, SCX_EV_DISPATCH_KEEP_LAST);
 
 	if (seq_buf_has_overflowed(&s) && dump_len >= sizeof(trunc_marker))
 		memcpy(ei->dump + dump_len - sizeof(trunc_marker),
@@ -7113,6 +7121,7 @@ __bpf_kfunc void scx_bpf_events(struct scx_event_stats *events,
 		e_cpu = per_cpu_ptr(&event_stats_cpu, cpu);
 		scx_agg_event(&e_sys, e_cpu, SCX_EV_SELECT_CPU_FALLBACK);
 		scx_agg_event(&e_sys, e_cpu, SCX_EV_DISPATCH_LOCAL_DSQ_OFFLINE);
+		scx_agg_event(&e_sys, e_cpu, SCX_EV_DISPATCH_KEEP_LAST);
 	}
 
 	/*
-- 
2.48.1

[PATCH v3 05/11] sched_ext: Add an event, SCX_EV_ENQ_SKIP_EXITING

[Changwoo Min]

Add a core event, SCX_EV_ENQ_SKIP_EXITING, which represents how many
times a task is enqueued to a local DSQ when exiting if
SCX_OPS_ENQ_EXITING is not set.

__scx_add_event() is used since the caller holds an rq lock,
so the preemption has already been disabled.

Signed-off-by: Changwoo Min <changwoo@igalia.com>
---
 kernel/sched/ext.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c
index 7147f730850b..37f86a84ac5d 100644
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -1461,6 +1461,12 @@ struct scx_event_stats {
 	 * continued to run because there were no other tasks on the CPU.
 	 */
 	u64		SCX_EV_DISPATCH_KEEP_LAST;
+
+	/*
+	 * If SCX_OPS_ENQ_EXITING is not set, the number of times that a task
+	 * is dispatched to a local DSQ when exiting.
+	 */
+	u64		SCX_EV_ENQ_SKIP_EXITING;
 };
 
 /*
@@ -2068,8 +2074,10 @@ static void do_enqueue_task(struct rq *rq, struct task_struct *p, u64 enq_flags,
 
 	/* see %SCX_OPS_ENQ_EXITING */
 	if (!static_branch_unlikely(&scx_ops_enq_exiting) &&
-	    unlikely(p->flags & PF_EXITING))
+	    unlikely(p->flags & PF_EXITING)) {
+		__scx_add_event(SCX_EV_ENQ_SKIP_EXITING, 1);
 		goto local;
+	}
 
 	if (!SCX_HAS_OP(enqueue))
 		goto global;
@@ -4986,6 +4994,7 @@ static void scx_dump_state(struct scx_exit_info *ei, size_t dump_len)
 	scx_dump_event(s, &events, SCX_EV_SELECT_CPU_FALLBACK);
 	scx_dump_event(s, &events, SCX_EV_DISPATCH_LOCAL_DSQ_OFFLINE);
 	scx_dump_event(s, &events, SCX_EV_DISPATCH_KEEP_LAST);
+	scx_dump_event(s, &events, SCX_EV_ENQ_SKIP_EXITING);
 
 	if (seq_buf_has_overflowed(&s) && dump_len >= sizeof(trunc_marker))
 		memcpy(ei->dump + dump_len - sizeof(trunc_marker),
-- 
2.48.1

[PATCH v3 06/11] sched_ext: Add an event, SCX_EV_BYPASS_ACTIVATE

[Changwoo Min]

Add a core event, SCX_EV_BYPASS_ACTIVATE, which represents how many
times the bypass mode has been triggered.

Signed-off-by: Changwoo Min <changwoo@igalia.com>
---
 kernel/sched/ext.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c
index 37f86a84ac5d..07d54b52e971 100644
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -1467,6 +1467,11 @@ struct scx_event_stats {
 	 * is dispatched to a local DSQ when exiting.
 	 */
 	u64		SCX_EV_ENQ_SKIP_EXITING;
+
+	/*
+	 * The number of times the bypassing mode has been activated.
+	 */
+	u64		SCX_EV_BYPASS_ACTIVATE;
 };
 
 /*
@@ -4400,6 +4405,7 @@ static void scx_ops_bypass(bool bypass)
 		WARN_ON_ONCE(scx_ops_bypass_depth <= 0);
 		if (scx_ops_bypass_depth != 1)
 			goto unlock;
+		scx_add_event(SCX_EV_BYPASS_ACTIVATE, 1);
 	} else {
 		scx_ops_bypass_depth--;
 		WARN_ON_ONCE(scx_ops_bypass_depth < 0);
@@ -4995,6 +5001,7 @@ static void scx_dump_state(struct scx_exit_info *ei, size_t dump_len)
 	scx_dump_event(s, &events, SCX_EV_DISPATCH_LOCAL_DSQ_OFFLINE);
 	scx_dump_event(s, &events, SCX_EV_DISPATCH_KEEP_LAST);
 	scx_dump_event(s, &events, SCX_EV_ENQ_SKIP_EXITING);
+	scx_dump_event(s, &events, SCX_EV_BYPASS_ACTIVATE);
 
 	if (seq_buf_has_overflowed(&s) && dump_len >= sizeof(trunc_marker))
 		memcpy(ei->dump + dump_len - sizeof(trunc_marker),
@@ -7131,6 +7138,7 @@ __bpf_kfunc void scx_bpf_events(struct scx_event_stats *events,
 		scx_agg_event(&e_sys, e_cpu, SCX_EV_SELECT_CPU_FALLBACK);
 		scx_agg_event(&e_sys, e_cpu, SCX_EV_DISPATCH_LOCAL_DSQ_OFFLINE);
 		scx_agg_event(&e_sys, e_cpu, SCX_EV_DISPATCH_KEEP_LAST);
+		scx_agg_event(&e_sys, e_cpu, SCX_EV_BYPASS_ACTIVATE);
 	}
 
 	/*
-- 
2.48.1

[PATCH v3 07/11] sched_ext: Add an event, SCX_EV_BYPASS_DISPATCH

[Changwoo Min]

Add a core event, SCX_EV_BYPASS_DISPATCH, which represents how many
tasks have been dispatched in the bypass mode.

__scx_add_event() is used since the caller holds an rq lock,
so the preemption has already been disabled.

Signed-off-by: Changwoo Min <changwoo@igalia.com>
---
 kernel/sched/ext.c | 27 +++++++++++++++++++++++----
 1 file changed, 23 insertions(+), 4 deletions(-)

diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c
index 07d54b52e971..236cdb0071eb 100644
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -1468,6 +1468,11 @@ struct scx_event_stats {
 	 */
 	u64		SCX_EV_ENQ_SKIP_EXITING;
 
+	/*
+	 * The number of tasks dispatched in the bypassing mode.
+	 */
+	u64		SCX_EV_BYPASS_DISPATCH;
+
 	/*
 	 * The number of times the bypassing mode has been activated.
 	 */
@@ -2869,11 +2874,17 @@ static int balance_one(struct rq *rq, struct task_struct *prev)
 	}
 
 	/* if there already are tasks to run, nothing to do */
-	if (rq->scx.local_dsq.nr)
+	if (rq->scx.local_dsq.nr) {
+		if (scx_rq_bypassing(rq))
+			__scx_add_event(SCX_EV_BYPASS_DISPATCH, 1);
 		goto has_tasks;
+	}
 
-	if (consume_global_dsq(rq))
+	if (consume_global_dsq(rq)) {
+		if (scx_rq_bypassing(rq))
+			__scx_add_event(SCX_EV_BYPASS_DISPATCH, 1);
 		goto has_tasks;
+	}
 
 	if (!SCX_HAS_OP(dispatch) || scx_rq_bypassing(rq) || !scx_rq_online(rq))
 		goto no_tasks;
@@ -2899,10 +2910,16 @@ static int balance_one(struct rq *rq, struct task_struct *prev)
 			rq->scx.flags |= SCX_RQ_BAL_KEEP;
 			goto has_tasks;
 		}
-		if (rq->scx.local_dsq.nr)
+		if (rq->scx.local_dsq.nr) {
+			if (scx_rq_bypassing(rq))
+				__scx_add_event(SCX_EV_BYPASS_DISPATCH, 1);
 			goto has_tasks;
-		if (consume_global_dsq(rq))
+		}
+		if (consume_global_dsq(rq)) {
+			if (scx_rq_bypassing(rq))
+				__scx_add_event(SCX_EV_BYPASS_DISPATCH, 1);
 			goto has_tasks;
+		}
 
 		/*
 		 * ops.dispatch() can trap us in this loop by repeatedly
@@ -5001,6 +5018,7 @@ static void scx_dump_state(struct scx_exit_info *ei, size_t dump_len)
 	scx_dump_event(s, &events, SCX_EV_DISPATCH_LOCAL_DSQ_OFFLINE);
 	scx_dump_event(s, &events, SCX_EV_DISPATCH_KEEP_LAST);
 	scx_dump_event(s, &events, SCX_EV_ENQ_SKIP_EXITING);
+	scx_dump_event(s, &events, SCX_EV_BYPASS_DISPATCH);
 	scx_dump_event(s, &events, SCX_EV_BYPASS_ACTIVATE);
 
 	if (seq_buf_has_overflowed(&s) && dump_len >= sizeof(trunc_marker))
@@ -7138,6 +7156,7 @@ __bpf_kfunc void scx_bpf_events(struct scx_event_stats *events,
 		scx_agg_event(&e_sys, e_cpu, SCX_EV_SELECT_CPU_FALLBACK);
 		scx_agg_event(&e_sys, e_cpu, SCX_EV_DISPATCH_LOCAL_DSQ_OFFLINE);
 		scx_agg_event(&e_sys, e_cpu, SCX_EV_DISPATCH_KEEP_LAST);
+		scx_agg_event(&e_sys, e_cpu, SCX_EV_BYPASS_DISPATCH);
 		scx_agg_event(&e_sys, e_cpu, SCX_EV_BYPASS_ACTIVATE);
 	}
 
-- 
2.48.1

[PATCH v3 08/11] sched_ext: Add an event, SCX_EV_BYPASS_DURATION

[Changwoo Min]

Add a core event, SCX_EV_BYPASS_DURATION, which represents the
total duration of bypass modes in nanoseconds.

Signed-off-by: Changwoo Min <changwoo@igalia.com>
---
 kernel/sched/ext.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c
index 236cdb0071eb..b4871409bd1c 100644
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -1468,6 +1468,11 @@ struct scx_event_stats {
 	 */
 	u64		SCX_EV_ENQ_SKIP_EXITING;
 
+	/*
+	 * The total duration of bypass modes in nanoseconds.
+	 */
+	u64		SCX_EV_BYPASS_DURATION;
+
 	/*
 	 * The number of tasks dispatched in the bypassing mode.
 	 */
@@ -1529,6 +1534,12 @@ static DEFINE_PER_CPU(struct scx_event_stats, event_stats_cpu);
 } while (0)
 
 
+/*
+ * The last time the bypass mode started.
+ * This is used to measure SCX_EV_BYPASS_DURATION.
+ */
+static unsigned long scx_bypass_timestamp;
+
 static void scx_bpf_events(struct scx_event_stats *events, size_t events__sz);
 
 static enum scx_ops_enable_state scx_ops_enable_state(void)
@@ -4422,12 +4433,15 @@ static void scx_ops_bypass(bool bypass)
 		WARN_ON_ONCE(scx_ops_bypass_depth <= 0);
 		if (scx_ops_bypass_depth != 1)
 			goto unlock;
+		scx_bypass_timestamp = ktime_get_ns();
 		scx_add_event(SCX_EV_BYPASS_ACTIVATE, 1);
 	} else {
 		scx_ops_bypass_depth--;
 		WARN_ON_ONCE(scx_ops_bypass_depth < 0);
 		if (scx_ops_bypass_depth != 0)
 			goto unlock;
+		scx_add_event(SCX_EV_BYPASS_DURATION,
+			      ktime_get_ns() - scx_bypass_timestamp);
 	}
 
 	atomic_inc(&scx_ops_breather_depth);
@@ -5018,6 +5032,7 @@ static void scx_dump_state(struct scx_exit_info *ei, size_t dump_len)
 	scx_dump_event(s, &events, SCX_EV_DISPATCH_LOCAL_DSQ_OFFLINE);
 	scx_dump_event(s, &events, SCX_EV_DISPATCH_KEEP_LAST);
 	scx_dump_event(s, &events, SCX_EV_ENQ_SKIP_EXITING);
+	scx_dump_event(s, &events, SCX_EV_BYPASS_DURATION);
 	scx_dump_event(s, &events, SCX_EV_BYPASS_DISPATCH);
 	scx_dump_event(s, &events, SCX_EV_BYPASS_ACTIVATE);
 
@@ -7156,6 +7171,7 @@ __bpf_kfunc void scx_bpf_events(struct scx_event_stats *events,
 		scx_agg_event(&e_sys, e_cpu, SCX_EV_SELECT_CPU_FALLBACK);
 		scx_agg_event(&e_sys, e_cpu, SCX_EV_DISPATCH_LOCAL_DSQ_OFFLINE);
 		scx_agg_event(&e_sys, e_cpu, SCX_EV_DISPATCH_KEEP_LAST);
+		scx_agg_event(&e_sys, e_cpu, SCX_EV_BYPASS_DURATION);
 		scx_agg_event(&e_sys, e_cpu, SCX_EV_BYPASS_DISPATCH);
 		scx_agg_event(&e_sys, e_cpu, SCX_EV_BYPASS_ACTIVATE);
 	}
-- 
2.48.1

[PATCH v3 09/11] sched_ext: Add scx_bpf_events() and scx_read_event() for BPF schedulers

[Changwoo Min]

scx_bpf_events() is added to the header files so the BPF scheduler
can use it. Also, scx_read_event() is added to read an event type in a
compatible way.

Signed-off-by: Changwoo Min <changwoo@igalia.com>
---
 tools/sched_ext/include/scx/common.bpf.h | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/tools/sched_ext/include/scx/common.bpf.h b/tools/sched_ext/include/scx/common.bpf.h
index f254a39b86a5..705540003024 100644
--- a/tools/sched_ext/include/scx/common.bpf.h
+++ b/tools/sched_ext/include/scx/common.bpf.h
@@ -78,6 +78,10 @@ struct rq *scx_bpf_cpu_rq(s32 cpu) __ksym;
 struct cgroup *scx_bpf_task_cgroup(struct task_struct *p) __ksym __weak;
 u64 scx_bpf_now(void) __ksym __weak;
 
+void scx_bpf_events(struct scx_event_stats *events, size_t events__sz) __ksym __weak;
+#define scx_read_event(e, name)							\
+	(bpf_core_field_exists((e)->name) ? (e)->name : 0)
+
 /*
  * Use the following as @it__iter when calling scx_bpf_dsq_move[_vtime]() from
  * within bpf_for_each() loops.
-- 
2.48.1

[PATCH v3 10/11] sched_ext: Print core event count in scx_central scheduler

[Changwoo Min]

Modify the scx_central scheduler to print the core event counter
every second.

Signed-off-by: Changwoo Min <changwoo@igalia.com>
---
 tools/sched_ext/scx_central.bpf.c | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/tools/sched_ext/scx_central.bpf.c b/tools/sched_ext/scx_central.bpf.c
index 50bc1737c167..376c14d5dd0d 100644
--- a/tools/sched_ext/scx_central.bpf.c
+++ b/tools/sched_ext/scx_central.bpf.c
@@ -256,6 +256,7 @@ static int central_timerfn(void *map, int *key, struct bpf_timer *timer)
 	u64 now = scx_bpf_now();
 	u64 nr_to_kick = nr_queued;
 	s32 i, curr_cpu;
+	struct scx_event_stats events;
 
 	curr_cpu = bpf_get_smp_processor_id();
 	if (timer_pinned && (curr_cpu != central_cpu)) {
@@ -291,6 +292,26 @@ static int central_timerfn(void *map, int *key, struct bpf_timer *timer)
 
 	bpf_timer_start(timer, TIMER_INTERVAL_NS, BPF_F_TIMER_CPU_PIN);
 	__sync_fetch_and_add(&nr_timers, 1);
+
+	/* print event counters every second */
+	if (nr_timers % 1000 == 0) {
+		scx_bpf_events(&events, sizeof(events));
+
+		bpf_printk("%35s: %llu\n", "SCX_EV_SELECT_CPU_FALLBACK",
+			   scx_read_event(&events, SCX_EV_SELECT_CPU_FALLBACK));
+		bpf_printk("%35s: %llu\n", "SCX_EV_DISPATCH_LOCAL_DSQ_OFFLINE",
+			   scx_read_event(&events, SCX_EV_DISPATCH_LOCAL_DSQ_OFFLINE));
+		bpf_printk("%35s: %llu\n", "SCX_EV_DISPATCH_KEEP_LAST",
+			   scx_read_event(&events, SCX_EV_DISPATCH_KEEP_LAST));
+		bpf_printk("%35s: %llu\n", "SCX_EV_ENQ_SKIP_EXITING",
+			   scx_read_event(&events, SCX_EV_ENQ_SKIP_EXITING));
+		bpf_printk("%35s: %llu\n", "SCX_EV_BYPASS_DURATION",
+			   scx_read_event(&events, SCX_EV_BYPASS_DURATION));
+		bpf_printk("%35s: %llu\n", "SCX_EV_BYPASS_DISPATCH",
+			   scx_read_event(&events, SCX_EV_BYPASS_DISPATCH));
+		bpf_printk("%35s: %llu\n", "SCX_EV_BYPASS_ACTIVATE",
+			   scx_read_event(&events, SCX_EV_BYPASS_ACTIVATE));
+	}
 	return 0;
 }
 
-- 
2.48.1

[PATCH v3 11/11] sched_ext: Print core event count in scx_qmap scheduler

[Changwoo Min]

Modify the scx_qmap scheduler to print the core event counter
every second.

Signed-off-by: Changwoo Min <changwoo@igalia.com>
---
 tools/sched_ext/scx_qmap.bpf.c | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/tools/sched_ext/scx_qmap.bpf.c b/tools/sched_ext/scx_qmap.bpf.c
index 3a20bb0c014a..5edb79742e37 100644
--- a/tools/sched_ext/scx_qmap.bpf.c
+++ b/tools/sched_ext/scx_qmap.bpf.c
@@ -763,6 +763,8 @@ static void dump_shared_dsq(void)
 
 static int monitor_timerfn(void *map, int *key, struct bpf_timer *timer)
 {
+	struct scx_event_stats events;
+
 	bpf_rcu_read_lock();
 	dispatch_highpri(true);
 	bpf_rcu_read_unlock();
@@ -772,6 +774,23 @@ static int monitor_timerfn(void *map, int *key, struct bpf_timer *timer)
 	if (print_shared_dsq)
 		dump_shared_dsq();
 
+	scx_bpf_events(&events, sizeof(events));
+
+	bpf_printk("%35s: %llu\n", "SCX_EV_SELECT_CPU_FALLBACK",
+		   scx_read_event(&events, SCX_EV_SELECT_CPU_FALLBACK));
+	bpf_printk("%35s: %llu\n", "SCX_EV_DISPATCH_LOCAL_DSQ_OFFLINE",
+		   scx_read_event(&events, SCX_EV_DISPATCH_LOCAL_DSQ_OFFLINE));
+	bpf_printk("%35s: %llu\n", "SCX_EV_DISPATCH_KEEP_LAST",
+		   scx_read_event(&events, SCX_EV_DISPATCH_KEEP_LAST));
+	bpf_printk("%35s: %llu\n", "SCX_EV_ENQ_SKIP_EXITING",
+		   scx_read_event(&events, SCX_EV_ENQ_SKIP_EXITING));
+	bpf_printk("%35s: %llu\n", "SCX_EV_BYPASS_DURATION",
+		   scx_read_event(&events, SCX_EV_BYPASS_DURATION));
+	bpf_printk("%35s: %llu\n", "SCX_EV_BYPASS_DISPATCH",
+		   scx_read_event(&events, SCX_EV_BYPASS_DISPATCH));
+	bpf_printk("%35s: %llu\n", "SCX_EV_BYPASS_ACTIVATE",
+		   scx_read_event(&events, SCX_EV_BYPASS_ACTIVATE));
+
 	bpf_timer_start(timer, ONE_SEC_IN_NS, 0);
 	return 0;
 }
-- 
2.48.1

Re: [PATCH v3 05/11] sched_ext: Add an event, SCX_EV_ENQ_SKIP_EXITING

[Tejun Heo]

On Fri, Jan 31, 2025 at 04:09:32PM +0900, Changwoo Min wrote:
> Add a core event, SCX_EV_ENQ_SKIP_EXITING, which represents how many
> times a task is enqueued to a local DSQ when exiting if
> SCX_OPS_ENQ_EXITING is not set.
> 
> __scx_add_event() is used since the caller holds an rq lock,
> so the preemption has already been disabled.
> 
> Signed-off-by: Changwoo Min <changwoo@igalia.com>
> ---
>  kernel/sched/ext.c | 11 ++++++++++-
>  1 file changed, 10 insertions(+), 1 deletion(-)
> 
> diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c
> index 7147f730850b..37f86a84ac5d 100644
> --- a/kernel/sched/ext.c
> +++ b/kernel/sched/ext.c
> @@ -1461,6 +1461,12 @@ struct scx_event_stats {
>  	 * continued to run because there were no other tasks on the CPU.
>  	 */
>  	u64		SCX_EV_DISPATCH_KEEP_LAST;
> +
> +	/*
> +	 * If SCX_OPS_ENQ_EXITING is not set, the number of times that a task
> +	 * is dispatched to a local DSQ when exiting.
> +	 */
> +	u64		SCX_EV_ENQ_SKIP_EXITING;
>  };
>  
>  /*
> @@ -2068,8 +2074,10 @@ static void do_enqueue_task(struct rq *rq, struct task_struct *p, u64 enq_flags,
>  
>  	/* see %SCX_OPS_ENQ_EXITING */
>  	if (!static_branch_unlikely(&scx_ops_enq_exiting) &&
> -	    unlikely(p->flags & PF_EXITING))
> +	    unlikely(p->flags & PF_EXITING)) {
> +		__scx_add_event(SCX_EV_ENQ_SKIP_EXITING, 1);
>  		goto local;
> +	}
>  
>  	if (!SCX_HAS_OP(enqueue))
>  		goto global;
> @@ -4986,6 +4994,7 @@ static void scx_dump_state(struct scx_exit_info *ei, size_t dump_len)
>  	scx_dump_event(s, &events, SCX_EV_SELECT_CPU_FALLBACK);
>  	scx_dump_event(s, &events, SCX_EV_DISPATCH_LOCAL_DSQ_OFFLINE);
>  	scx_dump_event(s, &events, SCX_EV_DISPATCH_KEEP_LAST);
> +	scx_dump_event(s, &events, SCX_EV_ENQ_SKIP_EXITING);
>  
>  	if (seq_buf_has_overflowed(&s) && dump_len >= sizeof(trunc_marker))
>  		memcpy(ei->dump + dump_len - sizeof(trunc_marker),

Missing agg call?

-- 
tejun

Re: [PATCH v3 04/11] sched_ext: Add an event, SCX_EV_DISPATCH_KEEP_LAST

[Tejun Heo]

On Fri, Jan 31, 2025 at 04:09:31PM +0900, Changwoo Min wrote:
> Add a core event, SCX_EV_DISPATCH_KEEP_LAST, which represents how many
> times a task is continued to run without ops.enqueue() when
> SCX_OPS_ENQ_LAST is not set.
> 
> __scx_add_event() is used since the caller holds an rq lock,
> so the preemption has already been disabled.
> 
> Signed-off-by: Changwoo Min <changwoo@igalia.com>

Applied 1-4 to sched_ext/for-6.15.

Thanks.

-- 
tejun

Re: [PATCH v3 07/11] sched_ext: Add an event, SCX_EV_BYPASS_DISPATCH

[Tejun Heo]

Hello,

On Fri, Jan 31, 2025 at 04:09:34PM +0900, Changwoo Min wrote:
...
> @@ -2869,11 +2874,17 @@ static int balance_one(struct rq *rq, struct task_struct *prev)
>  	}
>  
>  	/* if there already are tasks to run, nothing to do */
> -	if (rq->scx.local_dsq.nr)
> +	if (rq->scx.local_dsq.nr) {
> +		if (scx_rq_bypassing(rq))
> +			__scx_add_event(SCX_EV_BYPASS_DISPATCH, 1);
>  		goto has_tasks;
> +	}
>  
> -	if (consume_global_dsq(rq))
> +	if (consume_global_dsq(rq)) {
> +		if (scx_rq_bypassing(rq))
> +			__scx_add_event(SCX_EV_BYPASS_DISPATCH, 1);

Hmm... Wouldn't it be easier to count it from select_task_rq_scx() and
do_enqueue_task()? The latter already has scx_rq_bypassing() condition and
the former one can easily cache bypassing test result and use that in the
else block.

> @@ -2899,10 +2910,16 @@ static int balance_one(struct rq *rq, struct task_struct *prev)
>  			rq->scx.flags |= SCX_RQ_BAL_KEEP;
>  			goto has_tasks;
>  		}
> -		if (rq->scx.local_dsq.nr)
> +		if (rq->scx.local_dsq.nr) {
> +			if (scx_rq_bypassing(rq))
> +				__scx_add_event(SCX_EV_BYPASS_DISPATCH, 1);
>  			goto has_tasks;
> -		if (consume_global_dsq(rq))
> +		}
> +		if (consume_global_dsq(rq)) {
> +			if (scx_rq_bypassing(rq))
> +				__scx_add_event(SCX_EV_BYPASS_DISPATCH, 1);
>  			goto has_tasks;
> +		}

The above can happen while bypass mode is being turned on but once on
control doesn't even reach here, right?

Thanks.

-- 
tejun

Re: [PATCH v3 02/11] sched_ext: Add an event, SCX_EV_SELECT_CPU_FALLBACK

[Tejun Heo]

On Fri, Jan 31, 2025 at 04:09:29PM +0900, Changwoo Min wrote:
...
> @@ -2170,6 +2175,11 @@ static void enqueue_task_scx(struct rq *rq, struct task_struct *p, int enq_flags
>  	do_enqueue_task(rq, p, enq_flags, sticky_cpu);
>  out:
>  	rq->scx.flags &= ~SCX_RQ_IN_WAKEUP;
> +
> +	if ((enq_flags & SCX_ENQ_CPU_SELECTED) &&
> +	    (cpu_of(rq) != p->scx.selected_cpu)) {
> +		__scx_add_event(SCX_EV_SELECT_CPU_FALLBACK, 1);
> +	}
>  }

I made minor edits to the above block while applying:

	if ((enq_flags & SCX_ENQ_CPU_SELECTED) &&
	    unlikely(cpu_of(rq) != p->scx.selected_cpu))
		__scx_add_event(SCX_EV_SELECT_CPU_FALLBACK, 1);

Thanks.

-- 
tejun

Re: [PATCH v3 08/11] sched_ext: Add an event, SCX_EV_BYPASS_DURATION

[Tejun Heo]

On Fri, Jan 31, 2025 at 04:09:35PM +0900, Changwoo Min wrote:
> +/*
> + * The last time the bypass mode started.
> + * This is used to measure SCX_EV_BYPASS_DURATION.
> + */
> +static unsigned long scx_bypass_timestamp;

Move it inside the function?

Thanks.

-- 
tejun

Re: [PATCH v3 00/11] sched_ext: Implement core event counters

[Tejun Heo]

On Fri, Jan 31, 2025 at 04:09:27PM +0900, Changwoo Min wrote:
> The sched_ext core often has to override the BPF scheduler decisions,
> and some events could be interesting but not easily visible.

Applied 1-4. Will wait for refresh on the rest.

Thanks.

-- 
tejun

Re: [PATCH v3 05/11] sched_ext: Add an event, SCX_EV_ENQ_SKIP_EXITING

[Changwoo Min]

Hello,

On 25. 2. 3. 02:20, Tejun Heo wrote:
> On Fri, Jan 31, 2025 at 04:09:32PM +0900, Changwoo Min wrote:
> Missing agg call?
Ah, you are right. Will fix it.

Regards,
Changwoo Min

Re: [PATCH v3 08/11] sched_ext: Add an event, SCX_EV_BYPASS_DURATION

[Changwoo Min]

Hello,

On 25. 2. 3. 02:35, Tejun Heo wrote:
> On Fri, Jan 31, 2025 at 04:09:35PM +0900, Changwoo Min wrote:
>> +/*
>> + * The last time the bypass mode started.
>> + * This is used to measure SCX_EV_BYPASS_DURATION.
>> + */
>> +static unsigned long scx_bypass_timestamp;
> 
> Move it inside the function?

That will look cleaner. I will move it inside scx_ops_bypass().

Regards,
Changwoo Min

Re: [PATCH v3 07/11] sched_ext: Add an event, SCX_EV_BYPASS_DISPATCH

[Changwoo Min]

Hello,

On 25. 2. 3. 02:33, Tejun Heo wrote:
> Hello,
> 
> On Fri, Jan 31, 2025 at 04:09:34PM +0900, Changwoo Min wrote:
> ...
>> @@ -2869,11 +2874,17 @@ static int balance_one(struct rq *rq, struct task_struct *prev)
>>   	}
>>   
>>   	/* if there already are tasks to run, nothing to do */
>> -	if (rq->scx.local_dsq.nr)
>> +	if (rq->scx.local_dsq.nr) {
>> +		if (scx_rq_bypassing(rq))
>> +			__scx_add_event(SCX_EV_BYPASS_DISPATCH, 1);
>>   		goto has_tasks;
>> +	}
>>   
>> -	if (consume_global_dsq(rq))
>> +	if (consume_global_dsq(rq)) {
>> +		if (scx_rq_bypassing(rq))
>> +			__scx_add_event(SCX_EV_BYPASS_DISPATCH, 1);
> 
> Hmm... Wouldn't it be easier to count it from select_task_rq_scx() and
> do_enqueue_task()? The latter already has scx_rq_bypassing() condition and
> the former one can easily cache bypassing test result and use that in the
> else block.

That makes sense. I will change the code as you suggested.

> 
>> @@ -2899,10 +2910,16 @@ static int balance_one(struct rq *rq, struct task_struct *prev)
>>   			rq->scx.flags |= SCX_RQ_BAL_KEEP;
>>   			goto has_tasks;
>>   		}
>> -		if (rq->scx.local_dsq.nr)
>> +		if (rq->scx.local_dsq.nr) {
>> +			if (scx_rq_bypassing(rq))
>> +				__scx_add_event(SCX_EV_BYPASS_DISPATCH, 1);
>>   			goto has_tasks;
>> -		if (consume_global_dsq(rq))
>> +		}
>> +		if (consume_global_dsq(rq)) {
>> +			if (scx_rq_bypassing(rq))
>> +				__scx_add_event(SCX_EV_BYPASS_DISPATCH, 1);
>>   			goto has_tasks;
>> +		}
> 
> The above can happen while bypass mode is being turned on but once on
> control doesn't even reach here, right?

You are right. I will clean this up.

Regards,
Changwoo Min