[PATCH RFC 0/7] rcutorture: Rewrite RCU priority-boost test around a CPU-hungry reader

[PATCH RFC 0/7] rcutorture: Rewrite RCU priority-boost test around a CPU-hungry reader

[Joel Fernandes]

Hi,

This RFC rewrites rcutorture's RCU priority-boost test so that it no longer
depends on disabling RT throttling and no longer infers boost failure from a
grace period merely taking too long, which is fragile.

The idea: make the to-be-boosted reader need a *lot* of CPU time, more than
throttling could ever give it within the test window.  Each test interval,
one CPU runs a SCHED_NORMAL reader that sits in an RCU read-side critical
section and spins until it has consumed test_boost_reader_ms of CPU time
(default 2000 ms), measured with the task's sum_exec_runtime so time spent
preempted does not count.  A SCHED_FIFO hog on the same CPU tries to starve
it, and a grace period is started so the reader blocks it.  An unboosted
reader only gets what the throttling leaves it, about 5% of the CPU
(~200 ms over the 4 s test_boost_duration), so it cannot finish.  If it does
finish in time, boosting worked.  If not, it is counted as a boost failure.

Related work: Sebastian's recent series on the DL fair_server taking over RT
throttling [1], with its RFC [2], also touches rcutorture's handling of RT
throttling. This patch should simplify or make those patches unnecessary.

Testing, all with kvm.sh on an x86 host, --duration 10 and
--bootargs "rcutorture.test_boost=2" (which forces the test on in scenarios
whose .boot file disables it), 10 minutes per scenario:

  Scenario      Result    Boost failures / intervals
  TREE01        SUCCESS   0 / 70
  TREE02        SUCCESS   0 / 73
  TREE03        SUCCESS   0 / 71
  TREE08        SUCCESS   0 / 74
  TREE09        SUCCESS   0 / 59
  BUSTED-BOOST  FAILURE   82 / 84   (expected, boosting disabled)

BUSTED-BOOST boots with rcutree.boost_disable=1 (patches 5-7), so its failures
show that the test catches broken boosting.  Each failure reports e.g. "Boost
inversion: reader got only 203 of 2000 ms CPU in 4003 jiffies", the CPU time
throttling leaves an unboosted task.

The series applies on top of the rcu/next tree.

[1] https://lore.kernel.org/all/20261110-sched_dl-v2-0-cc9399f7aeb1@linutronix.de/
[2] https://lore.kernel.org/all/20261023161348.t3lpPLQZ@linutronix.de/

Joel Fernandes (7):
  rcutorture: Rewrite RCU priority-boost test around a CPU-hungry reader
  rcutorture: Stop disabling RT throttling for boost testing
  rcutorture: Remove unused check_boost_failed hook
  doc: Document rcutorture.test_boost_reader_ms
  rcu: Add rcutree.boost_disable to disable priority boosting at runtime
  doc: Document rcutree.boost_disable
  rcutorture: Make BUSTED-BOOST disable RCU priority boosting

 .../admin-guide/kernel-parameters.txt         |  14 ++
 kernel/rcu/rcutorture.c                       | 192 +++++++++---------
 kernel/rcu/tree_plugin.h                      |   5 +-
 .../rcutorture/configs/rcu/BUSTED-BOOST.boot  |   1 +
 4 files changed, 116 insertions(+), 96 deletions(-)


base-commit: 44646ae309330530a23e1e4c44fd47d274cfd0ea
-- 
2.54.0

[PATCH RFC 1/7] rcutorture: Rewrite RCU priority-boost test around a CPU-hungry reader

[Joel Fernandes]

The old boost test relies on RT throttling being disabled and on grace
periods taking too long, which is fragile.  Instead, each interval one
CPU runs a normal-priority reader that spins inside an RCU read-side
critical section until it has consumed test_boost_reader_ms of CPU time
(measured via sum_exec_runtime, so time spent preempted does not count),
while a SCHED_FIFO hog on the same CPU tries to starve it.  With RT
throttling enabled, an unboosted reader gets only about 5% of the CPU,
far less than it needs within test_boost_duration.  If the reader
finishes, boosting worked, otherwise it failed.

Only one CPU is tested per interval because RCU boosts the readers of an
rcu_node one at a time, so a second slow reader sharing the rcu_node
would wait for the first one's whole CPU burn and falsely fail.

The reader publishes completion before leaving its critical section.
Otherwise the deboost on rcu_read_unlock() lets the hog preempt the
reader before it can set its flag, which falsely fails the test.

Testing methodology:

The test must pass when boosting works and fail when it does not, so
both directions were exercised.  All runs used kvm.sh with
--duration 10 --bootargs "rcutorture.test_boost=2" (which also forces
the test on in scenarios whose .boot file disables it).

Success testing: with the series applied and no other changes, run
TREE01 TREE02 TREE03 TREE08 TREE09 BUSTED-BOOST for 10 minutes each.
Every scenario ended "SUCCESS" with rtbf (boost failures) of 0 in 59 to
83 boost intervals.  (BUSTED-BOOST can no longer fail by construction
because CONFIG_RCU_BOOST is now enabled automatically for preemptible
RCU.)

Failure testing: break boosting on purpose by making the early-return
check in rcu_initiate_boost() (kernel/rcu/tree_plugin.h) always true:

	-	if (!rnp->boost_kthread_task ||
	+	if (1 || !rnp->boost_kthread_task ||

then run TREE03 and TREE02 for 10 minutes each.  Both ended "FAILURE",
with 58 of 72 (TREE03) and 62 of 72 (TREE02) intervals reporting:

	rcu_torture_boost boosting failed
	Boost inversion: reader got only 203 of 2000 ms CPU in 4003 jiffies

The 203 ms is the CPU time that RT throttling leaves a SCHED_NORMAL task
over test_boost_duration, confirming that the reader finishes only when
boosted.  The remaining intervals were discarded because of hotplug or
stutter.  Setting CONFIG_RCU_BOOST_DELAY=3000 also produces failures,
roughly half of the intervals.

Pitfall found while testing: a first version set the reader's finished
flag after rcu_read_unlock(), which failed about half of the intervals
in TREE01 (readers on CPUs brought online by its nr_cpus=17 hotplug)
even with boosting working, hence the ordering above.

test_boost_reader_ms should be well below test_boost_duration minus
RCU_BOOST_DELAY and slack: a boosted reader needs about
RCU_BOOST_DELAY + test_boost_reader_ms / 0.95 to finish.

Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
---
 kernel/rcu/rcutorture.c | 180 +++++++++++++++++++++++-----------------
 1 file changed, 106 insertions(+), 74 deletions(-)

diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c
index 924705b7759d..2152906a807e 100644
--- a/kernel/rcu/rcutorture.c
+++ b/kernel/rcu/rcutorture.c
@@ -149,6 +149,7 @@ torture_param(int, test_boost, 1, "Test RCU prio boost: 0=no, 1=maybe, 2=yes.");
 torture_param(int, test_boost_duration, 4, "Duration of each boost test, seconds.");
 torture_param(int, test_boost_holdoff, 0, "Holdoff time from rcutorture start, seconds.");
 torture_param(int, test_boost_interval, 7, "Interval between boost tests, seconds.");
+torture_param(int, test_boost_reader_ms, 2000, "Boost-test reader CPU time, ms.");
 torture_param(int, test_nmis, 0, "End-test NMI tests, 0 to disable.");
 torture_param(bool, test_no_idle_hz, true, "Test support for tickless idle CPUs");
 torture_param(int, test_srcu_lockdep, 0, "Test specified SRCU deadlock scenario.");
@@ -333,6 +334,7 @@ static bool shutdown_time_arrived(void)
 }
 
 static unsigned long boost_starttime;	/* jiffies of next boost test start. */
+static unsigned long boost_claimed;	/* ->boost_starttime of claimed test. */
 static DEFINE_MUTEX(boost_mutex);	/* protect setting boost_starttime */
 					/*  and boost task create/destroy. */
 static atomic_t barrier_cbs_count;	/* Barrier callbacks registered. */
@@ -1334,10 +1336,17 @@ static unsigned long rcutorture_seq_diff(unsigned long new, unsigned long old)
 }
 
 /*
- * RCU torture priority-boost testing.  Runs one real-time thread per
- * CPU for moderate bursts, repeatedly starting grace periods and waiting
- * for them to complete.  If a given grace period takes too long, we assume
- * that priority inversion has occurred.
+ * RCU torture priority-boost testing.  Each test interval, one CPU
+ * runs a normal-priority reader that spins inside an RCU read-side
+ * critical section until it has consumed test_boost_reader_ms of CPU
+ * time, while a SCHED_FIFO hog on the same CPU tries to starve it.  RT
+ * throttling is left enabled, so an unboosted reader gets only a few
+ * percent of the CPU, far too little to finish within test_boost_duration.
+ * If the reader finishes anyway, boosting lifted it above the hog.
+ *
+ * Only one CPU is tested per interval because RCU boosts the readers
+ * of a given rcu_node one at a time, so a second slow reader sharing the
+ * rcu_node would wait for the first one's whole CPU burn.
  */
 
 static int old_rt_runtime = -1;
@@ -1366,61 +1375,111 @@ static void rcu_torture_enable_rt_throttle(void)
 	old_rt_runtime = -1;
 }
 
-static bool rcu_torture_boost_failed(unsigned long gp_state, unsigned long *start)
+struct rcu_torture_boost_reader {
+	struct completion done;
+	bool in_cs;	// Reader is inside its RCU read-side critical section.
+	bool finished;	// Reader consumed all of its CPU time.
+	bool abort;	// Hog asks the reader to give up.
+	u64 ran_ns;	// CPU time consumed so far.
+};
+
+// Reader CPU time so far, retrying to avoid torn 64-bit loads on 32-bit.
+static u64 rcu_torture_boost_runtime(void)
+{
+	u64 a, b;
+
+	do {
+		a = READ_ONCE(current->se.sum_exec_runtime);
+		b = READ_ONCE(current->se.sum_exec_runtime);
+	} while (a != b);
+	return a;
+}
+
+static int rcu_torture_boost_reader(void *arg)
+{
+	struct rcu_torture_boost_reader *br = arg;
+	u64 goal = (u64)test_boost_reader_ms * NSEC_PER_MSEC;
+	u64 ran;
+	u64 start;
+	int idx;
+
+	idx = cur_ops->readlock();
+	start = rcu_torture_boost_runtime();
+	WRITE_ONCE(br->in_cs, true);
+	do {
+		cpu_relax();
+		ran = rcu_torture_boost_runtime() - start;
+		WRITE_ONCE(br->ran_ns, ran);
+	} while (ran < goal && !READ_ONCE(br->abort) && !torture_must_stop());
+	// Publish before unlocking: deboosting lets the hog preempt us at once.
+	WRITE_ONCE(br->finished, ran >= goal);
+	cur_ops->readunlock(idx);
+	kthread_complete_and_exit(&br->done, 0);
+}
+
+// Run one boost test on this CPU.
+static void rcu_torture_boost_once(void)
 {
-	int cpu;
 	static int dbg_done;
-	unsigned long end = jiffies;
-	bool gp_done;
-	unsigned long j;
-	static unsigned long last_persist;
-	unsigned long lp;
-	unsigned long mininterval = test_boost_duration * HZ - HZ / 2;
+	struct rcu_torture_boost_reader br = { };
+	unsigned long endtime;
+	unsigned long gp_state = 0;
+	unsigned long start;
+	struct task_struct *t;
+	bool stuttered = false;
 
-	if (end - *start > mininterval) {
-		// Recheck after checking time to avoid false positives.
-		smp_mb(); // Time check before grace-period check.
-		if (cur_ops->poll_gp_state(gp_state))
-			return false; // passed, though perhaps just barely
-		if (cur_ops->check_boost_failed && !cur_ops->check_boost_failed(gp_state, &cpu)) {
-			// At most one persisted message per boost test.
-			j = jiffies;
-			lp = READ_ONCE(last_persist);
-			if (time_after(j, lp + mininterval) &&
-			    cmpxchg(&last_persist, lp, j) == lp) {
-				if (cpu < 0)
-					pr_info("Boost inversion persisted: QS from all CPUs\n");
-				else
-					pr_info("Boost inversion persisted: No QS from CPU %d\n", cpu);
-			}
-			return false; // passed on a technicality
+	init_completion(&br.done);
+	t = kthread_create(rcu_torture_boost_reader, &br, "rcu_torture_boost_rdr");
+	if (IS_ERR(t)) {
+		n_rcu_torture_boost_ktrerror++;
+		return;
+	}
+	kthread_bind(t, raw_smp_processor_id());
+	wake_up_process(t);
+
+	// Wait for the reader to enter its critical section, then start
+	// a grace period that it will block.
+	start = jiffies;
+	while (!READ_ONCE(br.in_cs) && !torture_must_stop() &&
+	       time_before(jiffies, start + HZ))
+		schedule_timeout_uninterruptible(1);
+	if (!READ_ONCE(br.in_cs))
+		goto out;
+	gp_state = cur_ops->start_gp_poll();
+
+	// Hog the CPU until the reader finishes or the interval ends.
+	start = jiffies;
+	endtime = start + test_boost_duration * HZ;
+	while (!READ_ONCE(br.finished) && time_before(jiffies, endtime) &&
+	       !torture_must_stop()) {
+		if (stutter_wait("rcu_torture_boost")) {
+			sched_set_fifo_low(current);
+			stuttered = true;
+			break;
 		}
+		cpu_relax();
+	}
+
+	smp_mb(); // Time check before reader check.
+	if (!READ_ONCE(br.finished) && !stuttered && !torture_must_stop()) {
 		VERBOSE_TOROUT_STRING("rcu_torture_boost boosting failed");
 		n_rcu_torture_boost_failure++;
+		pr_info("Boost inversion: reader got only %llu of %d ms CPU in %lu jiffies, GP %s.\n",
+			READ_ONCE(br.ran_ns) / NSEC_PER_MSEC, test_boost_reader_ms, jiffies - start,
+			cur_ops->poll_gp_state(gp_state) ? "ended" : "still pending");
 		if (!xchg(&dbg_done, 1) && cur_ops->gp_kthread_dbg) {
 			pr_info("Boost inversion thread ->rt_priority %u gp_state %lu jiffies %lu\n",
-				current->rt_priority, gp_state, end - *start);
+				current->rt_priority, gp_state, jiffies - start);
 			cur_ops->gp_kthread_dbg();
-			// Recheck after print to flag grace period ending during splat.
-			gp_done = cur_ops->poll_gp_state(gp_state);
-			pr_info("Boost inversion: GP %lu %s.\n", gp_state,
-				gp_done ? "ended already" : "still pending");
-
 		}
-
-		return true; // failed
-	} else if (cur_ops->check_boost_failed && !cur_ops->check_boost_failed(gp_state, NULL)) {
-		*start = jiffies;
 	}
-
-	return false; // passed
+out:
+	WRITE_ONCE(br.abort, true);
+	wait_for_completion(&br.done);
 }
 
 static int rcu_torture_boost(void *arg)
 {
-	unsigned long endtime;
-	unsigned long gp_state;
-	unsigned long gp_state_time;
 	unsigned long oldstarttime;
 	unsigned long booststarttime = get_torture_init_jiffies() + test_boost_holdoff * HZ;
 
@@ -1441,9 +1500,6 @@ static int rcu_torture_boost(void *arg)
 
 	/* Each pass through the following loop does one boost-test cycle. */
 	do {
-		bool failed = false; // Test failed already in this test interval
-		bool gp_initiated = false;
-
 		if (kthread_should_stop())
 			goto checkwait;
 
@@ -1457,33 +1513,9 @@ static int rcu_torture_boost(void *arg)
 				goto checkwait;
 		}
 
-		// Do one boost-test interval.
-		endtime = oldstarttime + test_boost_duration * HZ;
-		while (time_before(jiffies, endtime)) {
-			// Has current GP gone too long?
-			if (gp_initiated && !failed && !cur_ops->poll_gp_state(gp_state))
-				failed = rcu_torture_boost_failed(gp_state, &gp_state_time);
-			// If we don't have a grace period in flight, start one.
-			if (!gp_initiated || cur_ops->poll_gp_state(gp_state)) {
-				gp_state = cur_ops->start_gp_poll();
-				gp_initiated = true;
-				gp_state_time = jiffies;
-			}
-			if (stutter_wait("rcu_torture_boost")) {
-				sched_set_fifo_low(current);
-				// If the grace period already ended,
-				// we don't know when that happened, so
-				// start over.
-				if (cur_ops->poll_gp_state(gp_state))
-					gp_initiated = false;
-			}
-			if (torture_must_stop())
-				goto checkwait;
-		}
-
-		// In case the grace period extended beyond the end of the loop.
-		if (gp_initiated && !failed && !cur_ops->poll_gp_state(gp_state))
-			rcu_torture_boost_failed(gp_state, &gp_state_time);
+		// Do one boost-test interval, if no other CPU claimed it.
+		if (xchg(&boost_claimed, oldstarttime) != oldstarttime)
+			rcu_torture_boost_once();
 
 		/*
 		 * Set the start time of the next test interval.
-- 
2.54.0

[PATCH RFC 2/7] rcutorture: Stop disabling RT throttling for boost testing

[Joel Fernandes]

The CPU-hungry-reader boost test depends on RT throttling to bound the
CPU time an unboosted reader can obtain, so there is no longer any
reason to disable it.  Remove the code that does so, along with the
now-unneeded sched/sysctl.h include.

Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
---
 kernel/rcu/rcutorture.c | 29 -----------------------------
 1 file changed, 29 deletions(-)

diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c
index 2152906a807e..ad980e9257dd 100644
--- a/kernel/rcu/rcutorture.c
+++ b/kernel/rcu/rcutorture.c
@@ -43,7 +43,6 @@
 #include <linux/torture.h>
 #include <linux/vmalloc.h>
 #include <linux/sched/debug.h>
-#include <linux/sched/sysctl.h>
 #include <linux/oom.h>
 #include <linux/tick.h>
 #include <linux/rcupdate_trace.h>
@@ -1349,32 +1348,6 @@ static unsigned long rcutorture_seq_diff(unsigned long new, unsigned long old)
  * rcu_node would wait for the first one's whole CPU burn.
  */
 
-static int old_rt_runtime = -1;
-
-static void rcu_torture_disable_rt_throttle(void)
-{
-	/*
-	 * Disable RT throttling so that rcutorture's boost threads don't get
-	 * throttled. Only possible if rcutorture is built-in otherwise the
-	 * user should manually do this by setting the sched_rt_period_us and
-	 * sched_rt_runtime sysctls.
-	 */
-	if (!IS_BUILTIN(CONFIG_RCU_TORTURE_TEST) || old_rt_runtime != -1)
-		return;
-
-	old_rt_runtime = sysctl_sched_rt_runtime;
-	sysctl_sched_rt_runtime = -1;
-}
-
-static void rcu_torture_enable_rt_throttle(void)
-{
-	if (!IS_BUILTIN(CONFIG_RCU_TORTURE_TEST) || old_rt_runtime == -1)
-		return;
-
-	sysctl_sched_rt_runtime = old_rt_runtime;
-	old_rt_runtime = -1;
-}
-
 struct rcu_torture_boost_reader {
 	struct completion done;
 	bool in_cs;	// Reader is inside its RCU read-side critical section.
@@ -3422,7 +3395,6 @@ static int rcutorture_booster_cleanup(unsigned int cpu)
 	mutex_lock(&boost_mutex);
 	t = boost_tasks[cpu];
 	boost_tasks[cpu] = NULL;
-	rcu_torture_enable_rt_throttle();
 	mutex_unlock(&boost_mutex);
 
 	/* This must be outside of the mutex, otherwise deadlock! */
@@ -3460,7 +3432,6 @@ static int rcutorture_booster_init(unsigned int cpu)
 
 	/* Don't allow time recalculation while creating a new task. */
 	mutex_lock(&boost_mutex);
-	rcu_torture_disable_rt_throttle();
 	VERBOSE_TOROUT_STRING("Creating rcu_torture_boost task");
 	boost_tasks[cpu] = kthread_run_on_cpu(rcu_torture_boost, NULL,
 					      cpu, "rcu_torture_boost_%u");
-- 
2.54.0

[PATCH RFC 3/7] rcutorture: Remove unused check_boost_failed hook

[Joel Fernandes]

Nothing in rcutorture calls ->check_boost_failed() now that boost
failure is decided by whether the reader finished, so drop the hook
from struct rcu_torture_ops and its initializer.

Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
---
 kernel/rcu/rcutorture.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c
index ad980e9257dd..b13b25b686f7 100644
--- a/kernel/rcu/rcutorture.c
+++ b/kernel/rcu/rcutorture.c
@@ -43,6 +43,7 @@
 #include <linux/torture.h>
 #include <linux/vmalloc.h>
 #include <linux/sched/debug.h>
+#include <linux/sched/sysctl.h>
 #include <linux/oom.h>
 #include <linux/tick.h>
 #include <linux/rcupdate_trace.h>
@@ -425,7 +426,6 @@ struct rcu_torture_ops {
 	void (*fqs)(void);
 	void (*stats)(void);
 	void (*gp_kthread_dbg)(void);
-	bool (*check_boost_failed)(unsigned long gp_state, int *cpup);
 	int (*stall_dur)(void);
 	void (*get_gp_data)(int *flags, unsigned long *gp_seq);
 	void (*gp_slow_register)(atomic_t *rgssp);
@@ -637,7 +637,6 @@ static struct rcu_torture_ops rcu_ops = {
 	.cb_barrier		= rcu_barrier,
 	.fqs			= rcu_force_quiescent_state,
 	.gp_kthread_dbg		= show_rcu_gp_kthreads,
-	.check_boost_failed	= rcu_check_boost_fail,
 	.stall_dur		= rcu_jiffies_till_stall_check,
 	.get_gp_data		= rcutorture_get_gp_data,
 	.gp_slow_register	= rcu_gp_slow_register,
-- 
2.54.0

[PATCH RFC 4/7] doc: Document rcutorture.test_boost_reader_ms

[Joel Fernandes]

Add the new priority-boost reader CPU-time parameter to
kernel-parameters.txt.

Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
---
 Documentation/admin-guide/kernel-parameters.txt | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 176648941aec..ac97cbd98b46 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -6283,6 +6283,13 @@ Kernel parameters
 	rcutorture.test_boost_interval= [KNL]
 			Interval (s) between each boost test.
 
+	rcutorture.test_boost_reader_ms= [KNL]
+			CPU time (ms) that each boost-test reader must
+			consume within the test_boost_duration interval.
+			The reader runs in an RCU read-side critical section
+			while a real-time CPU hog tries to starve it, so it
+			can finish in time only if it is priority boosted.
+
 	rcutorture.test_no_idle_hz= [KNL]
 			Test RCU's dyntick-idle handling.  See also the
 			rcutorture.shuffle_interval parameter.
-- 
2.54.0

[PATCH RFC 5/7] rcu: Add rcutree.boost_disable to disable priority boosting at runtime

[Joel Fernandes]

CONFIG_RCU_BOOST is now enabled automatically for preemptible RCU and
cannot be turned off, so rcutorture's BUSTED-BOOST scenario, which is
meant to show that rcutorture detects broken priority boosting, can no
longer break it.  Add a boot parameter that makes rcu_initiate_boost()
do nothing, giving that scenario a way to break boosting.

Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
---
 kernel/rcu/tree_plugin.h | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index ae5d21bae7bc..04a58dd003f4 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -1270,6 +1270,9 @@ static int rcu_boost_kthread(void *arg)
 	return 0;
 }
 
+static bool boost_disable;
+module_param(boost_disable, bool, 0444);
+
 /*
  * Check to see if it is time to start boosting RCU readers that are
  * blocking the current grace period, and, if so, tell the per-rcu_node
@@ -1284,7 +1287,7 @@ static void rcu_initiate_boost(struct rcu_node *rnp, unsigned long flags)
 	__releases(rnp->lock)
 {
 	raw_lockdep_assert_held_rcu_node(rnp);
-	if (!rnp->boost_kthread_task ||
+	if (boost_disable || !rnp->boost_kthread_task ||
 	    (!rcu_preempt_blocked_readers_cgp(rnp) && !rnp->exp_tasks)) {
 		raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
 		return;
-- 
2.54.0

[PATCH RFC 6/7] doc: Document rcutree.boost_disable

[Joel Fernandes]

Add the new rcutree.boost_disable parameter to kernel-parameters.txt.

Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
---
 Documentation/admin-guide/kernel-parameters.txt | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index ac97cbd98b46..8f91499b3488 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -5651,6 +5651,13 @@ Kernel parameters
 			Set maximum number of finished RCU callbacks to
 			process in one batch.
 
+	rcutree.boost_disable=	[KNL]
+			Disable RCU priority boosting at runtime, even in
+			kernels built with CONFIG_RCU_BOOST=y.  This is
+			for testing the boost tests, for example by the
+			rcutorture BUSTED-BOOST scenario, and should not
+			be used otherwise.
+
 	rcutree.csd_lock_suppress_rcu_stall=	[KNL]
 			Do only a one-line RCU CPU stall warning when
 			there is an ongoing too-long CSD-lock wait.
-- 
2.54.0

[PATCH RFC 7/7] rcutorture: Make BUSTED-BOOST disable RCU priority boosting

[Joel Fernandes]

BUSTED-BOOST is supposed to fail because priority boosting is broken,
but boosting is now always built in for preemptible RCU, so the
scenario passed.  Boot it with rcutree.boost_disable=1 so that it again
exercises rcutorture's ability to detect missing priority boosting.

Testing: kvm.sh --configs BUSTED-BOOST --duration 10 now ends FAILURE,
with 81 of 84 boost intervals reporting:

	rcu_torture_boost boosting failed
	Boost inversion: reader got only 203 of 2000 ms CPU in 4003 jiffies

Without the parameter the same scenario passed with no boost failures.

Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
---
 tools/testing/selftests/rcutorture/configs/rcu/BUSTED-BOOST.boot | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tools/testing/selftests/rcutorture/configs/rcu/BUSTED-BOOST.boot b/tools/testing/selftests/rcutorture/configs/rcu/BUSTED-BOOST.boot
index 84f6bb98ce99..5aa92357958a 100644
--- a/tools/testing/selftests/rcutorture/configs/rcu/BUSTED-BOOST.boot
+++ b/tools/testing/selftests/rcutorture/configs/rcu/BUSTED-BOOST.boot
@@ -6,3 +6,4 @@ rcutree.gp_cleanup_delay=3
 rcutree.kthread_prio=2
 threadirqs
 rcutree.use_softirq=0
+rcutree.boost_disable=1
-- 
2.54.0

Re: [PATCH RFC 0/7] rcutorture: Rewrite RCU priority-boost test around a CPU-hungry reader

[Sebastian Andrzej Siewior]

On 2026-11-11 16:38:18 [-0500], Joel Fernandes wrote:
> Related work: Sebastian's recent series on the DL fair_server taking over RT
> throttling [1], with its RFC [2], also touches rcutorture's handling of RT
> throttling. This patch should simplify or make those patches unnecessary.

Let me look at this. If this works out then the disabling of the
fair_server becomes no longer required which would be my favorite, too.

Sebastian

Re: [PATCH RFC 1/7] rcutorture: Rewrite RCU priority-boost test around a CPU-hungry reader

[Sebastian Andrzej Siewior]

On 2026-11-11 16:38:19 [-0500], Joel Fernandes wrote:
> --- a/kernel/rcu/rcutorture.c
> +++ b/kernel/rcu/rcutorture.c
> +	init_completion(&br.done);
> +	t = kthread_create(rcu_torture_boost_reader, &br, "rcu_torture_boost_rdr");
> +	if (IS_ERR(t)) {
> +		n_rcu_torture_boost_ktrerror++;
> +		return;
> +	}
> +	kthread_bind(t, raw_smp_processor_id());
> +	wake_up_process(t);

=> kthread_run_on_cpu()

Sebastian

Re: [PATCH RFC 1/7] rcutorture: Rewrite RCU priority-boost test around a CPU-hungry reader

[Sebastian Andrzej Siewior]

On 2026-11-11 16:38:19 [-0500], Joel Fernandes wrote:
> The old boost test relies on RT throttling being disabled and on grace
> periods taking too long, which is fragile.  Instead, each interval one
> CPU runs a normal-priority reader that spins inside an RCU read-side
> critical section until it has consumed test_boost_reader_ms of CPU time
> (measured via sum_exec_runtime, so time spent preempted does not count),
> while a SCHED_FIFO hog on the same CPU tries to starve it.  With RT
> throttling enabled, an unboosted reader gets only about 5% of the CPU,
> far less than it needs within test_boost_duration.  If the reader
> finishes, boosting worked, otherwise it failed.

What I see from tracing is:

| rcu_torture_boo-2066    [001] d..2.   237.448002: sched_switch: prev_comm=rcu_torture_boo prev_pid=2066 prev_prio=120 prev_state=R+ ==> next_comm=rcu_torture_boo next_pid=93 next_prio=98

preempted as SCHED_OTHER

|          rcub/0-16      [003] d..3.   237.460085: sched_pi_setprio: comm=rcu_torture_boo pid=2066 oldprio=120 newprio=97

boosted shortly after

| rcu_torture_boo-93      [001] d..2.   237.460108: sched_switch: prev_comm=rcu_torture_boo prev_pid=93 prev_prio=98 prev_state=R+ ==> next_comm=rcu_torture_boo next_pid=2066 next_prio=97
| rcu_torture_boo-2066    [001] d.h1.   237.463994: sched_stat_runtime: comm=rcu_torture_boo pid=2066 runtime=3908069 [ns]
| rcu_torture_boo-2066    [001] d.h1.   237.467985: sched_stat_runtime: comm=rcu_torture_boo pid=2066 runtime=3991212 [ns]
| rcu_torture_boo-2066    [001] d.h1.   237.471995: sched_stat_runtime: comm=rcu_torture_boo pid=2066 runtime=4009726 [ns]

back on CPU and not pushed away.
|  rcu_torture_boo-2066    [001] dN.2.   238.397969: sched_stat_runtime: comm=rcu_torture_boo pid=2066 runtime=5311 [ns]
|  rcu_torture_boo-2066    [001] d..2.   238.397975: sched_switch: prev_comm=rcu_torture_boo prev_pid=2066 prev_prio=97 prev_state=R+ ==> next_comm=kworker/1:3 next_pid=1386 next_prio=120
|     kworker/1:1H-745     [001] d..2.   238.398067: sched_switch: prev_comm=kworker/1:1H prev_pid=745 prev_prio=100 prev_state=I ==> next_comm=rcu_torture_boo next_pid=2066 next_prio=97
|  rcu_torture_boo-2066    [001] d.s3.   238.398122: sched_waking: comm=kworker/1:3 pid=1386 prio=120 target_cpu=001
|  rcu_torture_boo-2066    [001] d.s4.   238.398124: sched_stat_runtime: comm=rcu_torture_boo pid=2066 runtime=63431 [ns]
|  rcu_torture_boo-2066    [001] dNs4.   238.398128: sched_wakeup: comm=kworker/1:3 pid=1386 prio=120 target_cpu=001
|  rcu_torture_boo-2066    [001] d..2.   238.398130: sched_switch: prev_comm=rcu_torture_boo prev_pid=2066 prev_prio=97 prev_state=R+ ==> next_comm=kworker/1:3 next_pid=1386 next_prio=120
|      kworker/1:3-1386    [001] d..2.   238.398137: sched_switch: prev_comm=kworker/1:3 prev_pid=1386 prev_prio=120 prev_state=I ==> next_comm=rcu_torture_boo next_pid=2066 next_prio=97
|  rcu_torture_boo-2066    [001] d.h1.   238.399986: sched_stat_runtime: comm=rcu_torture_boo pid=2066 runtime=1851337 [ns]
|  rcu_torture_boo-2066    [001] d.h1.   238.400081: sched_stat_runtime: comm=rcu_torture_boo pid=2066 runtime=94860 [ns]

The fair_server preempts now and then.
|  rcu_torture_boo-2066    [001] d.h1.   239.455988: sched_stat_runtime: comm=rcu_torture_boo pid=2066 runtime=4002590 [ns]
|  rcu_torture_boo-2066    [001] d..3.   239.455999: sched_waking: comm=rcu_preempt pid=15 prio=97 target_cpu=000
|  rcu_torture_boo-2066    [001] d..3.   239.456012: sched_waking: comm=rcu_exp_gp_kthr pid=18 prio=120 target_cpu=002
|  rcu_torture_boo-2066    [001] d..3.   239.456023: sched_pi_setprio: comm=rcu_torture_boo pid=2066 oldprio=97 newprio=120
|  rcu_torture_boo-2066    [001] d..3.   239.456024: sched_stat_runtime: comm=rcu_torture_boo pid=2066 runtime=35773 [ns]
|  rcu_torture_boo-2066    [001] d..2.   239.456032: sched_switch: prev_comm=rcu_torture_boo prev_pid=2066 prev_prio=120 prev_state=R+ ==> next_comm=rcu_torture_boo next_pid=93 next_prio=98

Leaving the RCU section and getting de-boosted.

This looks good.

…

> Success testing: with the series applied and no other changes, run
> TREE01 TREE02 TREE03 TREE08 TREE09 BUSTED-BOOST for 10 minutes each.
> Every scenario ended "SUCCESS" with rtbf (boost failures) of 0 in 59 to
> 83 boost intervals.  (BUSTED-BOOST can no longer fail by construction
> because CONFIG_RCU_BOOST is now enabled automatically for preemptible
> RCU.)

I didn't run the individual tests but checked in my rcu-tree setup that
boosting works as it should and that it complains once it does not. And
the throttling of the RT tasks has no visible impact.

Sebastian

Re: [PATCH RFC 2/7] rcutorture: Stop disabling RT throttling for boost testing

[Sebastian Andrzej Siewior]

On 2026-11-11 16:38:20 [-0500], Joel Fernandes wrote:
> The CPU-hungry-reader boost test depends on RT throttling to bound the
> CPU time an unboosted reader can obtain, so there is no longer any
> reason to disable it.  Remove the code that does so, along with the
> now-unneeded sched/sysctl.h include.

Can I talk you into replacing this with
  20261113-sched_dl-v3-3-a76ffa1cdeee@linutronix.de ?

It removes an additional declaration in RCU's header file plus (the more
important part) it explains that it is not doing anything rather than
saying it is no longer needed due to the rework.

> Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>

Sebastian

Re: [PATCH RFC 3/7] rcutorture: Remove unused check_boost_failed hook

[Sebastian Andrzej Siewior]

On 2026-11-11 16:38:21 [-0500], Joel Fernandes wrote:
> --- a/kernel/rcu/rcutorture.c
> +++ b/kernel/rcu/rcutorture.c
> @@ -43,6 +43,7 @@
>  #include <linux/torture.h>
>  #include <linux/vmalloc.h>
>  #include <linux/sched/debug.h>
> +#include <linux/sched/sysctl.h>
>  #include <linux/oom.h>
>  #include <linux/tick.h>
>  #include <linux/rcupdate_trace.h>

This looks like folding and rebase went wrong.

> @@ -425,7 +426,6 @@ struct rcu_torture_ops {
>  	void (*fqs)(void);
>  	void (*stats)(void);
>  	void (*gp_kthread_dbg)(void);
> -	bool (*check_boost_failed)(unsigned long gp_state, int *cpup);
>  	int (*stall_dur)(void);
>  	void (*get_gp_data)(int *flags, unsigned long *gp_seq);
>  	void (*gp_slow_register)(atomic_t *rgssp);
> @@ -637,7 +637,6 @@ static struct rcu_torture_ops rcu_ops = {
>  	.cb_barrier		= rcu_barrier,
>  	.fqs			= rcu_force_quiescent_state,
>  	.gp_kthread_dbg		= show_rcu_gp_kthreads,
> -	.check_boost_failed	= rcu_check_boost_fail,
>  	.stall_dur		= rcu_jiffies_till_stall_check,
>  	.get_gp_data		= rcutorture_get_gp_data,
>  	.gp_slow_register	= rcu_gp_slow_register,

if I grep for rcu_check_boost_fail at this point I see the function, its
declaration but no users. That means it could be removed, too.

Sebastian

Re: [PATCH RFC 1/7] rcutorture: Rewrite RCU priority-boost test around a CPU-hungry reader

[Sebastian Andrzej Siewior]

On 2026-11-11 16:38:19 [-0500], Joel Fernandes wrote:
> --- a/kernel/rcu/rcutorture.c
> +++ b/kernel/rcu/rcutorture.c
> @@ -149,6 +149,7 @@ torture_param(int, test_boost, 1, "Test RCU prio boost: 0=no, 1=maybe, 2=yes.");
>  torture_param(int, test_boost_duration, 4, "Duration of each boost test, seconds.");
>  torture_param(int, test_boost_holdoff, 0, "Holdoff time from rcutorture start, seconds.");
>  torture_param(int, test_boost_interval, 7, "Interval between boost tests, seconds.");
> +torture_param(int, test_boost_reader_ms, 2000, "Boost-test reader CPU time, ms.");

That that I look again later where you add the Documentation for this:
You should check that this parameter is in range (not too small (<0) not
to too large maybe). rcu_torture_init() checks also other.

>  torture_param(int, test_nmis, 0, "End-test NMI tests, 0 to disable.");
>  torture_param(bool, test_no_idle_hz, true, "Test support for tickless idle CPUs");
>  torture_param(int, test_srcu_lockdep, 0, "Test specified SRCU deadlock scenario.");
> @@ -1366,61 +1375,111 @@ static void rcu_torture_enable_rt_throttle(void)
…
> +	smp_mb(); // Time check before reader check.
> +	if (!READ_ONCE(br.finished) && !stuttered && !torture_must_stop()) {
>  		VERBOSE_TOROUT_STRING("rcu_torture_boost boosting failed");
>  		n_rcu_torture_boost_failure++;
> +		pr_info("Boost inversion: reader got only %llu of %d ms CPU in %lu jiffies, GP %s.\n",
> +			READ_ONCE(br.ran_ns) / NSEC_PER_MSEC, test_boost_reader_ms, jiffies - start,

rcu_torture_boost_reader::ran_ns is u64 so this needs to be something
like div64_ul().

Sebastian

Re: [PATCH RFC 5/7] rcu: Add rcutree.boost_disable to disable priority boosting at runtime

[Sebastian Andrzej Siewior]

On 2026-11-11 16:38:23 [-0500], Joel Fernandes wrote:
> CONFIG_RCU_BOOST is now enabled automatically for preemptible RCU and
> cannot be turned off, so rcutorture's BUSTED-BOOST scenario, which is
> meant to show that rcutorture detects broken priority boosting, can no
> longer break it.  Add a boot parameter that makes rcu_initiate_boost()
> do nothing, giving that scenario a way to break boosting.

It has:
|         depends on (RT_MUTEXES && PREEMPT_RCU && RCU_EXPERT) || PREEMPT_RT
|         default y if PREEMPT_RT

which I read as always enable on PREEMPT_RT and on PREEMPT_RCU only with
RCU_EXPERT enabled which off by default.
But BUSTED-BOOST always enabled RCU_EXPERT which means it is always
enabled. Hmm.

> --- a/kernel/rcu/tree_plugin.h
> +++ b/kernel/rcu/tree_plugin.h
> @@ -1270,6 +1270,9 @@ static int rcu_boost_kthread(void *arg)
>  	return 0;
>  }
>  
> +static bool boost_disable;
> +module_param(boost_disable, bool, 0444);
> +
>  /*
>   * Check to see if it is time to start boosting RCU readers that are
>   * blocking the current grace period, and, if so, tell the per-rcu_node
> @@ -1284,7 +1287,7 @@ static void rcu_initiate_boost(struct rcu_node *rnp, unsigned long flags)
>  	__releases(rnp->lock)
>  {
>  	raw_lockdep_assert_held_rcu_node(rnp);
> -	if (!rnp->boost_kthread_task ||
> +	if (boost_disable || !rnp->boost_kthread_task ||

Could you instead make this a parameter of rcutorture (and not rcu's)
and then check it behind IS_ENABLED(CONFIG_RCU_TORTURE_TEST)?
That way non-testing code does not have this additional read and the
variable parameter and so on.

Maybe tell rcu_spawn_one_boost_kthread() not to populate the tread this
thread in case.

>  	    (!rcu_preempt_blocked_readers_cgp(rnp) && !rnp->exp_tasks)) {
>  		raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
>  		return;

Sebastian