[PATCH 00/13] srcu: Round out atomic SRCU support

[PATCH 00/13] srcu: Round out atomic SRCU support

[Kunwu Chan]

From: Kunwu Chan <kunwu.chan@gmail.com>

This series rounds out atomic SRCU support with a reader-free fastpath,
some atomic-context fixes, and additional test coverage.

The core change is a reader-free fastpath for synchronize_srcu_atomic().
Atomic SRCU's read-side critical sections disable preemption, so when
the summed lock and unlock counts match for both ranks,
synchronize_srcu_atomic() can end the grace period immediately, without
the index flip or the srcu_advance_state() scans.  Correctness depends
on the grace-period anchor written by srcu_gp_start() preceding the
lock-count scan; two LKMM litmus tests (patches 1-2) verify that
ordering.

The series is:

 1-2  LKMM litmus tests for the reader-free fastpath ordering.
   3  srcutree: add the reader-free fastpath.
   4  rcutorture: add --do-atomic-srcu to torture.sh.
 5-7  srcutree: honor and propagate is_atomic so atomic SRCU never
      transitions to big.
   8  srcutree: forbid srcu_expedite_current() on atomic SRCU.
   9  rcutorture: disable srcu_expedite_current() for atomic SRCU.
  10  srcutree: skip callback scheduling for atomic SRCU grace periods.
  11  srcutree: drop the srcu_barrier() sleep for atomic SRCU.
  12  srcutree: remove leftover debug pr_alert()s.
  13  srcu: restrict the non_block annotation to task context, fixing a
      KCSAN data race.

The KCSAN race in patch 13 is a data race on the interrupted task's
->non_block_count when a timer callback runs atomic-SRCU readers in
inline softirq context.

Tested with herd7 7.58 (litmus tests), torture.sh --do-atomic-srcu, and
torture.sh --do-atomic-srcu --do-kcsan (KCSAN+PROVE_LOCKING).

Kunwu Chan (13):
  litmus: Add SRCU fastpath anchor-before-scan test
  litmus: Add SRCU fastpath scan-before-anchor test
  srcutree: Add reader-free fastpath to synchronize_srcu_atomic()
  rcutorture: Add atomic-SRCU support to torture.sh
  srcutree: Honor is_atomic in check_init_srcu_struct()
  srcutree: Make init_srcu_struct_atomic() prevent transition to big
  srcutree: Don't transition atomic SRCU to big in srcu_gp_end()
  srcutree: Forbid srcu_expedite_current() on atomic SRCU
  rcutorture: Disable srcu_expedite_current() for atomic SRCU
  srcutree: Skip callback scheduling for atomic SRCU grace periods
  srcutree: Remove srcu_barrier() sleep for atomic SRCU
  srcutree: Remove debug pr_alert()s
  srcu: Restrict atomic-SRCU non_block annotation to task context

 include/linux/srcu.h                          | 10 +-
 kernel/rcu/rcutorture.c                       |  1 +
 kernel/rcu/srcutree.c                         | 95 ++++++++++++++-----
 tools/memory-model/litmus-tests/README        | 16 ++++
 .../SRCU-fastpath-anchor-before-scan.litmus   | 56 +++++++++++
 .../SRCU-fastpath-scan-before-anchor.litmus   | 53 +++++++++++
 .../selftests/rcutorture/bin/torture.sh       | 24 +++++
 7 files changed, 229 insertions(+), 26 deletions(-)
 create mode 100644 tools/memory-model/litmus-tests/SRCU-fastpath-anchor-before-scan.litmus
 create mode 100644 tools/memory-model/litmus-tests/SRCU-fastpath-scan-before-anchor.litmus


base-commit: ef15070f96c5f07de3a9b593c36f6e83bc9e3d58
-- 
2.43.0

[PATCH 01/13] litmus: Add SRCU fastpath anchor-before-scan test

[Kunwu Chan]

From: Kunwu Chan <kunwu.chan@gmail.com>

synchronize_srcu_atomic() may end its grace period immediately when
its scan of the per-CPU lock counters finds no readers.  Correctness
requires the grace-period anchor written by srcu_gp_start() to precede
the smp_mb() ordering the lock scan.  This ordering ensures that any
reader whose lock increment is missed by the scan cannot have
incremented its lock counter before the grace-period anchor, and
therefore cannot be a pre-existing reader of this grace period.

This litmus test models the key ordering between the grace-period
anchor and the lock counter scan, where "seq" models the
grace-period anchor in ->srcu_gp_seq and "ctr" models the per-CPU
->srcu_ctrs[].srcu_locks counter.  P0 writes the anchor before the
smp_mb() and the lock scan.  P1 models the reader-side counter
increment, with the smp_mb() of __srcu_read_lock() following the
increment.  P2 models an observer that sees the reader's increment
before seeing the anchor.

The outcome is forbidden by LKMM, and herd7 reports "Never".  See
SRCU-fastpath-scan-before-anchor.litmus for the reversed ordering,
which permits this outcome.

Tested with herd7 7.58 using linux-kernel.cfg.

Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
---
 .../SRCU-fastpath-anchor-before-scan.litmus   | 56 +++++++++++++++++++
 1 file changed, 56 insertions(+)
 create mode 100644 tools/memory-model/litmus-tests/SRCU-fastpath-anchor-before-scan.litmus

diff --git a/tools/memory-model/litmus-tests/SRCU-fastpath-anchor-before-scan.litmus b/tools/memory-model/litmus-tests/SRCU-fastpath-anchor-before-scan.litmus
new file mode 100644
index 000000000000..8200a75e15ef
--- /dev/null
+++ b/tools/memory-model/litmus-tests/SRCU-fastpath-anchor-before-scan.litmus
@@ -0,0 +1,56 @@
+C SRCU-fastpath-anchor-before-scan
+
+(*
+ * Result: Never
+ *
+ * The synchronize_srcu_atomic() fastpath may end its grace period
+ * immediately when its scan of the per-CPU lock counters finds no
+ * readers.  Correctness requires the grace-period anchor written by
+ * srcu_gp_start() to precede the smp_mb() ordering the lock scan.
+ * This ordering ensures that any reader whose lock increment is missed
+ * by the scan cannot have incremented its lock counter before the
+ * grace-period anchor, and therefore cannot be a pre-existing reader
+ * of this grace period.
+ *
+ * This litmus test models the key ordering between the grace-period
+ * anchor and the lock counter scan, where "seq" models the
+ * grace-period anchor in ->srcu_gp_seq and "ctr" models the per-CPU
+ * ->srcu_ctrs[].srcu_locks counter.  P0 writes the anchor before the
+ * smp_mb() and the lock scan.  P1 models the reader-side counter
+ * increment, with the smp_mb() of __srcu_read_lock() following the
+ * increment.  P2 models an observer that sees the reader's increment
+ * before seeing the anchor.
+ *
+ * The outcome is forbidden by LKMM, and herd7 reports "Never".  See
+ * SRCU-fastpath-scan-before-anchor.litmus for the reversed ordering,
+ * which permits this outcome.
+ *)
+
+{}
+
+P0(int *seq, int *ctr)
+{
+	int r2;
+
+	WRITE_ONCE(*seq, 1);
+	smp_mb();
+	r2 = READ_ONCE(*ctr);
+}
+
+P1(int *ctr)
+{
+	WRITE_ONCE(*ctr, 1);
+	smp_mb();
+}
+
+P2(int *seq, int *ctr)
+{
+	int r3;
+	int r4;
+
+	r3 = READ_ONCE(*ctr);
+	smp_mb();
+	r4 = READ_ONCE(*seq);
+}
+
+exists (0:r2 = 0 /\ 2:r3 = 1 /\ 2:r4 = 0)
-- 
2.43.0

[PATCH 02/13] litmus: Add SRCU fastpath scan-before-anchor test

[Kunwu Chan]

From: Kunwu Chan <kunwu.chan@gmail.com>

If the synchronize_srcu_atomic() fastpath instead places its lock scan
before the grace-period anchor, the scan can miss a reader whose
increment was already visible before the anchor.  That reader already
existed when the grace period started, so completing the grace period
without waiting for it would violate the SRCU grace-period guarantee.

This litmus test models the reversed ordering, with the lock scan placed
before the grace-period anchor.  "seq" models the grace-period anchor in
->srcu_gp_seq and "ctr" models the per-CPU ->srcu_ctrs[].srcu_locks
counter.  P0 scans the lock counter before writing the anchor, with an
smp_mb() between them.  P1 models the reader-side counter increment,
with the smp_mb() of __srcu_read_lock() following the increment.  P2
models an observer that sees the reader's increment before seeing the
anchor.

The same outcome is allowed with this ordering, and herd7 reports
"Sometimes".  The litmus-tests README is also updated to describe both
SRCU fastpath tests.

Tested with herd7 7.58 using linux-kernel.cfg.

Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
---
 tools/memory-model/litmus-tests/README        | 16 ++++++
 .../SRCU-fastpath-scan-before-anchor.litmus   | 53 +++++++++++++++++++
 2 files changed, 69 insertions(+)
 create mode 100644 tools/memory-model/litmus-tests/SRCU-fastpath-scan-before-anchor.litmus

diff --git a/tools/memory-model/litmus-tests/README b/tools/memory-model/litmus-tests/README
index d311a0ff1ae6..449747c6db9e 100644
--- a/tools/memory-model/litmus-tests/README
+++ b/tools/memory-model/litmus-tests/README
@@ -137,6 +137,22 @@ S+fencewmbonceonce+poacquireonce.litmus
 	Can a smp_wmb(), instead of a release, and an acquire order
 	a prior store against a subsequent store?
 
+SRCU-fastpath-anchor-before-scan.litmus
+	This models the synchronize_srcu_atomic() fastpath with the
+	grace-period anchor ordered before the lock-counter scan.  This
+	ordering prevents readers that existed before the grace period
+	from being missed by the scan.  See
+	SRCU-fastpath-scan-before-anchor.litmus for the reversed
+	ordering.
+
+SRCU-fastpath-scan-before-anchor.litmus
+	This models the synchronize_srcu_atomic() fastpath with the
+	lock-counter scan ordered before the grace-period anchor.  This
+	permits the scan to miss readers that existed before the grace
+	period, violating the SRCU grace-period guarantee.  See
+	SRCU-fastpath-anchor-before-scan.litmus for the opposite
+	ordering.
+
 WRC+poonceonces+Once.litmus
 WRC+pooncerelease+fencermbonceonce+Once.litmus
 	These two are members of an extension of the MP litmus-test
diff --git a/tools/memory-model/litmus-tests/SRCU-fastpath-scan-before-anchor.litmus b/tools/memory-model/litmus-tests/SRCU-fastpath-scan-before-anchor.litmus
new file mode 100644
index 000000000000..931a41014de7
--- /dev/null
+++ b/tools/memory-model/litmus-tests/SRCU-fastpath-scan-before-anchor.litmus
@@ -0,0 +1,53 @@
+C SRCU-fastpath-scan-before-anchor
+
+(*
+ * Result: Sometimes
+ *
+ * If the synchronize_srcu_atomic() fastpath instead places its lock
+ * scan before the grace-period anchor, the scan can miss a reader whose
+ * increment was already visible before the anchor.  That reader already
+ * existed when the grace period started, so completing the grace period
+ * without waiting for it would violate the SRCU grace-period guarantee.
+ *
+ * This litmus test models the reversed ordering, with the lock scan
+ * placed before the grace-period anchor.  "seq" models the grace-period
+ * anchor in ->srcu_gp_seq and "ctr" models the per-CPU
+ * ->srcu_ctrs[].srcu_locks counter.  P0 scans the lock counter before
+ * writing the anchor, with an smp_mb() between them.  P1 models the
+ * reader-side counter increment, with the smp_mb() of __srcu_read_lock()
+ * following the increment.  P2 models an observer that sees the reader's
+ * increment before seeing the anchor.
+ *
+ * The same outcome is allowed with this ordering, and herd7 reports
+ * "Sometimes".  See SRCU-fastpath-anchor-before-scan.litmus for the
+ * opposite ordering, which forbids this outcome.
+ *)
+
+{}
+
+P0(int *seq, int *ctr)
+{
+	int r2;
+
+	r2 = READ_ONCE(*ctr);
+	smp_mb();
+	WRITE_ONCE(*seq, 1);
+}
+
+P1(int *ctr)
+{
+	WRITE_ONCE(*ctr, 1);
+	smp_mb();
+}
+
+P2(int *seq, int *ctr)
+{
+	int r3;
+	int r4;
+
+	r3 = READ_ONCE(*ctr);
+	smp_mb();
+	r4 = READ_ONCE(*seq);
+}
+
+exists (0:r2 = 0 /\ 2:r3 = 1 /\ 2:r4 = 0)
-- 
2.43.0

[PATCH 03/13] srcutree: Add reader-free fastpath to synchronize_srcu_atomic()

[Kunwu Chan]

From: Kunwu Chan <kunwu.chan@gmail.com>

synchronize_srcu_atomic() is restricted to srcu_read_lock_atomic() and
srcu_read_unlock_atomic(), whose read-side critical sections disable
preemption.  In the common case where there are no readers at all, the
grace period therefore need not do the index flip.  Add a fastpath
that sums both ranks of the per-CPU ->srcu_ctrs[] counters and, if the
lock counts match the unlock counts on both ranks, ends the grace
period immediately, skipping the srcu_advance_state() scans, mirroring
the similar Tiny SRCU fastpath.

Correctness requires the counter-sum proof to follow the grace-period
anchor written by srcu_gp_start(); placing it before the anchor could
let this grace period miss a pre-existing reader and return without
waiting for it.  The smp_mb() between the unlock and lock sums pairs
with the smp_mb() in __srcu_read_lock().  The grace period is ended
manually under ->lock and ->srcu_atomic_gp_flag.

Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
---
 kernel/rcu/srcutree.c | 48 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 48 insertions(+)

diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c
index 7dd705eec573..533607de5728 100644
--- a/kernel/rcu/srcutree.c
+++ b/kernel/rcu/srcutree.c
@@ -2113,6 +2113,8 @@ void synchronize_srcu_atomic(struct srcu_struct *ssp)
 {
 	unsigned long srcu_state;
 	struct srcu_usage *sup = ssp->srcu_sup;
+	unsigned long rdm0, rdm1;
+	unsigned long unlocks0, unlocks1;
 
 	// Initialize.	Either init_srcu_struct() was invoked or
 	// DEFINE_SRCU() or similar was used.  Therefore, no allocation
@@ -2149,6 +2151,52 @@ void synchronize_srcu_atomic(struct srcu_struct *ssp)
 	srcu_gp_start(ssp);
 	raw_spin_unlock_irq_rcu_node(sup);
 
+	//
+	// Fastpath:  If there are no readers at all, neither grace-period
+	// scan need wait, so both can be satisfied at once without doing
+	// the index flip.  The counter-sum proof is the same as that of
+	// srcu_readers_active_idx_check(), but spanning both indices.
+	// Atomic SRCU guarantees that all readers are of
+	// SRCU_READ_FLAVOR_ATOMIC, so the SLOWGP check never triggers and
+	// the ->srcu_reader_flavor masks returned by
+	// srcu_readers_unlock_idx() are unused.
+	//
+	// This proof must follow the grace-period anchor written by the
+	// srcu_gp_start() above, never precede it.  With the anchor first,
+	// a reader whose lock increment is missed by the sums below cannot
+	// have incremented its lock counter before the anchor, and therefore
+	// cannot be a pre-existing reader of this grace period.  Placing the
+	// proof before the anchor would let this grace period miss a
+	// pre-existing reader and return without waiting for it.
+	//
+	// The smp_mb() pairs with the smp_mb() in __srcu_read_lock()
+	// (store-buffering pattern), which guarantees that a lock is always
+	// counted if the corresponding unlock is counted, the same
+	// memory-ordering guarantee as is provided by
+	// srcu_readers_active_idx_check().
+	//
+	unlocks0 = srcu_readers_unlock_idx(ssp, 0, &rdm0);
+	unlocks1 = srcu_readers_unlock_idx(ssp, 1, &rdm1);
+	smp_mb(); /* A */
+	if (srcu_readers_lock_idx(ssp, 0, false, unlocks0) &&
+	    srcu_readers_lock_idx(ssp, 1, false, unlocks1)) {
+		// No readers, so end this grace period manually, skipping
+		// the index flip.  Advancing the sequence number via
+		// rcu_seq_start() in srcu_gp_start() above and rcu_seq_end()
+		// below keeps get_state_synchronize_srcu() and
+		// poll_state_synchronize_srcu() working, all under ->lock
+		// and ->srcu_atomic_gp_flag, which excludes concurrent
+		// sequence-number updates.
+		raw_spin_lock_irq_rcu_node(sup);
+		rcu_seq_end(&sup->srcu_gp_seq);
+		raw_spin_unlock_irq_rcu_node(sup);
+		WARN_ON_ONCE(!poll_state_synchronize_srcu(ssp, srcu_state));
+		atomic_set_release(&sup->srcu_atomic_gp_flag, 0);
+		preempt_enable();
+		non_block_end();
+		return;
+	}
+
 	// Wait for it to complete, helping it along.
 	while (!poll_state_synchronize_srcu(ssp, srcu_state)) {
 		cpu_relax();
-- 
2.43.0

[PATCH 04/13] rcutorture: Add atomic-SRCU support to torture.sh

[Kunwu Chan]

From: Kunwu Chan <kunwu.chan@gmail.com>

Add the --do-atomic-srcu argument to torture.sh, which runs the
SRCU-N, SRCU-P, and SRCU-T scenarios, thus covering both Tree SRCU
(SRCU-N and SRCU-P) and Tiny SRCU (SRCU-T), with
rcutorture.reader_flavor=0x10 appended to the boot parameters so
that it takes precedence over each scenario's own reader-flavor
setting.  This exercises srcu_read_lock_atomic(),
srcu_read_unlock_atomic(), and synchronize_srcu_atomic().

As with other torture.sh tests, the --do-kcsan argument runs a
KCSAN+PROVE_LOCKING variant of this test.

Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
---
 .../selftests/rcutorture/bin/torture.sh       | 24 +++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/tools/testing/selftests/rcutorture/bin/torture.sh b/tools/testing/selftests/rcutorture/bin/torture.sh
index f0083891ee81..8885812d866a 100755
--- a/tools/testing/selftests/rcutorture/bin/torture.sh
+++ b/tools/testing/selftests/rcutorture/bin/torture.sh
@@ -68,6 +68,7 @@ do_clocksourcewd="${ifnotaarch64}"
 do_rt=yes
 do_rcutasksflavors="${ifnotaarch64}" # FIXME: Back to "yes" when SMP=n auto-avoided
 do_srcu_lockdep=yes
+do_atomic_srcu=no
 do_rcu_rust=no
 
 # doyesno - Helper function for yes/no arguments
@@ -103,6 +104,7 @@ usage () {
 	echo "       --do-rcu-rust / --do-no-rcu-rust / --no-rcu-rust"
 	echo "       --do-scftorture / --do-no-scftorture / --no-scftorture"
 	echo "       --do-srcu-lockdep / --do-no-srcu-lockdep / --no-srcu-lockdep"
+	echo "       --do-atomic-srcu / --do-no-atomic-srcu / --no-atomic-srcu"
 	echo "       --duration [ <minutes> | <hours>h | <days>d ]"
 	echo "       --guest-cpu-limit N"
 	echo "       --kcsan-kmake-arg kernel-make-arguments"
@@ -148,6 +150,7 @@ do
 		do_kcsan=yes
 		do_clocksourcewd="${ifnotaarch64}"
 		do_srcu_lockdep=yes
+		do_atomic_srcu=yes
 		;;
 	--do-allmodconfig|--do-no-allmodconfig|--no-allmodconfig)
 		do_allmodconfig=`doyesno "$1" --do-allmodconfig`
@@ -183,6 +186,7 @@ do
 		do_kcsan=no
 		do_clocksourcewd=no
 		do_srcu_lockdep=no
+		do_atomic_srcu=no
 		;;
 	--do-normal|--do-norm|--do-no-normal|--do-no-norm|--no-normal|--no-norm)
 		do_normal=`doyesno "$1" --do-normal`
@@ -212,6 +216,9 @@ do
 	--do-srcu-lockdep|--do-no-srcu-lockdep|--no-srcu-lockdep)
 		do_srcu_lockdep=`doyesno "$1" --do-srcu-lockdep`
 		;;
+	--do-atomic-srcu|--do-no-atomic-srcu|--no-atomic-srcu)
+		do_atomic_srcu=`doyesno "$1" --do-atomic-srcu`
+		;;
 	--duration)
 		checkarg --duration "(minutes)" $# "$2" '^[0-9][0-9]*\(m\|h\|d\|\)$' '^error'
 		mult=1
@@ -497,6 +504,23 @@ then
 	torture_set "rcutorture" tools/testing/selftests/rcutorture/bin/kvm.sh --allcpus --duration "$duration_rcutorture" --configs "$configs_rcutorture" --trust-make
 fi
 
+# Test atomic SRCU across Tree SRCU (SRCU-N and SRCU-P) and Tiny SRCU
+# (SRCU-T).  The reader flavor selects srcu_read_lock_atomic() and
+# synchronize_srcu_atomic().  Tiny SRCU requires SMP=n, which aarch64
+# does not support.
+if test "$do_atomic_srcu" = "yes"
+then
+	torture_bootargs="rcutorture.reader_flavor=0x10"
+	configs_atomic_srcu="SRCU-N SRCU-P"
+	if test "$ifnotaarch64" = yes
+	then
+		configs_atomic_srcu="$configs_atomic_srcu SRCU-T"
+	fi
+	torture_set "atomic-srcu" tools/testing/selftests/rcutorture/bin/kvm.sh \
+		--allcpus --duration "$duration_rcutorture" \
+		--configs "$configs_atomic_srcu" --trust-make
+fi
+
 if test "$do_locktorture" = "yes"
 then
 	torture_bootargs="torture.disable_onoff_at_boot"
-- 
2.43.0

[PATCH 05/13] srcutree: Honor is_atomic in check_init_srcu_struct()

[Kunwu Chan]

From: Kunwu Chan <kunwu.chan@gmail.com>

check_init_srcu_struct() drops its is_atomic argument, passing
hard-coded false to init_srcu_struct_fields().  Pass is_atomic through
instead.

Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
---
 kernel/rcu/srcutree.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c
index 533607de5728..2af36db37fa9 100644
--- a/kernel/rcu/srcutree.c
+++ b/kernel/rcu/srcutree.c
@@ -522,7 +522,7 @@ static void check_init_srcu_struct(struct srcu_struct *ssp, bool is_atomic)
 		raw_spin_unlock_irqrestore_rcu_node(ssp->srcu_sup, flags);
 		return;
 	}
-	init_srcu_struct_fields(ssp, true, false);
+	init_srcu_struct_fields(ssp, true, is_atomic);
 	raw_spin_unlock_irqrestore_rcu_node(ssp->srcu_sup, flags);
 }
 
-- 
2.43.0

[PATCH 06/13] srcutree: Make init_srcu_struct_atomic() prevent transition to big

[Kunwu Chan]

From: Kunwu Chan <kunwu.chan@gmail.com>

The is_atomic parameter of init_srcu_struct_fields() exists so that
atomic SRCU never transitions to big, but neither
init_srcu_struct_atomic() nor its lockdep counterpart
__init_srcu_struct_atomic() sets it.

On systems where srcutree.convert_to_big selects SRCU_SIZING_INIT,
this needlessly allocates a full srcu_node combining tree for any
dynamically initialized atomic srcu_struct, despite atomic SRCU
having neither callbacks nor srcu_barrier() operations.

Pass true from both atomic entry points, adding an is_atomic parameter
to __init_srcu_struct_common().

Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
---
 kernel/rcu/srcutree.c | 15 ++++++++-------
 1 file changed, 8 insertions(+), 7 deletions(-)

diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c
index 2af36db37fa9..01f224a56b41 100644
--- a/kernel/rcu/srcutree.c
+++ b/kernel/rcu/srcutree.c
@@ -305,26 +305,27 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp, bool is_static, bool
 #ifdef CONFIG_DEBUG_LOCK_ALLOC
 
 static int
-__init_srcu_struct_common(struct srcu_struct *ssp, const char *name, struct lock_class_key *key)
+__init_srcu_struct_common(struct srcu_struct *ssp, const char *name,
+			  struct lock_class_key *key, bool is_atomic)
 {
 	/* Don't re-initialize a lock while it is held. */
 	debug_check_no_locks_freed((void *)ssp, sizeof(*ssp));
 	lockdep_init_map(&ssp->dep_map, name, key, 0);
-	return init_srcu_struct_fields(ssp, false, false);
+	return init_srcu_struct_fields(ssp, false, is_atomic);
 }
 
 int init_srcu_struct_lockdep(struct srcu_struct *ssp, const char *name,
 			     struct lock_class_key *key)
 {
 	ssp->srcu_reader_flavor = 0;
-	return __init_srcu_struct_common(ssp, name, key);
+	return __init_srcu_struct_common(ssp, name, key, false);
 }
 EXPORT_SYMBOL_GPL(init_srcu_struct_lockdep);
 
 int __init_srcu_struct_fast(struct srcu_struct *ssp, const char *name, struct lock_class_key *key)
 {
 	ssp->srcu_reader_flavor = SRCU_READ_FLAVOR_FAST;
-	return __init_srcu_struct_common(ssp, name, key);
+	return __init_srcu_struct_common(ssp, name, key, false);
 }
 EXPORT_SYMBOL_GPL(__init_srcu_struct_fast);
 
@@ -332,14 +333,14 @@ int __init_srcu_struct_fast_updown(struct srcu_struct *ssp, const char *name,
 				   struct lock_class_key *key)
 {
 	ssp->srcu_reader_flavor = SRCU_READ_FLAVOR_FAST_UPDOWN;
-	return __init_srcu_struct_common(ssp, name, key);
+	return __init_srcu_struct_common(ssp, name, key, false);
 }
 EXPORT_SYMBOL_GPL(__init_srcu_struct_fast_updown);
 
 int __init_srcu_struct_atomic(struct srcu_struct *ssp, const char *name, struct lock_class_key *key)
 {
 	ssp->srcu_reader_flavor = SRCU_READ_FLAVOR_ATOMIC;
-	return __init_srcu_struct_common(ssp, name, key);
+	return __init_srcu_struct_common(ssp, name, key, true);
 }
 EXPORT_SYMBOL_GPL(__init_srcu_struct_atomic);
 
@@ -414,7 +415,7 @@ EXPORT_SYMBOL_GPL(init_srcu_struct_fast_updown);
 int init_srcu_struct_atomic(struct srcu_struct *ssp)
 {
 	ssp->srcu_reader_flavor = SRCU_READ_FLAVOR_ATOMIC;
-	return init_srcu_struct_fields(ssp, false, false);
+	return init_srcu_struct_fields(ssp, false, true);
 }
 EXPORT_SYMBOL_GPL(init_srcu_struct_atomic);
 
-- 
2.43.0

[PATCH 07/13] srcutree: Don't transition atomic SRCU to big in srcu_gp_end()

[Kunwu Chan]

From: Kunwu Chan <kunwu.chan@gmail.com>

The transition-to-big code path in srcu_gp_end() calls
init_srcu_struct_nodes() with GFP_KERNEL, which is illegal in the
atomic context reachable from synchronize_srcu_atomic().  Atomic SRCU
has no use for the srcu_node combining tree: it has neither callbacks
nor srcu_barrier() operations, and its grace periods are serialized
by ->srcu_atomic_gp_flag instead.

Skip this transition entirely for atomic SRCU, which is also defense
in depth against any path that might wrongly set ->srcu_size_state for
an atomic srcu_struct.

Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
---
 kernel/rcu/srcutree.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c
index 01f224a56b41..74acd5645d4c 100644
--- a/kernel/rcu/srcutree.c
+++ b/kernel/rcu/srcutree.c
@@ -1074,8 +1074,9 @@ static void srcu_gp_end(struct srcu_struct *ssp, bool is_atomic)
 		raw_spin_unlock_irq_rcu_node(sup);
 	}
 
-	/* Transition to big if needed. */
-	if (ss_state != SRCU_SIZE_SMALL && ss_state != SRCU_SIZE_BIG) {
+	/* Transition to big if needed, but never for atomic SRCU. */
+	if (!is_atomic && ss_state != SRCU_SIZE_SMALL &&
+	    ss_state != SRCU_SIZE_BIG) {
 		if (ss_state == SRCU_SIZE_ALLOC)
 			init_srcu_struct_nodes(ssp, GFP_KERNEL);
 		else
-- 
2.43.0

[PATCH 08/13] srcutree: Forbid srcu_expedite_current() on atomic SRCU

[Kunwu Chan]

From: Kunwu Chan <kunwu.chan@gmail.com>

srcu_expedite_current() has no early guard for atomic SRCU, so calling
it on an atomic srcu_struct leaks the expediting callback in
__call_srcu(), leaving ->srcu_ec_state stuck at SRCU_EC_PENDING.

Add the same WARN_ON_ONCE()-guarded early return used by the other
forbidden APIs, and list srcu_expedite_current() and srcu_barrier() in
synchronize_srcu_atomic()'s documentation.

Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
---
 kernel/rcu/srcutree.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c
index 74acd5645d4c..2d2a12f86115 100644
--- a/kernel/rcu/srcutree.c
+++ b/kernel/rcu/srcutree.c
@@ -1972,6 +1972,9 @@ static void srcu_expedite_current_cb(struct rcu_head *rhp)
  * no current grace period, one might be created.  If the current grace
  * period is currently sleeping, that sleep will complete before expediting
  * will take effect.
+ *
+ * This function must not be invoked on srcu_struct structures that are
+ * used with srcu_read_lock_atomic() and synchronize_srcu_atomic().
  */
 void srcu_expedite_current(struct srcu_struct *ssp)
 {
@@ -1979,6 +1982,9 @@ void srcu_expedite_current(struct srcu_struct *ssp)
 	bool needcb = false;
 	struct srcu_data *sdp;
 
+	// Atomic SRCU has no callbacks, so there is nothing to expedite.
+	if (WARN_ON_ONCE(ssp->srcu_reader_flavor == SRCU_READ_FLAVOR_ATOMIC))
+		return;
 	migrate_disable();
 	sdp = this_cpu_ptr(ssp->sda);
 	raw_spin_lock_irqsave_sdp_contention(sdp, &flags);
@@ -2104,8 +2110,9 @@ static void srcu_advance_state(struct srcu_struct *ssp, bool is_atomic)
  *
  * If synchronize_srcu_atomic() is invoked on a given srcu_struct
  * structure, then none of call_srcu(), synchronize_srcu(),
- * synchronize_srcu_expedited(), or start_poll_synchronize_srcu() may be
- * invoked on that same structure.
+ * synchronize_srcu_expedited(), start_poll_synchronize_srcu(),
+ * srcu_barrier(), or srcu_expedite_current() may be invoked on that
+ * same structure.
  *
  * Because synchronize_srcu_atomic() is even more expedited than is
  * synchronize_srcu_expedited(), there is no expedited counterpart to
-- 
2.43.0

[PATCH 09/13] rcutorture: Disable srcu_expedite_current() for atomic SRCU

[Kunwu Chan]

From: Kunwu Chan <kunwu.chan@gmail.com>

srcu_expedite_current() is now forbidden for atomic SRCU, so null out
->exp_current in srcu_torture_init_forbidden_apis().

Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
---
 kernel/rcu/rcutorture.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c
index a9672fe88cbc..d795e07a12ad 100644
--- a/kernel/rcu/rcutorture.c
+++ b/kernel/rcu/rcutorture.c
@@ -721,6 +721,7 @@ static void srcu_torture_init_forbidden_apis(void)
 	cur_ops->call = NULL;
 	cur_ops->cb_barrier = NULL;
 	cur_ops->deferred_free = NULL;
+	cur_ops->exp_current = NULL;
 	cur_ops->start_gp_poll = NULL;
 }
 
-- 
2.43.0

[PATCH 10/13] srcutree: Skip callback scheduling for atomic SRCU grace periods

[Kunwu Chan]

From: Kunwu Chan <kunwu.chan@gmail.com>

call_srcu() is forbidden on atomic srcu_struct, so srcu_gp_end() never
has callbacks to invoke for them.  Yet it schedules callback invocation,
which for atomic SRCU's SRCU_SIZE_SMALL state arms the boot CPU's
->delay_work timer every grace period, only for srcu_invoke_callbacks()
to find nothing to do.

Skip this for atomic SRCU.

Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
---
 kernel/rcu/srcutree.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c
index 2d2a12f86115..c19f59725706 100644
--- a/kernel/rcu/srcutree.c
+++ b/kernel/rcu/srcutree.c
@@ -1018,10 +1018,10 @@ static void srcu_gp_end(struct srcu_struct *ssp, bool is_atomic)
 
 	/* Initiate callback invocation as needed. */
 	ss_state = smp_load_acquire(&sup->srcu_size_state);
-	if (ss_state < SRCU_SIZE_WAIT_BARRIER) {
+	if (!is_atomic && ss_state < SRCU_SIZE_WAIT_BARRIER) {
 		srcu_schedule_cbs_sdp(per_cpu_ptr(ssp->sda, get_boot_cpu_id()),
 					cbdelay);
-	} else {
+	} else if (!is_atomic) {
 		idx = rcu_seq_ctr(gpseq) % ARRAY_SIZE(snp->srcu_have_cbs);
 		srcu_for_each_node_breadth_first(ssp, snp) {
 			raw_spin_lock_irq_rcu_node(snp);
-- 
2.43.0

[PATCH 11/13] srcutree: Remove srcu_barrier() sleep for atomic SRCU

[Kunwu Chan]

From: Kunwu Chan <kunwu.chan@gmail.com>

The atomic-SRCU path in srcu_barrier() sleeps for 100 milliseconds just
in case there are callbacks to wait for.  But call_srcu() refuses
atomic SRCU with a WARN_ON_ONCE() before reaching the deferred-enqueue
path, so there can be no callbacks, deferred or otherwise.

Drop the sleep.

Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
---
 kernel/rcu/srcutree.c | 7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)

diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c
index c19f59725706..a93bf803e2f7 100644
--- a/kernel/rcu/srcutree.c
+++ b/kernel/rcu/srcutree.c
@@ -1895,12 +1895,9 @@ void srcu_barrier(struct srcu_struct *ssp)
 	unsigned long s;
 
 	check_init_srcu_struct(ssp, false);
-	if (WARN_ON_ONCE(ssp->srcu_reader_flavor == SRCU_READ_FLAVOR_ATOMIC)) {
-		// There shouldn't be any callbacks for atomic SRCU,
-		// but just in case.
-		schedule_timeout_uninterruptible(HZ/10);
+	// Atomic SRCU has no callbacks, so there is nothing to wait on.
+	if (WARN_ON_ONCE(ssp->srcu_reader_flavor == SRCU_READ_FLAVOR_ATOMIC))
 		return;
-	}
 
 	/*
 	 * Register any deferred callbacks before snapshotting the sequence.  The
-- 
2.43.0

[PATCH 12/13] srcutree: Remove debug pr_alert()s

[Kunwu Chan]

From: Kunwu Chan <kunwu.chan@gmail.com>

Remove the debug pr_alert()s from srcu_gp_start() and
synchronize_srcu_atomic().

Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
---
 kernel/rcu/srcutree.c | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c
index a93bf803e2f7..a016c93d65cb 100644
--- a/kernel/rcu/srcutree.c
+++ b/kernel/rcu/srcutree.c
@@ -916,7 +916,6 @@ static void srcu_gp_start(struct srcu_struct *ssp)
 {
 	int state;
 
-	/*&&&&*/pr_alert("%s() start: ->srcu_gp_seq: %lx ->srcu_gp_seq_needed: %lx\n", __func__, ssp->srcu_sup->srcu_gp_seq, ssp->srcu_sup->srcu_gp_seq_needed);
 	lockdep_assert_held(&ACCESS_PRIVATE(ssp->srcu_sup, lock));
 	WARN_ON_ONCE(ULONG_CMP_GE(ssp->srcu_sup->srcu_gp_seq, ssp->srcu_sup->srcu_gp_seq_needed));
 	WRITE_ONCE(ssp->srcu_sup->srcu_gp_start, jiffies);
@@ -2152,7 +2151,6 @@ void synchronize_srcu_atomic(struct srcu_struct *ssp)
 	// OK, we really have to do it ourselves.  Start the grace period.
 	non_block_start();  // We must not voluntarily block!
 	smp_store_release(&sup->srcu_gp_seq_needed, srcu_state); // See srcu_funnel_gp_start().
-	/*&&&&*/pr_alert("%s() start: ->srcu_gp_seq: %lx ->srcu_gp_seq_needed: %lx\n", __func__, ssp->srcu_sup->srcu_gp_seq, ssp->srcu_sup->srcu_gp_seq_needed);
 	ASSERT_EXCLUSIVE_WRITER(ssp->srcu_sup->srcu_gp_seq);
 	srcu_gp_start(ssp);
 	raw_spin_unlock_irq_rcu_node(sup);
@@ -2208,7 +2206,6 @@ void synchronize_srcu_atomic(struct srcu_struct *ssp)
 		cpu_relax();
 		srcu_advance_state(ssp, true);
 	}
-	/*&&&&*/pr_alert("%s() end: ->srcu_gp_seq: %lx ->srcu_gp_seq_needed: %lx\n", __func__, ssp->srcu_sup->srcu_gp_seq, ssp->srcu_sup->srcu_gp_seq_needed);
 	ASSERT_EXCLUSIVE_WRITER(sup->srcu_atomic_gp_flag);
 	atomic_set_release(&sup->srcu_atomic_gp_flag, 0);
 	preempt_enable();
-- 
2.43.0

[PATCH 13/13] srcu: Restrict atomic-SRCU non_block annotation to task context

[Kunwu Chan]

From: Kunwu Chan <kunwu.chan@gmail.com>

srcu_read_lock_atomic() and srcu_read_unlock_atomic() arm and disarm
might_sleep() checks via non_block_start()/non_block_end(), skipping
hardirq so the update does not land on the interrupted task's
->non_block_count.

Inline softirqs run on the interrupted task's stack as well, so a
timer callback running atomic-SRCU readers races with the interrupted
task's own ->non_block_count updates, as KCSAN reports:

	BUG: KCSAN: data-race in srcu_torture_read_lock / srcu_torture_read_unlock

	write to 0xffffa00f818ea418 of 4 bytes by interrupt on cpu 0:
		srcu_torture_read_lock+0x422/0x470
		rcutorture_one_extend+0xdc/0x600
		rcu_torture_one_read+0xd1/0x330
		rcu_torture_timer+0x75/0x140
		call_timer_fn+0xe6/0x2f0
		...
		run_timer_softirq+0xb7/0x130
		handle_softirqs+0xfc/0x3f0
		__irq_exit_rcu+0x8e/0x100

Use in_task() so the annotation is applied only in task context; it
is redundant elsewhere because might_sleep() already warns about
sleeping from atomic context.

Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
---
 include/linux/srcu.h | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/include/linux/srcu.h b/include/linux/srcu.h
index 0de21155abc1..403fbe57ab15 100644
--- a/include/linux/srcu.h
+++ b/include/linux/srcu.h
@@ -346,11 +346,13 @@ static inline int srcu_read_lock_atomic(struct srcu_struct *ssp)
 	/*
 	 * Arm might_sleep() to catch even a *potentially* sleeping call
 	 * in the section, not just an actual schedule: the atomic-domain
-	 * promise must hold on every path, contended or not. In hardirq
-	 * the annotation would land on the interrupted task; it is also
+	 * promise must hold on every path, contended or not. In hardirq,
+	 * softirq, or NMI the annotation would land on the interrupted
+	 * task, and can also result in data races against that task's
+	 * own non_block_start()/non_block_end() invocations; it is also
 	 * redundant there, so skip it.
 	 */
-	if (!in_hardirq())
+	if (in_task())
 		non_block_start();
 	srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_ATOMIC);
 	retval = __srcu_read_lock(ssp);
@@ -562,7 +564,7 @@ static inline void srcu_read_unlock_atomic(struct srcu_struct *ssp, int idx)
 	srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_ATOMIC);
 	srcu_lock_release(&ssp->dep_map);
 	__srcu_read_unlock(ssp, idx);
-	if (!in_hardirq())
+	if (in_task())
 		non_block_end();
 	preempt_enable();
 }
-- 
2.43.0

Re: [PATCH 05/13] srcutree: Honor is_atomic in check_init_srcu_struct()

[Paul E. McKenney]

On Mon, Sep 07, 2026 at 03:58:21PM +0800, Kunwu Chan wrote:
> From: Kunwu Chan <kunwu.chan@gmail.com>
> 
> check_init_srcu_struct() drops its is_atomic argument, passing
> hard-coded false to init_srcu_struct_fields().  Pass is_atomic through
> instead.
> 
> Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>

Good catch, thank you!

I am folding this one into the following commit with attribution:

4e01d5320a2f ("srcutree: Suppress to-big transition for atomic SRCU")

This avoids potential bisection issues.

							Thanx, Paul

> ---
>  kernel/rcu/srcutree.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c
> index 533607de5728..2af36db37fa9 100644
> --- a/kernel/rcu/srcutree.c
> +++ b/kernel/rcu/srcutree.c
> @@ -522,7 +522,7 @@ static void check_init_srcu_struct(struct srcu_struct *ssp, bool is_atomic)
>  		raw_spin_unlock_irqrestore_rcu_node(ssp->srcu_sup, flags);
>  		return;
>  	}
> -	init_srcu_struct_fields(ssp, true, false);
> +	init_srcu_struct_fields(ssp, true, is_atomic);
>  	raw_spin_unlock_irqrestore_rcu_node(ssp->srcu_sup, flags);
>  }
>  
> -- 
> 2.43.0
> 

Re: [PATCH 03/13] srcutree: Add reader-free fastpath to synchronize_srcu_atomic()

[Paul E. McKenney]

On Mon, Sep 07, 2026 at 03:58:19PM +0800, Kunwu Chan wrote:
> From: Kunwu Chan <kunwu.chan@gmail.com>
> 
> synchronize_srcu_atomic() is restricted to srcu_read_lock_atomic() and
> srcu_read_unlock_atomic(), whose read-side critical sections disable
> preemption.  In the common case where there are no readers at all, the
> grace period therefore need not do the index flip.  Add a fastpath
> that sums both ranks of the per-CPU ->srcu_ctrs[] counters and, if the
> lock counts match the unlock counts on both ranks, ends the grace
> period immediately, skipping the srcu_advance_state() scans, mirroring
> the similar Tiny SRCU fastpath.
> 
> Correctness requires the counter-sum proof to follow the grace-period
> anchor written by srcu_gp_start(); placing it before the anchor could
> let this grace period miss a pre-existing reader and return without
> waiting for it.  The smp_mb() between the unlock and lock sums pairs
> with the smp_mb() in __srcu_read_lock().  The grace period is ended
> manually under ->lock and ->srcu_atomic_gp_flag.
> 
> Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>

Adding David Woodhouse on CC.

David, does this provide appropriate performance in your use case?

							Thanx, Paul

> ---
>  kernel/rcu/srcutree.c | 48 +++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 48 insertions(+)
> 
> diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c
> index 7dd705eec573..533607de5728 100644
> --- a/kernel/rcu/srcutree.c
> +++ b/kernel/rcu/srcutree.c
> @@ -2113,6 +2113,8 @@ void synchronize_srcu_atomic(struct srcu_struct *ssp)
>  {
>  	unsigned long srcu_state;
>  	struct srcu_usage *sup = ssp->srcu_sup;
> +	unsigned long rdm0, rdm1;
> +	unsigned long unlocks0, unlocks1;
>  
>  	// Initialize.	Either init_srcu_struct() was invoked or
>  	// DEFINE_SRCU() or similar was used.  Therefore, no allocation
> @@ -2149,6 +2151,52 @@ void synchronize_srcu_atomic(struct srcu_struct *ssp)
>  	srcu_gp_start(ssp);
>  	raw_spin_unlock_irq_rcu_node(sup);
>  
> +	//
> +	// Fastpath:  If there are no readers at all, neither grace-period
> +	// scan need wait, so both can be satisfied at once without doing
> +	// the index flip.  The counter-sum proof is the same as that of
> +	// srcu_readers_active_idx_check(), but spanning both indices.
> +	// Atomic SRCU guarantees that all readers are of
> +	// SRCU_READ_FLAVOR_ATOMIC, so the SLOWGP check never triggers and
> +	// the ->srcu_reader_flavor masks returned by
> +	// srcu_readers_unlock_idx() are unused.
> +	//
> +	// This proof must follow the grace-period anchor written by the
> +	// srcu_gp_start() above, never precede it.  With the anchor first,
> +	// a reader whose lock increment is missed by the sums below cannot
> +	// have incremented its lock counter before the anchor, and therefore
> +	// cannot be a pre-existing reader of this grace period.  Placing the
> +	// proof before the anchor would let this grace period miss a
> +	// pre-existing reader and return without waiting for it.
> +	//
> +	// The smp_mb() pairs with the smp_mb() in __srcu_read_lock()
> +	// (store-buffering pattern), which guarantees that a lock is always
> +	// counted if the corresponding unlock is counted, the same
> +	// memory-ordering guarantee as is provided by
> +	// srcu_readers_active_idx_check().
> +	//
> +	unlocks0 = srcu_readers_unlock_idx(ssp, 0, &rdm0);
> +	unlocks1 = srcu_readers_unlock_idx(ssp, 1, &rdm1);
> +	smp_mb(); /* A */
> +	if (srcu_readers_lock_idx(ssp, 0, false, unlocks0) &&
> +	    srcu_readers_lock_idx(ssp, 1, false, unlocks1)) {
> +		// No readers, so end this grace period manually, skipping
> +		// the index flip.  Advancing the sequence number via
> +		// rcu_seq_start() in srcu_gp_start() above and rcu_seq_end()
> +		// below keeps get_state_synchronize_srcu() and
> +		// poll_state_synchronize_srcu() working, all under ->lock
> +		// and ->srcu_atomic_gp_flag, which excludes concurrent
> +		// sequence-number updates.
> +		raw_spin_lock_irq_rcu_node(sup);
> +		rcu_seq_end(&sup->srcu_gp_seq);
> +		raw_spin_unlock_irq_rcu_node(sup);
> +		WARN_ON_ONCE(!poll_state_synchronize_srcu(ssp, srcu_state));
> +		atomic_set_release(&sup->srcu_atomic_gp_flag, 0);
> +		preempt_enable();
> +		non_block_end();
> +		return;
> +	}
> +
>  	// Wait for it to complete, helping it along.
>  	while (!poll_state_synchronize_srcu(ssp, srcu_state)) {
>  		cpu_relax();
> -- 
> 2.43.0
> 

Re: [PATCH 03/13] srcutree: Add reader-free fastpath to synchronize_srcu_atomic()

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 2128 bytes --]

On Tue, 2026-09-08 at 13:29 -0700, Paul E. McKenney wrote:
> On Mon, Sep 07, 2026 at 03:58:19PM +0800, Kunwu Chan wrote:
> > From: Kunwu Chan <kunwu.chan@gmail.com>
> > 
> > synchronize_srcu_atomic() is restricted to srcu_read_lock_atomic() and
> > srcu_read_unlock_atomic(), whose read-side critical sections disable
> > preemption.  In the common case where there are no readers at all, the
> > grace period therefore need not do the index flip.  Add a fastpath
> > that sums both ranks of the per-CPU ->srcu_ctrs[] counters and, if the
> > lock counts match the unlock counts on both ranks, ends the grace
> > period immediately, skipping the srcu_advance_state() scans, mirroring
> > the similar Tiny SRCU fastpath.
> > 
> > Correctness requires the counter-sum proof to follow the grace-period
> > anchor written by srcu_gp_start(); placing it before the anchor could
> > let this grace period miss a pre-existing reader and return without
> > waiting for it.  The smp_mb() between the unlock and lock sums pairs
> > with the smp_mb() in __srcu_read_lock().  The grace period is ended
> > manually under ->lock and ->srcu_atomic_gp_flag.
> >
> > Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
> 
> Adding David Woodhouse on CC.
> 
> David, does this provide appropriate performance in your use case?

Looks like it should; I've thrown it into the test. Thanks.

My version¹ didn't drive the GP at all, and just took the fast path if
there were no readers. And because it didn't drive the GP, I *think*
the "Correctness requires…" part of the commit message cited above
didn't apply?

Driving the GP does mean a certain amount of serialization that my
earlier bailout didn't incur, but I think that's probably only going to
show up on a microbenchmark. Let's see.

I *do* want the fast path used from synchronize_srcu_expedited()
though, if possible. That's what solves Sean's *other* problem. Can we
have that?

¹ https://git.infradead.org/?p=users/dwmw2/linux.git;a=commitdiff;h=0f1456f508
  https://git.infradead.org/?p=users/dwmw2/linux.git;a=commitdiff;h=62ff5ac4ca

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 6179 bytes --]

Re: [PATCH 03/13] srcutree: Add reader-free fastpath to synchronize_srcu_atomic()

[Paul E. McKenney]

On Tue, Sep 08, 2026 at 10:13:39PM +0100, David Woodhouse wrote:
> On Tue, 2026-09-08 at 13:29 -0700, Paul E. McKenney wrote:
> > On Mon, Sep 07, 2026 at 03:58:19PM +0800, Kunwu Chan wrote:
> > > From: Kunwu Chan <kunwu.chan@gmail.com>
> > > 
> > > synchronize_srcu_atomic() is restricted to srcu_read_lock_atomic() and
> > > srcu_read_unlock_atomic(), whose read-side critical sections disable
> > > preemption.  In the common case where there are no readers at all, the
> > > grace period therefore need not do the index flip.  Add a fastpath
> > > that sums both ranks of the per-CPU ->srcu_ctrs[] counters and, if the
> > > lock counts match the unlock counts on both ranks, ends the grace
> > > period immediately, skipping the srcu_advance_state() scans, mirroring
> > > the similar Tiny SRCU fastpath.
> > > 
> > > Correctness requires the counter-sum proof to follow the grace-period
> > > anchor written by srcu_gp_start(); placing it before the anchor could
> > > let this grace period miss a pre-existing reader and return without
> > > waiting for it.  The smp_mb() between the unlock and lock sums pairs
> > > with the smp_mb() in __srcu_read_lock().  The grace period is ended
> > > manually under ->lock and ->srcu_atomic_gp_flag.
> > >
> > > Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
> > 
> > Adding David Woodhouse on CC.
> > 
> > David, does this provide appropriate performance in your use case?
> 
> Looks like it should; I've thrown it into the test. Thanks.
> 
> My version¹ didn't drive the GP at all, and just took the fast path if
> there were no readers. And because it didn't drive the GP, I *think*
> the "Correctness requires…" part of the commit message cited above
> didn't apply?

And another option is to pull the fastpath up earlier, before
checking and/or acquiring ->srcu_atomic_gp_flag.  But this is a
bit more complicated from a concurrency viewpoint, at least if
we are to interact normally with get_state_synchronize_srcu() and
poll_state_synchronize_srcu().  For example:

	srcu_state = get_state_synchronize_srcu(ssp);
	synchronize_srcu_atomic(ssp);
	WARN_ON_ONCE(!poll_state_synchronize_srcu(ssp, srcu_state));

Without at least some mucking with the grace-period mechanism, that
WARN_ON_ONCE() could trigger, which just might not be universally
considered to be a friendly act.  ;-)

So it would be very good to keep the fastpath where Kunwu put it, if
that works reasonably.

> Driving the GP does mean a certain amount of serialization that my
> earlier bailout didn't incur, but I think that's probably only going to
> show up on a microbenchmark. Let's see.

Here is hoping!

> I *do* want the fast path used from synchronize_srcu_expedited()
> though, if possible. That's what solves Sean's *other* problem. Can we
> have that?

I believe so, but let's get this put to bed first.  Please note
that synchronize_srcu_expedited() has the same relationship with
get_state_synchronize_srcu() and poll_state_synchronize_srcu(), so there
will be some trickiness there as well.

							Thanx, Paul

> ¹ https://git.infradead.org/?p=users/dwmw2/linux.git;a=commitdiff;h=0f1456f508
>   https://git.infradead.org/?p=users/dwmw2/linux.git;a=commitdiff;h=62ff5ac4ca

Re: [PATCH 03/13] srcutree: Add reader-free fastpath to synchronize_srcu_atomic()

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 1250 bytes --]

On Tue, 2026-09-08 at 14:54 -0700, Paul E. McKenney wrote:
> 
> And another option is to pull the fastpath up earlier, before
> checking and/or acquiring ->srcu_atomic_gp_flag.  But this is a
> bit more complicated from a concurrency viewpoint, at least if
> we are to interact normally with get_state_synchronize_srcu() and
> poll_state_synchronize_srcu().  For example:
> 
> 	srcu_state = get_state_synchronize_srcu(ssp);
> 	synchronize_srcu_atomic(ssp);
> 	WARN_ON_ONCE(!poll_state_synchronize_srcu(ssp, srcu_state));
> 
> Without at least some mucking with the grace-period mechanism, that
> WARN_ON_ONCE() could trigger, which just might not be universally
> considered to be a friendly act.  ;-)
> 
> So it would be very good to keep the fastpath where Kunwu put it, if
> that works reasonably.

Could we not just declare that synchronize_srcu_atomic() *isn't*
guaranteed to drive a GP, so the above code isn't valid? Why poll for a
thing that's atomic? Is that a likely use case?

And you've already forbidden start_poll_synchronize_srcu() for atomic,
haven't you?

I concede that the same logic doesn't work quite as well for
synchronize_srcu_expedited(), as it's an established API. Could we
retcon that one?

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 6179 bytes --]

Re: [PATCH 03/13] srcutree: Add reader-free fastpath to synchronize_srcu_atomic()

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 5884 bytes --]

On Tue, 2026-09-08 at 13:29 -0700, Paul E. McKenney wrote:
> On Mon, Sep 07, 2026 at 03:58:19PM +0800, Kunwu Chan wrote:
> > From: Kunwu Chan <kunwu.chan@gmail.com>
> > 
> > synchronize_srcu_atomic() is restricted to srcu_read_lock_atomic() and
> > srcu_read_unlock_atomic(), whose read-side critical sections disable
> > preemption.  In the common case where there are no readers at all, the
> > grace period therefore need not do the index flip.  Add a fastpath
> > that sums both ranks of the per-CPU ->srcu_ctrs[] counters and, if the
> > lock counts match the unlock counts on both ranks, ends the grace
> > period immediately, skipping the srcu_advance_state() scans, mirroring
> > the similar Tiny SRCU fastpath.
> > 
> > Correctness requires the counter-sum proof to follow the grace-period
> > anchor written by srcu_gp_start(); placing it before the anchor could
> > let this grace period miss a pre-existing reader and return without
> > waiting for it.  The smp_mb() between the unlock and lock sums pairs
> > with the smp_mb() in __srcu_read_lock().  The grace period is ended
> > manually under ->lock and ->srcu_atomic_gp_flag.
> > 
> > Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
> 
> Adding David Woodhouse on CC.
> 
> David, does this provide appropriate performance in your use case?

Compared with the early bail that I had before, there's a *slight*
shift into the higher-latency buckets as expected, but definitely not
enough that I care. We're doing this whole thing to eliminate the
multi-millisecond p100 latencies that happen when we invoke the
workqueue, and the differences we're looking at in the table below are
*well* below what we care about:


  ┌─────────────┬───────────────────┬───────────────────┬───────┐
  │   bucket    │  try-first (#10)  │    Kunwu (#11)    │   Δ   │
  ├─────────────┼───────────────────┼───────────────────┼───────┤
  │ [0,1µs)     │ 5,338,474 (60.8%) │ 4,953,657 (56.8%) │ −7%   │
  ├─────────────┼───────────────────┼───────────────────┼───────┤
  │ [1,2)       │ 1,077,998         │ 1,087,133         │ ≈     │
  ├─────────────┼───────────────────┼───────────────────┼───────┤
  │ [2,4)       │ 803,244           │ 790,489           │ ≈     │
  ├─────────────┼───────────────────┼───────────────────┼───────┤
  │ [4,8)       │ 811,708           │ 880,673           │ +8%   │
  ├─────────────┼───────────────────┼───────────────────┼───────┤
  │ [8,16)      │ 366,183           │ 597,443           │ +63%  │
  ├─────────────┼───────────────────┼───────────────────┼───────┤
  │ [16,32)     │ 338,731           │ 361,243           │ +7%   │
  ├─────────────┼───────────────────┼───────────────────┼───────┤
  │ [32,64)     │ 14,290            │ 28,823            │ ×2.0  │
  ├─────────────┼───────────────────┼───────────────────┼───────┤
  │ [64,128)    │ 2,549             │ 3,655             │ +43%  │
  ├─────────────┼───────────────────┼───────────────────┼───────┤
  │ [128,256)   │ 555               │ 783               │ +41%  │
  ├─────────────┼───────────────────┼───────────────────┼───────┤
  │ [256,512)   │ 456               │ 625               │ +37%  │
  ├─────────────┼───────────────────┼───────────────────┼───────┤
  │ [512µs+)    │ 13                │ 8                 │ ≈     │
  ├─────────────┼───────────────────┼───────────────────┼───────┤
  │ max         │ 1.07ms            │ 1.56ms            │ +46%  │
  ├─────────────┼───────────────────┼───────────────────┼───────┤
  │ total walks │ 8,777,514         │ 8,723,453         │ −0.6% │
  └─────────────┴───────────────────┴───────────────────┴───────┘

  (192-CPU PREEMPT_RT host, 12 concurrent gfn_to_pfn_cache invalidation
  reproducers, 300-second windows; each "walk" is one invalidation drain
  which includes the synchronize_srcu_atomic() call. #10/#11 are just
  local build numbers.)

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 6179 bytes --]

Re: [PATCH 03/13] srcutree: Add reader-free fastpath to synchronize_srcu_atomic()

[Paul E. McKenney]

On Tue, Sep 08, 2026 at 11:26:06PM +0100, David Woodhouse wrote:
> On Tue, 2026-09-08 at 13:29 -0700, Paul E. McKenney wrote:
> > On Mon, Sep 07, 2026 at 03:58:19PM +0800, Kunwu Chan wrote:
> > > From: Kunwu Chan <kunwu.chan@gmail.com>
> > > 
> > > synchronize_srcu_atomic() is restricted to srcu_read_lock_atomic() and
> > > srcu_read_unlock_atomic(), whose read-side critical sections disable
> > > preemption.  In the common case where there are no readers at all, the
> > > grace period therefore need not do the index flip.  Add a fastpath
> > > that sums both ranks of the per-CPU ->srcu_ctrs[] counters and, if the
> > > lock counts match the unlock counts on both ranks, ends the grace
> > > period immediately, skipping the srcu_advance_state() scans, mirroring
> > > the similar Tiny SRCU fastpath.
> > > 
> > > Correctness requires the counter-sum proof to follow the grace-period
> > > anchor written by srcu_gp_start(); placing it before the anchor could
> > > let this grace period miss a pre-existing reader and return without
> > > waiting for it.  The smp_mb() between the unlock and lock sums pairs
> > > with the smp_mb() in __srcu_read_lock().  The grace period is ended
> > > manually under ->lock and ->srcu_atomic_gp_flag.
> > > 
> > > Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
> > 
> > Adding David Woodhouse on CC.
> > 
> > David, does this provide appropriate performance in your use case?
> 
> Compared with the early bail that I had before, there's a *slight*
> shift into the higher-latency buckets as expected, but definitely not
> enough that I care. We're doing this whole thing to eliminate the
> multi-millisecond p100 latencies that happen when we invoke the
> workqueue, and the differences we're looking at in the table below are
> *well* below what we care about:
> 
> 
>   ┌─────────────┬───────────────────┬───────────────────┬───────┐
>   │   bucket    │  try-first (#10)  │    Kunwu (#11)    │   Δ   │
>   ├─────────────┼───────────────────┼───────────────────┼───────┤
>   │ [0,1µs)     │ 5,338,474 (60.8%) │ 4,953,657 (56.8%) │ −7%   │
>   ├─────────────┼───────────────────┼───────────────────┼───────┤
>   │ [1,2)       │ 1,077,998         │ 1,087,133         │ ≈     │
>   ├─────────────┼───────────────────┼───────────────────┼───────┤
>   │ [2,4)       │ 803,244           │ 790,489           │ ≈     │
>   ├─────────────┼───────────────────┼───────────────────┼───────┤
>   │ [4,8)       │ 811,708           │ 880,673           │ +8%   │
>   ├─────────────┼───────────────────┼───────────────────┼───────┤
>   │ [8,16)      │ 366,183           │ 597,443           │ +63%  │
>   ├─────────────┼───────────────────┼───────────────────┼───────┤
>   │ [16,32)     │ 338,731           │ 361,243           │ +7%   │
>   ├─────────────┼───────────────────┼───────────────────┼───────┤
>   │ [32,64)     │ 14,290            │ 28,823            │ ×2.0  │
>   ├─────────────┼───────────────────┼───────────────────┼───────┤
>   │ [64,128)    │ 2,549             │ 3,655             │ +43%  │
>   ├─────────────┼───────────────────┼───────────────────┼───────┤
>   │ [128,256)   │ 555               │ 783               │ +41%  │
>   ├─────────────┼───────────────────┼───────────────────┼───────┤
>   │ [256,512)   │ 456               │ 625               │ +37%  │
>   ├─────────────┼───────────────────┼───────────────────┼───────┤
>   │ [512µs+)    │ 13                │ 8                 │ ≈     │
>   ├─────────────┼───────────────────┼───────────────────┼───────┤
>   │ max         │ 1.07ms            │ 1.56ms            │ +46%  │
>   ├─────────────┼───────────────────┼───────────────────┼───────┤
>   │ total walks │ 8,777,514         │ 8,723,453         │ −0.6% │
>   └─────────────┴───────────────────┴───────────────────┴───────┘
> 
>   (192-CPU PREEMPT_RT host, 12 concurrent gfn_to_pfn_cache invalidation
>   reproducers, 300-second windows; each "walk" is one invalidation drain
>   which includes the synchronize_srcu_atomic() call. #10/#11 are just
>   local build numbers.)

Thank you very much!!!

I am interpreting this to mean that Kunwu's current approach does what
you need, so that we can keep current get_state_synchronize_srcu()
and poll_state_synchronize_srcu() semantics.

							Thanx, Paul

Re: [PATCH 03/13] srcutree: Add reader-free fastpath to synchronize_srcu_atomic()

[Paul E. McKenney]

On Tue, Sep 08, 2026 at 11:09:34PM +0100, David Woodhouse wrote:
> On Tue, 2026-09-08 at 14:54 -0700, Paul E. McKenney wrote:
> > 
> > And another option is to pull the fastpath up earlier, before
> > checking and/or acquiring ->srcu_atomic_gp_flag.  But this is a
> > bit more complicated from a concurrency viewpoint, at least if
> > we are to interact normally with get_state_synchronize_srcu() and
> > poll_state_synchronize_srcu().  For example:
> > 
> > 	srcu_state = get_state_synchronize_srcu(ssp);
> > 	synchronize_srcu_atomic(ssp);
> > 	WARN_ON_ONCE(!poll_state_synchronize_srcu(ssp, srcu_state));
> > 
> > Without at least some mucking with the grace-period mechanism, that
> > WARN_ON_ONCE() could trigger, which just might not be universally
> > considered to be a friendly act.  ;-)
> > 
> > So it would be very good to keep the fastpath where Kunwu put it, if
> > that works reasonably.
> 
> Could we not just declare that synchronize_srcu_atomic() *isn't*
> guaranteed to drive a GP, so the above code isn't valid? Why poll for a
> thing that's atomic? Is that a likely use case?

We *could*, but one more thing to explain and one more thing for users
to get wrong.

> And you've already forbidden start_poll_synchronize_srcu() for atomic,
> haven't you?

Yes, because there is no such thing as an asynchronous atomic SRCU
grace period.

> I concede that the same logic doesn't work quite as well for
> synchronize_srcu_expedited(), as it's an established API. Could we
> retcon that one?

My hope is that this same sort of approach works there as well.

Hey, I can dream, can't I?  ;-)

							Thanx, Paul

Re: [PATCH 03/13] srcutree: Add reader-free fastpath to synchronize_srcu_atomic()

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 349 bytes --]

On Tue, 2026-09-08 at 15:53 -0700, Paul E. McKenney wrote:
> Thank you very much!!!
> 
> I am interpreting this to mean that Kunwu's current approach does what
> you need, so that we can keep current get_state_synchronize_srcu()
> and poll_state_synchronize_srcu() semantics.

Yes, thank you. I'll rebase the KVM parts on top when it lands.

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 6179 bytes --]

Re: [PATCH 03/13] srcutree: Add reader-free fastpath to synchronize_srcu_atomic()

[Paul E. McKenney]

On Tue, Sep 08, 2026 at 11:56:41PM +0100, David Woodhouse wrote:
> On Tue, 2026-09-08 at 15:53 -0700, Paul E. McKenney wrote:
> > Thank you very much!!!
> > 
> > I am interpreting this to mean that Kunwu's current approach does what
> > you need, so that we can keep current get_state_synchronize_srcu()
> > and poll_state_synchronize_srcu() semantics.
> 
> Yes, thank you. I'll rebase the KVM parts on top when it lands.

Queued for testing and further review, thank you both!

							Thanx, Paul

Re: [PATCH 04/13] rcutorture: Add atomic-SRCU support to torture.sh

[Paul E. McKenney]

On Mon, Sep 07, 2026 at 03:58:20PM +0800, Kunwu Chan wrote:
> From: Kunwu Chan <kunwu.chan@gmail.com>
> 
> Add the --do-atomic-srcu argument to torture.sh, which runs the
> SRCU-N, SRCU-P, and SRCU-T scenarios, thus covering both Tree SRCU
> (SRCU-N and SRCU-P) and Tiny SRCU (SRCU-T), with
> rcutorture.reader_flavor=0x10 appended to the boot parameters so
> that it takes precedence over each scenario's own reader-flavor
> setting.  This exercises srcu_read_lock_atomic(),
> srcu_read_unlock_atomic(), and synchronize_srcu_atomic().
> 
> As with other torture.sh tests, the --do-kcsan argument runs a
> KCSAN+PROVE_LOCKING variant of this test.
> 
> Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>

Queued for testing and review, thank you!

							Thanx, Paul

> ---
>  .../selftests/rcutorture/bin/torture.sh       | 24 +++++++++++++++++++
>  1 file changed, 24 insertions(+)
> 
> diff --git a/tools/testing/selftests/rcutorture/bin/torture.sh b/tools/testing/selftests/rcutorture/bin/torture.sh
> index f0083891ee81..8885812d866a 100755
> --- a/tools/testing/selftests/rcutorture/bin/torture.sh
> +++ b/tools/testing/selftests/rcutorture/bin/torture.sh
> @@ -68,6 +68,7 @@ do_clocksourcewd="${ifnotaarch64}"
>  do_rt=yes
>  do_rcutasksflavors="${ifnotaarch64}" # FIXME: Back to "yes" when SMP=n auto-avoided
>  do_srcu_lockdep=yes
> +do_atomic_srcu=no
>  do_rcu_rust=no
>  
>  # doyesno - Helper function for yes/no arguments
> @@ -103,6 +104,7 @@ usage () {
>  	echo "       --do-rcu-rust / --do-no-rcu-rust / --no-rcu-rust"
>  	echo "       --do-scftorture / --do-no-scftorture / --no-scftorture"
>  	echo "       --do-srcu-lockdep / --do-no-srcu-lockdep / --no-srcu-lockdep"
> +	echo "       --do-atomic-srcu / --do-no-atomic-srcu / --no-atomic-srcu"
>  	echo "       --duration [ <minutes> | <hours>h | <days>d ]"
>  	echo "       --guest-cpu-limit N"
>  	echo "       --kcsan-kmake-arg kernel-make-arguments"
> @@ -148,6 +150,7 @@ do
>  		do_kcsan=yes
>  		do_clocksourcewd="${ifnotaarch64}"
>  		do_srcu_lockdep=yes
> +		do_atomic_srcu=yes
>  		;;
>  	--do-allmodconfig|--do-no-allmodconfig|--no-allmodconfig)
>  		do_allmodconfig=`doyesno "$1" --do-allmodconfig`
> @@ -183,6 +186,7 @@ do
>  		do_kcsan=no
>  		do_clocksourcewd=no
>  		do_srcu_lockdep=no
> +		do_atomic_srcu=no
>  		;;
>  	--do-normal|--do-norm|--do-no-normal|--do-no-norm|--no-normal|--no-norm)
>  		do_normal=`doyesno "$1" --do-normal`
> @@ -212,6 +216,9 @@ do
>  	--do-srcu-lockdep|--do-no-srcu-lockdep|--no-srcu-lockdep)
>  		do_srcu_lockdep=`doyesno "$1" --do-srcu-lockdep`
>  		;;
> +	--do-atomic-srcu|--do-no-atomic-srcu|--no-atomic-srcu)
> +		do_atomic_srcu=`doyesno "$1" --do-atomic-srcu`
> +		;;
>  	--duration)
>  		checkarg --duration "(minutes)" $# "$2" '^[0-9][0-9]*\(m\|h\|d\|\)$' '^error'
>  		mult=1
> @@ -497,6 +504,23 @@ then
>  	torture_set "rcutorture" tools/testing/selftests/rcutorture/bin/kvm.sh --allcpus --duration "$duration_rcutorture" --configs "$configs_rcutorture" --trust-make
>  fi
>  
> +# Test atomic SRCU across Tree SRCU (SRCU-N and SRCU-P) and Tiny SRCU
> +# (SRCU-T).  The reader flavor selects srcu_read_lock_atomic() and
> +# synchronize_srcu_atomic().  Tiny SRCU requires SMP=n, which aarch64
> +# does not support.
> +if test "$do_atomic_srcu" = "yes"
> +then
> +	torture_bootargs="rcutorture.reader_flavor=0x10"
> +	configs_atomic_srcu="SRCU-N SRCU-P"
> +	if test "$ifnotaarch64" = yes
> +	then
> +		configs_atomic_srcu="$configs_atomic_srcu SRCU-T"
> +	fi
> +	torture_set "atomic-srcu" tools/testing/selftests/rcutorture/bin/kvm.sh \
> +		--allcpus --duration "$duration_rcutorture" \
> +		--configs "$configs_atomic_srcu" --trust-make
> +fi
> +
>  if test "$do_locktorture" = "yes"
>  then
>  	torture_bootargs="torture.disable_onoff_at_boot"
> -- 
> 2.43.0
> 

Re: [PATCH 06/13] srcutree: Make init_srcu_struct_atomic() prevent transition to big

[Paul E. McKenney]

On Mon, Sep 07, 2026 at 03:58:22PM +0800, Kunwu Chan wrote:
> From: Kunwu Chan <kunwu.chan@gmail.com>
> 
> The is_atomic parameter of init_srcu_struct_fields() exists so that
> atomic SRCU never transitions to big, but neither
> init_srcu_struct_atomic() nor its lockdep counterpart
> __init_srcu_struct_atomic() sets it.
> 
> On systems where srcutree.convert_to_big selects SRCU_SIZING_INIT,
> this needlessly allocates a full srcu_node combining tree for any
> dynamically initialized atomic srcu_struct, despite atomic SRCU
> having neither callbacks nor srcu_barrier() operations.
> 
> Pass true from both atomic entry points, adding an is_atomic parameter
> to __init_srcu_struct_common().
> 
> Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>

Given that we have this commit, is this patch needed?

4e01d5320a2f ("srcutree: Suppress to-big transition for atomic SRCU")

If so, please tell me what I am missing.

						Thanx, Paul

> ---
>  kernel/rcu/srcutree.c | 15 ++++++++-------
>  1 file changed, 8 insertions(+), 7 deletions(-)
> 
> diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c
> index 2af36db37fa9..01f224a56b41 100644
> --- a/kernel/rcu/srcutree.c
> +++ b/kernel/rcu/srcutree.c
> @@ -305,26 +305,27 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp, bool is_static, bool
>  #ifdef CONFIG_DEBUG_LOCK_ALLOC
>  
>  static int
> -__init_srcu_struct_common(struct srcu_struct *ssp, const char *name, struct lock_class_key *key)
> +__init_srcu_struct_common(struct srcu_struct *ssp, const char *name,
> +			  struct lock_class_key *key, bool is_atomic)
>  {
>  	/* Don't re-initialize a lock while it is held. */
>  	debug_check_no_locks_freed((void *)ssp, sizeof(*ssp));
>  	lockdep_init_map(&ssp->dep_map, name, key, 0);
> -	return init_srcu_struct_fields(ssp, false, false);
> +	return init_srcu_struct_fields(ssp, false, is_atomic);
>  }
>  
>  int init_srcu_struct_lockdep(struct srcu_struct *ssp, const char *name,
>  			     struct lock_class_key *key)
>  {
>  	ssp->srcu_reader_flavor = 0;
> -	return __init_srcu_struct_common(ssp, name, key);
> +	return __init_srcu_struct_common(ssp, name, key, false);
>  }
>  EXPORT_SYMBOL_GPL(init_srcu_struct_lockdep);
>  
>  int __init_srcu_struct_fast(struct srcu_struct *ssp, const char *name, struct lock_class_key *key)
>  {
>  	ssp->srcu_reader_flavor = SRCU_READ_FLAVOR_FAST;
> -	return __init_srcu_struct_common(ssp, name, key);
> +	return __init_srcu_struct_common(ssp, name, key, false);
>  }
>  EXPORT_SYMBOL_GPL(__init_srcu_struct_fast);
>  
> @@ -332,14 +333,14 @@ int __init_srcu_struct_fast_updown(struct srcu_struct *ssp, const char *name,
>  				   struct lock_class_key *key)
>  {
>  	ssp->srcu_reader_flavor = SRCU_READ_FLAVOR_FAST_UPDOWN;
> -	return __init_srcu_struct_common(ssp, name, key);
> +	return __init_srcu_struct_common(ssp, name, key, false);
>  }
>  EXPORT_SYMBOL_GPL(__init_srcu_struct_fast_updown);
>  
>  int __init_srcu_struct_atomic(struct srcu_struct *ssp, const char *name, struct lock_class_key *key)
>  {
>  	ssp->srcu_reader_flavor = SRCU_READ_FLAVOR_ATOMIC;
> -	return __init_srcu_struct_common(ssp, name, key);
> +	return __init_srcu_struct_common(ssp, name, key, true);
>  }
>  EXPORT_SYMBOL_GPL(__init_srcu_struct_atomic);
>  
> @@ -414,7 +415,7 @@ EXPORT_SYMBOL_GPL(init_srcu_struct_fast_updown);
>  int init_srcu_struct_atomic(struct srcu_struct *ssp)
>  {
>  	ssp->srcu_reader_flavor = SRCU_READ_FLAVOR_ATOMIC;
> -	return init_srcu_struct_fields(ssp, false, false);
> +	return init_srcu_struct_fields(ssp, false, true);
>  }
>  EXPORT_SYMBOL_GPL(init_srcu_struct_atomic);
>  
> -- 
> 2.43.0
> 

Re: [PATCH 07/13] srcutree: Don't transition atomic SRCU to big in srcu_gp_end()

[Paul E. McKenney]

On Mon, Sep 07, 2026 at 03:58:23PM +0800, Kunwu Chan wrote:
> From: Kunwu Chan <kunwu.chan@gmail.com>
> 
> The transition-to-big code path in srcu_gp_end() calls
> init_srcu_struct_nodes() with GFP_KERNEL, which is illegal in the
> atomic context reachable from synchronize_srcu_atomic().  Atomic SRCU
> has no use for the srcu_node combining tree: it has neither callbacks
> nor srcu_barrier() operations, and its grace periods are serialized
> by ->srcu_atomic_gp_flag instead.
> 
> Skip this transition entirely for atomic SRCU, which is also defense
> in depth against any path that might wrongly set ->srcu_size_state for
> an atomic srcu_struct.
> 
> Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>

Again, given that we have this commit, is this patch needed?

4e01d5320a2f ("srcutree: Suppress to-big transition for atomic SRCU")

And again, if so, please tell me what I am missing.

							Thanx, Paul

> ---
>  kernel/rcu/srcutree.c | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c
> index 01f224a56b41..74acd5645d4c 100644
> --- a/kernel/rcu/srcutree.c
> +++ b/kernel/rcu/srcutree.c
> @@ -1074,8 +1074,9 @@ static void srcu_gp_end(struct srcu_struct *ssp, bool is_atomic)
>  		raw_spin_unlock_irq_rcu_node(sup);
>  	}
>  
> -	/* Transition to big if needed. */
> -	if (ss_state != SRCU_SIZE_SMALL && ss_state != SRCU_SIZE_BIG) {
> +	/* Transition to big if needed, but never for atomic SRCU. */
> +	if (!is_atomic && ss_state != SRCU_SIZE_SMALL &&
> +	    ss_state != SRCU_SIZE_BIG) {
>  		if (ss_state == SRCU_SIZE_ALLOC)
>  			init_srcu_struct_nodes(ssp, GFP_KERNEL);
>  		else
> -- 
> 2.43.0
> 

Re: [PATCH 08/13] srcutree: Forbid srcu_expedite_current() on atomic SRCU

[Paul E. McKenney]

On Mon, Sep 07, 2026 at 03:58:24PM +0800, Kunwu Chan wrote:
> From: Kunwu Chan <kunwu.chan@gmail.com>
> 
> srcu_expedite_current() has no early guard for atomic SRCU, so calling
> it on an atomic srcu_struct leaks the expediting callback in
> __call_srcu(), leaving ->srcu_ec_state stuck at SRCU_EC_PENDING.
> 
> Add the same WARN_ON_ONCE()-guarded early return used by the other
> forbidden APIs, and list srcu_expedite_current() and srcu_barrier() in
> synchronize_srcu_atomic()'s documentation.
> 
> Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>

Excellent catch, thank you very much!!!  I have queued this to be folded
into the original with attribution, all in the name of bisectability.

							Thanx, Paul

> ---
>  kernel/rcu/srcutree.c | 11 +++++++++--
>  1 file changed, 9 insertions(+), 2 deletions(-)
> 
> diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c
> index 74acd5645d4c..2d2a12f86115 100644
> --- a/kernel/rcu/srcutree.c
> +++ b/kernel/rcu/srcutree.c
> @@ -1972,6 +1972,9 @@ static void srcu_expedite_current_cb(struct rcu_head *rhp)
>   * no current grace period, one might be created.  If the current grace
>   * period is currently sleeping, that sleep will complete before expediting
>   * will take effect.
> + *
> + * This function must not be invoked on srcu_struct structures that are
> + * used with srcu_read_lock_atomic() and synchronize_srcu_atomic().
>   */
>  void srcu_expedite_current(struct srcu_struct *ssp)
>  {
> @@ -1979,6 +1982,9 @@ void srcu_expedite_current(struct srcu_struct *ssp)
>  	bool needcb = false;
>  	struct srcu_data *sdp;
>  
> +	// Atomic SRCU has no callbacks, so there is nothing to expedite.
> +	if (WARN_ON_ONCE(ssp->srcu_reader_flavor == SRCU_READ_FLAVOR_ATOMIC))
> +		return;
>  	migrate_disable();
>  	sdp = this_cpu_ptr(ssp->sda);
>  	raw_spin_lock_irqsave_sdp_contention(sdp, &flags);
> @@ -2104,8 +2110,9 @@ static void srcu_advance_state(struct srcu_struct *ssp, bool is_atomic)
>   *
>   * If synchronize_srcu_atomic() is invoked on a given srcu_struct
>   * structure, then none of call_srcu(), synchronize_srcu(),
> - * synchronize_srcu_expedited(), or start_poll_synchronize_srcu() may be
> - * invoked on that same structure.
> + * synchronize_srcu_expedited(), start_poll_synchronize_srcu(),
> + * srcu_barrier(), or srcu_expedite_current() may be invoked on that
> + * same structure.
>   *
>   * Because synchronize_srcu_atomic() is even more expedited than is
>   * synchronize_srcu_expedited(), there is no expedited counterpart to
> -- 
> 2.43.0
> 

Re: [PATCH 09/13] rcutorture: Disable srcu_expedite_current() for atomic SRCU

[Paul E. McKenney]

On Mon, Sep 07, 2026 at 03:58:25PM +0800, Kunwu Chan wrote:
> From: Kunwu Chan <kunwu.chan@gmail.com>
> 
> srcu_expedite_current() is now forbidden for atomic SRCU, so null out
> ->exp_current in srcu_torture_init_forbidden_apis().
> 
> Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>

And you got the testing end as well, very good, thank you!  I have
again folded this into the original with attribution in the name of
bisectability.

							Thanx, Paul

> ---
>  kernel/rcu/rcutorture.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c
> index a9672fe88cbc..d795e07a12ad 100644
> --- a/kernel/rcu/rcutorture.c
> +++ b/kernel/rcu/rcutorture.c
> @@ -721,6 +721,7 @@ static void srcu_torture_init_forbidden_apis(void)
>  	cur_ops->call = NULL;
>  	cur_ops->cb_barrier = NULL;
>  	cur_ops->deferred_free = NULL;
> +	cur_ops->exp_current = NULL;
>  	cur_ops->start_gp_poll = NULL;
>  }
>  
> -- 
> 2.43.0
> 

Re: [PATCH 01/13] litmus: Add SRCU fastpath anchor-before-scan test

[Paul E. McKenney]

On Mon, Sep 07, 2026 at 03:58:17PM +0800, Kunwu Chan wrote:
> From: Kunwu Chan <kunwu.chan@gmail.com>
> 
> synchronize_srcu_atomic() may end its grace period immediately when
> its scan of the per-CPU lock counters finds no readers.  Correctness
> requires the grace-period anchor written by srcu_gp_start() to precede
> the smp_mb() ordering the lock scan.  This ordering ensures that any
> reader whose lock increment is missed by the scan cannot have
> incremented its lock counter before the grace-period anchor, and
> therefore cannot be a pre-existing reader of this grace period.
> 
> This litmus test models the key ordering between the grace-period
> anchor and the lock counter scan, where "seq" models the
> grace-period anchor in ->srcu_gp_seq and "ctr" models the per-CPU
> ->srcu_ctrs[].srcu_locks counter.  P0 writes the anchor before the
> smp_mb() and the lock scan.  P1 models the reader-side counter
> increment, with the smp_mb() of __srcu_read_lock() following the
> increment.  P2 models an observer that sees the reader's increment
> before seeing the anchor.
> 
> The outcome is forbidden by LKMM, and herd7 reports "Never".  See
> SRCU-fastpath-scan-before-anchor.litmus for the reversed ordering,
> which permits this outcome.
> 
> Tested with herd7 7.58 using linux-kernel.cfg.
> 
> Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>

Litmus tests!  Very nice!!!  

Could you please put both of these in Documentation/litmus-tests, in a
new "srcu" subdirectory?

One thing for your consideration is use of the "filter" clause for the
first term of your "exists" clause.  Not a big deal at all for this small
of a litmus test, but the idea is that this litmus test only cares about
the 0:r2=0 case:  If that condition does not hold, then P0() and P1()
aren't the beginning and end of a valid SRCU read-side critical section.

Use of the "filter" allows herd7 to abandon a given execution early,
so it is a big deal for larger litmus tests.

Again, what you have is fine (or will be when moved to the other
directory), just pointing out the additional feature.

If you would like to see a use case, please see:

Documentation/litmus-tests/locking/RM-fixed.litmus

							Thanx, Paul

> ---
>  .../SRCU-fastpath-anchor-before-scan.litmus   | 56 +++++++++++++++++++
>  1 file changed, 56 insertions(+)
>  create mode 100644 tools/memory-model/litmus-tests/SRCU-fastpath-anchor-before-scan.litmus
> 
> diff --git a/tools/memory-model/litmus-tests/SRCU-fastpath-anchor-before-scan.litmus b/tools/memory-model/litmus-tests/SRCU-fastpath-anchor-before-scan.litmus
> new file mode 100644
> index 000000000000..8200a75e15ef
> --- /dev/null
> +++ b/tools/memory-model/litmus-tests/SRCU-fastpath-anchor-before-scan.litmus
> @@ -0,0 +1,56 @@
> +C SRCU-fastpath-anchor-before-scan
> +
> +(*
> + * Result: Never
> + *
> + * The synchronize_srcu_atomic() fastpath may end its grace period
> + * immediately when its scan of the per-CPU lock counters finds no
> + * readers.  Correctness requires the grace-period anchor written by
> + * srcu_gp_start() to precede the smp_mb() ordering the lock scan.
> + * This ordering ensures that any reader whose lock increment is missed
> + * by the scan cannot have incremented its lock counter before the
> + * grace-period anchor, and therefore cannot be a pre-existing reader
> + * of this grace period.
> + *
> + * This litmus test models the key ordering between the grace-period
> + * anchor and the lock counter scan, where "seq" models the
> + * grace-period anchor in ->srcu_gp_seq and "ctr" models the per-CPU
> + * ->srcu_ctrs[].srcu_locks counter.  P0 writes the anchor before the
> + * smp_mb() and the lock scan.  P1 models the reader-side counter
> + * increment, with the smp_mb() of __srcu_read_lock() following the
> + * increment.  P2 models an observer that sees the reader's increment
> + * before seeing the anchor.
> + *
> + * The outcome is forbidden by LKMM, and herd7 reports "Never".  See
> + * SRCU-fastpath-scan-before-anchor.litmus for the reversed ordering,
> + * which permits this outcome.
> + *)
> +
> +{}
> +
> +P0(int *seq, int *ctr)
> +{
> +	int r2;
> +
> +	WRITE_ONCE(*seq, 1);
> +	smp_mb();
> +	r2 = READ_ONCE(*ctr);
> +}
> +
> +P1(int *ctr)
> +{
> +	WRITE_ONCE(*ctr, 1);
> +	smp_mb();
> +}
> +
> +P2(int *seq, int *ctr)
> +{
> +	int r3;
> +	int r4;
> +
> +	r3 = READ_ONCE(*ctr);
> +	smp_mb();
> +	r4 = READ_ONCE(*seq);
> +}
> +
> +exists (0:r2 = 0 /\ 2:r3 = 1 /\ 2:r4 = 0)
> -- 
> 2.43.0
> 

Re: [PATCH 10/13] srcutree: Skip callback scheduling for atomic SRCU grace periods

[Paul E. McKenney]

On Mon, Sep 07, 2026 at 03:58:26PM +0800, Kunwu Chan wrote:
> From: Kunwu Chan <kunwu.chan@gmail.com>
> 
> call_srcu() is forbidden on atomic srcu_struct, so srcu_gp_end() never
> has callbacks to invoke for them.  Yet it schedules callback invocation,
> which for atomic SRCU's SRCU_SIZE_SMALL state arms the boot CPU's
> ->delay_work timer every grace period, only for srcu_invoke_callbacks()
> to find nothing to do.
> 
> Skip this for atomic SRCU.
> 
> Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>

Nice optimization, thank you!  This does not matter for correctness,
and thus does not affect bisectability, so I have queued it as is for
testing and further review.

							Thanx, Paul

> ---
>  kernel/rcu/srcutree.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c
> index 2d2a12f86115..c19f59725706 100644
> --- a/kernel/rcu/srcutree.c
> +++ b/kernel/rcu/srcutree.c
> @@ -1018,10 +1018,10 @@ static void srcu_gp_end(struct srcu_struct *ssp, bool is_atomic)
>  
>  	/* Initiate callback invocation as needed. */
>  	ss_state = smp_load_acquire(&sup->srcu_size_state);
> -	if (ss_state < SRCU_SIZE_WAIT_BARRIER) {
> +	if (!is_atomic && ss_state < SRCU_SIZE_WAIT_BARRIER) {
>  		srcu_schedule_cbs_sdp(per_cpu_ptr(ssp->sda, get_boot_cpu_id()),
>  					cbdelay);
> -	} else {
> +	} else if (!is_atomic) {
>  		idx = rcu_seq_ctr(gpseq) % ARRAY_SIZE(snp->srcu_have_cbs);
>  		srcu_for_each_node_breadth_first(ssp, snp) {
>  			raw_spin_lock_irq_rcu_node(snp);
> -- 
> 2.43.0
> 

Re: [PATCH 11/13] srcutree: Remove srcu_barrier() sleep for atomic SRCU

[Paul E. McKenney]

On Mon, Sep 07, 2026 at 03:58:27PM +0800, Kunwu Chan wrote:
> From: Kunwu Chan <kunwu.chan@gmail.com>
> 
> The atomic-SRCU path in srcu_barrier() sleeps for 100 milliseconds just
> in case there are callbacks to wait for.  But call_srcu() refuses
> atomic SRCU with a WARN_ON_ONCE() before reaching the deferred-enqueue
> path, so there can be no callbacks, deferred or otherwise.
> 
> Drop the sleep.
> 
> Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>

Good catch!  As you say, now that call_srcu() leaks callbacks that are
for atomic srcu_struct structures, there really cannot be any callbacks.
Queued for further review and testing, thank you!

							Thanx, Paul

> ---
>  kernel/rcu/srcutree.c | 7 ++-----
>  1 file changed, 2 insertions(+), 5 deletions(-)
> 
> diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c
> index c19f59725706..a93bf803e2f7 100644
> --- a/kernel/rcu/srcutree.c
> +++ b/kernel/rcu/srcutree.c
> @@ -1895,12 +1895,9 @@ void srcu_barrier(struct srcu_struct *ssp)
>  	unsigned long s;
>  
>  	check_init_srcu_struct(ssp, false);
> -	if (WARN_ON_ONCE(ssp->srcu_reader_flavor == SRCU_READ_FLAVOR_ATOMIC)) {
> -		// There shouldn't be any callbacks for atomic SRCU,
> -		// but just in case.
> -		schedule_timeout_uninterruptible(HZ/10);
> +	// Atomic SRCU has no callbacks, so there is nothing to wait on.
> +	if (WARN_ON_ONCE(ssp->srcu_reader_flavor == SRCU_READ_FLAVOR_ATOMIC))
>  		return;
> -	}
>  
>  	/*
>  	 * Register any deferred callbacks before snapshotting the sequence.  The
> -- 
> 2.43.0
> 

Re: [PATCH 12/13] srcutree: Remove debug pr_alert()s

[Paul E. McKenney]

On Mon, Sep 07, 2026 at 03:58:28PM +0800, Kunwu Chan wrote:
> From: Kunwu Chan <kunwu.chan@gmail.com>
> 
> Remove the debug pr_alert()s from srcu_gp_start() and
> synchronize_srcu_atomic().
> 
> Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>

Good point, we have gotten beyond where this debug code would be useful
(famous last words!).  I have folded this into the original commit with
attribution to prevent "&&&&" confusion.

							Thanx, Paul

> ---
>  kernel/rcu/srcutree.c | 3 ---
>  1 file changed, 3 deletions(-)
> 
> diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c
> index a93bf803e2f7..a016c93d65cb 100644
> --- a/kernel/rcu/srcutree.c
> +++ b/kernel/rcu/srcutree.c
> @@ -916,7 +916,6 @@ static void srcu_gp_start(struct srcu_struct *ssp)
>  {
>  	int state;
>  
> -	/*&&&&*/pr_alert("%s() start: ->srcu_gp_seq: %lx ->srcu_gp_seq_needed: %lx\n", __func__, ssp->srcu_sup->srcu_gp_seq, ssp->srcu_sup->srcu_gp_seq_needed);
>  	lockdep_assert_held(&ACCESS_PRIVATE(ssp->srcu_sup, lock));
>  	WARN_ON_ONCE(ULONG_CMP_GE(ssp->srcu_sup->srcu_gp_seq, ssp->srcu_sup->srcu_gp_seq_needed));
>  	WRITE_ONCE(ssp->srcu_sup->srcu_gp_start, jiffies);
> @@ -2152,7 +2151,6 @@ void synchronize_srcu_atomic(struct srcu_struct *ssp)
>  	// OK, we really have to do it ourselves.  Start the grace period.
>  	non_block_start();  // We must not voluntarily block!
>  	smp_store_release(&sup->srcu_gp_seq_needed, srcu_state); // See srcu_funnel_gp_start().
> -	/*&&&&*/pr_alert("%s() start: ->srcu_gp_seq: %lx ->srcu_gp_seq_needed: %lx\n", __func__, ssp->srcu_sup->srcu_gp_seq, ssp->srcu_sup->srcu_gp_seq_needed);
>  	ASSERT_EXCLUSIVE_WRITER(ssp->srcu_sup->srcu_gp_seq);
>  	srcu_gp_start(ssp);
>  	raw_spin_unlock_irq_rcu_node(sup);
> @@ -2208,7 +2206,6 @@ void synchronize_srcu_atomic(struct srcu_struct *ssp)
>  		cpu_relax();
>  		srcu_advance_state(ssp, true);
>  	}
> -	/*&&&&*/pr_alert("%s() end: ->srcu_gp_seq: %lx ->srcu_gp_seq_needed: %lx\n", __func__, ssp->srcu_sup->srcu_gp_seq, ssp->srcu_sup->srcu_gp_seq_needed);
>  	ASSERT_EXCLUSIVE_WRITER(sup->srcu_atomic_gp_flag);
>  	atomic_set_release(&sup->srcu_atomic_gp_flag, 0);
>  	preempt_enable();
> -- 
> 2.43.0
> 

Re: [PATCH 13/13] srcu: Restrict atomic-SRCU non_block annotation to task context

[Paul E. McKenney]

On Mon, Sep 07, 2026 at 03:58:29PM +0800, Kunwu Chan wrote:
> From: Kunwu Chan <kunwu.chan@gmail.com>
> 
> srcu_read_lock_atomic() and srcu_read_unlock_atomic() arm and disarm
> might_sleep() checks via non_block_start()/non_block_end(), skipping
> hardirq so the update does not land on the interrupted task's
> ->non_block_count.
> 
> Inline softirqs run on the interrupted task's stack as well, so a
> timer callback running atomic-SRCU readers races with the interrupted
> task's own ->non_block_count updates, as KCSAN reports:
> 
> 	BUG: KCSAN: data-race in srcu_torture_read_lock / srcu_torture_read_unlock
> 
> 	write to 0xffffa00f818ea418 of 4 bytes by interrupt on cpu 0:
> 		srcu_torture_read_lock+0x422/0x470
> 		rcutorture_one_extend+0xdc/0x600
> 		rcu_torture_one_read+0xd1/0x330
> 		rcu_torture_timer+0x75/0x140
> 		call_timer_fn+0xe6/0x2f0
> 		...
> 		run_timer_softirq+0xb7/0x130
> 		handle_softirqs+0xfc/0x3f0
> 		__irq_exit_rcu+0x8e/0x100
> 
> Use in_task() so the annotation is applied only in task context; it
> is redundant elsewhere because might_sleep() already warns about
> sleeping from atomic context.
> 
> Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>

Again, good catch!  In theory, this should be squashed into the original,
but I don't know of many people bisecting with KCSAN.  I might need to
squash it later (of course with attribution), but at least for the time
being, queued for testing and further review.

							Thanx, Paul

> ---
>  include/linux/srcu.h | 10 ++++++----
>  1 file changed, 6 insertions(+), 4 deletions(-)
> 
> diff --git a/include/linux/srcu.h b/include/linux/srcu.h
> index 0de21155abc1..403fbe57ab15 100644
> --- a/include/linux/srcu.h
> +++ b/include/linux/srcu.h
> @@ -346,11 +346,13 @@ static inline int srcu_read_lock_atomic(struct srcu_struct *ssp)
>  	/*
>  	 * Arm might_sleep() to catch even a *potentially* sleeping call
>  	 * in the section, not just an actual schedule: the atomic-domain
> -	 * promise must hold on every path, contended or not. In hardirq
> -	 * the annotation would land on the interrupted task; it is also
> +	 * promise must hold on every path, contended or not. In hardirq,
> +	 * softirq, or NMI the annotation would land on the interrupted
> +	 * task, and can also result in data races against that task's
> +	 * own non_block_start()/non_block_end() invocations; it is also
>  	 * redundant there, so skip it.
>  	 */
> -	if (!in_hardirq())
> +	if (in_task())
>  		non_block_start();
>  	srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_ATOMIC);
>  	retval = __srcu_read_lock(ssp);
> @@ -562,7 +564,7 @@ static inline void srcu_read_unlock_atomic(struct srcu_struct *ssp, int idx)
>  	srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_ATOMIC);
>  	srcu_lock_release(&ssp->dep_map);
>  	__srcu_read_unlock(ssp, idx);
> -	if (!in_hardirq())
> +	if (in_task())
>  		non_block_end();
>  	preempt_enable();
>  }
> -- 
> 2.43.0
> 

Re: [PATCH 06/13] srcutree: Make init_srcu_struct_atomic() prevent transition to big

[Kunwu Chan]

On Tue, 8 Sep 2026 16:36:11 -0700 "Paul E. McKenney" <paulmck@kernel.org> wrote:

> On Mon, Sep 07, 2026 at 03:58:22PM +0800, Kunwu Chan wrote:
> > From: Kunwu Chan <kunwu.chan@gmail.com>
> > 
> > The is_atomic parameter of init_srcu_struct_fields() exists so that
> > atomic SRCU never transitions to big, but neither
> > init_srcu_struct_atomic() nor its lockdep counterpart
> > __init_srcu_struct_atomic() sets it.
> > 
> > On systems where srcutree.convert_to_big selects SRCU_SIZING_INIT,
> > this needlessly allocates a full srcu_node combining tree for any
> > dynamically initialized atomic srcu_struct, despite atomic SRCU
> > having neither callbacks nor srcu_barrier() operations.
> > 
> > Pass true from both atomic entry points, adding an is_atomic parameter
> > to __init_srcu_struct_common().
> > 
> > Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
> 
> Given that we have this commit, is this patch needed?
> 
> 4e01d5320a2f ("srcutree: Suppress to-big transition for atomic SRCU")
> 
> If so, please tell me what I am missing.

Yes, this patch is still needed. The is_atomic mechanism is there, but 
the atomic initialization paths still pass false.

In particular, init_srcu_struct_atomic() passes false directly, while the 
lockdep path goes through __init_srcu_struct_common(), which also passes false.

This patch makes both atomic entry points propagate true.

Thanks,
KunWu

> 
> 						Thanx, Paul
> 
> > ---
> >  kernel/rcu/srcutree.c | 15 ++++++++-------
> >  1 file changed, 8 insertions(+), 7 deletions(-)
> > 
> > diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c
> > index 2af36db37fa9..01f224a56b41 100644
> > --- a/kernel/rcu/srcutree.c
> > +++ b/kernel/rcu/srcutree.c
> > @@ -305,26 +305,27 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp, bool is_static, bool
> >  #ifdef CONFIG_DEBUG_LOCK_ALLOC
> >  
> >  static int
> > -__init_srcu_struct_common(struct srcu_struct *ssp, const char *name, struct lock_class_key *key)
> > +__init_srcu_struct_common(struct srcu_struct *ssp, const char *name,
> > +			  struct lock_class_key *key, bool is_atomic)
> >  {
> >  	/* Don't re-initialize a lock while it is held. */
> >  	debug_check_no_locks_freed((void *)ssp, sizeof(*ssp));
> >  	lockdep_init_map(&ssp->dep_map, name, key, 0);
> > -	return init_srcu_struct_fields(ssp, false, false);
> > +	return init_srcu_struct_fields(ssp, false, is_atomic);
> >  }
> >  
> >  int init_srcu_struct_lockdep(struct srcu_struct *ssp, const char *name,
> >  			     struct lock_class_key *key)
> >  {
> >  	ssp->srcu_reader_flavor = 0;
> > -	return __init_srcu_struct_common(ssp, name, key);
> > +	return __init_srcu_struct_common(ssp, name, key, false);
> >  }
> >  EXPORT_SYMBOL_GPL(init_srcu_struct_lockdep);
> >  
> >  int __init_srcu_struct_fast(struct srcu_struct *ssp, const char *name, struct lock_class_key *key)
> >  {
> >  	ssp->srcu_reader_flavor = SRCU_READ_FLAVOR_FAST;
> > -	return __init_srcu_struct_common(ssp, name, key);
> > +	return __init_srcu_struct_common(ssp, name, key, false);
> >  }
> >  EXPORT_SYMBOL_GPL(__init_srcu_struct_fast);
> >  
> > @@ -332,14 +333,14 @@ int __init_srcu_struct_fast_updown(struct srcu_struct *ssp, const char *name,
> >  				   struct lock_class_key *key)
> >  {
> >  	ssp->srcu_reader_flavor = SRCU_READ_FLAVOR_FAST_UPDOWN;
> > -	return __init_srcu_struct_common(ssp, name, key);
> > +	return __init_srcu_struct_common(ssp, name, key, false);
> >  }
> >  EXPORT_SYMBOL_GPL(__init_srcu_struct_fast_updown);
> >  
> >  int __init_srcu_struct_atomic(struct srcu_struct *ssp, const char *name, struct lock_class_key *key)
> >  {
> >  	ssp->srcu_reader_flavor = SRCU_READ_FLAVOR_ATOMIC;
> > -	return __init_srcu_struct_common(ssp, name, key);
> > +	return __init_srcu_struct_common(ssp, name, key, true);
> >  }
> >  EXPORT_SYMBOL_GPL(__init_srcu_struct_atomic);
> >  
> > @@ -414,7 +415,7 @@ EXPORT_SYMBOL_GPL(init_srcu_struct_fast_updown);
> >  int init_srcu_struct_atomic(struct srcu_struct *ssp)
> >  {
> >  	ssp->srcu_reader_flavor = SRCU_READ_FLAVOR_ATOMIC;
> > -	return init_srcu_struct_fields(ssp, false, false);
> > +	return init_srcu_struct_fields(ssp, false, true);
> >  }
> >  EXPORT_SYMBOL_GPL(init_srcu_struct_atomic);
> >  
> > -- 
> > 2.43.0
> > 
> 

Re: [PATCH 07/13] srcutree: Don't transition atomic SRCU to big in srcu_gp_end()

[Kunwu Chan]

On Tue, 8 Sep 2026 16:38:19 -0700 "Paul E. McKenney" <paulmck@kernel.org> wrote:

> On Mon, Sep 07, 2026 at 03:58:23PM +0800, Kunwu Chan wrote:
> > From: Kunwu Chan <kunwu.chan@gmail.com>
> > 
> > The transition-to-big code path in srcu_gp_end() calls
> > init_srcu_struct_nodes() with GFP_KERNEL, which is illegal in the
> > atomic context reachable from synchronize_srcu_atomic().  Atomic SRCU
> > has no use for the srcu_node combining tree: it has neither callbacks
> > nor srcu_barrier() operations, and its grace periods are serialized
> > by ->srcu_atomic_gp_flag instead.
> > 
> > Skip this transition entirely for atomic SRCU, which is also defense
> > in depth against any path that might wrongly set ->srcu_size_state for
> > an atomic srcu_struct.
> > 
> > Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
> 
> Again, given that we have this commit, is this patch needed?
> 
> 4e01d5320a2f ("srcutree: Suppress to-big transition for atomic SRCU")
> 
> And again, if so, please tell me what I am missing.

Yes, this is a separate transition path. That commit suppresses the 
initialization-time transition, while this patch prevents the transition 
from srcu_gp_end(), which can otherwise call 
init_srcu_struct_nodes(..., GFP_KERNEL) for an atomic SRCU.

So this is defense in depth for the atomic path.

Thanks,
KunWu

> 
> 							Thanx, Paul
> 
> > ---
> >  kernel/rcu/srcutree.c | 5 +++--
> >  1 file changed, 3 insertions(+), 2 deletions(-)
> > 
> > diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c
> > index 01f224a56b41..74acd5645d4c 100644
> > --- a/kernel/rcu/srcutree.c
> > +++ b/kernel/rcu/srcutree.c
> > @@ -1074,8 +1074,9 @@ static void srcu_gp_end(struct srcu_struct *ssp, bool is_atomic)
> >  		raw_spin_unlock_irq_rcu_node(sup);
> >  	}
> >  
> > -	/* Transition to big if needed. */
> > -	if (ss_state != SRCU_SIZE_SMALL && ss_state != SRCU_SIZE_BIG) {
> > +	/* Transition to big if needed, but never for atomic SRCU. */
> > +	if (!is_atomic && ss_state != SRCU_SIZE_SMALL &&
> > +	    ss_state != SRCU_SIZE_BIG) {
> >  		if (ss_state == SRCU_SIZE_ALLOC)
> >  			init_srcu_struct_nodes(ssp, GFP_KERNEL);
> >  		else
> > -- 
> > 2.43.0
> > 
> 

Sent using hkml (https://github.com/sjp38/hackermail)

Re: [PATCH 01/13] litmus: Add SRCU fastpath anchor-before-scan test

[Kunwu Chan]

On Tue, 8 Sep 2026 16:58:31 -0700 "Paul E. McKenney" <paulmck@kernel.org> wrote:

> On Mon, Sep 07, 2026 at 03:58:17PM +0800, Kunwu Chan wrote:
> > From: Kunwu Chan <kunwu.chan@gmail.com>
> > 
> > synchronize_srcu_atomic() may end its grace period immediately when
> > its scan of the per-CPU lock counters finds no readers.  Correctness
> > requires the grace-period anchor written by srcu_gp_start() to precede
> > the smp_mb() ordering the lock scan.  This ordering ensures that any
> > reader whose lock increment is missed by the scan cannot have
> > incremented its lock counter before the grace-period anchor, and
> > therefore cannot be a pre-existing reader of this grace period.
> > 
> > This litmus test models the key ordering between the grace-period
> > anchor and the lock counter scan, where "seq" models the
> > grace-period anchor in ->srcu_gp_seq and "ctr" models the per-CPU
> > ->srcu_ctrs[].srcu_locks counter.  P0 writes the anchor before the
> > smp_mb() and the lock scan.  P1 models the reader-side counter
> > increment, with the smp_mb() of __srcu_read_lock() following the
> > increment.  P2 models an observer that sees the reader's increment
> > before seeing the anchor.
> > 
> > The outcome is forbidden by LKMM, and herd7 reports "Never".  See
> > SRCU-fastpath-scan-before-anchor.litmus for the reversed ordering,
> > which permits this outcome.
> > 
> > Tested with herd7 7.58 using linux-kernel.cfg.
> > 
> > Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
> 
> Litmus tests!  Very nice!!!  
> 
> Could you please put both of these in Documentation/litmus-tests, in a
> new "srcu" subdirectory?
> 
> One thing for your consideration is use of the "filter" clause for the
> first term of your "exists" clause.  Not a big deal at all for this small
> of a litmus test, but the idea is that this litmus test only cares about
> the 0:r2=0 case:  If that condition does not hold, then P0() and P1()
> aren't the beginning and end of a valid SRCU read-side critical section.
> 
> Use of the "filter" allows herd7 to abandon a given execution early,
> so it is a big deal for larger litmus tests.
> 
> Again, what you have is fine (or will be when moved to the other
> directory), just pointing out the additional feature.
> 
> If you would like to see a use case, please see:
> 
> Documentation/litmus-tests/locking/RM-fixed.litmus

Thanks, Paul.
I’ll move both tests to Documentation/litmus-tests/srcu/ and use a "filter" 
clause for the first test as suggested.

I’ll send the two litmus tests as a separate follow-up series, so this won’t 
hold up the current atomic SRCU series.
I’d also like to continue maintaining the SRCU litmus tests as they evolve. 
If you think a MAINTAINERS entry for the SRCU litmus tests would be appropriate, 
I’d be happy to prepare that as well.

Thanks,
KunWu

> 
> 							Thanx, Paul
> 
> > ---
> >  .../SRCU-fastpath-anchor-before-scan.litmus   | 56 +++++++++++++++++++
> >  1 file changed, 56 insertions(+)
> >  create mode 100644 tools/memory-model/litmus-tests/SRCU-fastpath-anchor-before-scan.litmus
> > 
> > diff --git a/tools/memory-model/litmus-tests/SRCU-fastpath-anchor-before-scan.litmus b/tools/memory-model/litmus-tests/SRCU-fastpath-anchor-before-scan.litmus
> > new file mode 100644
> > index 000000000000..8200a75e15ef
> > --- /dev/null
> > +++ b/tools/memory-model/litmus-tests/SRCU-fastpath-anchor-before-scan.litmus
> > @@ -0,0 +1,56 @@
> > +C SRCU-fastpath-anchor-before-scan
> > +
> > +(*
> > + * Result: Never
> > + *
> > + * The synchronize_srcu_atomic() fastpath may end its grace period
> > + * immediately when its scan of the per-CPU lock counters finds no
> > + * readers.  Correctness requires the grace-period anchor written by
> > + * srcu_gp_start() to precede the smp_mb() ordering the lock scan.
> > + * This ordering ensures that any reader whose lock increment is missed
> > + * by the scan cannot have incremented its lock counter before the
> > + * grace-period anchor, and therefore cannot be a pre-existing reader
> > + * of this grace period.
> > + *
> > + * This litmus test models the key ordering between the grace-period
> > + * anchor and the lock counter scan, where "seq" models the
> > + * grace-period anchor in ->srcu_gp_seq and "ctr" models the per-CPU
> > + * ->srcu_ctrs[].srcu_locks counter.  P0 writes the anchor before the
> > + * smp_mb() and the lock scan.  P1 models the reader-side counter
> > + * increment, with the smp_mb() of __srcu_read_lock() following the
> > + * increment.  P2 models an observer that sees the reader's increment
> > + * before seeing the anchor.
> > + *
> > + * The outcome is forbidden by LKMM, and herd7 reports "Never".  See
> > + * SRCU-fastpath-scan-before-anchor.litmus for the reversed ordering,
> > + * which permits this outcome.
> > + *)
> > +
> > +{}
> > +
> > +P0(int *seq, int *ctr)
> > +{
> > +	int r2;
> > +
> > +	WRITE_ONCE(*seq, 1);
> > +	smp_mb();
> > +	r2 = READ_ONCE(*ctr);
> > +}
> > +
> > +P1(int *ctr)
> > +{
> > +	WRITE_ONCE(*ctr, 1);
> > +	smp_mb();
> > +}
> > +
> > +P2(int *seq, int *ctr)
> > +{
> > +	int r3;
> > +	int r4;
> > +
> > +	r3 = READ_ONCE(*ctr);
> > +	smp_mb();
> > +	r4 = READ_ONCE(*seq);
> > +}
> > +
> > +exists (0:r2 = 0 /\ 2:r3 = 1 /\ 2:r4 = 0)
> > -- 
> > 2.43.0
> > 
> 

Sent using hkml (https://github.com/sjp38/hackermail)

Re: [PATCH 03/13] srcutree: Add reader-free fastpath to synchronize_srcu_atomic()

[Kunwu Chan]

On Tue, 08 Sep 2026 23:26:06 +0100 David Woodhouse <dwmw2@infradead.org> wrote:

> On Tue, 2026-09-08 at 13:29 -0700, Paul E. McKenney wrote:
> > On Mon, Sep 07, 2026 at 03:58:19PM +0800, Kunwu Chan wrote:
> > > From: Kunwu Chan <kunwu.chan@gmail.com>
> > > 
> > > synchronize_srcu_atomic() is restricted to srcu_read_lock_atomic() and
> > > srcu_read_unlock_atomic(), whose read-side critical sections disable
> > > preemption.  In the common case where there are no readers at all, the
> > > grace period therefore need not do the index flip.  Add a fastpath
> > > that sums both ranks of the per-CPU ->srcu_ctrs[] counters and, if the
> > > lock counts match the unlock counts on both ranks, ends the grace
> > > period immediately, skipping the srcu_advance_state() scans, mirroring
> > > the similar Tiny SRCU fastpath.
> > > 
> > > Correctness requires the counter-sum proof to follow the grace-period
> > > anchor written by srcu_gp_start(); placing it before the anchor could
> > > let this grace period miss a pre-existing reader and return without
> > > waiting for it.  The smp_mb() between the unlock and lock sums pairs
> > > with the smp_mb() in __srcu_read_lock().  The grace period is ended
> > > manually under ->lock and ->srcu_atomic_gp_flag.
> > > 
> > > Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
> > 
> > Adding David Woodhouse on CC.
> > 
> > David, does this provide appropriate performance in your use case?
> 
> Compared with the early bail that I had before, there's a *slight*
> shift into the higher-latency buckets as expected, but definitely not
> enough that I care. We're doing this whole thing to eliminate the
> multi-millisecond p100 latencies that happen when we invoke the
> workqueue, and the differences we're looking at in the table below are
> *well* below what we care about:
> 
> 
>   ┌─────────────┬───────────────────┬───────────────────┬───────┐
>   │   bucket    │  try-first (#10)  │    Kunwu (#11)    │   Δ   │
>   ├─────────────┼───────────────────┼───────────────────┼───────┤
>   │ [0,1µs)     │ 5,338,474 (60.8%) │ 4,953,657 (56.8%) │ −7%   │
>   ├─────────────┼───────────────────┼───────────────────┼───────┤
>   │ [1,2)       │ 1,077,998         │ 1,087,133         │ ≈     │
>   ├─────────────┼───────────────────┼───────────────────┼───────┤
>   │ [2,4)       │ 803,244           │ 790,489           │ ≈     │
>   ├─────────────┼───────────────────┼───────────────────┼───────┤
>   │ [4,8)       │ 811,708           │ 880,673           │ +8%   │
>   ├─────────────┼───────────────────┼───────────────────┼───────┤
>   │ [8,16)      │ 366,183           │ 597,443           │ +63%  │
>   ├─────────────┼───────────────────┼───────────────────┼───────┤
>   │ [16,32)     │ 338,731           │ 361,243           │ +7%   │
>   ├─────────────┼───────────────────┼───────────────────┼───────┤
>   │ [32,64)     │ 14,290            │ 28,823            │ ×2.0  │
>   ├─────────────┼───────────────────┼───────────────────┼───────┤
>   │ [64,128)    │ 2,549             │ 3,655             │ +43%  │
>   ├─────────────┼───────────────────┼───────────────────┼───────┤
>   │ [128,256)   │ 555               │ 783               │ +41%  │
>   ├─────────────┼───────────────────┼───────────────────┼───────┤
>   │ [256,512)   │ 456               │ 625               │ +37%  │
>   ├─────────────┼───────────────────┼───────────────────┼───────┤
>   │ [512µs+)    │ 13                │ 8                 │ ≈     │
>   ├─────────────┼───────────────────┼───────────────────┼───────┤
>   │ max         │ 1.07ms            │ 1.56ms            │ +46%  │
>   ├─────────────┼───────────────────┼───────────────────┼───────┤
>   │ total walks │ 8,777,514         │ 8,723,453         │ −0.6% │
>   └─────────────┴───────────────────┴───────────────────┴───────┘
> 
>   (192-CPU PREEMPT_RT host, 12 concurrent gfn_to_pfn_cache invalidation
>   reproducers, 300-second windows; each "walk" is one invalidation drain
>   which includes the synchronize_srcu_atomic() call. #10/#11 are just
>   local build numbers.)
> 

Thanks, David, for testing this with your KVM workload and confirming 
the current approach works well.

I’ll keep looking into the fastpath for synchronize_srcu_expedited() 
as well, while preserving the existing SRCU semantics.

Thanks,
KunWu

Re: [PATCH 04/13] rcutorture: Add atomic-SRCU support to torture.sh

[Paul E. McKenney]

On Tue, Sep 08, 2026 at 04:34:41PM -0700, Paul E. McKenney wrote:
> On Mon, Sep 07, 2026 at 03:58:20PM +0800, Kunwu Chan wrote:
> > From: Kunwu Chan <kunwu.chan@gmail.com>
> > 
> > Add the --do-atomic-srcu argument to torture.sh, which runs the
> > SRCU-N, SRCU-P, and SRCU-T scenarios, thus covering both Tree SRCU
> > (SRCU-N and SRCU-P) and Tiny SRCU (SRCU-T), with
> > rcutorture.reader_flavor=0x10 appended to the boot parameters so
> > that it takes precedence over each scenario's own reader-flavor
> > setting.  This exercises srcu_read_lock_atomic(),
> > srcu_read_unlock_atomic(), and synchronize_srcu_atomic().
> > 
> > As with other torture.sh tests, the --do-kcsan argument runs a
> > KCSAN+PROVE_LOCKING variant of this test.
> > 
> > Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
> 
> Queued for testing and review, thank you!

I did take the liberty of changing "do_atomic_srcu=no" to
"do_atomic_srcu=yes" in order to increase test coverage.

							Thanx, Paul

> > ---
> >  .../selftests/rcutorture/bin/torture.sh       | 24 +++++++++++++++++++
> >  1 file changed, 24 insertions(+)
> > 
> > diff --git a/tools/testing/selftests/rcutorture/bin/torture.sh b/tools/testing/selftests/rcutorture/bin/torture.sh
> > index f0083891ee81..8885812d866a 100755
> > --- a/tools/testing/selftests/rcutorture/bin/torture.sh
> > +++ b/tools/testing/selftests/rcutorture/bin/torture.sh
> > @@ -68,6 +68,7 @@ do_clocksourcewd="${ifnotaarch64}"
> >  do_rt=yes
> >  do_rcutasksflavors="${ifnotaarch64}" # FIXME: Back to "yes" when SMP=n auto-avoided
> >  do_srcu_lockdep=yes
> > +do_atomic_srcu=no
> >  do_rcu_rust=no
> >  
> >  # doyesno - Helper function for yes/no arguments
> > @@ -103,6 +104,7 @@ usage () {
> >  	echo "       --do-rcu-rust / --do-no-rcu-rust / --no-rcu-rust"
> >  	echo "       --do-scftorture / --do-no-scftorture / --no-scftorture"
> >  	echo "       --do-srcu-lockdep / --do-no-srcu-lockdep / --no-srcu-lockdep"
> > +	echo "       --do-atomic-srcu / --do-no-atomic-srcu / --no-atomic-srcu"
> >  	echo "       --duration [ <minutes> | <hours>h | <days>d ]"
> >  	echo "       --guest-cpu-limit N"
> >  	echo "       --kcsan-kmake-arg kernel-make-arguments"
> > @@ -148,6 +150,7 @@ do
> >  		do_kcsan=yes
> >  		do_clocksourcewd="${ifnotaarch64}"
> >  		do_srcu_lockdep=yes
> > +		do_atomic_srcu=yes
> >  		;;
> >  	--do-allmodconfig|--do-no-allmodconfig|--no-allmodconfig)
> >  		do_allmodconfig=`doyesno "$1" --do-allmodconfig`
> > @@ -183,6 +186,7 @@ do
> >  		do_kcsan=no
> >  		do_clocksourcewd=no
> >  		do_srcu_lockdep=no
> > +		do_atomic_srcu=no
> >  		;;
> >  	--do-normal|--do-norm|--do-no-normal|--do-no-norm|--no-normal|--no-norm)
> >  		do_normal=`doyesno "$1" --do-normal`
> > @@ -212,6 +216,9 @@ do
> >  	--do-srcu-lockdep|--do-no-srcu-lockdep|--no-srcu-lockdep)
> >  		do_srcu_lockdep=`doyesno "$1" --do-srcu-lockdep`
> >  		;;
> > +	--do-atomic-srcu|--do-no-atomic-srcu|--no-atomic-srcu)
> > +		do_atomic_srcu=`doyesno "$1" --do-atomic-srcu`
> > +		;;
> >  	--duration)
> >  		checkarg --duration "(minutes)" $# "$2" '^[0-9][0-9]*\(m\|h\|d\|\)$' '^error'
> >  		mult=1
> > @@ -497,6 +504,23 @@ then
> >  	torture_set "rcutorture" tools/testing/selftests/rcutorture/bin/kvm.sh --allcpus --duration "$duration_rcutorture" --configs "$configs_rcutorture" --trust-make
> >  fi
> >  
> > +# Test atomic SRCU across Tree SRCU (SRCU-N and SRCU-P) and Tiny SRCU
> > +# (SRCU-T).  The reader flavor selects srcu_read_lock_atomic() and
> > +# synchronize_srcu_atomic().  Tiny SRCU requires SMP=n, which aarch64
> > +# does not support.
> > +if test "$do_atomic_srcu" = "yes"
> > +then
> > +	torture_bootargs="rcutorture.reader_flavor=0x10"
> > +	configs_atomic_srcu="SRCU-N SRCU-P"
> > +	if test "$ifnotaarch64" = yes
> > +	then
> > +		configs_atomic_srcu="$configs_atomic_srcu SRCU-T"
> > +	fi
> > +	torture_set "atomic-srcu" tools/testing/selftests/rcutorture/bin/kvm.sh \
> > +		--allcpus --duration "$duration_rcutorture" \
> > +		--configs "$configs_atomic_srcu" --trust-make
> > +fi
> > +
> >  if test "$do_locktorture" = "yes"
> >  then
> >  	torture_bootargs="torture.disable_onoff_at_boot"
> > -- 
> > 2.43.0
> > 

Re: [PATCH 06/13] srcutree: Make init_srcu_struct_atomic() prevent transition to big

[Paul E. McKenney]

On Wed, Sep 09, 2026 at 10:34:16AM +0800, Kunwu Chan wrote:
> On Tue, 8 Sep 2026 16:36:11 -0700 "Paul E. McKenney" <paulmck@kernel.org> wrote:
> 
> > On Mon, Sep 07, 2026 at 03:58:22PM +0800, Kunwu Chan wrote:
> > > From: Kunwu Chan <kunwu.chan@gmail.com>
> > > 
> > > The is_atomic parameter of init_srcu_struct_fields() exists so that
> > > atomic SRCU never transitions to big, but neither
> > > init_srcu_struct_atomic() nor its lockdep counterpart
> > > __init_srcu_struct_atomic() sets it.
> > > 
> > > On systems where srcutree.convert_to_big selects SRCU_SIZING_INIT,
> > > this needlessly allocates a full srcu_node combining tree for any
> > > dynamically initialized atomic srcu_struct, despite atomic SRCU
> > > having neither callbacks nor srcu_barrier() operations.
> > > 
> > > Pass true from both atomic entry points, adding an is_atomic parameter
> > > to __init_srcu_struct_common().
> > > 
> > > Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
> > 
> > Given that we have this commit, is this patch needed?
> > 
> > 4e01d5320a2f ("srcutree: Suppress to-big transition for atomic SRCU")
> > 
> > If so, please tell me what I am missing.
> 
> Yes, this patch is still needed. The is_atomic mechanism is there, but 
> the atomic initialization paths still pass false.
> 
> In particular, init_srcu_struct_atomic() passes false directly, while the 
> lockdep path goes through __init_srcu_struct_common(), which also passes false.
> 
> This patch makes both atomic entry points propagate true.

Very good, queued and pushed!

							Thanx, Paul

> Thanks,
> KunWu
> 
> > 
> > 						Thanx, Paul
> > 
> > > ---
> > >  kernel/rcu/srcutree.c | 15 ++++++++-------
> > >  1 file changed, 8 insertions(+), 7 deletions(-)
> > > 
> > > diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c
> > > index 2af36db37fa9..01f224a56b41 100644
> > > --- a/kernel/rcu/srcutree.c
> > > +++ b/kernel/rcu/srcutree.c
> > > @@ -305,26 +305,27 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp, bool is_static, bool
> > >  #ifdef CONFIG_DEBUG_LOCK_ALLOC
> > >  
> > >  static int
> > > -__init_srcu_struct_common(struct srcu_struct *ssp, const char *name, struct lock_class_key *key)
> > > +__init_srcu_struct_common(struct srcu_struct *ssp, const char *name,
> > > +			  struct lock_class_key *key, bool is_atomic)
> > >  {
> > >  	/* Don't re-initialize a lock while it is held. */
> > >  	debug_check_no_locks_freed((void *)ssp, sizeof(*ssp));
> > >  	lockdep_init_map(&ssp->dep_map, name, key, 0);
> > > -	return init_srcu_struct_fields(ssp, false, false);
> > > +	return init_srcu_struct_fields(ssp, false, is_atomic);
> > >  }
> > >  
> > >  int init_srcu_struct_lockdep(struct srcu_struct *ssp, const char *name,
> > >  			     struct lock_class_key *key)
> > >  {
> > >  	ssp->srcu_reader_flavor = 0;
> > > -	return __init_srcu_struct_common(ssp, name, key);
> > > +	return __init_srcu_struct_common(ssp, name, key, false);
> > >  }
> > >  EXPORT_SYMBOL_GPL(init_srcu_struct_lockdep);
> > >  
> > >  int __init_srcu_struct_fast(struct srcu_struct *ssp, const char *name, struct lock_class_key *key)
> > >  {
> > >  	ssp->srcu_reader_flavor = SRCU_READ_FLAVOR_FAST;
> > > -	return __init_srcu_struct_common(ssp, name, key);
> > > +	return __init_srcu_struct_common(ssp, name, key, false);
> > >  }
> > >  EXPORT_SYMBOL_GPL(__init_srcu_struct_fast);
> > >  
> > > @@ -332,14 +333,14 @@ int __init_srcu_struct_fast_updown(struct srcu_struct *ssp, const char *name,
> > >  				   struct lock_class_key *key)
> > >  {
> > >  	ssp->srcu_reader_flavor = SRCU_READ_FLAVOR_FAST_UPDOWN;
> > > -	return __init_srcu_struct_common(ssp, name, key);
> > > +	return __init_srcu_struct_common(ssp, name, key, false);
> > >  }
> > >  EXPORT_SYMBOL_GPL(__init_srcu_struct_fast_updown);
> > >  
> > >  int __init_srcu_struct_atomic(struct srcu_struct *ssp, const char *name, struct lock_class_key *key)
> > >  {
> > >  	ssp->srcu_reader_flavor = SRCU_READ_FLAVOR_ATOMIC;
> > > -	return __init_srcu_struct_common(ssp, name, key);
> > > +	return __init_srcu_struct_common(ssp, name, key, true);
> > >  }
> > >  EXPORT_SYMBOL_GPL(__init_srcu_struct_atomic);
> > >  
> > > @@ -414,7 +415,7 @@ EXPORT_SYMBOL_GPL(init_srcu_struct_fast_updown);
> > >  int init_srcu_struct_atomic(struct srcu_struct *ssp)
> > >  {
> > >  	ssp->srcu_reader_flavor = SRCU_READ_FLAVOR_ATOMIC;
> > > -	return init_srcu_struct_fields(ssp, false, false);
> > > +	return init_srcu_struct_fields(ssp, false, true);
> > >  }
> > >  EXPORT_SYMBOL_GPL(init_srcu_struct_atomic);
> > >  
> > > -- 
> > > 2.43.0
> > > 
> > 
> 

Re: [PATCH 07/13] srcutree: Don't transition atomic SRCU to big in srcu_gp_end()

[Paul E. McKenney]

On Wed, Sep 09, 2026 at 10:45:14AM +0800, Kunwu Chan wrote:
> On Tue, 8 Sep 2026 16:38:19 -0700 "Paul E. McKenney" <paulmck@kernel.org> wrote:
> 
> > On Mon, Sep 07, 2026 at 03:58:23PM +0800, Kunwu Chan wrote:
> > > From: Kunwu Chan <kunwu.chan@gmail.com>
> > > 
> > > The transition-to-big code path in srcu_gp_end() calls
> > > init_srcu_struct_nodes() with GFP_KERNEL, which is illegal in the
> > > atomic context reachable from synchronize_srcu_atomic().  Atomic SRCU
> > > has no use for the srcu_node combining tree: it has neither callbacks
> > > nor srcu_barrier() operations, and its grace periods are serialized
> > > by ->srcu_atomic_gp_flag instead.
> > > 
> > > Skip this transition entirely for atomic SRCU, which is also defense
> > > in depth against any path that might wrongly set ->srcu_size_state for
> > > an atomic srcu_struct.
> > > 
> > > Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
> > 
> > Again, given that we have this commit, is this patch needed?
> > 
> > 4e01d5320a2f ("srcutree: Suppress to-big transition for atomic SRCU")
> > 
> > And again, if so, please tell me what I am missing.
> 
> Yes, this is a separate transition path. That commit suppresses the 
> initialization-time transition, while this patch prevents the transition 
> from srcu_gp_end(), which can otherwise call 
> init_srcu_struct_nodes(..., GFP_KERNEL) for an atomic SRCU.
> 
> So this is defense in depth for the atomic path.

Fair point.  But is there any additional pathway other than the
SRCU_SIZING_IS_TORTURE() check at the end of srcu_torture_stats_print()?

							Thanx, Paul

> Thanks,
> KunWu
> 
> > 
> > 							Thanx, Paul
> > 
> > > ---
> > >  kernel/rcu/srcutree.c | 5 +++--
> > >  1 file changed, 3 insertions(+), 2 deletions(-)
> > > 
> > > diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c
> > > index 01f224a56b41..74acd5645d4c 100644
> > > --- a/kernel/rcu/srcutree.c
> > > +++ b/kernel/rcu/srcutree.c
> > > @@ -1074,8 +1074,9 @@ static void srcu_gp_end(struct srcu_struct *ssp, bool is_atomic)
> > >  		raw_spin_unlock_irq_rcu_node(sup);
> > >  	}
> > >  
> > > -	/* Transition to big if needed. */
> > > -	if (ss_state != SRCU_SIZE_SMALL && ss_state != SRCU_SIZE_BIG) {
> > > +	/* Transition to big if needed, but never for atomic SRCU. */
> > > +	if (!is_atomic && ss_state != SRCU_SIZE_SMALL &&
> > > +	    ss_state != SRCU_SIZE_BIG) {
> > >  		if (ss_state == SRCU_SIZE_ALLOC)
> > >  			init_srcu_struct_nodes(ssp, GFP_KERNEL);
> > >  		else
> > > -- 
> > > 2.43.0
> > > 
> > 
> 
> Sent using hkml (https://github.com/sjp38/hackermail)

Re: [PATCH 04/13] rcutorture: Add atomic-SRCU support to torture.sh

[KunWu Chan]

On Thu, Sep 10, 2026 at 6:35 AM Paul E. McKenney <paulmck@kernel.org> wrote:
>
> On Tue, Sep 08, 2026 at 04:34:41PM -0700, Paul E. McKenney wrote:
> > On Mon, Sep 07, 2026 at 03:58:20PM +0800, Kunwu Chan wrote:
> > > From: Kunwu Chan <kunwu.chan@gmail.com>
> > >
> > > Add the --do-atomic-srcu argument to torture.sh, which runs the
> > > SRCU-N, SRCU-P, and SRCU-T scenarios, thus covering both Tree SRCU
> > > (SRCU-N and SRCU-P) and Tiny SRCU (SRCU-T), with
> > > rcutorture.reader_flavor=0x10 appended to the boot parameters so
> > > that it takes precedence over each scenario's own reader-flavor
> > > setting.  This exercises srcu_read_lock_atomic(),
> > > srcu_read_unlock_atomic(), and synchronize_srcu_atomic().
> > >
> > > As with other torture.sh tests, the --do-kcsan argument runs a
> > > KCSAN+PROVE_LOCKING variant of this test.
> > >
> > > Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
> >
> > Queued for testing and review, thank you!
>
> I did take the liberty of changing "do_atomic_srcu=no" to
> "do_atomic_srcu=yes" in order to increase test coverage.
>

Thanks, no problem.

I also found a couple of data races in other subsystems while running
the tests. I’m still validating the fixes locally and will Cc you if I
send patches for them.

Thanks,
KunWu

>                                                         Thanx, Paul
>
> > > ---
> > >  .../selftests/rcutorture/bin/torture.sh       | 24 +++++++++++++++++++
> > >  1 file changed, 24 insertions(+)
> > >
> > > diff --git a/tools/testing/selftests/rcutorture/bin/torture.sh b/tools/testing/selftests/rcutorture/bin/torture.sh
> > > index f0083891ee81..8885812d866a 100755
> > > --- a/tools/testing/selftests/rcutorture/bin/torture.sh
> > > +++ b/tools/testing/selftests/rcutorture/bin/torture.sh
> > > @@ -68,6 +68,7 @@ do_clocksourcewd="${ifnotaarch64}"
> > >  do_rt=yes
> > >  do_rcutasksflavors="${ifnotaarch64}" # FIXME: Back to "yes" when SMP=n auto-avoided
> > >  do_srcu_lockdep=yes
> > > +do_atomic_srcu=no
> > >  do_rcu_rust=no
> > >
> > >  # doyesno - Helper function for yes/no arguments
> > > @@ -103,6 +104,7 @@ usage () {
> > >     echo "       --do-rcu-rust / --do-no-rcu-rust / --no-rcu-rust"
> > >     echo "       --do-scftorture / --do-no-scftorture / --no-scftorture"
> > >     echo "       --do-srcu-lockdep / --do-no-srcu-lockdep / --no-srcu-lockdep"
> > > +   echo "       --do-atomic-srcu / --do-no-atomic-srcu / --no-atomic-srcu"
> > >     echo "       --duration [ <minutes> | <hours>h | <days>d ]"
> > >     echo "       --guest-cpu-limit N"
> > >     echo "       --kcsan-kmake-arg kernel-make-arguments"
> > > @@ -148,6 +150,7 @@ do
> > >             do_kcsan=yes
> > >             do_clocksourcewd="${ifnotaarch64}"
> > >             do_srcu_lockdep=yes
> > > +           do_atomic_srcu=yes
> > >             ;;
> > >     --do-allmodconfig|--do-no-allmodconfig|--no-allmodconfig)
> > >             do_allmodconfig=`doyesno "$1" --do-allmodconfig`
> > > @@ -183,6 +186,7 @@ do
> > >             do_kcsan=no
> > >             do_clocksourcewd=no
> > >             do_srcu_lockdep=no
> > > +           do_atomic_srcu=no
> > >             ;;
> > >     --do-normal|--do-norm|--do-no-normal|--do-no-norm|--no-normal|--no-norm)
> > >             do_normal=`doyesno "$1" --do-normal`
> > > @@ -212,6 +216,9 @@ do
> > >     --do-srcu-lockdep|--do-no-srcu-lockdep|--no-srcu-lockdep)
> > >             do_srcu_lockdep=`doyesno "$1" --do-srcu-lockdep`
> > >             ;;
> > > +   --do-atomic-srcu|--do-no-atomic-srcu|--no-atomic-srcu)
> > > +           do_atomic_srcu=`doyesno "$1" --do-atomic-srcu`
> > > +           ;;
> > >     --duration)
> > >             checkarg --duration "(minutes)" $# "$2" '^[0-9][0-9]*\(m\|h\|d\|\)$' '^error'
> > >             mult=1
> > > @@ -497,6 +504,23 @@ then
> > >     torture_set "rcutorture" tools/testing/selftests/rcutorture/bin/kvm.sh --allcpus --duration "$duration_rcutorture" --configs "$configs_rcutorture" --trust-make
> > >  fi
> > >
> > > +# Test atomic SRCU across Tree SRCU (SRCU-N and SRCU-P) and Tiny SRCU
> > > +# (SRCU-T).  The reader flavor selects srcu_read_lock_atomic() and
> > > +# synchronize_srcu_atomic().  Tiny SRCU requires SMP=n, which aarch64
> > > +# does not support.
> > > +if test "$do_atomic_srcu" = "yes"
> > > +then
> > > +   torture_bootargs="rcutorture.reader_flavor=0x10"
> > > +   configs_atomic_srcu="SRCU-N SRCU-P"
> > > +   if test "$ifnotaarch64" = yes
> > > +   then
> > > +           configs_atomic_srcu="$configs_atomic_srcu SRCU-T"
> > > +   fi
> > > +   torture_set "atomic-srcu" tools/testing/selftests/rcutorture/bin/kvm.sh \
> > > +           --allcpus --duration "$duration_rcutorture" \
> > > +           --configs "$configs_atomic_srcu" --trust-make
> > > +fi
> > > +
> > >  if test "$do_locktorture" = "yes"
> > >  then
> > >     torture_bootargs="torture.disable_onoff_at_boot"
> > > --
> > > 2.43.0
> > >

Re: [PATCH 07/13] srcutree: Don't transition atomic SRCU to big in srcu_gp_end()

[KunWu Chan]

On Thu, Sep 10, 2026 at 8:13 AM Paul E. McKenney <paulmck@kernel.org> wrote:
>
> On Wed, Sep 09, 2026 at 10:45:14AM +0800, Kunwu Chan wrote:
> > On Tue, 8 Sep 2026 16:38:19 -0700 "Paul E. McKenney" <paulmck@kernel.org> wrote:
> >
> > > On Mon, Sep 07, 2026 at 03:58:23PM +0800, Kunwu Chan wrote:
> > > > From: Kunwu Chan <kunwu.chan@gmail.com>
> > > >
> > > > The transition-to-big code path in srcu_gp_end() calls
> > > > init_srcu_struct_nodes() with GFP_KERNEL, which is illegal in the
> > > > atomic context reachable from synchronize_srcu_atomic().  Atomic SRCU
> > > > has no use for the srcu_node combining tree: it has neither callbacks
> > > > nor srcu_barrier() operations, and its grace periods are serialized
> > > > by ->srcu_atomic_gp_flag instead.
> > > >
> > > > Skip this transition entirely for atomic SRCU, which is also defense
> > > > in depth against any path that might wrongly set ->srcu_size_state for
> > > > an atomic srcu_struct.
> > > >
> > > > Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
> > >
> > > Again, given that we have this commit, is this patch needed?
> > >
> > > 4e01d5320a2f ("srcutree: Suppress to-big transition for atomic SRCU")
> > >
> > > And again, if so, please tell me what I am missing.
> >
> > Yes, this is a separate transition path. That commit suppresses the
> > initialization-time transition, while this patch prevents the transition
> > from srcu_gp_end(), which can otherwise call
> > init_srcu_struct_nodes(..., GFP_KERNEL) for an atomic SRCU.
> >
> > So this is defense in depth for the atomic path.
>
> Fair point.  But is there any additional pathway other than the
> SRCU_SIZING_IS_TORTURE() check at the end of srcu_torture_stats_print()?

I checked all transition-to-big pathways for atomic SRCU.
Atomic SRCU can reach srcu_gp_end(), but its srcu_size_state remains
SRCU_SIZE_SMALL,
so the transition condition there cannot be satisfied.

The contention path is also unreachable:
synchronize_srcu_atomic() calls srcu_gp_start() directly and never
goes through srcu_gp_start_if_needed().
The other callers of that path reject atomic SRCU.

The only remaining explicit transition is SRCU_SIZING_IS_TORTURE() in
srcu_torture_stats_print(),
which currently lacks an atomic exclusion. I'll send a follow-up patch
to add a srcu_reader_flavor check there.

So I agree the srcu_gp_end() patch can be dropped.

Thanks,
Kunwu

>
>                                                         Thanx, Paul
>
> > Thanks,
> > KunWu
> >
> > >
> > >                                                     Thanx, Paul
> > >
> > > > ---
> > > >  kernel/rcu/srcutree.c | 5 +++--
> > > >  1 file changed, 3 insertions(+), 2 deletions(-)
> > > >
> > > > diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c
> > > > index 01f224a56b41..74acd5645d4c 100644
> > > > --- a/kernel/rcu/srcutree.c
> > > > +++ b/kernel/rcu/srcutree.c
> > > > @@ -1074,8 +1074,9 @@ static void srcu_gp_end(struct srcu_struct *ssp, bool is_atomic)
> > > >           raw_spin_unlock_irq_rcu_node(sup);
> > > >   }
> > > >
> > > > - /* Transition to big if needed. */
> > > > - if (ss_state != SRCU_SIZE_SMALL && ss_state != SRCU_SIZE_BIG) {
> > > > + /* Transition to big if needed, but never for atomic SRCU. */
> > > > + if (!is_atomic && ss_state != SRCU_SIZE_SMALL &&
> > > > +     ss_state != SRCU_SIZE_BIG) {
> > > >           if (ss_state == SRCU_SIZE_ALLOC)
> > > >                   init_srcu_struct_nodes(ssp, GFP_KERNEL);
> > > >           else
> > > > --
> > > > 2.43.0
> > > >
> > >
> >
> > Sent using hkml (https://github.com/sjp38/hackermail)

Re: [PATCH 07/13] srcutree: Don't transition atomic SRCU to big in srcu_gp_end()

[Paul E. McKenney]

On Thu, Sep 10, 2026 at 11:18:20AM +0800, KunWu Chan wrote:
> On Thu, Sep 10, 2026 at 8:13 AM Paul E. McKenney <paulmck@kernel.org> wrote:
> >
> > On Wed, Sep 09, 2026 at 10:45:14AM +0800, Kunwu Chan wrote:
> > > On Tue, 8 Sep 2026 16:38:19 -0700 "Paul E. McKenney" <paulmck@kernel.org> wrote:
> > >
> > > > On Mon, Sep 07, 2026 at 03:58:23PM +0800, Kunwu Chan wrote:
> > > > > From: Kunwu Chan <kunwu.chan@gmail.com>
> > > > >
> > > > > The transition-to-big code path in srcu_gp_end() calls
> > > > > init_srcu_struct_nodes() with GFP_KERNEL, which is illegal in the
> > > > > atomic context reachable from synchronize_srcu_atomic().  Atomic SRCU
> > > > > has no use for the srcu_node combining tree: it has neither callbacks
> > > > > nor srcu_barrier() operations, and its grace periods are serialized
> > > > > by ->srcu_atomic_gp_flag instead.
> > > > >
> > > > > Skip this transition entirely for atomic SRCU, which is also defense
> > > > > in depth against any path that might wrongly set ->srcu_size_state for
> > > > > an atomic srcu_struct.
> > > > >
> > > > > Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
> > > >
> > > > Again, given that we have this commit, is this patch needed?
> > > >
> > > > 4e01d5320a2f ("srcutree: Suppress to-big transition for atomic SRCU")
> > > >
> > > > And again, if so, please tell me what I am missing.
> > >
> > > Yes, this is a separate transition path. That commit suppresses the
> > > initialization-time transition, while this patch prevents the transition
> > > from srcu_gp_end(), which can otherwise call
> > > init_srcu_struct_nodes(..., GFP_KERNEL) for an atomic SRCU.
> > >
> > > So this is defense in depth for the atomic path.
> >
> > Fair point.  But is there any additional pathway other than the
> > SRCU_SIZING_IS_TORTURE() check at the end of srcu_torture_stats_print()?
> 
> I checked all transition-to-big pathways for atomic SRCU.
> Atomic SRCU can reach srcu_gp_end(), but its srcu_size_state remains
> SRCU_SIZE_SMALL,
> so the transition condition there cannot be satisfied.
> 
> The contention path is also unreachable:
> synchronize_srcu_atomic() calls srcu_gp_start() directly and never
> goes through srcu_gp_start_if_needed().
> The other callers of that path reject atomic SRCU.
> 
> The only remaining explicit transition is SRCU_SIZING_IS_TORTURE() in
> srcu_torture_stats_print(),
> which currently lacks an atomic exclusion. I'll send a follow-up patch
> to add a srcu_reader_flavor check there.

Very good!

> So I agree the srcu_gp_end() patch can be dropped.

It might not hurt to have a WARN_ON_ONCE() there, just in case someone
implements another pathway.

							Thanx, Paul

> Thanks,
> Kunwu
> 
> >
> >                                                         Thanx, Paul
> >
> > > Thanks,
> > > KunWu
> > >
> > > >
> > > >                                                     Thanx, Paul
> > > >
> > > > > ---
> > > > >  kernel/rcu/srcutree.c | 5 +++--
> > > > >  1 file changed, 3 insertions(+), 2 deletions(-)
> > > > >
> > > > > diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c
> > > > > index 01f224a56b41..74acd5645d4c 100644
> > > > > --- a/kernel/rcu/srcutree.c
> > > > > +++ b/kernel/rcu/srcutree.c
> > > > > @@ -1074,8 +1074,9 @@ static void srcu_gp_end(struct srcu_struct *ssp, bool is_atomic)
> > > > >           raw_spin_unlock_irq_rcu_node(sup);
> > > > >   }
> > > > >
> > > > > - /* Transition to big if needed. */
> > > > > - if (ss_state != SRCU_SIZE_SMALL && ss_state != SRCU_SIZE_BIG) {
> > > > > + /* Transition to big if needed, but never for atomic SRCU. */
> > > > > + if (!is_atomic && ss_state != SRCU_SIZE_SMALL &&
> > > > > +     ss_state != SRCU_SIZE_BIG) {
> > > > >           if (ss_state == SRCU_SIZE_ALLOC)
> > > > >                   init_srcu_struct_nodes(ssp, GFP_KERNEL);
> > > > >           else
> > > > > --
> > > > > 2.43.0
> > > > >
> > > >
> > >
> > > Sent using hkml (https://github.com/sjp38/hackermail)

Re: [PATCH 04/13] rcutorture: Add atomic-SRCU support to torture.sh

[Paul E. McKenney]

On Thu, Sep 10, 2026 at 09:30:48AM +0800, KunWu Chan wrote:
> On Thu, Sep 10, 2026 at 6:35 AM Paul E. McKenney <paulmck@kernel.org> wrote:
> >
> > On Tue, Sep 08, 2026 at 04:34:41PM -0700, Paul E. McKenney wrote:
> > > On Mon, Sep 07, 2026 at 03:58:20PM +0800, Kunwu Chan wrote:
> > > > From: Kunwu Chan <kunwu.chan@gmail.com>
> > > >
> > > > Add the --do-atomic-srcu argument to torture.sh, which runs the
> > > > SRCU-N, SRCU-P, and SRCU-T scenarios, thus covering both Tree SRCU
> > > > (SRCU-N and SRCU-P) and Tiny SRCU (SRCU-T), with
> > > > rcutorture.reader_flavor=0x10 appended to the boot parameters so
> > > > that it takes precedence over each scenario's own reader-flavor
> > > > setting.  This exercises srcu_read_lock_atomic(),
> > > > srcu_read_unlock_atomic(), and synchronize_srcu_atomic().
> > > >
> > > > As with other torture.sh tests, the --do-kcsan argument runs a
> > > > KCSAN+PROVE_LOCKING variant of this test.
> > > >
> > > > Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
> > >
> > > Queued for testing and review, thank you!
> >
> > I did take the liberty of changing "do_atomic_srcu=no" to
> > "do_atomic_srcu=yes" in order to increase test coverage.
> >
> 
> Thanks, no problem.
> 
> I also found a couple of data races in other subsystems while running
> the tests. I’m still validating the fixes locally and will Cc you if I
> send patches for them.

Sounds good!  Note that some maintainers are more friendly to such
patches than others.  But it shouldn't hurt to send them.  ;-)

Just please carefully check the design.  For example, blindly applying
READ_ONCE() and WRITE_ONCE() is not a strategy to win.

							Thanx, Paul

> Thanks,
> KunWu
> 
> >                                                         Thanx, Paul
> >
> > > > ---
> > > >  .../selftests/rcutorture/bin/torture.sh       | 24 +++++++++++++++++++
> > > >  1 file changed, 24 insertions(+)
> > > >
> > > > diff --git a/tools/testing/selftests/rcutorture/bin/torture.sh b/tools/testing/selftests/rcutorture/bin/torture.sh
> > > > index f0083891ee81..8885812d866a 100755
> > > > --- a/tools/testing/selftests/rcutorture/bin/torture.sh
> > > > +++ b/tools/testing/selftests/rcutorture/bin/torture.sh
> > > > @@ -68,6 +68,7 @@ do_clocksourcewd="${ifnotaarch64}"
> > > >  do_rt=yes
> > > >  do_rcutasksflavors="${ifnotaarch64}" # FIXME: Back to "yes" when SMP=n auto-avoided
> > > >  do_srcu_lockdep=yes
> > > > +do_atomic_srcu=no
> > > >  do_rcu_rust=no
> > > >
> > > >  # doyesno - Helper function for yes/no arguments
> > > > @@ -103,6 +104,7 @@ usage () {
> > > >     echo "       --do-rcu-rust / --do-no-rcu-rust / --no-rcu-rust"
> > > >     echo "       --do-scftorture / --do-no-scftorture / --no-scftorture"
> > > >     echo "       --do-srcu-lockdep / --do-no-srcu-lockdep / --no-srcu-lockdep"
> > > > +   echo "       --do-atomic-srcu / --do-no-atomic-srcu / --no-atomic-srcu"
> > > >     echo "       --duration [ <minutes> | <hours>h | <days>d ]"
> > > >     echo "       --guest-cpu-limit N"
> > > >     echo "       --kcsan-kmake-arg kernel-make-arguments"
> > > > @@ -148,6 +150,7 @@ do
> > > >             do_kcsan=yes
> > > >             do_clocksourcewd="${ifnotaarch64}"
> > > >             do_srcu_lockdep=yes
> > > > +           do_atomic_srcu=yes
> > > >             ;;
> > > >     --do-allmodconfig|--do-no-allmodconfig|--no-allmodconfig)
> > > >             do_allmodconfig=`doyesno "$1" --do-allmodconfig`
> > > > @@ -183,6 +186,7 @@ do
> > > >             do_kcsan=no
> > > >             do_clocksourcewd=no
> > > >             do_srcu_lockdep=no
> > > > +           do_atomic_srcu=no
> > > >             ;;
> > > >     --do-normal|--do-norm|--do-no-normal|--do-no-norm|--no-normal|--no-norm)
> > > >             do_normal=`doyesno "$1" --do-normal`
> > > > @@ -212,6 +216,9 @@ do
> > > >     --do-srcu-lockdep|--do-no-srcu-lockdep|--no-srcu-lockdep)
> > > >             do_srcu_lockdep=`doyesno "$1" --do-srcu-lockdep`
> > > >             ;;
> > > > +   --do-atomic-srcu|--do-no-atomic-srcu|--no-atomic-srcu)
> > > > +           do_atomic_srcu=`doyesno "$1" --do-atomic-srcu`
> > > > +           ;;
> > > >     --duration)
> > > >             checkarg --duration "(minutes)" $# "$2" '^[0-9][0-9]*\(m\|h\|d\|\)$' '^error'
> > > >             mult=1
> > > > @@ -497,6 +504,23 @@ then
> > > >     torture_set "rcutorture" tools/testing/selftests/rcutorture/bin/kvm.sh --allcpus --duration "$duration_rcutorture" --configs "$configs_rcutorture" --trust-make
> > > >  fi
> > > >
> > > > +# Test atomic SRCU across Tree SRCU (SRCU-N and SRCU-P) and Tiny SRCU
> > > > +# (SRCU-T).  The reader flavor selects srcu_read_lock_atomic() and
> > > > +# synchronize_srcu_atomic().  Tiny SRCU requires SMP=n, which aarch64
> > > > +# does not support.
> > > > +if test "$do_atomic_srcu" = "yes"
> > > > +then
> > > > +   torture_bootargs="rcutorture.reader_flavor=0x10"
> > > > +   configs_atomic_srcu="SRCU-N SRCU-P"
> > > > +   if test "$ifnotaarch64" = yes
> > > > +   then
> > > > +           configs_atomic_srcu="$configs_atomic_srcu SRCU-T"
> > > > +   fi
> > > > +   torture_set "atomic-srcu" tools/testing/selftests/rcutorture/bin/kvm.sh \
> > > > +           --allcpus --duration "$duration_rcutorture" \
> > > > +           --configs "$configs_atomic_srcu" --trust-make
> > > > +fi
> > > > +
> > > >  if test "$do_locktorture" = "yes"
> > > >  then
> > > >     torture_bootargs="torture.disable_onoff_at_boot"
> > > > --
> > > > 2.43.0
> > > >

Re: [PATCH 07/13] srcutree: Don't transition atomic SRCU to big in srcu_gp_end()

[KunWu Chan]

On Thu, Sep 10, 2026 at 11:46 AM Paul E. McKenney <paulmck@kernel.org> wrote:
>
> On Thu, Sep 10, 2026 at 11:18:20AM +0800, KunWu Chan wrote:
> > On Thu, Sep 10, 2026 at 8:13 AM Paul E. McKenney <paulmck@kernel.org> wrote:
> > >
> > > On Wed, Sep 09, 2026 at 10:45:14AM +0800, Kunwu Chan wrote:
> > > > On Tue, 8 Sep 2026 16:38:19 -0700 "Paul E. McKenney" <paulmck@kernel.org> wrote:
> > > >
> > > > > On Mon, Sep 07, 2026 at 03:58:23PM +0800, Kunwu Chan wrote:
> > > > > > From: Kunwu Chan <kunwu.chan@gmail.com>
> > > > > >
> > > > > > The transition-to-big code path in srcu_gp_end() calls
> > > > > > init_srcu_struct_nodes() with GFP_KERNEL, which is illegal in the
> > > > > > atomic context reachable from synchronize_srcu_atomic().  Atomic SRCU
> > > > > > has no use for the srcu_node combining tree: it has neither callbacks
> > > > > > nor srcu_barrier() operations, and its grace periods are serialized
> > > > > > by ->srcu_atomic_gp_flag instead.
> > > > > >
> > > > > > Skip this transition entirely for atomic SRCU, which is also defense
> > > > > > in depth against any path that might wrongly set ->srcu_size_state for
> > > > > > an atomic srcu_struct.
> > > > > >
> > > > > > Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
> > > > >
> > > > > Again, given that we have this commit, is this patch needed?
> > > > >
> > > > > 4e01d5320a2f ("srcutree: Suppress to-big transition for atomic SRCU")
> > > > >
> > > > > And again, if so, please tell me what I am missing.
> > > >
> > > > Yes, this is a separate transition path. That commit suppresses the
> > > > initialization-time transition, while this patch prevents the transition
> > > > from srcu_gp_end(), which can otherwise call
> > > > init_srcu_struct_nodes(..., GFP_KERNEL) for an atomic SRCU.
> > > >
> > > > So this is defense in depth for the atomic path.
> > >
> > > Fair point.  But is there any additional pathway other than the
> > > SRCU_SIZING_IS_TORTURE() check at the end of srcu_torture_stats_print()?
> >
> > I checked all transition-to-big pathways for atomic SRCU.
> > Atomic SRCU can reach srcu_gp_end(), but its srcu_size_state remains
> > SRCU_SIZE_SMALL,
> > so the transition condition there cannot be satisfied.
> >
> > The contention path is also unreachable:
> > synchronize_srcu_atomic() calls srcu_gp_start() directly and never
> > goes through srcu_gp_start_if_needed().
> > The other callers of that path reject atomic SRCU.
> >
> > The only remaining explicit transition is SRCU_SIZING_IS_TORTURE() in
> > srcu_torture_stats_print(),
> > which currently lacks an atomic exclusion. I'll send a follow-up patch
> > to add a srcu_reader_flavor check there.
>
> Very good!
>
> > So I agree the srcu_gp_end() patch can be dropped.
>
> It might not hurt to have a WARN_ON_ONCE() there, just in case someone
> implements another pathway.

Thanks, Paul. I'll send a v2 of this patch separately, dropping
this patch change and adding the WARN_ON_ONCE() before the
transition check:
+ WARN_ON_ONCE(ssp->srcu_reader_flavor == SRCU_READ_FLAVOR_ATOMIC &&
    ss_state != SRCU_SIZE_SMALL);

if (ss_state != SRCU_SIZE_SMALL &&
   ss_state != SRCU_SIZE_BIG) { ... ....

Thanks,
Kunwu

>
>                                                         Thanx, Paul
>
> > Thanks,
> > Kunwu
> >
> > >
> > >                                                         Thanx, Paul
> > >
> > > > Thanks,
> > > > KunWu
> > > >
> > > > >
> > > > >                                                     Thanx, Paul
> > > > >
> > > > > > ---
> > > > > >  kernel/rcu/srcutree.c | 5 +++--
> > > > > >  1 file changed, 3 insertions(+), 2 deletions(-)
> > > > > >
> > > > > > diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c
> > > > > > index 01f224a56b41..74acd5645d4c 100644
> > > > > > --- a/kernel/rcu/srcutree.c
> > > > > > +++ b/kernel/rcu/srcutree.c
> > > > > > @@ -1074,8 +1074,9 @@ static void srcu_gp_end(struct srcu_struct *ssp, bool is_atomic)
> > > > > >           raw_spin_unlock_irq_rcu_node(sup);
> > > > > >   }
> > > > > >
> > > > > > - /* Transition to big if needed. */
> > > > > > - if (ss_state != SRCU_SIZE_SMALL && ss_state != SRCU_SIZE_BIG) {
> > > > > > + /* Transition to big if needed, but never for atomic SRCU. */
> > > > > > + if (!is_atomic && ss_state != SRCU_SIZE_SMALL &&
> > > > > > +     ss_state != SRCU_SIZE_BIG) {
> > > > > >           if (ss_state == SRCU_SIZE_ALLOC)
> > > > > >                   init_srcu_struct_nodes(ssp, GFP_KERNEL);
> > > > > >           else
> > > > > > --
> > > > > > 2.43.0
> > > > > >
> > > > >
> > > >
> > > > Sent using hkml (https://github.com/sjp38/hackermail)

Re: [PATCH 04/13] rcutorture: Add atomic-SRCU support to torture.sh

[KunWu Chan]

On Thu, Sep 10, 2026 at 11:47 AM Paul E. McKenney <paulmck@kernel.org> wrote:
>
> On Thu, Sep 10, 2026 at 09:30:48AM +0800, KunWu Chan wrote:
> > On Thu, Sep 10, 2026 at 6:35 AM Paul E. McKenney <paulmck@kernel.org> wrote:
> > >
> > > On Tue, Sep 08, 2026 at 04:34:41PM -0700, Paul E. McKenney wrote:
> > > > On Mon, Sep 07, 2026 at 03:58:20PM +0800, Kunwu Chan wrote:
> > > > > From: Kunwu Chan <kunwu.chan@gmail.com>
> > > > >
> > > > > Add the --do-atomic-srcu argument to torture.sh, which runs the
> > > > > SRCU-N, SRCU-P, and SRCU-T scenarios, thus covering both Tree SRCU
> > > > > (SRCU-N and SRCU-P) and Tiny SRCU (SRCU-T), with
> > > > > rcutorture.reader_flavor=0x10 appended to the boot parameters so
> > > > > that it takes precedence over each scenario's own reader-flavor
> > > > > setting.  This exercises srcu_read_lock_atomic(),
> > > > > srcu_read_unlock_atomic(), and synchronize_srcu_atomic().
> > > > >
> > > > > As with other torture.sh tests, the --do-kcsan argument runs a
> > > > > KCSAN+PROVE_LOCKING variant of this test.
> > > > >
> > > > > Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
> > > >
> > > > Queued for testing and review, thank you!
> > >
> > > I did take the liberty of changing "do_atomic_srcu=no" to
> > > "do_atomic_srcu=yes" in order to increase test coverage.
> > >
> >
> > Thanks, no problem.
> >
> > I also found a couple of data races in other subsystems while running
> > the tests. I’m still validating the fixes locally and will Cc you if I
> > send patches for them.
>
> Sounds good!  Note that some maintainers are more friendly to such
> patches than others.  But it shouldn't hurt to send them.  ;-)
>
> Just please carefully check the design.  For example, blindly applying
> READ_ONCE() and WRITE_ONCE() is not a strategy to win.
>

Thanks for the reminder, Paul. I’ll keep that in mind and make sure to
check the design carefully.

Thanks,
Kunwu

>                                                         Thanx, Paul
>
> > Thanks,
> > KunWu
> >
> > >                                                         Thanx, Paul
> > >
> > > > > ---
> > > > >  .../selftests/rcutorture/bin/torture.sh       | 24 +++++++++++++++++++
> > > > >  1 file changed, 24 insertions(+)
> > > > >
> > > > > diff --git a/tools/testing/selftests/rcutorture/bin/torture.sh b/tools/testing/selftests/rcutorture/bin/torture.sh
> > > > > index f0083891ee81..8885812d866a 100755
> > > > > --- a/tools/testing/selftests/rcutorture/bin/torture.sh
> > > > > +++ b/tools/testing/selftests/rcutorture/bin/torture.sh
> > > > > @@ -68,6 +68,7 @@ do_clocksourcewd="${ifnotaarch64}"
> > > > >  do_rt=yes
> > > > >  do_rcutasksflavors="${ifnotaarch64}" # FIXME: Back to "yes" when SMP=n auto-avoided
> > > > >  do_srcu_lockdep=yes
> > > > > +do_atomic_srcu=no
> > > > >  do_rcu_rust=no
> > > > >
> > > > >  # doyesno - Helper function for yes/no arguments
> > > > > @@ -103,6 +104,7 @@ usage () {
> > > > >     echo "       --do-rcu-rust / --do-no-rcu-rust / --no-rcu-rust"
> > > > >     echo "       --do-scftorture / --do-no-scftorture / --no-scftorture"
> > > > >     echo "       --do-srcu-lockdep / --do-no-srcu-lockdep / --no-srcu-lockdep"
> > > > > +   echo "       --do-atomic-srcu / --do-no-atomic-srcu / --no-atomic-srcu"
> > > > >     echo "       --duration [ <minutes> | <hours>h | <days>d ]"
> > > > >     echo "       --guest-cpu-limit N"
> > > > >     echo "       --kcsan-kmake-arg kernel-make-arguments"
> > > > > @@ -148,6 +150,7 @@ do
> > > > >             do_kcsan=yes
> > > > >             do_clocksourcewd="${ifnotaarch64}"
> > > > >             do_srcu_lockdep=yes
> > > > > +           do_atomic_srcu=yes
> > > > >             ;;
> > > > >     --do-allmodconfig|--do-no-allmodconfig|--no-allmodconfig)
> > > > >             do_allmodconfig=`doyesno "$1" --do-allmodconfig`
> > > > > @@ -183,6 +186,7 @@ do
> > > > >             do_kcsan=no
> > > > >             do_clocksourcewd=no
> > > > >             do_srcu_lockdep=no
> > > > > +           do_atomic_srcu=no
> > > > >             ;;
> > > > >     --do-normal|--do-norm|--do-no-normal|--do-no-norm|--no-normal|--no-norm)
> > > > >             do_normal=`doyesno "$1" --do-normal`
> > > > > @@ -212,6 +216,9 @@ do
> > > > >     --do-srcu-lockdep|--do-no-srcu-lockdep|--no-srcu-lockdep)
> > > > >             do_srcu_lockdep=`doyesno "$1" --do-srcu-lockdep`
> > > > >             ;;
> > > > > +   --do-atomic-srcu|--do-no-atomic-srcu|--no-atomic-srcu)
> > > > > +           do_atomic_srcu=`doyesno "$1" --do-atomic-srcu`
> > > > > +           ;;
> > > > >     --duration)
> > > > >             checkarg --duration "(minutes)" $# "$2" '^[0-9][0-9]*\(m\|h\|d\|\)$' '^error'
> > > > >             mult=1
> > > > > @@ -497,6 +504,23 @@ then
> > > > >     torture_set "rcutorture" tools/testing/selftests/rcutorture/bin/kvm.sh --allcpus --duration "$duration_rcutorture" --configs "$configs_rcutorture" --trust-make
> > > > >  fi
> > > > >
> > > > > +# Test atomic SRCU across Tree SRCU (SRCU-N and SRCU-P) and Tiny SRCU
> > > > > +# (SRCU-T).  The reader flavor selects srcu_read_lock_atomic() and
> > > > > +# synchronize_srcu_atomic().  Tiny SRCU requires SMP=n, which aarch64
> > > > > +# does not support.
> > > > > +if test "$do_atomic_srcu" = "yes"
> > > > > +then
> > > > > +   torture_bootargs="rcutorture.reader_flavor=0x10"
> > > > > +   configs_atomic_srcu="SRCU-N SRCU-P"
> > > > > +   if test "$ifnotaarch64" = yes
> > > > > +   then
> > > > > +           configs_atomic_srcu="$configs_atomic_srcu SRCU-T"
> > > > > +   fi
> > > > > +   torture_set "atomic-srcu" tools/testing/selftests/rcutorture/bin/kvm.sh \
> > > > > +           --allcpus --duration "$duration_rcutorture" \
> > > > > +           --configs "$configs_atomic_srcu" --trust-make
> > > > > +fi
> > > > > +
> > > > >  if test "$do_locktorture" = "yes"
> > > > >  then
> > > > >     torture_bootargs="torture.disable_onoff_at_boot"
> > > > > --
> > > > > 2.43.0
> > > > >