[PATCH 3/4] rcu: Perform early sequence fetch for polling locklessly

[Frederic Weisbecker <frederic@kernel.org>]

The workqueue ordering guarantees that the work sees all the accesses
of the task prior to its call to the corresponding queue_work().

Therefore the sequence to poll can be retrieved locklessly.

The only downside is that it is then possible to miss the 0x1 flag set
by a prior work. But this could already happen concurrently anyway after
the exp_poll_lock is unlocked. In the worst case the slow path involving
synchronize_rcu_expedited() takes care of the situation.

Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Cc: Neeraj Upadhyay <quic_neeraju@quicinc.com>
Cc: Boqun Feng <boqun.feng@gmail.com>
Cc: Uladzislau Rezki <uladzislau.rezki@sony.com>
Cc: Joel Fernandes <joel@joelfernandes.org>
---
 kernel/rcu/tree_exp.h | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)

diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h
index 763ec35546ed..c4a19c6a83cf 100644
--- a/kernel/rcu/tree_exp.h
+++ b/kernel/rcu/tree_exp.h
@@ -909,9 +909,7 @@ static void sync_rcu_do_polled_gp(struct work_struct *wp)
 	struct rcu_node *rnp = container_of(wp, struct rcu_node, exp_poll_wq);
 	unsigned long s;
 
-	raw_spin_lock_irqsave(&rnp->exp_poll_lock, flags);
-	s = rnp->exp_seq_poll_rq;
-	raw_spin_unlock_irqrestore(&rnp->exp_poll_lock, flags);
+	s = READ_ONCE(rnp->exp_seq_poll_rq);
 	if (s & 0x1)
 		return;
 	while (!sync_exp_work_done(s))
@@ -919,7 +917,7 @@ static void sync_rcu_do_polled_gp(struct work_struct *wp)
 	raw_spin_lock_irqsave(&rnp->exp_poll_lock, flags);
 	s = rnp->exp_seq_poll_rq;
 	if (!(s & 0x1) && sync_exp_work_done(s))
-		rnp->exp_seq_poll_rq |= 0x1;
+		WRITE_ONCE(rnp->exp_seq_poll_rq, s | 0x1);
 	raw_spin_unlock_irqrestore(&rnp->exp_poll_lock, flags);
 }
 
@@ -949,7 +947,7 @@ unsigned long start_poll_synchronize_rcu_expedited(void)
 	if (rcu_init_invoked())
 		raw_spin_lock_irqsave(&rnp->exp_poll_lock, flags);
 	if ((rnp->exp_seq_poll_rq & 0x1) || ULONG_CMP_LT(rnp->exp_seq_poll_rq, s)) {
-		rnp->exp_seq_poll_rq = s;
+		WRITE_ONCE(rnp->exp_seq_poll_rq, s);
 		if (rcu_init_invoked())
 			queue_work(rcu_gp_wq, &rnp->exp_poll_wq);
 	}
-- 
2.25.1