From: Lai Jiangshan <jiangshanlai@gmail.com>
To: linux-kernel@vger.kernel.org
Cc: rcu@vger.kernel.org, x86@kernel.org,
Lai Jiangshan <jiangshan.ljs@antgroup.com>,
"Paul E. McKenney" <paulmck@kernel.org>,
Peter Zijlstra <peterz@infradead.org>,
Frederic Weisbecker <frederic@kernel.org>,
Neeraj Upadhyay <quic_neeraju@quicinc.com>,
Joel Fernandes <joel@joelfernandes.org>,
Josh Triplett <josh@joshtriplett.org>,
Boqun Feng <boqun.feng@gmail.com>,
Steven Rostedt <rostedt@goodmis.org>,
Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
Lai Jiangshan <jiangshanlai@gmail.com>,
Zqiang <qiang.zhang1211@gmail.com>
Subject: [PATCH 04/10] rcu: Add macros set_rcu_preempt_special() and clear_rcu_preempt_special()
Date: Thu, 28 Mar 2024 15:53:12 +0800 [thread overview]
Message-ID: <20240328075318.83039-5-jiangshanlai@gmail.com> (raw)
In-Reply-To: <20240328075318.83039-1-jiangshanlai@gmail.com>
From: Lai Jiangshan <jiangshan.ljs@antgroup.com>
Add wrappers accessing to t->rcu_read_unlock_special.b so that the
wrappers can be adapted for synching the value with the later-introduced
per-cpu rcu_preempt_count.
Cc: "Paul E. McKenney" <paulmck@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Frederic Weisbecker <frederic@kernel.org>
Signed-off-by: Lai Jiangshan <jiangshan.ljs@antgroup.com>
---
kernel/rcu/tree_exp.h | 2 +-
kernel/rcu/tree_plugin.h | 20 +++++++++++++-------
2 files changed, 14 insertions(+), 8 deletions(-)
diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h
index 6b83537480b1..4e91414552e6 100644
--- a/kernel/rcu/tree_exp.h
+++ b/kernel/rcu/tree_exp.h
@@ -725,7 +725,7 @@ static void rcu_exp_handler(void *unused)
raw_spin_lock_irqsave_rcu_node(rnp, flags);
if (rnp->expmask & rdp->grpmask) {
WRITE_ONCE(rdp->cpu_no_qs.b.exp, true);
- t->rcu_read_unlock_special.b.exp_hint = true;
+ set_rcu_preempt_special(exp_hint);
}
raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
return;
diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index d899b4afc21c..3dbd5609185e 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -106,6 +106,14 @@ static void rcu_report_exp_rnp(struct rcu_node *rnp, bool wake);
static bool sync_rcu_exp_done(struct rcu_node *rnp);
static void rcu_read_unlock_special(struct task_struct *t);
+#define set_rcu_preempt_special(reason) do { \
+ WRITE_ONCE(current->rcu_read_unlock_special.b.reason, true); \
+ } while (0)
+
+#define clear_rcu_preempt_special(reason) do { \
+ WRITE_ONCE(current->rcu_read_unlock_special.b.reason, false); \
+ } while (0)
+
/*
* Tell them what RCU they are running.
*/
@@ -293,7 +301,7 @@ static void rcu_qs(void)
TPS("cpuqs"));
__this_cpu_write(rcu_data.cpu_no_qs.b.norm, false);
barrier(); /* Coordinate with rcu_flavor_sched_clock_irq(). */
- WRITE_ONCE(current->rcu_read_unlock_special.b.need_qs, false);
+ clear_rcu_preempt_special(need_qs);
}
}
@@ -325,7 +333,7 @@ void rcu_note_context_switch(bool preempt)
/* Possibly blocking in an RCU read-side critical section. */
rnp = rdp->mynode;
raw_spin_lock_rcu_node(rnp);
- t->rcu_read_unlock_special.b.blocked = true;
+ set_rcu_preempt_special(blocked);
t->rcu_blocked_node = rnp;
/*
@@ -399,7 +407,7 @@ void __rcu_read_lock(void)
if (IS_ENABLED(CONFIG_PROVE_LOCKING))
WARN_ON_ONCE(rcu_preempt_depth() > RCU_NEST_PMAX);
if (IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) && rcu_state.gp_kthread)
- WRITE_ONCE(current->rcu_read_unlock_special.b.need_qs, true);
+ set_rcu_preempt_special(need_qs);
barrier(); /* critical section after entry code. */
}
EXPORT_SYMBOL_GPL(__rcu_read_lock);
@@ -738,7 +746,7 @@ static void rcu_flavor_sched_clock_irq(int user)
__this_cpu_read(rcu_data.cpu_no_qs.b.norm) &&
!t->rcu_read_unlock_special.b.need_qs &&
time_after(jiffies, rcu_state.gp_start + HZ))
- t->rcu_read_unlock_special.b.need_qs = true;
+ set_rcu_preempt_special(need_qs);
}
/*
@@ -751,12 +759,10 @@ static void rcu_flavor_sched_clock_irq(int user)
*/
void exit_rcu(void)
{
- struct task_struct *t = current;
-
if (unlikely(!list_empty(¤t->rcu_node_entry))) {
rcu_preempt_depth_set(1);
barrier();
- WRITE_ONCE(t->rcu_read_unlock_special.b.blocked, true);
+ set_rcu_preempt_special(blocked);
} else if (unlikely(rcu_preempt_depth())) {
rcu_preempt_depth_set(1);
} else {
--
2.19.1.6.gb485710b
next prev parent reply other threads:[~2024-03-28 7:51 UTC|newest]
Thread overview: 23+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-03-28 7:53 [PATCH 00/10] rcu/x86: Use per-cpu rcu preempt count Lai Jiangshan
2024-03-28 7:53 ` [PATCH 01/10] lib: Use rcu_preempt_depth() to replace current->rcu_read_lock_nesting Lai Jiangshan
2024-03-28 7:53 ` [PATCH 02/10] rcu: Move rcu_preempt_depth_set() to rcupdate.h Lai Jiangshan
2024-03-31 11:09 ` Joel Fernandes
2024-03-31 16:16 ` Lai Jiangshan
2024-04-01 2:27 ` Joel Fernandes
2024-04-01 11:40 ` Alan Huang
2024-04-01 11:57 ` Alan Huang
2024-03-28 7:53 ` [PATCH 03/10] rcu: Reorder tree_exp.h after tree_plugin.h Lai Jiangshan
2024-03-28 7:53 ` Lai Jiangshan [this message]
2024-03-28 7:53 ` [PATCH 05/10] rcu: Make rcu_read_unlock_special() global Lai Jiangshan
2024-03-28 7:53 ` [PATCH 06/10] rcu: Rename marco __LINUX_RCU_H to __KERNEL_RCU_H Lai Jiangshan
2024-03-28 7:53 ` [PATCH 07/10] sched/core: Add rcu_preempt_switch() Lai Jiangshan
2024-03-28 7:53 ` [PATCH 08/10] rcu: Implement PCPU_RCU_PREEMPT_COUNT framework Lai Jiangshan
2024-04-22 10:41 ` Frederic Weisbecker
2024-03-28 7:53 ` [PATCH 09/10] x86/rcu: Add rcu_preempt_count Lai Jiangshan
2024-03-29 7:37 ` kernel test robot
2024-03-29 7:48 ` kernel test robot
2024-04-22 11:05 ` Frederic Weisbecker
2024-04-23 9:02 ` Lai Jiangshan
2024-04-23 11:33 ` Frederic Weisbecker
2024-03-28 7:53 ` [PATCH 10/10] x86/rcu: Add THUNK rcu_read_unlock_special_thunk Lai Jiangshan
2024-03-29 14:46 ` kernel test robot
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20240328075318.83039-5-jiangshanlai@gmail.com \
--to=jiangshanlai@gmail.com \
--cc=boqun.feng@gmail.com \
--cc=frederic@kernel.org \
--cc=jiangshan.ljs@antgroup.com \
--cc=joel@joelfernandes.org \
--cc=josh@joshtriplett.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mathieu.desnoyers@efficios.com \
--cc=paulmck@kernel.org \
--cc=peterz@infradead.org \
--cc=qiang.zhang1211@gmail.com \
--cc=quic_neeraju@quicinc.com \
--cc=rcu@vger.kernel.org \
--cc=rostedt@goodmis.org \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.