From: Kuniyuki Iwashima <kuniyu@google.com>
To: Andrew Morton <akpm@linux-foundation.org>,
Johannes Weiner <hannes@cmpxchg.org>,
Michal Hocko <mhocko@kernel.org>,
Roman Gushchin <roman.gushchin@linux.dev>,
Shakeel Butt <shakeel.butt@linux.dev>
Cc: Muchun Song <muchun.song@linux.dev>,
Kuniyuki Iwashima <kuniyu@google.com>,
Kuniyuki Iwashima <kuni1840@gmail.com>,
linux-mm@kvack.org, cgroups@vger.kernel.org
Subject: [PATCH 1/5] memcg: Remove unnecessary READ_ONCE() under memcg_stock.lock.
Date: Fri, 16 Oct 2026 05:27:45 +0000 [thread overview]
Message-ID: <20261016052811.3085635-2-kuniyu@google.com> (raw)
In-Reply-To: <20261016052811.3085635-1-kuniyu@google.com>
this_cpu_ptr(&memcg_stock) is always modified under
local_lock(&memcg_stock.lock).
Unlike is_memcg_drain_needed() under RCU, consume_stock(),
drain_stock(), and refill_stock() do not need READ_ONCE().
Especially, in refill_stock(), READ_ONCE() is used twice for
stock->cached[i], which forces the compiler to emit two volatile
loads from stock->cached[i].
Let's remove the unnecessary READ_ONCE() and eliminate the
redundant loads in refill_stock():
Before:
cached = READ_ONCE(stock->cached[i]);
4c 8b 80 88 d7 bd 84 movq -0x7b422878(%rax), %r8
if (memcg == READ_ONCE(stock->cached[i])) {
48 3b 98 88 d7 bd 84 cmpq -0x7b422878(%rax), %rbx
0f 84 86 01 00 00 je 0xffffffff817d01e0 <refill_stock+0x210>
After:
cached = stock->cached[i];
4c 8b 88 88 d7 bd 84 movq -0x7b422878(%rax), %r9
if (memcg == stock->cached[i]) {
4c 39 cb cmpq %r9, %rbx
0f 84 7a 01 00 00 je 0xffffffff817d01d0 <refill_stock+0x200>
Signed-off-by: Kuniyuki Iwashima <kuniyu@google.com>
---
mm/memcontrol.c | 14 +++++++-------
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 856a7d07586c..c268a7864547 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -2135,10 +2135,10 @@ static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages)
stock = this_cpu_ptr(&memcg_stock);
for (i = 0; i < NR_MEMCG_STOCK; ++i) {
- if (memcg != READ_ONCE(stock->cached[i]))
+ if (memcg != stock->cached[i])
continue;
- stock_pages = READ_ONCE(stock->nr_pages[i]);
+ stock_pages = stock->nr_pages[i];
if (stock_pages >= nr_pages) {
stock_pages -= nr_pages;
WRITE_ONCE(stock->nr_pages[i], stock_pages);
@@ -2168,13 +2168,13 @@ static void memcg_uncharge(struct mem_cgroup *memcg, unsigned int nr_pages)
*/
static void drain_stock(struct memcg_stock_pcp *stock, int i)
{
- struct mem_cgroup *old = READ_ONCE(stock->cached[i]);
+ struct mem_cgroup *old = stock->cached[i];
uint8_t stock_pages;
if (!old)
return;
- stock_pages = READ_ONCE(stock->nr_pages[i]);
+ stock_pages = stock->nr_pages[i];
if (stock_pages) {
memcg_uncharge(old, stock_pages);
WRITE_ONCE(stock->nr_pages[i], 0);
@@ -2254,11 +2254,11 @@ static void refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages)
stock = this_cpu_ptr(&memcg_stock);
for (i = 0; i < NR_MEMCG_STOCK; ++i) {
- cached = READ_ONCE(stock->cached[i]);
+ cached = stock->cached[i];
if (!cached && empty_slot == -1)
empty_slot = i;
- if (memcg == READ_ONCE(stock->cached[i])) {
- stock_pages = READ_ONCE(stock->nr_pages[i]) + nr_pages;
+ if (memcg == stock->cached[i]) {
+ stock_pages = stock->nr_pages[i] + nr_pages;
WRITE_ONCE(stock->nr_pages[i], stock_pages);
if (stock_pages > MEMCG_CHARGE_BATCH)
drain_stock(stock, i);
--
2.56.0.212.g129ede20d9-goog
next prev parent reply other threads:[~2026-10-16 5:28 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-10-16 5:27 [PATCH 0/5] memcg: Batch mod_memcg_state() for MEMCG_SOCK Kuniyuki Iwashima
2026-10-16 5:27 ` Kuniyuki Iwashima [this message]
2026-10-16 5:27 ` [PATCH 2/5] memcg: Reduce nesting in consume_stock() and refill_stock() Kuniyuki Iwashima
2026-10-16 5:27 ` [PATCH 3/5] memcg: Annotate slow paths " Kuniyuki Iwashima
2026-10-16 5:27 ` [PATCH 4/5] memcg: Batch mod_memcg_state() for MEMCG_SOCK Kuniyuki Iwashima
2026-10-16 5:27 ` [PATCH 5/5] selftests: cgroup: Deflake test_memcg_sock() in test_memcontrol.c Kuniyuki Iwashima
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261016052811.3085635-2-kuniyu@google.com \
--to=kuniyu@google.com \
--cc=akpm@linux-foundation.org \
--cc=cgroups@vger.kernel.org \
--cc=hannes@cmpxchg.org \
--cc=kuni1840@gmail.com \
--cc=linux-mm@kvack.org \
--cc=mhocko@kernel.org \
--cc=muchun.song@linux.dev \
--cc=roman.gushchin@linux.dev \
--cc=shakeel.butt@linux.dev \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox