[PATCH v1] mm: move LRU size stat updates out of lru_lock in folio batches

[PATCH v1] mm: move LRU size stat updates out of lru_lock in folio batches

[Joanne Koong]

folio_batch_move_lru() and folios_put_refs() take the lruvec's lru_lock
once per batch, but they update the node and memcg lruvec stats for
every folio in the batch while holding the lock. Only the LRU lists and
the memcg lru_zone_size need lru_lock. Updating the stats while holding
the lru_lock is unnecessary, and they make up a large part of the
critical section, as seen in [1]. With many threads faulting and freeing
memory in a single memcg, about half of all cycles are spent waiting for
lru_lock, and the hold time of the lru_lock directly limits throughput.

Accumulate the stat changes per LRU list while holding the lock instead,
and only apply them after dropping the lock (while still inside
the RCU read section that keeps the lruvec's memcg alive). This moves
the stat updates out of the lru_lock critical section and turns the
per-folio updates into at most one update per LRU list per batch.

The batch is attached to the lruvec while lru_lock is held so that the
LRU add/del helpers don't need to pass it down. To keep struct lruvec
from growing, cost_lock is moved into the hole after lru_lock, which
leaves room for the batch pointer in the same cacheline as lru_lock.

The stats can now lag behind the LRU lists by one batch, which is well
within the per-cpu batching they already have.

On a 2-socket Skylake (80 CPUs, 2 NUMA nodes), with threads pinned to
CPUs repeatedly faulting in and freeing memory in a single memcg,
average pages per second over 7 freshly created memcgs per run, 2 boots
each:

                     before    after
  shmem, 20 threads  4.74M     5.47M   (+15%)
  shmem, 80 threads  7.94M    10.33M   (+30%)
  anon,  20 threads  3.04M     3.35M   (+10%)
  anon,  80 threads  3.23M     3.41M   (+5%)

Link: https://lore.kernel.org/linux-fsdevel/CAJnrk1ZezmmzrYdf8KP6re_a76izMS=qGPeC8ZZXiteHyZZC-Q@mail.gmail.com/ [1]
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
---
An alternative that avoids adding the lru_size_batch pointer to the lruvec
struct is to pass the batch down explicitly, but the plumbing for this is a
bit ugly (and doesn't cover mglru paths). The code looks like [2]. Happy to go
with this if that is preferred.

[2] https://github.com/joannekoong/linux/commit/2f20765b01f98113e3c83ea0d0f98580502bfb96

 include/linux/mm_inline.h | 12 ++++++-
 include/linux/mmzone.h    | 11 +++++--
 mm/folio.c                | 67 +++++++++++++++++++++++++++++++++------
 3 files changed, 77 insertions(+), 13 deletions(-)

diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
index ab69b9930893..b3d85fe91385 100644
--- a/include/linux/mm_inline.h
+++ b/include/linux/mm_inline.h
@@ -50,7 +50,17 @@ static __always_inline void __update_lru_size(struct lruvec *lruvec,
 	lockdep_assert_held(&lruvec->lru_lock);
 	WARN_ON_ONCE(nr_pages != (int)nr_pages);
 
-	mod_lruvec_state(lruvec, NR_LRU_BASE + lru, nr_pages);
+	/*
+	 * Batches in mm/folio.c attach an lru_size_batch while holding
+	 * lru_lock, to apply the node and memcg lruvec stats only after the
+	 * lock is dropped. The stats don't need lru_lock and updating them is
+	 * a large part of the critical section when batching. The zone stats
+	 * are still updated right away.
+	 */
+	if (lruvec->lru_size_batch)
+		lruvec->lru_size_batch->nr_pages[lru] += nr_pages;
+	else
+		mod_lruvec_state(lruvec, NR_LRU_BASE + lru, nr_pages);
 	__mod_zone_page_state(&pgdat->node_zones[zid],
 				NR_ZONE_LRU_BASE + lru, nr_pages);
 }
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 3b96d6c7123b..ac14f7118562 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -739,10 +739,19 @@ struct lru_cost {
 	unsigned long		last_io;
 };
 
+/* LRU size changes made under lru_lock, applied after the lock is dropped */
+struct lru_size_batch {
+	long nr_pages[NR_LRU_LISTS];
+};
+
 struct lruvec {
 	struct list_head		lists[NR_LRU_LISTS];
 	/* per lruvec lru_lock for memcg */
 	spinlock_t			lru_lock;
+	/* Protects cost[]. */
+	spinlock_t			cost_lock;
+	/* If set, defers lruvec stat updates until lru_lock is dropped */
+	struct lru_size_batch		*lru_size_batch;
 	/*
 	 * These track the cost of reclaiming one LRU - file or anon -
 	 * over the other. As the observed cost of reclaiming one LRU
@@ -751,8 +760,6 @@ struct lruvec {
 	 * serialises that update.
 	 */
 	struct lru_cost			cost[ANON_AND_FILE];
-	/* Protects cost[]. */
-	spinlock_t			cost_lock;
 	/* Non-resident age, driven by LRU movement */
 	atomic_long_t			nonresident_age;
 	/* Refaults at the time of last reclaim cycle */
diff --git a/mm/folio.c b/mm/folio.c
index 45c814f3857b..dc2c796a4ea7 100644
--- a/mm/folio.c
+++ b/mm/folio.c
@@ -64,11 +64,54 @@ static DEFINE_PER_CPU(struct cpu_fbatches, cpu_fbatches) = {
 	.lock_irq = INIT_LOCAL_LOCK(lock_irq),
 };
 
+/*
+ * lru_lock helpers for batches that defer LRU size stat updates until the
+ * lock is dropped, see __update_lru_size(). The deferred stats are applied
+ * while still in the RCU read section entered by locking, which keeps the
+ * lruvec's memcg alive.
+ */
+static void lruvec_unlock_irqrestore_batched(struct lruvec *lruvec,
+		unsigned long flags)
+{
+	struct lru_size_batch *batch = lruvec->lru_size_batch;
+	enum lru_list lru;
+
+	lruvec->lru_size_batch = NULL;
+	spin_unlock_irqrestore(&lruvec->lru_lock, flags);
+
+	for_each_lru(lru) {
+		long nr_pages = batch->nr_pages[lru];
+
+		if (nr_pages) {
+			mod_lruvec_state(lruvec, NR_LRU_BASE + lru, nr_pages);
+			batch->nr_pages[lru] = 0;
+		}
+	}
+	rcu_read_unlock();
+}
+
+static void folio_lruvec_relock_irqsave_batched(struct folio *folio,
+		struct lruvec **lruvecp, unsigned long *flagsp,
+		struct lru_size_batch *batch)
+{
+	if (*lruvecp) {
+		if (folio_matches_lruvec(folio, *lruvecp))
+			return;
+
+		lruvec_unlock_irqrestore_batched(*lruvecp, *flagsp);
+	}
+
+	*lruvecp = folio_lruvec_lock_irqsave(folio, flagsp);
+	VM_WARN_ON_ONCE((*lruvecp)->lru_size_batch);
+	(*lruvecp)->lru_size_batch = batch;
+}
+
 static void __page_cache_release(struct folio *folio, struct lruvec **lruvecp,
-		unsigned long *flagsp)
+		unsigned long *flagsp, struct lru_size_batch *batch)
 {
 	if (folio_test_lru(folio)) {
-		folio_lruvec_relock_irqsave(folio, lruvecp, flagsp);
+		folio_lruvec_relock_irqsave_batched(folio, lruvecp, flagsp,
+						    batch);
 		lruvec_del_folio(*lruvecp, folio);
 		__folio_clear_lru_flags(folio);
 	}
@@ -80,12 +123,13 @@ static void __page_cache_release(struct folio *folio, struct lruvec **lruvecp,
  */
 static void page_cache_release(struct folio *folio)
 {
+	struct lru_size_batch batch = {};
 	struct lruvec *lruvec = NULL;
 	unsigned long flags;
 
-	__page_cache_release(folio, &lruvec, &flags);
+	__page_cache_release(folio, &lruvec, &flags, &batch);
 	if (lruvec)
-		lruvec_unlock_irqrestore(lruvec, flags);
+		lruvec_unlock_irqrestore_batched(lruvec, flags);
 }
 
 void __folio_put(struct folio *folio)
@@ -165,6 +209,7 @@ static void lru_add(struct lruvec *lruvec, struct folio *folio)
 static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t move_fn)
 {
 	int i;
+	struct lru_size_batch batch = {};
 	struct lruvec *lruvec = NULL;
 	unsigned long flags = 0;
 	struct folio_batch free_fbatch;
@@ -199,14 +244,15 @@ static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t move_fn)
 			continue;
 		}
 
-		folio_lruvec_relock_irqsave(folio, &lruvec, &flags);
+		folio_lruvec_relock_irqsave_batched(folio, &lruvec, &flags,
+						    &batch);
 		move_fn(lruvec, folio);
 
 		folio_set_lru(folio);
 	}
 
 	if (lruvec)
-		lruvec_unlock_irqrestore(lruvec, flags);
+		lruvec_unlock_irqrestore_batched(lruvec, flags);
 
 	/* Cleanup filtered dead folios. */
 	if (is_lru_add) {
@@ -998,6 +1044,7 @@ void lru_cache_disable(void)
 void folios_put_refs(struct folio_batch *folios, unsigned int *refs)
 {
 	int i, j;
+	struct lru_size_batch batch = {};
 	struct lruvec *lruvec = NULL;
 	unsigned long flags = 0;
 
@@ -1014,7 +1061,7 @@ void folios_put_refs(struct folio_batch *folios, unsigned int *refs)
 
 		if (folio_is_zone_device(folio)) {
 			if (lruvec) {
-				lruvec_unlock_irqrestore(lruvec, flags);
+				lruvec_unlock_irqrestore_batched(lruvec, flags);
 				lruvec = NULL;
 			}
 			if (folio_ref_sub_and_test(folio, nr_refs))
@@ -1028,20 +1075,20 @@ void folios_put_refs(struct folio_batch *folios, unsigned int *refs)
 		/* hugetlb has its own memcg */
 		if (folio_test_hugetlb(folio)) {
 			if (lruvec) {
-				lruvec_unlock_irqrestore(lruvec, flags);
+				lruvec_unlock_irqrestore_batched(lruvec, flags);
 				lruvec = NULL;
 			}
 			free_huge_folio(folio);
 			continue;
 		}
-		__page_cache_release(folio, &lruvec, &flags);
+		__page_cache_release(folio, &lruvec, &flags, &batch);
 
 		if (j != i)
 			folios->folios[j] = folio;
 		j++;
 	}
 	if (lruvec)
-		lruvec_unlock_irqrestore(lruvec, flags);
+		lruvec_unlock_irqrestore_batched(lruvec, flags);
 	if (!j) {
 		folio_batch_reinit(folios);
 		return;
-- 
2.52.0

Re: [PATCH v1] mm: move LRU size stat updates out of lru_lock in folio batches

[Shakeel Butt]

On Thu, Oct 22, 2026 at 01:03:13PM -0700, Joanne Koong wrote:
> folio_batch_move_lru() and folios_put_refs() take the lruvec's lru_lock
> once per batch, but they update the node and memcg lruvec stats for
> every folio in the batch while holding the lock. Only the LRU lists and
> the memcg lru_zone_size need lru_lock. Updating the stats while holding
> the lru_lock is unnecessary, and they make up a large part of the
> critical section, as seen in [1]. 

Can you please add a tldr from [1] here? Too much text in that link.

> With many threads faulting and freeing
> memory in a single memcg, about half of all cycles are spent waiting for
> lru_lock, and the hold time of the lru_lock directly limits throughput.

Do you have perf profile showing this? Mainly I want to know where in stats
update we are spending the CPU most.

I wonder if we can find a lower bound here by removing the stats update
altogether in this code path (just for experimentation) and measure the
throughput we get.

> 
> Accumulate the stat changes per LRU list while holding the lock instead,
> and only apply them after dropping the lock (while still inside
> the RCU read section that keeps the lruvec's memcg alive). This moves
> the stat updates out of the lru_lock critical section and turns the
> per-folio updates into at most one update per LRU list per batch.
> 
> The batch is attached to the lruvec while lru_lock is held so that the
> LRU add/del helpers don't need to pass it down. To keep struct lruvec
> from growing, cost_lock is moved into the hole after lru_lock, which
> leaves room for the batch pointer in the same cacheline as lru_lock.
> 
> The stats can now lag behind the LRU lists by one batch, which is well
> within the per-cpu batching they already have.
> 
> On a 2-socket Skylake (80 CPUs, 2 NUMA nodes), with threads pinned to
> CPUs repeatedly faulting in and freeing memory in a single memcg,
> average pages per second over 7 freshly created memcgs per run, 2 boots
> each:
> 
>                      before    after
>   shmem, 20 threads  4.74M     5.47M   (+15%)
>   shmem, 80 threads  7.94M    10.33M   (+30%)
>   anon,  20 threads  3.04M     3.35M   (+10%)
>   anon,  80 threads  3.23M     3.41M   (+5%)
> 
> Link: https://lore.kernel.org/linux-fsdevel/CAJnrk1ZezmmzrYdf8KP6re_a76izMS=qGPeC8ZZXiteHyZZC-Q@mail.gmail.com/ [1]
> Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
> ---
> An alternative that avoids adding the lru_size_batch pointer to the lruvec
> struct is to pass the batch down explicitly, but the plumbing for this is a
> bit ugly (and doesn't cover mglru paths). The code looks like [2]. Happy to go
> with this if that is preferred.

Do you expect there might be opportunity to batch the stats other than LRU stats
you batched in this patch? If there is then I think we should at least explore
if a more general stat batching can be done.

This code looks good though.

Re: [PATCH v1] mm: move LRU size stat updates out of lru_lock in folio batches

[Joanne Koong]

On Thu, Oct 22, 2026 at 2:17 PM Shakeel Butt <shakeel.butt@linux.dev> wrote:
>
> On Thu, Oct 22, 2026 at 01:03:13PM -0700, Joanne Koong wrote:
> > folio_batch_move_lru() and folios_put_refs() take the lruvec's lru_lock
> > once per batch, but they update the node and memcg lruvec stats for
> > every folio in the batch while holding the lock. Only the LRU lists and
> > the memcg lru_zone_size need lru_lock. Updating the stats while holding
> > the lru_lock is unnecessary, and they make up a large part of the
> > critical section, as seen in [1].
>
> Can you please add a tldr from [1] here? Too much text in that link.

Will do in v2. THe relevant part of [1] is that a change that added
just a few instructions to the memcg stat update path cost about 2%
throughput because the LRU stat updates run under lru_lock and the
overhead went away once they were moved outside of the lock, which is
what motivated this patch.

>
> > With many threads faulting and freeing
> > memory in a single memcg, about half of all cycles are spent waiting for
> > lru_lock, and the hold time of the lru_lock directly limits throughput.
>
> Do you have perf profile showing this? Mainly I want to know where in stats
> update we are spending the CPU most.

The perf profile run on the top commit of mm-new (commit fcec4f4bcc
"riscv: mm: fix concurrency in mark_new_valid_map()) using the same
benchmark setup (80 threads, shmem) shows:
- 52.8% of all cycles are spent spinning on the lru_lock (evenly split
between folio_batch_move_lru() and folios_put_refs())
- About 25% of the time spent holding lru_lock in those 2 paths goes
to the LRU stat updates (the ones this patch moves to outside the
lock) and the rest is the list manipulation itself
- Most of the cost in __mod_memcg_lruvec_state() is cache misses
(loading pn->memcg, pn->lruvec_stats_percpu, and the per-cpu
counters).

The stat updates are only 0.7% of total cycles but I think it's the
lock contention on callers waiting for the lock that ends up hurting
throughput.

>
> I wonder if we can find a lower bound here by removing the stats update
> altogether in this code path (just for experimentation) and measure the
> throughput we get.

With the LRU stat updates removed from these paths entirely, I'm
seeing that throughput is essentially the same as with this patch
(within noise):

                       mm-new   this patch   no stat updates
    shmem, 80 threads  7.94M    10.33M       10.31M
    anon,  20 threads  3.04M     3.35M        3.33M

I think once the updates are moved out of the critical section
entirely, they don't cost much to do.

>
> >
> > Accumulate the stat changes per LRU list while holding the lock instead,
> > and only apply them after dropping the lock (while still inside
> > the RCU read section that keeps the lruvec's memcg alive). This moves
> > the stat updates out of the lru_lock critical section and turns the
> > per-folio updates into at most one update per LRU list per batch.
> >
> > The batch is attached to the lruvec while lru_lock is held so that the
> > LRU add/del helpers don't need to pass it down. To keep struct lruvec
> > from growing, cost_lock is moved into the hole after lru_lock, which
> > leaves room for the batch pointer in the same cacheline as lru_lock.
> >
> > The stats can now lag behind the LRU lists by one batch, which is well
> > within the per-cpu batching they already have.
> >
> > On a 2-socket Skylake (80 CPUs, 2 NUMA nodes), with threads pinned to
> > CPUs repeatedly faulting in and freeing memory in a single memcg,
> > average pages per second over 7 freshly created memcgs per run, 2 boots
> > each:
> >
> >                      before    after
> >   shmem, 20 threads  4.74M     5.47M   (+15%)
> >   shmem, 80 threads  7.94M    10.33M   (+30%)
> >   anon,  20 threads  3.04M     3.35M   (+10%)
> >   anon,  80 threads  3.23M     3.41M   (+5%)
> >
> > Link: https://lore.kernel.org/linux-fsdevel/CAJnrk1ZezmmzrYdf8KP6re_a76izMS=qGPeC8ZZXiteHyZZC-Q@mail.gmail.com/ [1]
> > Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
> > ---
> > An alternative that avoids adding the lru_size_batch pointer to the lruvec
> > struct is to pass the batch down explicitly, but the plumbing for this is a
> > bit ugly (and doesn't cover mglru paths). The code looks like [2]. Happy to go
> > with this if that is preferred.
>
> Do you expect there might be opportunity to batch the stats other than LRU stats
> you batched in this patch? If there is then I think we should at least explore
> if a more general stat batching can be done.

I had experimented a little with doing this for LRU size stats in the
reclaim paths in vmscan and deferring the count_memcg_events() calls
in the same batch paths, but none of them made a measurable difference
to throughput. For the reclaim paths, I think it's because there
reclaim rather than the lru_lock is the bottleneck, and my benchmark
doesn't really exercise the paths that call count_memcg_events(). As
far as I can tell, the rmap stats (NR_ANON_MAPPED, NR_FILE_MAPPED) are
under the page table lock but that lock isn't really contended. I
think the other maybe-potentially-promising candidate is the page
cache stats (NR_FILE_PAGES, NR_SHMEM, ...) that are updated per folio
under the mapping's i_pages lock but that would matter only for
threads working on a single shared file. I can look more into the page
cache stats and see whether the complexity would be worth it there.

Thanks for taking a look at this patch!

Thanks,
Joanne