Linux cgroups development
 help / color / mirror / Atom feed
* [PATCH] mm: workingset: compare shadow nodes node-wide without kmem accounting
@ 2026-11-06 23:59 T.J. Mercier
  2026-11-16 17:39 ` T.J. Mercier
  2026-11-19  0:09 ` Johannes Weiner
  0 siblings, 2 replies; 5+ messages in thread
From: T.J. Mercier @ 2026-11-06 23:59 UTC (permalink / raw)
  To: Andrew Morton, Johannes Weiner
  Cc: Shakeel Butt, Michal Hocko, Roman Gushchin, Muchun Song, Qi Zheng,
	Dave Chinner, Michal Koutný, David Hildenbrand,
	Lorenzo Stoakes, Kairui Song, Barry Song, Axel Rasmussen,
	Yuanchu Xie, Wei Xu, Baoquan He, Baolin Wang, linux-mm, cgroups,
	linux-kernel, T.J. Mercier, Jaewon Kim, Jaeseon Sim,
	Youngchan Kim, Dmitry Skiba

count_shadow_nodes() limits the number of xarray nodes holding shadow
entries to 1/8th of the pages that could be using them. For a memcg, it
counts the LRU and slab pages local to that memcg.

With kmem accounting disabled (cgroup.memory=nokmem), shadow nodes are
not charged to any memcg and __list_lru_init() makes the shadow_nodes
list_lru non-memcg-aware, so all shadow nodes of a NUMA node are on a
single list. shrinker_alloc() falls back to a non-memcg-aware shadow
node shrinker, which shrink_slab() only invokes for the root memcg.
count_shadow_nodes() therefore compares every shadow node on the NUMA
node against the pages local to the root memcg.

When most tasks run in child memcgs, the root memcg only holds a small
fraction of all pages, so the shadow node limit derived from the root
memcg page count is far below the number of shadow nodes on the NUMA
node. Since the shrinker has seeks == 0, every root shrink_slab() call
trims about half of the "excess" shadow nodes. Most refault information
is thrown away, so refaults of recently evicted file folios are not
detected. Refaulted folios lose PG_workingset and the workingset_refault
and workingset_restore stats drop.

In testing on devices with kmem accounting disabled, moving tasks from
the root memcg into per-application memcgs dropped workingset_nodes from
~37,000 to ~900 and raised workingset_nodereclaim 4.6x while
pgsteal_file rose by ~12%.

When kmem accounting is disabled, compare the shadow nodes against the
whole NUMA node, as is done without memcg. Behavior with kmem
accounting enabled is unchanged.

Cc: Jaewon Kim <jaewon31.kim@samsung.com>
Tested-by: Jaeseon Sim <jason.sim@samsung.com> # Galaxy A18 LTE
Tested-by: Youngchan Kim <ycshark.kim@samsung.com> # Galaxy S27
Tested-by: Dmitry Skiba <dskiba@google.com> # Pixel 10
Signed-off-by: T.J. Mercier <tjmercier@google.com>
---
 mm/workingset.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/mm/workingset.c b/mm/workingset.c
index 1504f91cdca5..87d5e55dd684 100644
--- a/mm/workingset.c
+++ b/mm/workingset.c
@@ -682,7 +682,16 @@ static unsigned long count_shadow_nodes(struct shrinker *shrinker,
 	 * PAGE_SIZE / xa_nodes / node_entries * 8 / PAGE_SIZE
 	 */
 #ifdef CONFIG_MEMCG
-	if (sc->memcg) {
+	/*
+	 * Without kmem accounting the shadow_nodes list_lru is not
+	 * memcg-aware. So the count above covers the whole NUMA node, and the
+	 * shrinker only runs for the root memcg, whose local pages can be a
+	 * small fraction of the node's when most tasks run in child memcgs.
+	 * Compare the shadow node count against the memcg page count only when
+	 * the shadow_nodes list_lru is memcg-aware. Otherwise use the NUMA node
+	 * page count as is done without memcg.
+	 */
+	if (sc->memcg && shadow_nodes.memcg_aware) {
 		struct lruvec *lruvec;
 		int i;
 
-- 
2.56.0.455.g6c5a503b38-goog


^ permalink raw reply related	[flat|nested] 5+ messages in thread

* Re: [PATCH] mm: workingset: compare shadow nodes node-wide without kmem accounting
  2026-11-06 23:59 [PATCH] mm: workingset: compare shadow nodes node-wide without kmem accounting T.J. Mercier
@ 2026-11-16 17:39 ` T.J. Mercier
  2026-11-16 20:04   ` Barry Song
  2026-11-19  0:09 ` Johannes Weiner
  1 sibling, 1 reply; 5+ messages in thread
From: T.J. Mercier @ 2026-11-16 17:39 UTC (permalink / raw)
  To: Andrew Morton, Johannes Weiner
  Cc: Shakeel Butt, Michal Hocko, Roman Gushchin, Muchun Song, Qi Zheng,
	Dave Chinner, Michal Koutný, David Hildenbrand,
	Lorenzo Stoakes, Kairui Song, Barry Song, Axel Rasmussen,
	Yuanchu Xie, Wei Xu, Baoquan He, Baolin Wang, linux-mm, cgroups,
	linux-kernel, Jaewon Kim, Jaeseon Sim, Youngchan Kim,
	Dmitry Skiba

On Fri, Nov 6, 2026 at 3:59 PM T.J. Mercier <tjmercier@google.com> wrote:
>
> count_shadow_nodes() limits the number of xarray nodes holding shadow
> entries to 1/8th of the pages that could be using them. For a memcg, it
> counts the LRU and slab pages local to that memcg.
>
> With kmem accounting disabled (cgroup.memory=nokmem), shadow nodes are
> not charged to any memcg and __list_lru_init() makes the shadow_nodes
> list_lru non-memcg-aware, so all shadow nodes of a NUMA node are on a
> single list. shrinker_alloc() falls back to a non-memcg-aware shadow
> node shrinker, which shrink_slab() only invokes for the root memcg.
> count_shadow_nodes() therefore compares every shadow node on the NUMA
> node against the pages local to the root memcg.
>
> When most tasks run in child memcgs, the root memcg only holds a small
> fraction of all pages, so the shadow node limit derived from the root
> memcg page count is far below the number of shadow nodes on the NUMA
> node. Since the shrinker has seeks == 0, every root shrink_slab() call
> trims about half of the "excess" shadow nodes. Most refault information
> is thrown away, so refaults of recently evicted file folios are not
> detected. Refaulted folios lose PG_workingset and the workingset_refault
> and workingset_restore stats drop.
>
> In testing on devices with kmem accounting disabled, moving tasks from
> the root memcg into per-application memcgs dropped workingset_nodes from
> ~37,000 to ~900 and raised workingset_nodereclaim 4.6x while
> pgsteal_file rose by ~12%.
>
> When kmem accounting is disabled, compare the shadow nodes against the
> whole NUMA node, as is done without memcg. Behavior with kmem
> accounting enabled is unchanged.
>
> Cc: Jaewon Kim <jaewon31.kim@samsung.com>
> Tested-by: Jaeseon Sim <jason.sim@samsung.com> # Galaxy A18 LTE
> Tested-by: Youngchan Kim <ycshark.kim@samsung.com> # Galaxy S27
> Tested-by: Dmitry Skiba <dskiba@google.com> # Pixel 10
> Signed-off-by: T.J. Mercier <tjmercier@google.com>

I'm guessing there aren't many other nokmem users?

> ---
>
>  mm/workingset.c | 11 ++++++++++-
>  1 file changed, 10 insertions(+), 1 deletion(-)
>
> diff --git a/mm/workingset.c b/mm/workingset.c
> index 1504f91cdca5..87d5e55dd684 100644
> --- a/mm/workingset.c
> +++ b/mm/workingset.c
> @@ -682,7 +682,16 @@ static unsigned long count_shadow_nodes(struct shrinker *shrinker,
>          * PAGE_SIZE / xa_nodes / node_entries * 8 / PAGE_SIZE
>          */
>  #ifdef CONFIG_MEMCG
> -       if (sc->memcg) {
> +       /*
> +        * Without kmem accounting the shadow_nodes list_lru is not
> +        * memcg-aware. So the count above covers the whole NUMA node, and the
> +        * shrinker only runs for the root memcg, whose local pages can be a
> +        * small fraction of the node's when most tasks run in child memcgs.
> +        * Compare the shadow node count against the memcg page count only when
> +        * the shadow_nodes list_lru is memcg-aware. Otherwise use the NUMA node
> +        * page count as is done without memcg.
> +        */
> +       if (sc->memcg && shadow_nodes.memcg_aware) {
>                 struct lruvec *lruvec;
>                 int i;
>
> --
> 2.56.0.455.g6c5a503b38-goog
>

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [PATCH] mm: workingset: compare shadow nodes node-wide without kmem accounting
  2026-11-16 17:39 ` T.J. Mercier
@ 2026-11-16 20:04   ` Barry Song
  2026-11-16 22:30     ` T.J. Mercier
  0 siblings, 1 reply; 5+ messages in thread
From: Barry Song @ 2026-11-16 20:04 UTC (permalink / raw)
  To: T.J. Mercier
  Cc: Andrew Morton, Johannes Weiner, Shakeel Butt, Michal Hocko,
	Roman Gushchin, Muchun Song, Qi Zheng, Dave Chinner,
	Michal Koutný, David Hildenbrand, Lorenzo Stoakes,
	Kairui Song, Axel Rasmussen, Yuanchu Xie, Wei Xu, Baoquan He,
	Baolin Wang, linux-mm, cgroups, linux-kernel, Jaewon Kim,
	Jaeseon Sim, Youngchan Kim, Dmitry Skiba

On Tue, Nov 17, 2026 at 1:40 AM T.J. Mercier <tjmercier@google.com> wrote:
>
> On Fri, Nov 6, 2026 at 3:59 PM T.J. Mercier <tjmercier@google.com> wrote:
> >
> > count_shadow_nodes() limits the number of xarray nodes holding shadow
> > entries to 1/8th of the pages that could be using them. For a memcg, it
> > counts the LRU and slab pages local to that memcg.
> >
> > With kmem accounting disabled (cgroup.memory=nokmem), shadow nodes are
> > not charged to any memcg and __list_lru_init() makes the shadow_nodes
> > list_lru non-memcg-aware, so all shadow nodes of a NUMA node are on a
> > single list. shrinker_alloc() falls back to a non-memcg-aware shadow
> > node shrinker, which shrink_slab() only invokes for the root memcg.
> > count_shadow_nodes() therefore compares every shadow node on the NUMA
> > node against the pages local to the root memcg.
> >
> > When most tasks run in child memcgs, the root memcg only holds a small
> > fraction of all pages, so the shadow node limit derived from the root
> > memcg page count is far below the number of shadow nodes on the NUMA
> > node. Since the shrinker has seeks == 0, every root shrink_slab() call
> > trims about half of the "excess" shadow nodes. Most refault information
> > is thrown away, so refaults of recently evicted file folios are not
> > detected. Refaulted folios lose PG_workingset and the workingset_refault
> > and workingset_restore stats drop.
> >
> > In testing on devices with kmem accounting disabled, moving tasks from
> > the root memcg into per-application memcgs dropped workingset_nodes from
> > ~37,000 to ~900 and raised workingset_nodereclaim 4.6x while
> > pgsteal_file rose by ~12%.
> >
> > When kmem accounting is disabled, compare the shadow nodes against the
> > whole NUMA node, as is done without memcg. Behavior with kmem
> > accounting enabled is unchanged.
> >
> > Cc: Jaewon Kim <jaewon31.kim@samsung.com>
> > Tested-by: Jaeseon Sim <jason.sim@samsung.com> # Galaxy A18 LTE
> > Tested-by: Youngchan Kim <ycshark.kim@samsung.com> # Galaxy S27
> > Tested-by: Dmitry Skiba <dskiba@google.com> # Pixel 10
> > Signed-off-by: T.J. Mercier <tjmercier@google.com>
>
> I'm guessing there aren't many other nokmem users?
>

Out of curiosity, is `cgroup.memory=nokmem` commonly used
on Android devices? If so, what's the rationale behind it?

As for the patch itself, it makes sense to me.

Reviewed-by: Barry Song <baohua@kernel.org>

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [PATCH] mm: workingset: compare shadow nodes node-wide without kmem accounting
  2026-11-16 20:04   ` Barry Song
@ 2026-11-16 22:30     ` T.J. Mercier
  0 siblings, 0 replies; 5+ messages in thread
From: T.J. Mercier @ 2026-11-16 22:30 UTC (permalink / raw)
  To: Barry Song
  Cc: Andrew Morton, Johannes Weiner, Shakeel Butt, Michal Hocko,
	Roman Gushchin, Muchun Song, Qi Zheng, Dave Chinner,
	Michal Koutný, David Hildenbrand, Lorenzo Stoakes,
	Kairui Song, Axel Rasmussen, Yuanchu Xie, Wei Xu, Baoquan He,
	Baolin Wang, linux-mm, cgroups, linux-kernel, Jaewon Kim,
	Jaeseon Sim, Youngchan Kim, Dmitry Skiba

On Mon, Nov 16, 2026 at 12:04 PM Barry Song <baohua@kernel.org> wrote:
>
> On Tue, Nov 17, 2026 at 1:40 AM T.J. Mercier <tjmercier@google.com> wrote:
> >
> > On Fri, Nov 6, 2026 at 3:59 PM T.J. Mercier <tjmercier@google.com> wrote:
> > >
> > > count_shadow_nodes() limits the number of xarray nodes holding shadow
> > > entries to 1/8th of the pages that could be using them. For a memcg, it
> > > counts the LRU and slab pages local to that memcg.
> > >
> > > With kmem accounting disabled (cgroup.memory=nokmem), shadow nodes are
> > > not charged to any memcg and __list_lru_init() makes the shadow_nodes
> > > list_lru non-memcg-aware, so all shadow nodes of a NUMA node are on a
> > > single list. shrinker_alloc() falls back to a non-memcg-aware shadow
> > > node shrinker, which shrink_slab() only invokes for the root memcg.
> > > count_shadow_nodes() therefore compares every shadow node on the NUMA
> > > node against the pages local to the root memcg.
> > >
> > > When most tasks run in child memcgs, the root memcg only holds a small
> > > fraction of all pages, so the shadow node limit derived from the root
> > > memcg page count is far below the number of shadow nodes on the NUMA
> > > node. Since the shrinker has seeks == 0, every root shrink_slab() call
> > > trims about half of the "excess" shadow nodes. Most refault information
> > > is thrown away, so refaults of recently evicted file folios are not
> > > detected. Refaulted folios lose PG_workingset and the workingset_refault
> > > and workingset_restore stats drop.
> > >
> > > In testing on devices with kmem accounting disabled, moving tasks from
> > > the root memcg into per-application memcgs dropped workingset_nodes from
> > > ~37,000 to ~900 and raised workingset_nodereclaim 4.6x while
> > > pgsteal_file rose by ~12%.
> > >
> > > When kmem accounting is disabled, compare the shadow nodes against the
> > > whole NUMA node, as is done without memcg. Behavior with kmem
> > > accounting enabled is unchanged.
> > >
> > > Cc: Jaewon Kim <jaewon31.kim@samsung.com>
> > > Tested-by: Jaeseon Sim <jason.sim@samsung.com> # Galaxy A18 LTE
> > > Tested-by: Youngchan Kim <ycshark.kim@samsung.com> # Galaxy S27
> > > Tested-by: Dmitry Skiba <dskiba@google.com> # Pixel 10
> > > Signed-off-by: T.J. Mercier <tjmercier@google.com>
> >
> > I'm guessing there aren't many other nokmem users?
> >
>
> Out of curiosity, is `cgroup.memory=nokmem` commonly used
> on Android devices? If so, what's the rationale behind it?
>
> As for the patch itself, it makes sense to me.
>
> Reviewed-by: Barry Song <baohua@kernel.org>

Thanks Barry.

It's a mix since we don't currently require kmem accounting. The main
reasons vendors use nokmem are to avoid kmem-specific overhead, and to
avoid bugs they've hit with kmem enabled. We had one report of kmem
increasing clone latency by ~25% because of kmem charging in
dup_mmap(). Another report of a NULL pointer dereference in
list_lru_add() during eviction on 6.6 also resulted in kmem accounting
being disabled on the affected device. Those issues were from over a
year ago, and I haven't seen anything else recently.

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [PATCH] mm: workingset: compare shadow nodes node-wide without kmem accounting
  2026-11-06 23:59 [PATCH] mm: workingset: compare shadow nodes node-wide without kmem accounting T.J. Mercier
  2026-11-16 17:39 ` T.J. Mercier
@ 2026-11-19  0:09 ` Johannes Weiner
  1 sibling, 0 replies; 5+ messages in thread
From: Johannes Weiner @ 2026-11-19  0:09 UTC (permalink / raw)
  To: T.J. Mercier
  Cc: Andrew Morton, Shakeel Butt, Michal Hocko, Roman Gushchin,
	Muchun Song, Qi Zheng, Dave Chinner, Michal Koutný,
	David Hildenbrand, Lorenzo Stoakes, Kairui Song, Barry Song,
	Axel Rasmussen, Yuanchu Xie, Wei Xu, Baoquan He, Baolin Wang,
	linux-mm, cgroups, linux-kernel, Jaewon Kim, Jaeseon Sim,
	Youngchan Kim, Dmitry Skiba

On Fri, Nov 06, 2026 at 03:59:04PM -0800, T.J. Mercier wrote:
> count_shadow_nodes() limits the number of xarray nodes holding shadow
> entries to 1/8th of the pages that could be using them. For a memcg, it
> counts the LRU and slab pages local to that memcg.
> 
> With kmem accounting disabled (cgroup.memory=nokmem), shadow nodes are
> not charged to any memcg and __list_lru_init() makes the shadow_nodes
> list_lru non-memcg-aware, so all shadow nodes of a NUMA node are on a
> single list. shrinker_alloc() falls back to a non-memcg-aware shadow
> node shrinker, which shrink_slab() only invokes for the root memcg.
> count_shadow_nodes() therefore compares every shadow node on the NUMA
> node against the pages local to the root memcg.
> 
> When most tasks run in child memcgs, the root memcg only holds a small
> fraction of all pages, so the shadow node limit derived from the root
> memcg page count is far below the number of shadow nodes on the NUMA
> node. Since the shrinker has seeks == 0, every root shrink_slab() call
> trims about half of the "excess" shadow nodes. Most refault information
> is thrown away, so refaults of recently evicted file folios are not
> detected. Refaulted folios lose PG_workingset and the workingset_refault
> and workingset_restore stats drop.
> 
> In testing on devices with kmem accounting disabled, moving tasks from
> the root memcg into per-application memcgs dropped workingset_nodes from
> ~37,000 to ~900 and raised workingset_nodereclaim 4.6x while
> pgsteal_file rose by ~12%.
> 
> When kmem accounting is disabled, compare the shadow nodes against the
> whole NUMA node, as is done without memcg. Behavior with kmem
> accounting enabled is unchanged.
> 
> Cc: Jaewon Kim <jaewon31.kim@samsung.com>
> Tested-by: Jaeseon Sim <jason.sim@samsung.com> # Galaxy A18 LTE
> Tested-by: Youngchan Kim <ycshark.kim@samsung.com> # Galaxy S27
> Tested-by: Dmitry Skiba <dskiba@google.com> # Pixel 10
> Signed-off-by: T.J. Mercier <tjmercier@google.com>

Reviewed-by: Johannes Weiner <hannes@cmpxchg.org>

Looks good to me. Yeah I don't think nokmem is used by too many
different environments!

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2026-11-19  0:09 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-06 23:59 [PATCH] mm: workingset: compare shadow nodes node-wide without kmem accounting T.J. Mercier
2026-11-16 17:39 ` T.J. Mercier
2026-11-16 20:04   ` Barry Song
2026-11-16 22:30     ` T.J. Mercier
2026-11-19  0:09 ` Johannes Weiner

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox