* Re: [PATCH v5 12/12] mm, swap: merge zeromap into swap table
From: Kairui Song @ 2026-05-22 2:39 UTC (permalink / raw)
To: Andrew Morton
Cc: kasong, Kairui Song via B4 Relay, linux-mm, David Hildenbrand,
Zi Yan, Baolin Wang, Barry Song, Hugh Dickins, Chris Li,
Kemeng Shi, Nhat Pham, Baoquan He, Johannes Weiner, Youngjun Park,
Chengming Zhou, Roman Gushchin, Shakeel Butt, Muchun Song,
Usama Arif, linux-kernel, cgroups, Lorenzo Stoakes, Yosry Ahmed,
Qi Zheng
In-Reply-To: <20260521185204.a109bfcd1e0e8f52135c5ed5@linux-foundation.org>
On Thu, May 21, 2026 at 06:52:04PM +0800, Andrew Morton wrote:
> On Sun, 17 May 2026 23:39:51 +0800 Kairui Song via B4 Relay <devnull+kasong.tencent.com@kernel.org> wrote:
>
> > From: Kairui Song <kasong@tencent.com>
> >
> > By allocating one additional bit in the swap table entry's flags field
> > alongside the count, we can store the zeromap inline
> >
> > For 64 bit systems, zeromap will store in the swap table, avoiding zeromap
> > allocation. It reduces the allocated memory. That is the happy path.
> >
> > For certain 32-bit archs, there might not be enough bits in the swap
> > table to contain both PFN and flags. Therefore, conditionally let each
> > cluster have a zeromap field at build time, and use that instead.
> > If the swapfile cluster is not fully used, it will still save memory for
> > zeromap. The empty cluster does not allocate a zeromap. In the worst case,
> > all cluster are fully populated. We will use memory similar to the
> > previous zeromap implementation.
> >
> > A few macros were moved to different headers for build time struct
> > definition.
> >
> > ...
> >
> > @@ -469,13 +474,21 @@ static int swap_cluster_alloc_table(struct swap_cluster_info *ci, gfp_t gfp)
> > VM_WARN_ON_ONCE(ci->memcg_table);
> > ci->memcg_table = kzalloc_obj(*ci->memcg_table, gfp);
> > if (!ci->memcg_table)
> > - ret = -ENOMEM;
> > + goto err_free;
> > }
> > #endif
> > - if (ret)
> > - swap_cluster_free_table(ci);
> >
> > - return ret;
> > +#if !SWAP_TABLE_HAS_ZEROFLAG
> > + VM_WARN_ON_ONCE(ci->zero_bitmap);
> > + ci->zero_bitmap = bitmap_zalloc(SWAPFILE_CLUSTER, gfp);
> > + if (!ci->zero_bitmap)
> > + goto err_free;
> > +#endif
> > + return 0;
> > +
> > +err_free:
> > + swap_cluster_free_table(ci);
> > + return -ENOMEM;
> > }
>
> My m68k defconfig warned. I'll do the below, which looks good enough.
> Please check.
>
> Perhaps a custom guard() handler would clean things up here.
>
>
> From: Andrew Morton <akpm@linux-foundation.org>
> Subject: mm-swap-merge-zeromap-into-swap-table-fix-2
> Date: Thu May 21 06:39:20 PM PDT 2026
>
> mm/swapfile.c: In function 'swap_cluster_alloc_table':
> mm/swapfile.c:488:1: warning: label 'err_free' defined but not used [-Wunused-label]
> 488 | err_free:
> | ^~~~~~~~
>
> Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
> Cc: Baoquan He <bhe@redhat.com>
> Cc: Barry Song <baohua@kernel.org>
> Cc: Chengming Zhou <chengming.zhou@linux.dev>
> Cc: Chris Li <chrisl@kernel.org>
> Cc: David Hildenbrand <david@kernel.org>
> Cc: Hugh Dickins <hughd@google.com>
> Cc: Johannes Weiner <hannes@cmpxchg.org>
> Cc: Kairui Song <kasong@tencent.com>
> Cc: Kemeng Shi <shikemeng@huaweicloud.com>
> Cc: Lorenzo Stoakes <ljs@kernel.org>
> Cc: Muchun Song <muchun.song@linux.dev>
> Cc: Nhat Pham <nphamcs@gmail.com>
> Cc: Roman Gushchin <roman.gushchin@linux.dev>
> Cc: Shakeel Butt <shakeel.butt@linux.dev>
> Cc: Youngjun Park <youngjun.park@lge.com>
> Cc: Zi Yan <ziy@nvidia.com>
> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
> ---
>
> mm/swapfile.c | 16 ++++++++--------
> 1 file changed, 8 insertions(+), 8 deletions(-)
>
> --- a/mm/swapfile.c~mm-swap-merge-zeromap-into-swap-table-fix-2
> +++ a/mm/swapfile.c
> @@ -472,22 +472,22 @@ static int swap_cluster_alloc_table(stru
> if (!mem_cgroup_disabled()) {
> VM_WARN_ON_ONCE(ci->memcg_table);
> ci->memcg_table = kzalloc_obj(*ci->memcg_table, gfp);
> - if (!ci->memcg_table)
> - goto err_free;
> + if (!ci->memcg_table) {
> + swap_cluster_free_table(ci);
> + return -ENOMEM;
> + }
> }
> #endif
>
> #if !SWAP_TABLE_HAS_ZEROFLAG
> VM_WARN_ON_ONCE(ci->zero_bitmap);
> ci->zero_bitmap = bitmap_zalloc(SWAPFILE_CLUSTER, gfp);
> - if (!ci->zero_bitmap)
> - goto err_free;
> + if (!ci->zero_bitmap) {
> + swap_cluster_free_table(ci);
> + return -ENOMEM;
> + }
> #endif
> return 0;
> -
> -err_free:
> - swap_cluster_free_table(ci);
> - return -ENOMEM;
> }
>
> /*
> _
>
Looks good, thank you. The error path is still simple and
straight at the moment.
It will be even better if we also remove the now unused ret
variable as well (this isn't triggering warning yet since Kbuild
don't set unused-but-set-variable by default):
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 9712cc862c9c..615d90867111 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -450,7 +450,6 @@ static int swap_cluster_alloc_table(struct swap_cluster_info *ci, gfp_t gfp)
{
struct swap_table *table = NULL;
struct folio *folio;
- int ret = 0;
/* The cluster must be empty and not on any list during allocation. */
VM_WARN_ON_ONCE(ci->flags || !cluster_is_empty(ci));
^ permalink raw reply related
* Re: [PATCH v2 3/4] memcg: int16_t for cached slab stats
From: Qi Zheng @ 2026-05-22 2:30 UTC (permalink / raw)
To: Shakeel Butt, Andrew Morton
Cc: Johannes Weiner, Michal Hocko, Roman Gushchin, Muchun Song,
Alexandre Ghiti, Joshua Hahn, Harry Yoo, Meta kernel team,
linux-mm, cgroups, linux-kernel, kernel test robot
In-Reply-To: <20260522011908.1669332-4-shakeel.butt@linux.dev>
On 5/22/26 9:19 AM, Shakeel Butt wrote:
> Currently struct obj_stock_pcp stores cached slab stats in 'int' which
> is 4 bytes per counter on 64-bit machines. Switch them to int16_t to
> shrink the cached metadata.
>
> The existing PAGE_SIZE flush in __account_obj_stock() bounds *bytes at
> PAGE_SIZE on 4KiB and 16KiB page archs, well within int16_t. On 64KiB
> pages PAGE_SIZE is well above S16_MAX so that flush never fires, and a
> sufficiently long run of accumulations would overflow the cache. Add
> an explicit S16_MAX guard before each add: when the next add would
> push abs(*bytes) past S16_MAX, fold the cached value into @nr and
> flush directly via mod_objcg_mlstate() before the accumulation.
>
> Fixes: 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg per-node type")
> Tested-by: kernel test robot <oliver.sang@intel.com>
> Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
> Reviewed-by: Harry Yoo (Oracle) <harry@kernel.org>
> ---
>
> Changes since v2:
> - Collected tags
>
> mm/memcontrol.c | 16 +++++++++++++---
> 1 file changed, 13 insertions(+), 3 deletions(-)
>
Acked-by: Qi Zheng <qi.zheng@linux.dev>
Thanks!
^ permalink raw reply
* Re: [PATCH v2 1/4] memcg: store node_id instead of pglist_data pointer
From: Qi Zheng @ 2026-05-22 2:27 UTC (permalink / raw)
To: Shakeel Butt, Andrew Morton
Cc: Johannes Weiner, Michal Hocko, Roman Gushchin, Muchun Song,
Alexandre Ghiti, Joshua Hahn, Harry Yoo, Meta kernel team,
linux-mm, cgroups, linux-kernel, kernel test robot
In-Reply-To: <20260522011908.1669332-2-shakeel.butt@linux.dev>
On 5/22/26 9:19 AM, Shakeel Butt wrote:
> The struct obj_stock_pcp stores a pointer to pglist_data for the slab
> stats cached on the cpu. On 64-bit machines, this costs 8 bytes. The
> pointer is not strictly required: NODE_DATA() can recover it from the
> node id. Replace cached_pgdat with int16_t node_id and use NUMA_NO_NODE
> as the "no stats cached" sentinel.
>
> At the moment all the archs limit MAX_NUMNODES to 1024 so int16_t is
> plenty; a BUILD_BUG_ON() makes sure we notice if that ever changes.
>
> Fixes: 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg per-node type")
> Tested-by: kernel test robot <oliver.sang@intel.com>
> Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
> Acked-by: Muchun Song <muchun.song@linux.dev>
> Reviewed-by: Harry Yoo (Oracle) <harry@kernel.org>
> ---
>
> Changes since v1:
> - Added tags in the commit message
>
> mm/memcontrol.c | 26 +++++++++++++++++++-------
> 1 file changed, 19 insertions(+), 7 deletions(-)
>
Acked-by: Qi Zheng <qi.zheng@linux.dev>
Thanks!
^ permalink raw reply
* Re: [PATCH v2 2/4] memcg: uint16_t for nr_bytes in obj_stock_pcp
From: Qi Zheng @ 2026-05-22 2:23 UTC (permalink / raw)
To: Shakeel Butt, Andrew Morton
Cc: Johannes Weiner, Michal Hocko, Roman Gushchin, Muchun Song,
Alexandre Ghiti, Joshua Hahn, Harry Yoo, Meta kernel team,
linux-mm, cgroups, linux-kernel, kernel test robot
In-Reply-To: <20260522011908.1669332-3-shakeel.butt@linux.dev>
On 5/22/26 9:19 AM, Shakeel Butt wrote:
> Currently struct obj_stock_pcp stores nr_bytes in an 'unsigned int'
> which is 4 bytes on 64-bit machines. Switch the field to uint16_t to
> shrink the per-CPU cache.
>
> The kernel supports PAGE_SIZE_4KB, _8KB, _16KB, _32KB, _64KB and
> _256KB (see HAVE_PAGE_SIZE_* in arch/Kconfig). After the
> PAGE_SIZE-aligned flush in __refill_obj_stock(), the sub-page
> remainder fits in uint16_t up through 64KiB pages where PAGE_SIZE - 1
> == U16_MAX, but on 256KiB pages PAGE_SIZE - 1 == 0x3FFFF exceeds
> U16_MAX. The accumulator also needs to stay within uint16_t between
> page-aligned flushes on 64KiB pages where PAGE_SIZE itself is
> U16_MAX + 1.
>
> Accumulate the new total in an 'unsigned int' local, then:
>
> 1. Flush whenever the accumulator would hit U16_MAX. Together with
> the existing allow_uncharge flush at PAGE_SIZE, this keeps the
> uint16_t safe on PAGE_SIZE <= 64KiB.
>
> 2. On configs with PAGE_SHIFT > 16 (PAGE_SIZE_256KB on hexagon and
> powerpc 44x), push any sub-page remainder above U16_MAX into
> objcg->nr_charged_bytes via atomic_add before storing back, so
> the store cannot silently truncate. The PAGE_SHIFT > 16 guard
> folds the branch out at compile time on smaller page sizes.
>
> Fixes: 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg per-node type")
> Tested-by: kernel test robot <oliver.sang@intel.com>
> Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
> Reviewed-by: Harry Yoo (Oracle) <harry@kernel.org>
> ---
>
> Changes since v1:
> - Collected tags
> - Rearrange fields of obj_stock_pcp (David Laight)
> - Fix comparison operator (Harry)
>
> mm/memcontrol.c | 33 +++++++++++++++++++++++++++------
> 1 file changed, 27 insertions(+), 6 deletions(-)
>
> diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> index d7c162946719..e4f00a8159d5 100644
> --- a/mm/memcontrol.c
> +++ b/mm/memcontrol.c
> @@ -2019,8 +2019,8 @@ static DEFINE_PER_CPU_ALIGNED(struct memcg_stock_pcp, memcg_stock) = {
>
> struct obj_stock_pcp {
> local_trylock_t lock;
> - unsigned int nr_bytes;
> struct obj_cgroup *cached_objcg;
> + uint16_t nr_bytes;
> int16_t node_id;
> int nr_slab_reclaimable_b;
> int nr_slab_unreclaimable_b;
> @@ -3331,6 +3331,7 @@ static void __refill_obj_stock(struct obj_cgroup *objcg,
> bool allow_uncharge)
> {
> unsigned int nr_pages = 0;
> + unsigned int stock_nr_bytes;
>
> if (!stock) {
> nr_pages = nr_bytes >> PAGE_SHIFT;
> @@ -3339,21 +3340,41 @@ static void __refill_obj_stock(struct obj_cgroup *objcg,
> goto out;
> }
>
> + stock_nr_bytes = stock->nr_bytes;
> if (READ_ONCE(stock->cached_objcg) != objcg) { /* reset if necessary */
> drain_obj_stock(stock);
> obj_cgroup_get(objcg);
> - stock->nr_bytes = atomic_read(&objcg->nr_charged_bytes)
> + stock_nr_bytes = atomic_read(&objcg->nr_charged_bytes)
> ? atomic_xchg(&objcg->nr_charged_bytes, 0) : 0;
> WRITE_ONCE(stock->cached_objcg, objcg);
>
> allow_uncharge = true; /* Allow uncharge when objcg changes */
> }
> - stock->nr_bytes += nr_bytes;
> + stock_nr_bytes += nr_bytes;
> +
> + /* Since stock->nr_bytes is uint16_t, don't refill >= U16_MAX */
^
should also be changed to: don't refill > U16_MAX ?
Otherwise:
Acked-by: Qi Zheng <qi.zheng@linux.dev>
Thanks!
> + if ((allow_uncharge && (stock_nr_bytes > PAGE_SIZE)) ||
> + stock_nr_bytes > U16_MAX) {
> + nr_pages = stock_nr_bytes >> PAGE_SHIFT;
> + stock_nr_bytes &= (PAGE_SIZE - 1);
> +
> + /*
> + * On configs with PAGE_SHIFT > 16 (PAGE_SIZE_256KB on
> + * hexagon and powerpc 44x), the sub-page remainder can
> + * still exceed U16_MAX. Push the excess back to
> + * objcg->nr_charged_bytes so the store into uint16_t
> + * cannot silently truncate; folded out at compile time
> + * on smaller page sizes.
> + */
> + if (PAGE_SHIFT > 16 && stock_nr_bytes > U16_MAX) {
> + unsigned int kept = stock_nr_bytes & U16_MAX;
>
> - if (allow_uncharge && (stock->nr_bytes > PAGE_SIZE)) {
> - nr_pages = stock->nr_bytes >> PAGE_SHIFT;
> - stock->nr_bytes &= (PAGE_SIZE - 1);
> + atomic_add(stock_nr_bytes - kept,
> + &objcg->nr_charged_bytes);
> + stock_nr_bytes = kept;
> + }
> }
> + stock->nr_bytes = stock_nr_bytes;
>
> out:
> if (nr_pages)
^ permalink raw reply
* Re: [PATCH v5 12/12] mm, swap: merge zeromap into swap table
From: Andrew Morton @ 2026-05-22 1:52 UTC (permalink / raw)
To: kasong
Cc: Kairui Song via B4 Relay, linux-mm, David Hildenbrand, Zi Yan,
Baolin Wang, Barry Song, Hugh Dickins, Chris Li, Kemeng Shi,
Nhat Pham, Baoquan He, Johannes Weiner, Youngjun Park,
Chengming Zhou, Roman Gushchin, Shakeel Butt, Muchun Song,
Usama Arif, linux-kernel, cgroups, Lorenzo Stoakes, Yosry Ahmed,
Qi Zheng
In-Reply-To: <20260517-swap-table-p4-v5-12-88ae43e064c7@tencent.com>
On Sun, 17 May 2026 23:39:51 +0800 Kairui Song via B4 Relay <devnull+kasong.tencent.com@kernel.org> wrote:
> From: Kairui Song <kasong@tencent.com>
>
> By allocating one additional bit in the swap table entry's flags field
> alongside the count, we can store the zeromap inline
>
> For 64 bit systems, zeromap will store in the swap table, avoiding zeromap
> allocation. It reduces the allocated memory. That is the happy path.
>
> For certain 32-bit archs, there might not be enough bits in the swap
> table to contain both PFN and flags. Therefore, conditionally let each
> cluster have a zeromap field at build time, and use that instead.
> If the swapfile cluster is not fully used, it will still save memory for
> zeromap. The empty cluster does not allocate a zeromap. In the worst case,
> all cluster are fully populated. We will use memory similar to the
> previous zeromap implementation.
>
> A few macros were moved to different headers for build time struct
> definition.
>
> ...
>
> @@ -469,13 +474,21 @@ static int swap_cluster_alloc_table(struct swap_cluster_info *ci, gfp_t gfp)
> VM_WARN_ON_ONCE(ci->memcg_table);
> ci->memcg_table = kzalloc_obj(*ci->memcg_table, gfp);
> if (!ci->memcg_table)
> - ret = -ENOMEM;
> + goto err_free;
> }
> #endif
> - if (ret)
> - swap_cluster_free_table(ci);
>
> - return ret;
> +#if !SWAP_TABLE_HAS_ZEROFLAG
> + VM_WARN_ON_ONCE(ci->zero_bitmap);
> + ci->zero_bitmap = bitmap_zalloc(SWAPFILE_CLUSTER, gfp);
> + if (!ci->zero_bitmap)
> + goto err_free;
> +#endif
> + return 0;
> +
> +err_free:
> + swap_cluster_free_table(ci);
> + return -ENOMEM;
> }
My m68k defconfig warned. I'll do the below, which looks good enough.
Please check.
Perhaps a custom guard() handler would clean things up here.
From: Andrew Morton <akpm@linux-foundation.org>
Subject: mm-swap-merge-zeromap-into-swap-table-fix-2
Date: Thu May 21 06:39:20 PM PDT 2026
mm/swapfile.c: In function 'swap_cluster_alloc_table':
mm/swapfile.c:488:1: warning: label 'err_free' defined but not used [-Wunused-label]
488 | err_free:
| ^~~~~~~~
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Baoquan He <bhe@redhat.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Chengming Zhou <chengming.zhou@linux.dev>
Cc: Chris Li <chrisl@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Hugh Dickins <hughd@google.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Youngjun Park <youngjun.park@lge.com>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
mm/swapfile.c | 16 ++++++++--------
1 file changed, 8 insertions(+), 8 deletions(-)
--- a/mm/swapfile.c~mm-swap-merge-zeromap-into-swap-table-fix-2
+++ a/mm/swapfile.c
@@ -472,22 +472,22 @@ static int swap_cluster_alloc_table(stru
if (!mem_cgroup_disabled()) {
VM_WARN_ON_ONCE(ci->memcg_table);
ci->memcg_table = kzalloc_obj(*ci->memcg_table, gfp);
- if (!ci->memcg_table)
- goto err_free;
+ if (!ci->memcg_table) {
+ swap_cluster_free_table(ci);
+ return -ENOMEM;
+ }
}
#endif
#if !SWAP_TABLE_HAS_ZEROFLAG
VM_WARN_ON_ONCE(ci->zero_bitmap);
ci->zero_bitmap = bitmap_zalloc(SWAPFILE_CLUSTER, gfp);
- if (!ci->zero_bitmap)
- goto err_free;
+ if (!ci->zero_bitmap) {
+ swap_cluster_free_table(ci);
+ return -ENOMEM;
+ }
#endif
return 0;
-
-err_free:
- swap_cluster_free_table(ci);
- return -ENOMEM;
}
/*
_
^ permalink raw reply
* [PATCH v2 4/4] memcg: multi objcg charge support
From: Shakeel Butt @ 2026-05-22 1:19 UTC (permalink / raw)
To: Andrew Morton
Cc: Johannes Weiner, Michal Hocko, Roman Gushchin, Muchun Song,
Qi Zheng, Alexandre Ghiti, Joshua Hahn, Harry Yoo,
Meta kernel team, linux-mm, cgroups, linux-kernel,
kernel test robot
In-Reply-To: <20260522011908.1669332-1-shakeel.butt@linux.dev>
Commit 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg
per-node type") split a memcg's single obj_cgroup into one per NUMA
node so that reparenting LRU folios can take per-node lru locks. As a
side effect, the per-CPU obj_stock_pcp -- which caches exactly one
cached_objcg -- thrashes on workloads where threads of the same memcg
run on different NUMA nodes. The kernel test robot reported a 67.7%
regression on stress-ng.switch.ops_per_sec from this pattern.
Mirror the multi-slot pattern already used by memcg_stock_pcp: turn
nr_bytes and cached_objcg into NR_OBJ_STOCK-element arrays, scan all
slots on consume/refill/account, prefer empty slots when inserting,
and evict a random slot only when full. With multiple slots a CPU can
hold the per-node objcg variants of one memcg plus a few siblings
without ever forcing a drain.
A single int8_t index records which slot the cached slab stats belong
to; the stats are flushed on slot or pgdat change. With NR_OBJ_STOCK
= 5 the layout (verified with pahole) is:
offset 0 : lock(1) + index(1) + node_id(2) + slab stats(4) = 8B
offset 8 : nr_bytes[5] = 10B
offset 18 : padding = 6B
offset 24 : cached[5] = 40B
offset 64 : (line 2) work_struct + flags (cold)
so consume_obj_stock, refill_obj_stock and the slab account path each
touch exactly one 64-byte cache line on non-debug 64-bit builds.
Reported-by: kernel test robot <oliver.sang@intel.com>
Closes: https://lore.kernel.org/oe-lkp/202605121641.b6a60cb0-lkp@intel.com
Fixes: 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg per-node type")
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
Tested-by: kernel test robot <oliver.sang@intel.com>
---
Changes since v1:
- Use round robin for drain
mm/memcontrol.c | 188 ++++++++++++++++++++++++++++++++++--------------
1 file changed, 136 insertions(+), 52 deletions(-)
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 78c02451312b..ba17633b0bd0 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -150,14 +150,14 @@ static void obj_cgroup_release(struct percpu_ref *ref)
* However, it can be PAGE_SIZE or (x * PAGE_SIZE).
*
* The following sequence can lead to it:
- * 1) CPU0: objcg == stock->cached_objcg
+ * 1) CPU0: objcg cached in one of stock->cached[i]
* 2) CPU1: we do a small allocation (e.g. 92 bytes),
* PAGE_SIZE bytes are charged
* 3) CPU1: a process from another memcg is allocating something,
* the stock if flushed,
* objcg->nr_charged_bytes = PAGE_SIZE - 92
* 5) CPU0: we do release this object,
- * 92 bytes are added to stock->nr_bytes
+ * 92 bytes are added to stock->nr_bytes[i]
* 6) CPU0: stock is flushed,
* 92 bytes are added to objcg->nr_charged_bytes
*
@@ -2017,25 +2017,40 @@ static DEFINE_PER_CPU_ALIGNED(struct memcg_stock_pcp, memcg_stock) = {
.lock = INIT_LOCAL_TRYLOCK(lock),
};
+/*
+ * NR_OBJ_STOCK is sized so the entire hot path of obj_stock_pcp
+ * (lock, accounting metadata, nr_bytes[] and cached[]) fits within a
+ * single 64-byte cache line on non-debug 64-bit builds. With 5 slots:
+ * lock(1) + index(1) + node_id(2) + slab stats(4) + nr_bytes(10)
+ * + pad(6) + cached(40) == 64 bytes.
+ * A CPU can thus consume/refill/account against five different objcgs
+ * (typically per-node variants of the same memcg) while incurring at
+ * most one cache miss on the stock.
+ */
+#define NR_OBJ_STOCK 5
struct obj_stock_pcp {
local_trylock_t lock;
- struct obj_cgroup *cached_objcg;
- uint16_t nr_bytes;
+ int8_t index;
int16_t node_id;
int16_t nr_slab_reclaimable_b;
int16_t nr_slab_unreclaimable_b;
+ uint16_t nr_bytes[NR_OBJ_STOCK];
+ struct obj_cgroup *cached[NR_OBJ_STOCK];
struct work_struct work;
unsigned long flags;
+ uint8_t drain_idx;
};
static DEFINE_PER_CPU_ALIGNED(struct obj_stock_pcp, obj_stock) = {
.lock = INIT_LOCAL_TRYLOCK(lock),
+ .index = -1,
.node_id = NUMA_NO_NODE,
};
static DEFINE_MUTEX(percpu_charge_mutex);
+static void drain_obj_stock_slot(struct obj_stock_pcp *stock, int i);
static void drain_obj_stock(struct obj_stock_pcp *stock);
static bool obj_stock_flush_required(struct obj_stock_pcp *stock,
struct mem_cgroup *root_memcg);
@@ -3153,12 +3168,13 @@ static void unlock_stock(struct obj_stock_pcp *stock)
local_unlock(&obj_stock.lock);
}
-/* Call after __refill_obj_stock() to ensure stock->cached_objg == objcg */
+/* Call after __refill_obj_stock() so a slot for objcg exists in the stock */
static void __account_obj_stock(struct obj_cgroup *objcg,
struct obj_stock_pcp *stock, int nr,
struct pglist_data *pgdat, enum node_stat_item idx)
{
int16_t *bytes;
+ int i;
/*
* Though at the moment MAX_NUMNODES <= 1024 in all archs but let's make
@@ -3167,29 +3183,39 @@ static void __account_obj_stock(struct obj_cgroup *objcg,
*/
BUILD_BUG_ON(MAX_NUMNODES >= S16_MAX);
- if (!stock || READ_ONCE(stock->cached_objcg) != objcg)
+ if (!stock)
+ goto direct;
+
+ for (i = 0; i < NR_OBJ_STOCK; ++i) {
+ if (READ_ONCE(stock->cached[i]) == objcg)
+ break;
+ }
+ if (i == NR_OBJ_STOCK)
goto direct;
/*
* Save vmstat data in stock and skip vmstat array update unless
- * accumulating over a page of vmstat data or when pgdat changes.
+ * accumulating over a page of vmstat data or when the objcg slot or
+ * pgdat the stats belong to changes.
*/
- if (stock->node_id == NUMA_NO_NODE) {
+ if (stock->index < 0) {
+ stock->index = i;
stock->node_id = pgdat->node_id;
- } else if (stock->node_id != pgdat->node_id) {
- /* Flush the existing cached vmstat data */
+ } else if (stock->index != i || stock->node_id != pgdat->node_id) {
+ struct obj_cgroup *old = READ_ONCE(stock->cached[stock->index]);
struct pglist_data *oldpg = NODE_DATA(stock->node_id);
if (stock->nr_slab_reclaimable_b) {
- mod_objcg_mlstate(objcg, oldpg, NR_SLAB_RECLAIMABLE_B,
+ mod_objcg_mlstate(old, oldpg, NR_SLAB_RECLAIMABLE_B,
stock->nr_slab_reclaimable_b);
stock->nr_slab_reclaimable_b = 0;
}
if (stock->nr_slab_unreclaimable_b) {
- mod_objcg_mlstate(objcg, oldpg, NR_SLAB_UNRECLAIMABLE_B,
+ mod_objcg_mlstate(old, oldpg, NR_SLAB_UNRECLAIMABLE_B,
stock->nr_slab_unreclaimable_b);
stock->nr_slab_unreclaimable_b = 0;
}
+ stock->index = i;
stock->node_id = pgdat->node_id;
}
@@ -3230,10 +3256,16 @@ static bool __consume_obj_stock(struct obj_cgroup *objcg,
struct obj_stock_pcp *stock,
unsigned int nr_bytes)
{
- if (objcg == READ_ONCE(stock->cached_objcg) &&
- stock->nr_bytes >= nr_bytes) {
- stock->nr_bytes -= nr_bytes;
- return true;
+ int i;
+
+ for (i = 0; i < NR_OBJ_STOCK; ++i) {
+ if (READ_ONCE(stock->cached[i]) != objcg)
+ continue;
+ if (stock->nr_bytes[i] >= nr_bytes) {
+ stock->nr_bytes[i] -= nr_bytes;
+ return true;
+ }
+ return false;
}
return false;
@@ -3254,16 +3286,42 @@ static bool consume_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes)
return ret;
}
-static void drain_obj_stock(struct obj_stock_pcp *stock)
+/* Flush the cached slab stats (if any) back to their owning objcg/pgdat. */
+static void drain_obj_stock_stats(struct obj_stock_pcp *stock)
{
- struct obj_cgroup *old = READ_ONCE(stock->cached_objcg);
+ struct obj_cgroup *old;
+ struct pglist_data *oldpg;
+
+ if (stock->index < 0)
+ return;
+
+ old = READ_ONCE(stock->cached[stock->index]);
+ oldpg = NODE_DATA(stock->node_id);
+
+ if (stock->nr_slab_reclaimable_b) {
+ mod_objcg_mlstate(old, oldpg, NR_SLAB_RECLAIMABLE_B,
+ stock->nr_slab_reclaimable_b);
+ stock->nr_slab_reclaimable_b = 0;
+ }
+ if (stock->nr_slab_unreclaimable_b) {
+ mod_objcg_mlstate(old, oldpg, NR_SLAB_UNRECLAIMABLE_B,
+ stock->nr_slab_unreclaimable_b);
+ stock->nr_slab_unreclaimable_b = 0;
+ }
+ stock->index = -1;
+ stock->node_id = NUMA_NO_NODE;
+}
+
+static void drain_obj_stock_slot(struct obj_stock_pcp *stock, int i)
+{
+ struct obj_cgroup *old = READ_ONCE(stock->cached[i]);
if (!old)
return;
- if (stock->nr_bytes) {
- unsigned int nr_pages = stock->nr_bytes >> PAGE_SHIFT;
- unsigned int nr_bytes = stock->nr_bytes & (PAGE_SIZE - 1);
+ if (stock->nr_bytes[i]) {
+ unsigned int nr_pages = stock->nr_bytes[i] >> PAGE_SHIFT;
+ unsigned int nr_bytes = stock->nr_bytes[i] & (PAGE_SIZE - 1);
if (nr_pages) {
struct mem_cgroup *memcg;
@@ -3289,46 +3347,43 @@ static void drain_obj_stock(struct obj_stock_pcp *stock)
* so it might be changed in the future.
*/
atomic_add(nr_bytes, &old->nr_charged_bytes);
- stock->nr_bytes = 0;
+ stock->nr_bytes[i] = 0;
}
- /*
- * Flush the vmstat data in current stock
- */
- if (stock->nr_slab_reclaimable_b || stock->nr_slab_unreclaimable_b) {
- struct pglist_data *oldpg = NODE_DATA(stock->node_id);
-
- if (stock->nr_slab_reclaimable_b) {
- mod_objcg_mlstate(old, oldpg,
- NR_SLAB_RECLAIMABLE_B,
- stock->nr_slab_reclaimable_b);
- stock->nr_slab_reclaimable_b = 0;
- }
- if (stock->nr_slab_unreclaimable_b) {
- mod_objcg_mlstate(old, oldpg,
- NR_SLAB_UNRECLAIMABLE_B,
- stock->nr_slab_unreclaimable_b);
- stock->nr_slab_unreclaimable_b = 0;
- }
- stock->node_id = NUMA_NO_NODE;
- }
+ /* Flush vmstat data when its owning slot is being drained. */
+ if (stock->index == i)
+ drain_obj_stock_stats(stock);
- WRITE_ONCE(stock->cached_objcg, NULL);
+ WRITE_ONCE(stock->cached[i], NULL);
obj_cgroup_put(old);
}
+static void drain_obj_stock(struct obj_stock_pcp *stock)
+{
+ int i;
+
+ for (i = 0; i < NR_OBJ_STOCK; ++i)
+ drain_obj_stock_slot(stock, i);
+}
+
static bool obj_stock_flush_required(struct obj_stock_pcp *stock,
struct mem_cgroup *root_memcg)
{
- struct obj_cgroup *objcg = READ_ONCE(stock->cached_objcg);
+ struct obj_cgroup *objcg;
struct mem_cgroup *memcg;
bool flush = false;
+ int i;
rcu_read_lock();
- if (objcg) {
+ for (i = 0; i < NR_OBJ_STOCK; ++i) {
+ objcg = READ_ONCE(stock->cached[i]);
+ if (!objcg)
+ continue;
memcg = obj_cgroup_memcg(objcg);
- if (memcg && mem_cgroup_is_descendant(memcg, root_memcg))
+ if (memcg && mem_cgroup_is_descendant(memcg, root_memcg)) {
flush = true;
+ break;
+ }
}
rcu_read_unlock();
@@ -3342,6 +3397,7 @@ static void __refill_obj_stock(struct obj_cgroup *objcg,
{
unsigned int nr_pages = 0;
unsigned int stock_nr_bytes;
+ int i, slot = -1, empty_slot = -1;
if (!stock) {
nr_pages = nr_bytes >> PAGE_SHIFT;
@@ -3350,19 +3406,47 @@ static void __refill_obj_stock(struct obj_cgroup *objcg,
goto out;
}
- stock_nr_bytes = stock->nr_bytes;
- if (READ_ONCE(stock->cached_objcg) != objcg) { /* reset if necessary */
- drain_obj_stock(stock);
+ for (i = 0; i < NR_OBJ_STOCK; ++i) {
+ struct obj_cgroup *cached = READ_ONCE(stock->cached[i]);
+
+ if (!cached) {
+ if (empty_slot == -1)
+ empty_slot = i;
+ continue;
+ }
+ if (cached == objcg) {
+ slot = i;
+ break;
+ }
+ }
+
+ if (slot == -1) {
+ slot = empty_slot;
+ if (slot == -1) {
+ slot = stock->drain_idx++;
+ if (stock->drain_idx == NR_OBJ_STOCK)
+ stock->drain_idx = 0;
+ drain_obj_stock_slot(stock, slot);
+ }
obj_cgroup_get(objcg);
+ /*
+ * Keep the xchg result in the unsigned int local; storing
+ * it directly into stock->nr_bytes[slot] (uint16_t) would
+ * silently truncate values >= U16_MAX and bypass the flush
+ * guard below, leaking page-counter charges.
+ */
stock_nr_bytes = atomic_read(&objcg->nr_charged_bytes)
? atomic_xchg(&objcg->nr_charged_bytes, 0) : 0;
- WRITE_ONCE(stock->cached_objcg, objcg);
+ WRITE_ONCE(stock->cached[slot], objcg);
allow_uncharge = true; /* Allow uncharge when objcg changes */
+ } else {
+ stock_nr_bytes = stock->nr_bytes[slot];
}
+
stock_nr_bytes += nr_bytes;
- /* Since stock->nr_bytes is uint16_t, don't refill >= U16_MAX */
+ /* nr_bytes[] is uint16_t; flush if we would refill >= U16_MAX. */
if ((allow_uncharge && (stock_nr_bytes > PAGE_SIZE)) ||
stock_nr_bytes > U16_MAX) {
nr_pages = stock_nr_bytes >> PAGE_SHIFT;
@@ -3384,7 +3468,7 @@ static void __refill_obj_stock(struct obj_cgroup *objcg,
stock_nr_bytes = kept;
}
}
- stock->nr_bytes = stock_nr_bytes;
+ stock->nr_bytes[slot] = stock_nr_bytes;
out:
if (nr_pages)
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH v2 3/4] memcg: int16_t for cached slab stats
From: Shakeel Butt @ 2026-05-22 1:19 UTC (permalink / raw)
To: Andrew Morton
Cc: Johannes Weiner, Michal Hocko, Roman Gushchin, Muchun Song,
Qi Zheng, Alexandre Ghiti, Joshua Hahn, Harry Yoo,
Meta kernel team, linux-mm, cgroups, linux-kernel,
kernel test robot
In-Reply-To: <20260522011908.1669332-1-shakeel.butt@linux.dev>
Currently struct obj_stock_pcp stores cached slab stats in 'int' which
is 4 bytes per counter on 64-bit machines. Switch them to int16_t to
shrink the cached metadata.
The existing PAGE_SIZE flush in __account_obj_stock() bounds *bytes at
PAGE_SIZE on 4KiB and 16KiB page archs, well within int16_t. On 64KiB
pages PAGE_SIZE is well above S16_MAX so that flush never fires, and a
sufficiently long run of accumulations would overflow the cache. Add
an explicit S16_MAX guard before each add: when the next add would
push abs(*bytes) past S16_MAX, fold the cached value into @nr and
flush directly via mod_objcg_mlstate() before the accumulation.
Fixes: 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg per-node type")
Tested-by: kernel test robot <oliver.sang@intel.com>
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
Reviewed-by: Harry Yoo (Oracle) <harry@kernel.org>
---
Changes since v2:
- Collected tags
mm/memcontrol.c | 16 +++++++++++++---
1 file changed, 13 insertions(+), 3 deletions(-)
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index e4f00a8159d5..78c02451312b 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -2022,8 +2022,8 @@ struct obj_stock_pcp {
struct obj_cgroup *cached_objcg;
uint16_t nr_bytes;
int16_t node_id;
- int nr_slab_reclaimable_b;
- int nr_slab_unreclaimable_b;
+ int16_t nr_slab_reclaimable_b;
+ int16_t nr_slab_unreclaimable_b;
struct work_struct work;
unsigned long flags;
@@ -3158,7 +3158,7 @@ static void __account_obj_stock(struct obj_cgroup *objcg,
struct obj_stock_pcp *stock, int nr,
struct pglist_data *pgdat, enum node_stat_item idx)
{
- int *bytes;
+ int16_t *bytes;
/*
* Though at the moment MAX_NUMNODES <= 1024 in all archs but let's make
@@ -3195,6 +3195,16 @@ static void __account_obj_stock(struct obj_cgroup *objcg,
bytes = (idx == NR_SLAB_RECLAIMABLE_B) ? &stock->nr_slab_reclaimable_b
: &stock->nr_slab_unreclaimable_b;
+ /*
+ * To avoid overflow or underflow, flush directly if accumulating @nr
+ * would push the cached value past S16_MAX.
+ */
+ if (abs(nr + *bytes) > S16_MAX) {
+ nr += *bytes;
+ *bytes = 0;
+ goto direct;
+ }
+
/*
* Even for large object >= PAGE_SIZE, the vmstat data will still be
* cached locally at least once before pushing it out.
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH v2 2/4] memcg: uint16_t for nr_bytes in obj_stock_pcp
From: Shakeel Butt @ 2026-05-22 1:19 UTC (permalink / raw)
To: Andrew Morton
Cc: Johannes Weiner, Michal Hocko, Roman Gushchin, Muchun Song,
Qi Zheng, Alexandre Ghiti, Joshua Hahn, Harry Yoo,
Meta kernel team, linux-mm, cgroups, linux-kernel,
kernel test robot
In-Reply-To: <20260522011908.1669332-1-shakeel.butt@linux.dev>
Currently struct obj_stock_pcp stores nr_bytes in an 'unsigned int'
which is 4 bytes on 64-bit machines. Switch the field to uint16_t to
shrink the per-CPU cache.
The kernel supports PAGE_SIZE_4KB, _8KB, _16KB, _32KB, _64KB and
_256KB (see HAVE_PAGE_SIZE_* in arch/Kconfig). After the
PAGE_SIZE-aligned flush in __refill_obj_stock(), the sub-page
remainder fits in uint16_t up through 64KiB pages where PAGE_SIZE - 1
== U16_MAX, but on 256KiB pages PAGE_SIZE - 1 == 0x3FFFF exceeds
U16_MAX. The accumulator also needs to stay within uint16_t between
page-aligned flushes on 64KiB pages where PAGE_SIZE itself is
U16_MAX + 1.
Accumulate the new total in an 'unsigned int' local, then:
1. Flush whenever the accumulator would hit U16_MAX. Together with
the existing allow_uncharge flush at PAGE_SIZE, this keeps the
uint16_t safe on PAGE_SIZE <= 64KiB.
2. On configs with PAGE_SHIFT > 16 (PAGE_SIZE_256KB on hexagon and
powerpc 44x), push any sub-page remainder above U16_MAX into
objcg->nr_charged_bytes via atomic_add before storing back, so
the store cannot silently truncate. The PAGE_SHIFT > 16 guard
folds the branch out at compile time on smaller page sizes.
Fixes: 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg per-node type")
Tested-by: kernel test robot <oliver.sang@intel.com>
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
Reviewed-by: Harry Yoo (Oracle) <harry@kernel.org>
---
Changes since v1:
- Collected tags
- Rearrange fields of obj_stock_pcp (David Laight)
- Fix comparison operator (Harry)
mm/memcontrol.c | 33 +++++++++++++++++++++++++++------
1 file changed, 27 insertions(+), 6 deletions(-)
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index d7c162946719..e4f00a8159d5 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -2019,8 +2019,8 @@ static DEFINE_PER_CPU_ALIGNED(struct memcg_stock_pcp, memcg_stock) = {
struct obj_stock_pcp {
local_trylock_t lock;
- unsigned int nr_bytes;
struct obj_cgroup *cached_objcg;
+ uint16_t nr_bytes;
int16_t node_id;
int nr_slab_reclaimable_b;
int nr_slab_unreclaimable_b;
@@ -3331,6 +3331,7 @@ static void __refill_obj_stock(struct obj_cgroup *objcg,
bool allow_uncharge)
{
unsigned int nr_pages = 0;
+ unsigned int stock_nr_bytes;
if (!stock) {
nr_pages = nr_bytes >> PAGE_SHIFT;
@@ -3339,21 +3340,41 @@ static void __refill_obj_stock(struct obj_cgroup *objcg,
goto out;
}
+ stock_nr_bytes = stock->nr_bytes;
if (READ_ONCE(stock->cached_objcg) != objcg) { /* reset if necessary */
drain_obj_stock(stock);
obj_cgroup_get(objcg);
- stock->nr_bytes = atomic_read(&objcg->nr_charged_bytes)
+ stock_nr_bytes = atomic_read(&objcg->nr_charged_bytes)
? atomic_xchg(&objcg->nr_charged_bytes, 0) : 0;
WRITE_ONCE(stock->cached_objcg, objcg);
allow_uncharge = true; /* Allow uncharge when objcg changes */
}
- stock->nr_bytes += nr_bytes;
+ stock_nr_bytes += nr_bytes;
+
+ /* Since stock->nr_bytes is uint16_t, don't refill >= U16_MAX */
+ if ((allow_uncharge && (stock_nr_bytes > PAGE_SIZE)) ||
+ stock_nr_bytes > U16_MAX) {
+ nr_pages = stock_nr_bytes >> PAGE_SHIFT;
+ stock_nr_bytes &= (PAGE_SIZE - 1);
+
+ /*
+ * On configs with PAGE_SHIFT > 16 (PAGE_SIZE_256KB on
+ * hexagon and powerpc 44x), the sub-page remainder can
+ * still exceed U16_MAX. Push the excess back to
+ * objcg->nr_charged_bytes so the store into uint16_t
+ * cannot silently truncate; folded out at compile time
+ * on smaller page sizes.
+ */
+ if (PAGE_SHIFT > 16 && stock_nr_bytes > U16_MAX) {
+ unsigned int kept = stock_nr_bytes & U16_MAX;
- if (allow_uncharge && (stock->nr_bytes > PAGE_SIZE)) {
- nr_pages = stock->nr_bytes >> PAGE_SHIFT;
- stock->nr_bytes &= (PAGE_SIZE - 1);
+ atomic_add(stock_nr_bytes - kept,
+ &objcg->nr_charged_bytes);
+ stock_nr_bytes = kept;
+ }
}
+ stock->nr_bytes = stock_nr_bytes;
out:
if (nr_pages)
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH v2 1/4] memcg: store node_id instead of pglist_data pointer
From: Shakeel Butt @ 2026-05-22 1:19 UTC (permalink / raw)
To: Andrew Morton
Cc: Johannes Weiner, Michal Hocko, Roman Gushchin, Muchun Song,
Qi Zheng, Alexandre Ghiti, Joshua Hahn, Harry Yoo,
Meta kernel team, linux-mm, cgroups, linux-kernel,
kernel test robot
In-Reply-To: <20260522011908.1669332-1-shakeel.butt@linux.dev>
The struct obj_stock_pcp stores a pointer to pglist_data for the slab
stats cached on the cpu. On 64-bit machines, this costs 8 bytes. The
pointer is not strictly required: NODE_DATA() can recover it from the
node id. Replace cached_pgdat with int16_t node_id and use NUMA_NO_NODE
as the "no stats cached" sentinel.
At the moment all the archs limit MAX_NUMNODES to 1024 so int16_t is
plenty; a BUILD_BUG_ON() makes sure we notice if that ever changes.
Fixes: 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg per-node type")
Tested-by: kernel test robot <oliver.sang@intel.com>
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
Acked-by: Muchun Song <muchun.song@linux.dev>
Reviewed-by: Harry Yoo (Oracle) <harry@kernel.org>
---
Changes since v1:
- Added tags in the commit message
mm/memcontrol.c | 26 +++++++++++++++++++-------
1 file changed, 19 insertions(+), 7 deletions(-)
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index b8caeb7ccaa3..d7c162946719 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -2021,7 +2021,7 @@ struct obj_stock_pcp {
local_trylock_t lock;
unsigned int nr_bytes;
struct obj_cgroup *cached_objcg;
- struct pglist_data *cached_pgdat;
+ int16_t node_id;
int nr_slab_reclaimable_b;
int nr_slab_unreclaimable_b;
@@ -2031,6 +2031,7 @@ struct obj_stock_pcp {
static DEFINE_PER_CPU_ALIGNED(struct obj_stock_pcp, obj_stock) = {
.lock = INIT_LOCAL_TRYLOCK(lock),
+ .node_id = NUMA_NO_NODE,
};
static DEFINE_MUTEX(percpu_charge_mutex);
@@ -3159,6 +3160,13 @@ static void __account_obj_stock(struct obj_cgroup *objcg,
{
int *bytes;
+ /*
+ * Though at the moment MAX_NUMNODES <= 1024 in all archs but let's make
+ * sure it does not exceed S16_MAX otherwise we need to fix node_id type
+ * in struct obj_stock_pcp.
+ */
+ BUILD_BUG_ON(MAX_NUMNODES >= S16_MAX);
+
if (!stock || READ_ONCE(stock->cached_objcg) != objcg)
goto direct;
@@ -3166,9 +3174,11 @@ static void __account_obj_stock(struct obj_cgroup *objcg,
* Save vmstat data in stock and skip vmstat array update unless
* accumulating over a page of vmstat data or when pgdat changes.
*/
- if (stock->cached_pgdat != pgdat) {
+ if (stock->node_id == NUMA_NO_NODE) {
+ stock->node_id = pgdat->node_id;
+ } else if (stock->node_id != pgdat->node_id) {
/* Flush the existing cached vmstat data */
- struct pglist_data *oldpg = stock->cached_pgdat;
+ struct pglist_data *oldpg = NODE_DATA(stock->node_id);
if (stock->nr_slab_reclaimable_b) {
mod_objcg_mlstate(objcg, oldpg, NR_SLAB_RECLAIMABLE_B,
@@ -3180,7 +3190,7 @@ static void __account_obj_stock(struct obj_cgroup *objcg,
stock->nr_slab_unreclaimable_b);
stock->nr_slab_unreclaimable_b = 0;
}
- stock->cached_pgdat = pgdat;
+ stock->node_id = pgdat->node_id;
}
bytes = (idx == NR_SLAB_RECLAIMABLE_B) ? &stock->nr_slab_reclaimable_b
@@ -3276,19 +3286,21 @@ static void drain_obj_stock(struct obj_stock_pcp *stock)
* Flush the vmstat data in current stock
*/
if (stock->nr_slab_reclaimable_b || stock->nr_slab_unreclaimable_b) {
+ struct pglist_data *oldpg = NODE_DATA(stock->node_id);
+
if (stock->nr_slab_reclaimable_b) {
- mod_objcg_mlstate(old, stock->cached_pgdat,
+ mod_objcg_mlstate(old, oldpg,
NR_SLAB_RECLAIMABLE_B,
stock->nr_slab_reclaimable_b);
stock->nr_slab_reclaimable_b = 0;
}
if (stock->nr_slab_unreclaimable_b) {
- mod_objcg_mlstate(old, stock->cached_pgdat,
+ mod_objcg_mlstate(old, oldpg,
NR_SLAB_UNRECLAIMABLE_B,
stock->nr_slab_unreclaimable_b);
stock->nr_slab_unreclaimable_b = 0;
}
- stock->cached_pgdat = NULL;
+ stock->node_id = NUMA_NO_NODE;
}
WRITE_ONCE(stock->cached_objcg, NULL);
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH v2 0/4] memcg: shrink obj_stock_pcp and cache multiple objcgs
From: Shakeel Butt @ 2026-05-22 1:19 UTC (permalink / raw)
To: Andrew Morton
Cc: Johannes Weiner, Michal Hocko, Roman Gushchin, Muchun Song,
Qi Zheng, Alexandre Ghiti, Joshua Hahn, Harry Yoo,
Meta kernel team, linux-mm, cgroups, linux-kernel,
kernel test robot
Commit 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg
per-node type") split a memcg's single obj_cgroup into one per NUMA
node so that reparenting LRU folios can take per-node lru locks. As a
side effect, the per-CPU obj_stock_pcp -- which caches a single
cached_objcg pointer -- thrashes on workloads where threads of the
same memcg run on different NUMA nodes. The kernel test robot reported
a 67.7% regression on stress-ng.switch.ops_per_sec from this pattern.
Commit d0211878ce06 ("memcg: cache obj_stock by memcg, not by objcg
pointer") landed as a temporary fix by treating sibling per-node
objcgs as equivalent for the cache lookup, intended to be reverted
once per-node kmem accounting is introduced. This series takes a more
general approach: cache multiple objcgs per CPU using the multi-slot
pattern memcg_stock_pcp already uses, so the per-node objcg variants
of one memcg can all coexist in the stock without ever forcing a
drain. The temporary fix can then be reverted.
To avoid increasing the per-CPU cache footprint, the first three
patches shrink the existing single-slot obj_stock_pcp fields.
The final patch converts cached_objcg and nr_bytes into
NR_OBJ_STOCK=5 slot arrays and reorders the struct so the entire
consume/refill/account hot path fits within a single 64-byte cache
line on non-debug 64-bit builds (verified with pahole).
Reported-by: kernel test robot <oliver.sang@intel.com>
Closes: https://lore.kernel.org/oe-lkp/202605121641.b6a60cb0-lkp@intel.com
Fixes: 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg per-node type")
Tested-by: kernel test robot <oliver.sang@intel.com>
Shakeel Butt (4):
memcg: store node_id instead of pglist_data pointer
memcg: uint16_t for nr_bytes in obj_stock_pcp
memcg: int16_t for cached slab stats
memcg: multi objcg charge support
mm/memcontrol.c | 214 +++++++++++++++++++++++++++++++++++-------------
1 file changed, 157 insertions(+), 57 deletions(-)
--
Changes since v1:
http://lore.kernel.org/20260520053123.2709959-1-shakeel.butt@linux.dev
- Collected review tags (Harry & Muchun)
- Fix comparison operators (Harry)
- Use round robin for drain
2.53.0-Meta
^ permalink raw reply
* Re: [PATCH] memcg: use round-robin victim selection in refill_stock
From: Harry Yoo @ 2026-05-22 0:54 UTC (permalink / raw)
To: Shakeel Butt, Andrew Morton
Cc: Johannes Weiner, Michal Hocko, Roman Gushchin, Muchun Song,
Meta kernel team, linux-mm, cgroups, linux-kernel
In-Reply-To: <20260521223751.3794625-1-shakeel.butt@linux.dev>
On 5/22/26 7:37 AM, Shakeel Butt wrote:
> Harry Yoo reported that get_random_u32_below() is not safe to call in
> the nmi context and memcg charge draining can happen in nmi context.
>
> More specifically get_random_u32_below() is neither reentrant- nor
> NMI-safe: it acquires a per-cpu local_lock via local_lock_irqsave() on
> the batched_entropy_u32 state. An NMI that lands on a CPU mid-update of
> the ChaCha batch state and recurses into the random subsystem would
> corrupt that state. The memcg_stock local_trylock prevents re-entry
> on the percpu stock itself, but cannot protect an unrelated
> subsystem's per-cpu lock.
>
> Replace the random pick with a per-cpu round-robin counter stored in
> memcg_stock_pcp and serialized by the same local_trylock that already
> guards cached[] and nr_pages[]. No atomics, no random calls, no extra
> locks needed.
>
> Fixes: f735eebe55f8f ("memcg: multi-memcg percpu charge cache")
Acked-by: Harry Yoo (Oracle) <harry@kernel.org>
and perhaps
Cc: <stable@vger.kernel.org>
as it affects v6.18 (the latest LTS).
Thanks a lot for fixing it, Shakeel!
> Reported-by: Harry Yoo <harry@kernel.org>
> Closes: https://lore.kernel.org/4e20f643-6983-4b6e-b12d-c6c4eb20ae0c@kernel.org/
> Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
> ---
> mm/memcontrol.c | 5 ++++-
> 1 file changed, 4 insertions(+), 1 deletion(-)
>
> diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> index 0eb50e639f0a..6392a2704441 100644
> --- a/mm/memcontrol.c
> +++ b/mm/memcontrol.c
> @@ -2031,6 +2031,7 @@ struct memcg_stock_pcp {
>
> struct work_struct work;
> unsigned long flags;
> + uint8_t drain_idx;
> };
>
> static DEFINE_PER_CPU_ALIGNED(struct memcg_stock_pcp, memcg_stock) = {
> @@ -2214,7 +2215,9 @@ static void refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages)
> if (!success) {
> i = empty_slot;
> if (i == -1) {
> - i = get_random_u32_below(NR_MEMCG_STOCK);
> + i = stock->drain_idx++;
> + if (stock->drain_idx == NR_MEMCG_STOCK)
> + stock->drain_idx = 0;
> drain_stock(stock, i);
> }
> css_get(&memcg->css);
--
Cheers,
Harry / Hyeonggon
^ permalink raw reply
* [PATCH] memcg: use round-robin victim selection in refill_stock
From: Shakeel Butt @ 2026-05-21 22:37 UTC (permalink / raw)
To: Andrew Morton
Cc: Johannes Weiner, Michal Hocko, Roman Gushchin, Muchun Song,
Harry Yoo, Meta kernel team, linux-mm, cgroups, linux-kernel
Harry Yoo reported that get_random_u32_below() is not safe to call in
the nmi context and memcg charge draining can happen in nmi context.
More specifically get_random_u32_below() is neither reentrant- nor
NMI-safe: it acquires a per-cpu local_lock via local_lock_irqsave() on
the batched_entropy_u32 state. An NMI that lands on a CPU mid-update of
the ChaCha batch state and recurses into the random subsystem would
corrupt that state. The memcg_stock local_trylock prevents re-entry
on the percpu stock itself, but cannot protect an unrelated
subsystem's per-cpu lock.
Replace the random pick with a per-cpu round-robin counter stored in
memcg_stock_pcp and serialized by the same local_trylock that already
guards cached[] and nr_pages[]. No atomics, no random calls, no extra
locks needed.
Fixes: f735eebe55f8f ("memcg: multi-memcg percpu charge cache")
Reported-by: Harry Yoo <harry@kernel.org>
Closes: https://lore.kernel.org/4e20f643-6983-4b6e-b12d-c6c4eb20ae0c@kernel.org/
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
---
mm/memcontrol.c | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 0eb50e639f0a..6392a2704441 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -2031,6 +2031,7 @@ struct memcg_stock_pcp {
struct work_struct work;
unsigned long flags;
+ uint8_t drain_idx;
};
static DEFINE_PER_CPU_ALIGNED(struct memcg_stock_pcp, memcg_stock) = {
@@ -2214,7 +2215,9 @@ static void refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages)
if (!success) {
i = empty_slot;
if (i == -1) {
- i = get_random_u32_below(NR_MEMCG_STOCK);
+ i = stock->drain_idx++;
+ if (stock->drain_idx == NR_MEMCG_STOCK)
+ stock->drain_idx = 0;
drain_stock(stock, i);
}
css_get(&memcg->css);
--
2.53.0-Meta
^ permalink raw reply related
* Re: [PATCH 4/4] memcg: multi objcg charge support
From: Shakeel Butt @ 2026-05-21 20:19 UTC (permalink / raw)
To: Harry Yoo
Cc: Andrew Morton, Johannes Weiner, Michal Hocko, Roman Gushchin,
Muchun Song, Qi Zheng, Alexandre Ghiti, Joshua Hahn,
Meta kernel team, linux-mm, cgroups, linux-kernel,
kernel test robot
In-Reply-To: <5b09f618-3b84-4163-84f9-f3adc0f1cc97@kernel.org>
On Thu, May 21, 2026 at 10:43:11AM +0900, Harry Yoo wrote:
>
>
> On 5/21/26 10:05 AM, Shakeel Butt wrote:
> > On Wed, May 20, 2026 at 06:35:30PM +0900, Harry Yoo wrote:
> > > > @@ -3350,19 +3405,45 @@ static void __refill_obj_stock(struct obj_cgroup *objcg,
> > > > goto out;
> > > > }
> > > > - stock_nr_bytes = stock->nr_bytes;
> > > > - if (READ_ONCE(stock->cached_objcg) != objcg) { /* reset if necessary */
> > > > - drain_obj_stock(stock);
> > > > + for (i = 0; i < NR_OBJ_STOCK; ++i) {
> > > > + struct obj_cgroup *cached = READ_ONCE(stock->cached[i]);
> > > > +
> > > > + if (!cached) {
> > > > + if (empty_slot == -1)
> > > > + empty_slot = i;
> > > > + continue;
> > > > + }
> > > > + if (cached == objcg) {
> > > > + slot = i;
> > > > + break;
> > > > + }
> > > > + }
> > > > +
> > > > + if (slot == -1) {
> > > > + slot = empty_slot;
> > > > + if (slot == -1) {
> > > > + slot = get_random_u32_below(NR_OBJ_STOCK);
> > >
> > > It would break kmalloc_nolock() because _get_random_bytes() uses a spinlock.
> > > perhaps prandom_u32_state() should be sufficient in this case.
>
> s/spinlock/local_lock/
I do see spinlock in crng_make_state() for some code paths.
>
> > > Is there a reason why it uses random eviction, unlike multi-memcg percpu
> > > charge cache?
> >
> > Oh I didn't know and actually we are already using get_random_u32_below() in
> > refill_stock(). So, it need fixing as well. That would be a separate patch.
>
> Ouch, I see.
> > I will explore prandom_u32_state().
>
> Thanks!
>
> FYI, SLUB had a similar issue that was recently fixed:
> commit a1e244a9f1778 ("mm/slab: use prandom if !allow_spin").
>
> It uses prandom if spinning is not allowed when shuffling slab freelist.
The drain does not really need a random number. Fixing an index like 0 makes it
much simpler but it might expose some corner cases. Round robin might be enough
for this though. I will see what would be the easiest way forward.
^ permalink raw reply
* Re: [PATCH v4 5/8] mm: list_lru: introduce caller locking for additions and deletions
From: Liam R . Howlett @ 2026-05-21 17:30 UTC (permalink / raw)
To: Johannes Weiner
Cc: Andrew Morton, David Hildenbrand, Lorenzo Stoakes, Shakeel Butt,
Michal Hocko, Dave Chinner, Roman Gushchin, Muchun Song, Qi Zheng,
Yosry Ahmed, Zi Yan, Usama Arif, Kiryl Shutsemau, Vlastimil Babka,
Kairui Song, Mikhail Zaslonko, Vasily Gorbik, Baolin Wang,
Barry Song, Dev Jain, Lance Yang, Nico Pache, Ryan Roberts,
cgroups, linux-mm, linux-kernel
In-Reply-To: <20260521150330.1955924-6-hannes@cmpxchg.org>
On 26/05/21 11:02AM, Johannes Weiner wrote:
> Locking is currently internal to the list_lru API. However, a caller
> might want to keep auxiliary state synchronized with the LRU state.
>
> For example, the THP shrinker uses the lock of its custom LRU to keep
> PG_partially_mapped and vmstats consistent.
>
> To allow the THP shrinker to switch to list_lru, provide normal and
> irqsafe locking primitives as well as caller-locked variants of the
> addition and deletion functions.
>
> Reviewed-by: David Hildenbrand (Arm) <david@kernel.org>
> Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
> Reviewed-by: Lorenzo Stoakes (Oracle) <ljs@kernel.org>
> Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
Reviewed-by: Liam R. Howlett (Oracle) <liam@infradead.org>
> ---
> include/linux/list_lru.h | 41 ++++++++++++
> mm/list_lru.c | 131 ++++++++++++++++++++++++++++++---------
> 2 files changed, 141 insertions(+), 31 deletions(-)
>
> diff --git a/include/linux/list_lru.h b/include/linux/list_lru.h
> index fe739d35a864..c79ed378311f 100644
> --- a/include/linux/list_lru.h
> +++ b/include/linux/list_lru.h
> @@ -83,6 +83,44 @@ int memcg_list_lru_alloc(struct mem_cgroup *memcg, struct list_lru *lru,
> gfp_t gfp);
> void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup *parent);
>
> +/**
> + * list_lru_lock: lock the sublist for the given node and memcg
> + * @lru: the lru pointer
> + * @nid: the node id of the sublist to lock.
> + * @memcg: the cgroup of the sublist to lock.
> + *
> + * Returns the locked list_lru_one sublist. The caller must call
> + * list_lru_unlock() when done.
> + *
> + * You must ensure that the memcg is not freed during this call (e.g., with
> + * rcu or by taking a css refcnt).
> + *
> + * Return: the locked list_lru_one, or NULL on failure
> + */
> +struct list_lru_one *list_lru_lock(struct list_lru *lru, int nid,
> + struct mem_cgroup *memcg);
> +
> +/**
> + * list_lru_unlock: unlock a sublist locked by list_lru_lock()
> + * @l: the list_lru_one to unlock
> + */
> +void list_lru_unlock(struct list_lru_one *l);
> +
> +struct list_lru_one *list_lru_lock_irq(struct list_lru *lru, int nid,
> + struct mem_cgroup *memcg);
> +void list_lru_unlock_irq(struct list_lru_one *l);
> +
> +struct list_lru_one *list_lru_lock_irqsave(struct list_lru *lru, int nid,
> + struct mem_cgroup *memcg, unsigned long *irq_flags);
> +void list_lru_unlock_irqrestore(struct list_lru_one *l,
> + unsigned long *irq_flags);
> +
> +/* Caller-locked variants, see list_lru_add() etc for documentation */
> +bool __list_lru_add(struct list_lru *lru, struct list_lru_one *l,
> + struct list_head *item, int nid, struct mem_cgroup *memcg);
> +bool __list_lru_del(struct list_lru *lru, struct list_lru_one *l,
> + struct list_head *item, int nid);
> +
> /**
> * list_lru_add: add an element to the lru list's tail
> * @lru: the lru pointer
> @@ -115,6 +153,9 @@ void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup *paren
> bool list_lru_add(struct list_lru *lru, struct list_head *item, int nid,
> struct mem_cgroup *memcg);
>
> +bool list_lru_add_irq(struct list_lru *lru, struct list_head *item, int nid,
> + struct mem_cgroup *memcg);
> +
> /**
> * list_lru_add_obj: add an element to the lru list's tail
> * @lru: the lru pointer
> diff --git a/mm/list_lru.c b/mm/list_lru.c
> index 65962dbf6dda..df58226eea8c 100644
> --- a/mm/list_lru.c
> +++ b/mm/list_lru.c
> @@ -15,17 +15,23 @@
> #include "slab.h"
> #include "internal.h"
>
> -static inline void lock_list_lru(struct list_lru_one *l, bool irq)
> +static inline void lock_list_lru(struct list_lru_one *l, bool irq,
> + unsigned long *irq_flags)
> {
> - if (irq)
> + if (irq_flags)
> + spin_lock_irqsave(&l->lock, *irq_flags);
> + else if (irq)
> spin_lock_irq(&l->lock);
> else
> spin_lock(&l->lock);
> }
>
> -static inline void unlock_list_lru(struct list_lru_one *l, bool irq_off)
> +static inline void unlock_list_lru(struct list_lru_one *l, bool irq_off,
> + unsigned long *irq_flags)
> {
> - if (irq_off)
> + if (irq_flags)
> + spin_unlock_irqrestore(&l->lock, *irq_flags);
> + else if (irq_off)
> spin_unlock_irq(&l->lock);
> else
> spin_unlock(&l->lock);
> @@ -78,7 +84,7 @@ list_lru_from_memcg_idx(struct list_lru *lru, int nid, int idx)
>
> static inline struct list_lru_one *
> lock_list_lru_of_memcg(struct list_lru *lru, int nid, struct mem_cgroup *memcg,
> - bool irq, bool skip_empty)
> + bool irq, unsigned long *irq_flags, bool skip_empty)
> {
> struct list_lru_one *l;
>
> @@ -86,12 +92,12 @@ lock_list_lru_of_memcg(struct list_lru *lru, int nid, struct mem_cgroup *memcg,
> again:
> l = list_lru_from_memcg_idx(lru, nid, memcg_kmem_id(memcg));
> if (likely(l)) {
> - lock_list_lru(l, irq);
> + lock_list_lru(l, irq, irq_flags);
> if (likely(READ_ONCE(l->nr_items) != LONG_MIN)) {
> rcu_read_unlock();
> return l;
> }
> - unlock_list_lru(l, irq);
> + unlock_list_lru(l, irq, irq_flags);
> }
> /*
> * Caller may simply bail out if raced with reparenting or
> @@ -132,38 +138,106 @@ list_lru_from_memcg_idx(struct list_lru *lru, int nid, int idx)
>
> static inline struct list_lru_one *
> lock_list_lru_of_memcg(struct list_lru *lru, int nid, struct mem_cgroup *memcg,
> - bool irq, bool skip_empty)
> + bool irq, unsigned long *irq_flags, bool skip_empty)
> {
> struct list_lru_one *l = &lru->node[nid].lru;
>
> - lock_list_lru(l, irq);
> + lock_list_lru(l, irq, irq_flags);
>
> return l;
> }
> #endif /* CONFIG_MEMCG */
>
> -/* The caller must ensure the memcg lifetime. */
> -bool list_lru_add(struct list_lru *lru, struct list_head *item, int nid,
> - struct mem_cgroup *memcg)
> +struct list_lru_one *list_lru_lock(struct list_lru *lru, int nid,
> + struct mem_cgroup *memcg)
> {
> - struct list_lru_node *nlru = &lru->node[nid];
> - struct list_lru_one *l;
> + return lock_list_lru_of_memcg(lru, nid, memcg, /*irq=*/false,
> + /*irq_flags=*/NULL, /*skip_empty=*/false);
> +}
> +
> +void list_lru_unlock(struct list_lru_one *l)
> +{
> + unlock_list_lru(l, /*irq_off=*/false, /*irq_flags=*/NULL);
> +}
> +
> +struct list_lru_one *list_lru_lock_irq(struct list_lru *lru, int nid,
> + struct mem_cgroup *memcg)
> +{
> + return lock_list_lru_of_memcg(lru, nid, memcg, /*irq=*/true,
> + /*irq_flags=*/NULL, /*skip_empty=*/false);
> +}
> +
> +void list_lru_unlock_irq(struct list_lru_one *l)
> +{
> + unlock_list_lru(l, /*irq_off=*/true, /*irq_flags=*/NULL);
> +}
>
> - l = lock_list_lru_of_memcg(lru, nid, memcg, false, false);
> +struct list_lru_one *list_lru_lock_irqsave(struct list_lru *lru, int nid,
> + struct mem_cgroup *memcg,
> + unsigned long *flags)
> +{
> + return lock_list_lru_of_memcg(lru, nid, memcg, /*irq=*/true,
> + /*irq_flags=*/flags, /*skip_empty=*/false);
> +}
> +
> +void list_lru_unlock_irqrestore(struct list_lru_one *l, unsigned long *flags)
> +{
> + unlock_list_lru(l, /*irq_off=*/true, /*irq_flags=*/flags);
> +}
> +
> +bool __list_lru_add(struct list_lru *lru, struct list_lru_one *l,
> + struct list_head *item, int nid,
> + struct mem_cgroup *memcg)
> +{
> if (list_empty(item)) {
> list_add_tail(item, &l->list);
> /* Set shrinker bit if the first element was added */
> if (!l->nr_items++)
> set_shrinker_bit(memcg, nid, lru_shrinker_id(lru));
> - unlock_list_lru(l, false);
> - atomic_long_inc(&nlru->nr_items);
> + atomic_long_inc(&lru->node[nid].nr_items);
> return true;
> }
> - unlock_list_lru(l, false);
> return false;
> }
> EXPORT_SYMBOL_GPL(list_lru_add);
>
> +bool __list_lru_del(struct list_lru *lru, struct list_lru_one *l,
> + struct list_head *item, int nid)
> +{
> + if (!list_empty(item)) {
> + list_del_init(item);
> + l->nr_items--;
> + atomic_long_dec(&lru->node[nid].nr_items);
> + return true;
> + }
> + return false;
> +}
> +
> +/* The caller must ensure the memcg lifetime. */
> +bool list_lru_add(struct list_lru *lru, struct list_head *item, int nid,
> + struct mem_cgroup *memcg)
> +{
> + struct list_lru_one *l;
> + bool ret;
> +
> + l = list_lru_lock(lru, nid, memcg);
> + ret = __list_lru_add(lru, l, item, nid, memcg);
> + list_lru_unlock(l);
> + return ret;
> +}
> +
> +bool list_lru_add_irq(struct list_lru *lru, struct list_head *item,
> + int nid, struct mem_cgroup *memcg)
> +{
> + struct list_lru_one *l;
> + bool ret;
> +
> + l = list_lru_lock_irq(lru, nid, memcg);
> + ret = __list_lru_add(lru, l, item, nid, memcg);
> + list_lru_unlock_irq(l);
> + return ret;
> +}
> +
> bool list_lru_add_obj(struct list_lru *lru, struct list_head *item)
> {
> bool ret;
> @@ -185,19 +259,13 @@ EXPORT_SYMBOL_GPL(list_lru_add_obj);
> bool list_lru_del(struct list_lru *lru, struct list_head *item, int nid,
> struct mem_cgroup *memcg)
> {
> - struct list_lru_node *nlru = &lru->node[nid];
> struct list_lru_one *l;
> + bool ret;
>
> - l = lock_list_lru_of_memcg(lru, nid, memcg, false, false);
> - if (!list_empty(item)) {
> - list_del_init(item);
> - l->nr_items--;
> - unlock_list_lru(l, false);
> - atomic_long_dec(&nlru->nr_items);
> - return true;
> - }
> - unlock_list_lru(l, false);
> - return false;
> + l = list_lru_lock(lru, nid, memcg);
> + ret = __list_lru_del(lru, l, item, nid);
> + list_lru_unlock(l);
> + return ret;
> }
>
> bool list_lru_del_obj(struct list_lru *lru, struct list_head *item)
> @@ -270,7 +338,8 @@ __list_lru_walk_one(struct list_lru *lru, int nid, struct mem_cgroup *memcg,
> unsigned long isolated = 0;
>
> restart:
> - l = lock_list_lru_of_memcg(lru, nid, memcg, irq_off, true);
> + l = lock_list_lru_of_memcg(lru, nid, memcg, /*irq=*/irq_off,
> + /*irq_flags=*/NULL, /*skip_empty=*/true);
> if (!l)
> return isolated;
> list_for_each_safe(item, n, &l->list) {
> @@ -311,7 +380,7 @@ __list_lru_walk_one(struct list_lru *lru, int nid, struct mem_cgroup *memcg,
> BUG();
> }
> }
> - unlock_list_lru(l, irq_off);
> + unlock_list_lru(l, irq_off, NULL);
> out:
> return isolated;
> }
> --
> 2.54.0
>
^ permalink raw reply
* Re: [PATCH 4/8] mm: memcontrol: track MEMCG_KMEM per NUMA node
From: Shakeel Butt @ 2026-05-21 17:28 UTC (permalink / raw)
To: Alexandre Ghiti
Cc: Andrew Morton, Johannes Weiner, Michal Hocko, Roman Gushchin,
Muchun Song, Dennis Zhou, Tejun Heo, Christoph Lameter,
Vlastimil Babka, Yosry Ahmed, Nhat Pham, Sergey Senozhatsky,
Chengming Zhou, Suren Baghdasaryan, Qi Zheng, David Hildenbrand,
Lorenzo Stoakes, Minchan Kim, Mike Rapoport, Axel Rasmussen,
Barry Song, Kairui Song, Wei Xu, Yuanchu Xie, Liam R . Howlett,
Joshua Hahn, linux-mm, linux-kernel, cgroups
In-Reply-To: <20260511202136.330358-5-alex@ghiti.fr>
On Mon, May 11, 2026 at 10:20:39PM +0200, Alexandre Ghiti wrote:
> This patch gets rid of MEMCG_KMEM and wires all the "generic" functions
> by introducing per-node obj_cgroup objects.
>
> Note that it does not convert the kmem users to proper per-memcg-per-node
> accounting now, this is done in upcoming patches.
>
> Signed-off-by: Alexandre Ghiti <alex@ghiti.fr>
> ---
> include/linux/memcontrol.h | 23 ++++++++++----
> include/linux/mmzone.h | 1 +
> mm/memcontrol.c | 64 ++++++++++++++++++++++++--------------
> mm/vmstat.c | 1 +
> 4 files changed, 59 insertions(+), 30 deletions(-)
>
> diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
> index 568ab08f42af..17cf823160e4 100644
> --- a/include/linux/memcontrol.h
> +++ b/include/linux/memcontrol.h
> @@ -35,7 +35,6 @@ enum memcg_stat_item {
> MEMCG_SWAP = NR_VM_NODE_STAT_ITEMS,
> MEMCG_SOCK,
> MEMCG_PERCPU_B,
> - MEMCG_KMEM,
> MEMCG_ZSWAP_B,
> MEMCG_ZSWAPPED,
> MEMCG_ZSWAP_INCOMP,
> @@ -126,9 +125,10 @@ struct mem_cgroup_per_node {
> struct list_head objcg_list;
>
> #ifdef CONFIG_MEMCG_NMI_SAFETY_REQUIRES_ATOMIC
> - /* slab stats for nmi context */
> + /* slab and kmem stats for nmi context */
> atomic_t slab_reclaimable;
> atomic_t slab_unreclaimable;
> + atomic_t kmem;
> #endif
> };
>
> @@ -190,6 +190,7 @@ struct obj_cgroup {
> struct rcu_head rcu;
> };
> bool is_root;
> + int nid;
> };
>
> /*
> @@ -254,10 +255,6 @@ struct mem_cgroup {
> atomic_long_t memory_events[MEMCG_NR_MEMORY_EVENTS];
> atomic_long_t memory_events_local[MEMCG_NR_MEMORY_EVENTS];
>
> -#ifdef CONFIG_MEMCG_NMI_SAFETY_REQUIRES_ATOMIC
> - /* MEMCG_KMEM for nmi context */
> - atomic_t kmem_stat;
> -#endif
> /*
> * Hint of reclaim pressure for socket memroy management. Note
> * that this indicator should NOT be used in legacy cgroup mode
> @@ -776,6 +773,20 @@ static inline void obj_cgroup_put(struct obj_cgroup *objcg)
> percpu_ref_put(&objcg->refcnt);
> }
>
> +static inline struct obj_cgroup *obj_cgroup_get_nid(struct obj_cgroup *objcg,
> + int nid)
> +{
> + struct obj_cgroup *nid_objcg;
> + struct mem_cgroup *memcg;
> +
> + rcu_read_lock();
> + memcg = obj_cgroup_memcg(objcg);
> + nid_objcg = rcu_dereference(memcg->nodeinfo[nid]->objcg);
> + rcu_read_unlock();
> +
> + return nid_objcg;
What is guarating the life of nid_objcg?
^ permalink raw reply
* Re: [PATCH 2/8] mm: percpu: charge obj_exts allocation with __GFP_ACCOUNT
From: Shakeel Butt @ 2026-05-21 17:25 UTC (permalink / raw)
To: Alexandre Ghiti
Cc: Andrew Morton, Johannes Weiner, Michal Hocko, Roman Gushchin,
Muchun Song, Dennis Zhou, Tejun Heo, Christoph Lameter,
Vlastimil Babka, Yosry Ahmed, Nhat Pham, Sergey Senozhatsky,
Chengming Zhou, Suren Baghdasaryan, Qi Zheng, David Hildenbrand,
Lorenzo Stoakes, Minchan Kim, Mike Rapoport, Axel Rasmussen,
Barry Song, Kairui Song, Wei Xu, Yuanchu Xie, Liam R . Howlett,
Joshua Hahn, linux-mm, linux-kernel, cgroups
In-Reply-To: <20260511202136.330358-3-alex@ghiti.fr>
On Mon, May 11, 2026 at 10:20:37PM +0200, Alexandre Ghiti wrote:
> This is a preparatory patch for upcoming per-memcg-per-node kmem
> accounting.
>
> pcpu allocations are always fully charged at once using
> pcpu_obj_full_size(), which returns the size of the pcpu "metadata" +
> pcpu "payload". But metadata and payload may not be allocated on the
> same numa node, so charge the metadata independently from the payload.
>
> Do this by explicitly passing __GFP_ACCOUNT to the obj_exts allocation
> and remove its accounting in pcpu_memcg_pre_alloc_hook().
Will all the entries in obj_exts array be for the same memcg? If not then why we
are charging the whole array to the one which happen to allocate the array?
Sorry I don't know the details of percpu allocator, so asking some dumb
questions:
1. Does the alloc_percpu() (& similar functions) allocate the underlying on a
single node or does it allocate memory for each cpu on their local node?
For slub, it is on the same node, so the situation is easier to handle.
2. On a typical system how much memory is consumed by obj_exts for the percpu
allocator chunks? I am wondering if we don't charge it, how much will we
loose?
3. What would be side effect on assuming that obj_exts is on the same node as
the given chunk?
^ permalink raw reply
* Re: [PATCH v4 4/8] mm: list_lru: deduplicate lock_list_lru()
From: Liam R . Howlett @ 2026-05-21 17:18 UTC (permalink / raw)
To: Johannes Weiner
Cc: Andrew Morton, David Hildenbrand, Lorenzo Stoakes, Shakeel Butt,
Michal Hocko, Dave Chinner, Roman Gushchin, Muchun Song, Qi Zheng,
Yosry Ahmed, Zi Yan, Usama Arif, Kiryl Shutsemau, Vlastimil Babka,
Kairui Song, Mikhail Zaslonko, Vasily Gorbik, Baolin Wang,
Barry Song, Dev Jain, Lance Yang, Nico Pache, Ryan Roberts,
cgroups, linux-mm, linux-kernel
In-Reply-To: <20260521150330.1955924-5-hannes@cmpxchg.org>
On 26/05/21 11:02AM, Johannes Weiner wrote:
> The MEMCG and !MEMCG paths have the same pattern. Share the code.
>
> Reviewed-by: David Hildenbrand (Arm) <david@kernel.org>
> Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
> Reviewed-by: Lorenzo Stoakes (Oracle) <ljs@kernel.org>
> Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
Reviewed-by: Liam R. Howlett (Oracle) <liam@infradead.org>
> ---
> mm/list_lru.c | 21 +++++++++------------
> 1 file changed, 9 insertions(+), 12 deletions(-)
>
> diff --git a/mm/list_lru.c b/mm/list_lru.c
> index 9da6fce19832..65962dbf6dda 100644
> --- a/mm/list_lru.c
> +++ b/mm/list_lru.c
> @@ -15,6 +15,14 @@
> #include "slab.h"
> #include "internal.h"
>
> +static inline void lock_list_lru(struct list_lru_one *l, bool irq)
> +{
> + if (irq)
> + spin_lock_irq(&l->lock);
> + else
> + spin_lock(&l->lock);
> +}
> +
> static inline void unlock_list_lru(struct list_lru_one *l, bool irq_off)
> {
> if (irq_off)
> @@ -68,14 +76,6 @@ list_lru_from_memcg_idx(struct list_lru *lru, int nid, int idx)
> return &lru->node[nid].lru;
> }
>
> -static inline void lock_list_lru(struct list_lru_one *l, bool irq)
> -{
> - if (irq)
> - spin_lock_irq(&l->lock);
> - else
> - spin_lock(&l->lock);
> -}
> -
> static inline struct list_lru_one *
> lock_list_lru_of_memcg(struct list_lru *lru, int nid, struct mem_cgroup *memcg,
> bool irq, bool skip_empty)
> @@ -136,10 +136,7 @@ lock_list_lru_of_memcg(struct list_lru *lru, int nid, struct mem_cgroup *memcg,
> {
> struct list_lru_one *l = &lru->node[nid].lru;
>
> - if (irq)
> - spin_lock_irq(&l->lock);
> - else
> - spin_lock(&l->lock);
> + lock_list_lru(l, irq);
>
> return l;
> }
> --
> 2.54.0
>
^ permalink raw reply
* Re: [PATCH v4 3/8] mm: list_lru: move list dead check to lock_list_lru_of_memcg()
From: Liam R . Howlett @ 2026-05-21 17:17 UTC (permalink / raw)
To: Johannes Weiner
Cc: Andrew Morton, David Hildenbrand, Lorenzo Stoakes, Shakeel Butt,
Michal Hocko, Dave Chinner, Roman Gushchin, Muchun Song, Qi Zheng,
Yosry Ahmed, Zi Yan, Usama Arif, Kiryl Shutsemau, Vlastimil Babka,
Kairui Song, Mikhail Zaslonko, Vasily Gorbik, Baolin Wang,
Barry Song, Dev Jain, Lance Yang, Nico Pache, Ryan Roberts,
cgroups, linux-mm, linux-kernel
In-Reply-To: <20260521150330.1955924-4-hannes@cmpxchg.org>
On 26/05/21 11:02AM, Johannes Weiner wrote:
> Only the MEMCG variant of lock_list_lru() needs to check if there is a
> race with cgroup deletion and list reparenting. Move the check to the
> caller, so that the next patch can unify the lock_list_lru() variants.
>
> Reviewed-by: David Hildenbrand (Arm) <david@kernel.org>
> Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
> Reviewed-by: Lorenzo Stoakes (Oracle) <ljs@kernel.org>
> Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
Reviewed-by: Liam R. Howlett (Oracle) <liam@infradead.org>
> ---
> mm/list_lru.c | 17 ++++++++---------
> 1 file changed, 8 insertions(+), 9 deletions(-)
>
> diff --git a/mm/list_lru.c b/mm/list_lru.c
> index 9a68177619bf..9da6fce19832 100644
> --- a/mm/list_lru.c
> +++ b/mm/list_lru.c
> @@ -68,17 +68,12 @@ list_lru_from_memcg_idx(struct list_lru *lru, int nid, int idx)
> return &lru->node[nid].lru;
> }
>
> -static inline bool lock_list_lru(struct list_lru_one *l, bool irq)
> +static inline void lock_list_lru(struct list_lru_one *l, bool irq)
> {
> if (irq)
> spin_lock_irq(&l->lock);
> else
> spin_lock(&l->lock);
> - if (unlikely(READ_ONCE(l->nr_items) == LONG_MIN)) {
> - unlock_list_lru(l, irq);
> - return false;
> - }
> - return true;
> }
>
> static inline struct list_lru_one *
> @@ -90,9 +85,13 @@ lock_list_lru_of_memcg(struct list_lru *lru, int nid, struct mem_cgroup *memcg,
> rcu_read_lock();
> again:
> l = list_lru_from_memcg_idx(lru, nid, memcg_kmem_id(memcg));
> - if (likely(l) && lock_list_lru(l, irq)) {
> - rcu_read_unlock();
> - return l;
> + if (likely(l)) {
> + lock_list_lru(l, irq);
> + if (likely(READ_ONCE(l->nr_items) != LONG_MIN)) {
> + rcu_read_unlock();
> + return l;
> + }
> + unlock_list_lru(l, irq);
> }
> /*
> * Caller may simply bail out if raced with reparenting or
> --
> 2.54.0
>
^ permalink raw reply
* Re: [PATCH v4 2/8] mm: list_lru: deduplicate unlock_list_lru()
From: Liam R . Howlett @ 2026-05-21 17:16 UTC (permalink / raw)
To: Johannes Weiner
Cc: Andrew Morton, David Hildenbrand, Lorenzo Stoakes, Shakeel Butt,
Michal Hocko, Dave Chinner, Roman Gushchin, Muchun Song, Qi Zheng,
Yosry Ahmed, Zi Yan, Usama Arif, Kiryl Shutsemau, Vlastimil Babka,
Kairui Song, Mikhail Zaslonko, Vasily Gorbik, Baolin Wang,
Barry Song, Dev Jain, Lance Yang, Nico Pache, Ryan Roberts,
cgroups, linux-mm, linux-kernel
In-Reply-To: <20260521150330.1955924-3-hannes@cmpxchg.org>
On 26/05/21 11:02AM, Johannes Weiner wrote:
> The MEMCG and !MEMCG variants are the same. lock_list_lru() has the
> same pattern when bailing. Consolidate into a common implementation.
>
> Reviewed-by: David Hildenbrand (Arm) <david@kernel.org>
> Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
> Reviewed-by: Lorenzo Stoakes (Oracle) <ljs@kernel.org>
> Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
Reviewed-by: Liam R. Howlett (Oracle) <liam@infradead.org>
> ---
> mm/list_lru.c | 29 +++++++++--------------------
> 1 file changed, 9 insertions(+), 20 deletions(-)
>
> diff --git a/mm/list_lru.c b/mm/list_lru.c
> index d3619961a7ac..9a68177619bf 100644
> --- a/mm/list_lru.c
> +++ b/mm/list_lru.c
> @@ -15,6 +15,14 @@
> #include "slab.h"
> #include "internal.h"
>
> +static inline void unlock_list_lru(struct list_lru_one *l, bool irq_off)
> +{
> + if (irq_off)
> + spin_unlock_irq(&l->lock);
> + else
> + spin_unlock(&l->lock);
> +}
> +
> #ifdef CONFIG_MEMCG
> static LIST_HEAD(memcg_list_lrus);
> static DEFINE_MUTEX(list_lrus_mutex);
> @@ -67,10 +75,7 @@ static inline bool lock_list_lru(struct list_lru_one *l, bool irq)
> else
> spin_lock(&l->lock);
> if (unlikely(READ_ONCE(l->nr_items) == LONG_MIN)) {
> - if (irq)
> - spin_unlock_irq(&l->lock);
> - else
> - spin_unlock(&l->lock);
> + unlock_list_lru(l, irq);
> return false;
> }
> return true;
> @@ -101,14 +106,6 @@ lock_list_lru_of_memcg(struct list_lru *lru, int nid, struct mem_cgroup *memcg,
> memcg = parent_mem_cgroup(memcg);
> goto again;
> }
> -
> -static inline void unlock_list_lru(struct list_lru_one *l, bool irq_off)
> -{
> - if (irq_off)
> - spin_unlock_irq(&l->lock);
> - else
> - spin_unlock(&l->lock);
> -}
> #else
> static void list_lru_register(struct list_lru *lru)
> {
> @@ -147,14 +144,6 @@ lock_list_lru_of_memcg(struct list_lru *lru, int nid, struct mem_cgroup *memcg,
>
> return l;
> }
> -
> -static inline void unlock_list_lru(struct list_lru_one *l, bool irq_off)
> -{
> - if (irq_off)
> - spin_unlock_irq(&l->lock);
> - else
> - spin_unlock(&l->lock);
> -}
> #endif /* CONFIG_MEMCG */
>
> /* The caller must ensure the memcg lifetime. */
> --
> 2.54.0
>
^ permalink raw reply
* Re: [PATCH v4 1/8] mm: list_lru: lock_list_lru_of_memcg() cannot return NULL if !skip_empty
From: Liam R . Howlett @ 2026-05-21 17:16 UTC (permalink / raw)
To: Johannes Weiner
Cc: Andrew Morton, David Hildenbrand, Lorenzo Stoakes, Shakeel Butt,
Michal Hocko, Dave Chinner, Roman Gushchin, Muchun Song, Qi Zheng,
Yosry Ahmed, Zi Yan, Usama Arif, Kiryl Shutsemau, Vlastimil Babka,
Kairui Song, Mikhail Zaslonko, Vasily Gorbik, Baolin Wang,
Barry Song, Dev Jain, Lance Yang, Nico Pache, Ryan Roberts,
cgroups, linux-mm, linux-kernel
In-Reply-To: <20260521150330.1955924-2-hannes@cmpxchg.org>
On 26/05/21 11:02AM, Johannes Weiner wrote:
> skip_empty is only for the shrinker to abort and skip a list that's
> empty or whose cgroup is being deleted.
>
> For list additions and deletions, the cgroup hierarchy is walked
> upwards until a valid list_lru head is found, or it will fall back to
> the node list. Acquiring the lock won't fail. Remove the NULL checks
> in those callers.
>
> Reviewed-by: David Hildenbrand (Arm) <david@kernel.org>
> Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
> Reviewed-by: Lorenzo Stoakes (Oracle) <ljs@kernel.org>
> Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
Reviewed-by: Liam R. Howlett (Oracle) <liam@infradead.org>
> ---
> mm/list_lru.c | 5 +----
> 1 file changed, 1 insertion(+), 4 deletions(-)
>
> diff --git a/mm/list_lru.c b/mm/list_lru.c
> index dd29bcf8eb5f..d3619961a7ac 100644
> --- a/mm/list_lru.c
> +++ b/mm/list_lru.c
> @@ -165,8 +165,6 @@ bool list_lru_add(struct list_lru *lru, struct list_head *item, int nid,
> struct list_lru_one *l;
>
> l = lock_list_lru_of_memcg(lru, nid, memcg, false, false);
> - if (!l)
> - return false;
> if (list_empty(item)) {
> list_add_tail(item, &l->list);
> /* Set shrinker bit if the first element was added */
> @@ -204,9 +202,8 @@ bool list_lru_del(struct list_lru *lru, struct list_head *item, int nid,
> {
> struct list_lru_node *nlru = &lru->node[nid];
> struct list_lru_one *l;
> +
> l = lock_list_lru_of_memcg(lru, nid, memcg, false, false);
> - if (!l)
> - return false;
> if (!list_empty(item)) {
> list_del_init(item);
> l->nr_items--;
> --
> 2.54.0
>
^ permalink raw reply
* Re: [PATCH v4 7/8] mm/memory: flatten folio allocation retry loops
From: Shakeel Butt @ 2026-05-21 17:09 UTC (permalink / raw)
To: Johannes Weiner
Cc: Andrew Morton, David Hildenbrand, Lorenzo Stoakes, Michal Hocko,
Dave Chinner, Roman Gushchin, Muchun Song, Qi Zheng, Yosry Ahmed,
Zi Yan, Liam R . Howlett, Usama Arif, Kiryl Shutsemau,
Vlastimil Babka, Kairui Song, Mikhail Zaslonko, Vasily Gorbik,
Baolin Wang, Barry Song, Dev Jain, Lance Yang, Nico Pache,
Ryan Roberts, cgroups, linux-mm, linux-kernel
In-Reply-To: <20260521150330.1955924-8-hannes@cmpxchg.org>
On Thu, May 21, 2026 at 11:02:13AM -0400, Johannes Weiner wrote:
> alloc_swap_folio() and alloc_anon_folio() use a top-level if (folio)
> that buries the success path four levels deep. This makes for awkward
> long lines and wrapping. The next patch will add more code here, so
> flatten this now to keep things clean and simple.
>
> alloc_anon_folio() already has a next label, use it for !folio. Add
> the equivalent to alloc_swap_folio().
>
> No functional change intended.
>
> Suggested-by: Lorenzo Stoakes (Oracle) <ljs@kernel.org>
> Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
^ permalink raw reply
* [PATCH v4 8/8] mm: switch deferred split shrinker to list_lru
From: Johannes Weiner @ 2026-05-21 15:02 UTC (permalink / raw)
To: Andrew Morton
Cc: David Hildenbrand, Lorenzo Stoakes, Shakeel Butt, Michal Hocko,
Dave Chinner, Roman Gushchin, Muchun Song, Qi Zheng, Yosry Ahmed,
Zi Yan, Liam R . Howlett, Usama Arif, Kiryl Shutsemau,
Vlastimil Babka, Kairui Song, Mikhail Zaslonko, Vasily Gorbik,
Baolin Wang, Barry Song, Dev Jain, Lance Yang, Nico Pache,
Ryan Roberts, cgroups, linux-mm, linux-kernel
In-Reply-To: <20260521150330.1955924-1-hannes@cmpxchg.org>
The deferred split queue handles cgroups in a suboptimal fashion. The
queue is per-NUMA node or per-cgroup, not the intersection. That means
on a cgrouped system, a node-restricted allocation entering reclaim
can end up splitting large pages on other nodes:
alloc/unmap
deferred_split_folio()
list_add_tail(memcg->split_queue)
set_shrinker_bit(memcg, node, deferred_shrinker_id)
for_each_zone_zonelist_nodemask(restricted_nodes)
mem_cgroup_iter()
shrink_slab(node, memcg)
shrink_slab_memcg(node, memcg)
if test_shrinker_bit(memcg, node, deferred_shrinker_id)
deferred_split_scan()
walks memcg->split_queue
The shrinker bit adds an imperfect guard rail. As soon as the cgroup
has a single large page on the node of interest, all large pages owned
by that memcg, including those on other nodes, will be split.
list_lru properly sets up per-node, per-cgroup lists. As a bonus, it
streamlines a lot of the list operations and reclaim walks. It's used
widely by other major shrinkers already. Convert the deferred split
queue as well.
The list_lru per-memcg heads are instantiated on demand when the first
object of interest is allocated for a cgroup, by calling
folio_memcg_alloc_deferred(). Add calls to where splittable pages are
created: anon faults, swapin faults, khugepaged collapse.
These calls create all possible node heads for the cgroup at once, so
the migration code (between nodes) doesn't need any special care.
Reported-by: Mikhail Zaslonko <zaslonko@linux.ibm.com>
Tested-by: Mikhail Zaslonko <zaslonko@linux.ibm.com>
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Reviewed-by: Lorenzo Stoakes (Oracle) <ljs@kernel.org>
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
---
include/linux/huge_mm.h | 7 +-
include/linux/memcontrol.h | 4 -
include/linux/mmzone.h | 12 --
mm/huge_memory.c | 355 ++++++++++++-------------------------
mm/internal.h | 2 +-
mm/khugepaged.c | 3 +
mm/memcontrol.c | 12 +-
mm/memory.c | 8 +
mm/mm_init.c | 15 --
9 files changed, 133 insertions(+), 285 deletions(-)
diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h
index 127f9e1e7604..dc939873f5df 100644
--- a/include/linux/huge_mm.h
+++ b/include/linux/huge_mm.h
@@ -398,10 +398,10 @@ static inline int split_huge_page(struct page *page)
{
return split_huge_page_to_list_to_order(page, NULL, 0);
}
+
+int folio_memcg_alloc_deferred(struct folio *folio);
+
void deferred_split_folio(struct folio *folio, bool partially_mapped);
-#ifdef CONFIG_MEMCG
-void reparent_deferred_split_queue(struct mem_cgroup *memcg);
-#endif
void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,
unsigned long address, bool freeze);
@@ -634,7 +634,6 @@ static inline int folio_split(struct folio *folio, unsigned int new_order,
}
static inline void deferred_split_folio(struct folio *folio, bool partially_mapped) {}
-static inline void reparent_deferred_split_queue(struct mem_cgroup *memcg) {}
#define split_huge_pmd(__vma, __pmd, __address) \
do { } while (0)
diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index dc3fa687759b..4a7d8c4f55b4 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -277,10 +277,6 @@ struct mem_cgroup {
struct memcg_cgwb_frn cgwb_frn[MEMCG_CGWB_FRN_CNT];
#endif
-#ifdef CONFIG_TRANSPARENT_HUGEPAGE
- struct deferred_split deferred_split_queue;
-#endif
-
#ifdef CONFIG_LRU_GEN_WALKS_MMU
/* per-memcg mm_struct list */
struct lru_gen_mm_list mm_list;
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 1331a7b93f33..8e449f524f26 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -1431,14 +1431,6 @@ struct zonelist {
*/
extern struct page *mem_map;
-#ifdef CONFIG_TRANSPARENT_HUGEPAGE
-struct deferred_split {
- spinlock_t split_queue_lock;
- struct list_head split_queue;
- unsigned long split_queue_len;
-};
-#endif
-
#ifdef CONFIG_MEMORY_FAILURE
/*
* Per NUMA node memory failure handling statistics.
@@ -1564,10 +1556,6 @@ typedef struct pglist_data {
unsigned long first_deferred_pfn;
#endif /* CONFIG_DEFERRED_STRUCT_PAGE_INIT */
-#ifdef CONFIG_TRANSPARENT_HUGEPAGE
- struct deferred_split deferred_split_queue;
-#endif
-
#ifdef CONFIG_NUMA_BALANCING
/* start time in ms of current promote rate limit period */
unsigned int nbp_rl_start;
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index c565b2a651e0..67be09a58d5a 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -14,6 +14,7 @@
#include <linux/mmu_notifier.h>
#include <linux/rmap.h>
#include <linux/swap.h>
+#include <linux/list_lru.h>
#include <linux/shrinker.h>
#include <linux/mm_inline.h>
#include <linux/swapops.h>
@@ -67,6 +68,8 @@ unsigned long transparent_hugepage_flags __read_mostly =
(1<<TRANSPARENT_HUGEPAGE_DEFRAG_KHUGEPAGED_FLAG)|
(1<<TRANSPARENT_HUGEPAGE_USE_ZERO_PAGE_FLAG);
+static struct lock_class_key deferred_split_key;
+static struct list_lru deferred_split_lru;
static struct shrinker *deferred_split_shrinker;
static unsigned long deferred_split_count(struct shrinker *shrink,
struct shrink_control *sc);
@@ -943,6 +946,13 @@ static inline void hugepage_exit_sysfs(struct kobject *hugepage_kobj)
}
#endif /* CONFIG_SYSFS */
+int folio_memcg_alloc_deferred(struct folio *folio)
+{
+ if (mem_cgroup_disabled())
+ return 0;
+ return folio_memcg_list_lru_alloc(folio, &deferred_split_lru, GFP_KERNEL);
+}
+
static int __init thp_shrinker_init(void)
{
deferred_split_shrinker = shrinker_alloc(SHRINKER_NUMA_AWARE |
@@ -952,6 +962,13 @@ static int __init thp_shrinker_init(void)
if (!deferred_split_shrinker)
return -ENOMEM;
+ if (list_lru_init_memcg_key(&deferred_split_lru,
+ deferred_split_shrinker,
+ &deferred_split_key)) {
+ shrinker_free(deferred_split_shrinker);
+ return -ENOMEM;
+ }
+
deferred_split_shrinker->count_objects = deferred_split_count;
deferred_split_shrinker->scan_objects = deferred_split_scan;
shrinker_register(deferred_split_shrinker);
@@ -973,6 +990,7 @@ static int __init thp_shrinker_init(void)
huge_zero_folio_shrinker = shrinker_alloc(0, "thp-zero");
if (!huge_zero_folio_shrinker) {
shrinker_free(deferred_split_shrinker);
+ list_lru_destroy(&deferred_split_lru);
return -ENOMEM;
}
@@ -987,6 +1005,7 @@ static void __init thp_shrinker_exit(void)
{
shrinker_free(huge_zero_folio_shrinker);
shrinker_free(deferred_split_shrinker);
+ list_lru_destroy(&deferred_split_lru);
}
static int __init hugepage_init(void)
@@ -1166,119 +1185,6 @@ pmd_t maybe_pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma)
return pmd;
}
-static struct deferred_split *split_queue_node(int nid)
-{
- struct pglist_data *pgdata = NODE_DATA(nid);
-
- return &pgdata->deferred_split_queue;
-}
-
-#ifdef CONFIG_MEMCG
-static inline
-struct mem_cgroup *folio_split_queue_memcg(struct folio *folio,
- struct deferred_split *queue)
-{
- if (mem_cgroup_disabled())
- return NULL;
- if (split_queue_node(folio_nid(folio)) == queue)
- return NULL;
- return container_of(queue, struct mem_cgroup, deferred_split_queue);
-}
-
-static struct deferred_split *memcg_split_queue(int nid, struct mem_cgroup *memcg)
-{
- return memcg ? &memcg->deferred_split_queue : split_queue_node(nid);
-}
-#else
-static inline
-struct mem_cgroup *folio_split_queue_memcg(struct folio *folio,
- struct deferred_split *queue)
-{
- return NULL;
-}
-
-static struct deferred_split *memcg_split_queue(int nid, struct mem_cgroup *memcg)
-{
- return split_queue_node(nid);
-}
-#endif
-
-static struct deferred_split *split_queue_lock(int nid, struct mem_cgroup *memcg)
-{
- struct deferred_split *queue;
-
-retry:
- queue = memcg_split_queue(nid, memcg);
- spin_lock(&queue->split_queue_lock);
- /*
- * There is a period between setting memcg to dying and reparenting
- * deferred split queue, and during this period the THPs in the deferred
- * split queue will be hidden from the shrinker side.
- */
- if (unlikely(memcg_is_dying(memcg))) {
- spin_unlock(&queue->split_queue_lock);
- memcg = parent_mem_cgroup(memcg);
- goto retry;
- }
-
- return queue;
-}
-
-static struct deferred_split *
-split_queue_lock_irqsave(int nid, struct mem_cgroup *memcg, unsigned long *flags)
-{
- struct deferred_split *queue;
-
-retry:
- queue = memcg_split_queue(nid, memcg);
- spin_lock_irqsave(&queue->split_queue_lock, *flags);
- if (unlikely(memcg_is_dying(memcg))) {
- spin_unlock_irqrestore(&queue->split_queue_lock, *flags);
- memcg = parent_mem_cgroup(memcg);
- goto retry;
- }
-
- return queue;
-}
-
-static struct deferred_split *folio_split_queue_lock(struct folio *folio)
-{
- struct deferred_split *queue;
-
- rcu_read_lock();
- queue = split_queue_lock(folio_nid(folio), folio_memcg(folio));
- /*
- * The memcg destruction path is acquiring the split queue lock for
- * reparenting. Once you have it locked, it's safe to drop the rcu lock.
- */
- rcu_read_unlock();
-
- return queue;
-}
-
-static struct deferred_split *
-folio_split_queue_lock_irqsave(struct folio *folio, unsigned long *flags)
-{
- struct deferred_split *queue;
-
- rcu_read_lock();
- queue = split_queue_lock_irqsave(folio_nid(folio), folio_memcg(folio), flags);
- rcu_read_unlock();
-
- return queue;
-}
-
-static inline void split_queue_unlock(struct deferred_split *queue)
-{
- spin_unlock(&queue->split_queue_lock);
-}
-
-static inline void split_queue_unlock_irqrestore(struct deferred_split *queue,
- unsigned long flags)
-{
- spin_unlock_irqrestore(&queue->split_queue_lock, flags);
-}
-
static inline bool is_transparent_hugepage(const struct folio *folio)
{
if (!folio_test_large(folio))
@@ -1379,6 +1285,14 @@ static struct folio *vma_alloc_anon_folio_pmd(struct vm_area_struct *vma,
count_mthp_stat(order, MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE);
return NULL;
}
+
+ if (folio_memcg_alloc_deferred(folio)) {
+ folio_put(folio);
+ count_vm_event(THP_FAULT_FALLBACK);
+ count_mthp_stat(order, MTHP_STAT_ANON_FAULT_FALLBACK);
+ return NULL;
+ }
+
folio_throttle_swaprate(folio, gfp);
/*
@@ -3888,34 +3802,40 @@ static int __folio_freeze_and_split_unmapped(struct folio *folio, unsigned int n
struct folio *end_folio = folio_next(folio);
struct folio *new_folio, *next;
int old_order = folio_order(folio);
+ struct list_lru_one *lru;
+ bool dequeue_deferred;
int ret = 0;
- struct deferred_split *ds_queue;
VM_WARN_ON_ONCE(!mapping && end);
- /* Prevent deferred_split_scan() touching ->_refcount */
- ds_queue = folio_split_queue_lock(folio);
+ /*
+ * If this folio can be on the deferred split queue, lock out
+ * the shrinker before freezing the ref. If the shrinker sees
+ * a 0-ref folio, it assumes it beat folio_put() to the list
+ * lock and must clean up the LRU state - the same dequeue we
+ * will do below as part of the split.
+ */
+ dequeue_deferred = folio_test_anon(folio) && old_order > 1;
+ if (dequeue_deferred) {
+ rcu_read_lock();
+ lru = list_lru_lock(&deferred_split_lru,
+ folio_nid(folio), folio_memcg(folio));
+ }
if (folio_ref_freeze(folio, folio_cache_ref_count(folio) + 1)) {
struct swap_cluster_info *ci = NULL;
struct lruvec *lruvec;
- if (old_order > 1) {
- if (!list_empty(&folio->_deferred_list)) {
- ds_queue->split_queue_len--;
- /*
- * Reinitialize page_deferred_list after removing the
- * page from the split_queue, otherwise a subsequent
- * split will see list corruption when checking the
- * page_deferred_list.
- */
- list_del_init(&folio->_deferred_list);
- }
+ if (dequeue_deferred) {
+ __list_lru_del(&deferred_split_lru, lru,
+ &folio->_deferred_list, folio_nid(folio));
if (folio_test_partially_mapped(folio)) {
folio_clear_partially_mapped(folio);
mod_mthp_stat(old_order,
MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, -1);
}
+ list_lru_unlock(lru);
+ rcu_read_unlock();
}
- split_queue_unlock(ds_queue);
+
if (mapping) {
int nr = folio_nr_pages(folio);
@@ -4015,7 +3935,10 @@ static int __folio_freeze_and_split_unmapped(struct folio *folio, unsigned int n
if (ci)
swap_cluster_unlock(ci);
} else {
- split_queue_unlock(ds_queue);
+ if (dequeue_deferred) {
+ list_lru_unlock(lru);
+ rcu_read_unlock();
+ }
return -EAGAIN;
}
@@ -4381,33 +4304,35 @@ int split_folio_to_list(struct folio *folio, struct list_head *list)
* queueing THP splits, and that list is (racily observed to be) non-empty.
*
* It is unsafe to call folio_unqueue_deferred_split() until folio refcount is
- * zero: because even when split_queue_lock is held, a non-empty _deferred_list
- * might be in use on deferred_split_scan()'s unlocked on-stack list.
+ * zero: because even when the list_lru lock is held, a non-empty
+ * _deferred_list might be in use on deferred_split_scan()'s unlocked
+ * on-stack list.
*
- * If memory cgroups are enabled, split_queue_lock is in the mem_cgroup: it is
- * therefore important to unqueue deferred split before changing folio memcg.
+ * The list_lru sublist is determined by folio's memcg: it is therefore
+ * important to unqueue deferred split before changing folio memcg.
*/
bool __folio_unqueue_deferred_split(struct folio *folio)
{
- struct deferred_split *ds_queue;
+ struct list_lru_one *lru;
+ int nid = folio_nid(folio);
unsigned long flags;
bool unqueued = false;
WARN_ON_ONCE(folio_ref_count(folio));
WARN_ON_ONCE(!mem_cgroup_disabled() && !folio_memcg_charged(folio));
- ds_queue = folio_split_queue_lock_irqsave(folio, &flags);
- if (!list_empty(&folio->_deferred_list)) {
- ds_queue->split_queue_len--;
+ rcu_read_lock();
+ lru = list_lru_lock_irqsave(&deferred_split_lru, nid, folio_memcg(folio), &flags);
+ if (__list_lru_del(&deferred_split_lru, lru, &folio->_deferred_list, nid)) {
if (folio_test_partially_mapped(folio)) {
folio_clear_partially_mapped(folio);
mod_mthp_stat(folio_order(folio),
MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, -1);
}
- list_del_init(&folio->_deferred_list);
unqueued = true;
}
- split_queue_unlock_irqrestore(ds_queue, flags);
+ list_lru_unlock_irqrestore(lru, &flags);
+ rcu_read_unlock();
return unqueued; /* useful for debug warnings */
}
@@ -4415,7 +4340,9 @@ bool __folio_unqueue_deferred_split(struct folio *folio)
/* partially_mapped=false won't clear PG_partially_mapped folio flag */
void deferred_split_folio(struct folio *folio, bool partially_mapped)
{
- struct deferred_split *ds_queue;
+ struct list_lru_one *lru;
+ int nid;
+ struct mem_cgroup *memcg;
unsigned long flags;
/*
@@ -4438,7 +4365,11 @@ void deferred_split_folio(struct folio *folio, bool partially_mapped)
if (folio_test_swapcache(folio))
return;
- ds_queue = folio_split_queue_lock_irqsave(folio, &flags);
+ nid = folio_nid(folio);
+
+ rcu_read_lock();
+ memcg = folio_memcg(folio);
+ lru = list_lru_lock_irqsave(&deferred_split_lru, nid, memcg, &flags);
if (partially_mapped) {
if (!folio_test_partially_mapped(folio)) {
folio_set_partially_mapped(folio);
@@ -4446,36 +4377,20 @@ void deferred_split_folio(struct folio *folio, bool partially_mapped)
count_vm_event(THP_DEFERRED_SPLIT_PAGE);
count_mthp_stat(folio_order(folio), MTHP_STAT_SPLIT_DEFERRED);
mod_mthp_stat(folio_order(folio), MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, 1);
-
}
} else {
/* partially mapped folios cannot become non-partially mapped */
VM_WARN_ON_FOLIO(folio_test_partially_mapped(folio), folio);
}
- if (list_empty(&folio->_deferred_list)) {
- struct mem_cgroup *memcg;
-
- memcg = folio_split_queue_memcg(folio, ds_queue);
- list_add_tail(&folio->_deferred_list, &ds_queue->split_queue);
- ds_queue->split_queue_len++;
- if (memcg)
- set_shrinker_bit(memcg, folio_nid(folio),
- shrinker_id(deferred_split_shrinker));
- }
- split_queue_unlock_irqrestore(ds_queue, flags);
+ __list_lru_add(&deferred_split_lru, lru, &folio->_deferred_list, nid, memcg);
+ list_lru_unlock_irqrestore(lru, &flags);
+ rcu_read_unlock();
}
static unsigned long deferred_split_count(struct shrinker *shrink,
struct shrink_control *sc)
{
- struct pglist_data *pgdata = NODE_DATA(sc->nid);
- struct deferred_split *ds_queue = &pgdata->deferred_split_queue;
-
-#ifdef CONFIG_MEMCG
- if (sc->memcg)
- ds_queue = &sc->memcg->deferred_split_queue;
-#endif
- return READ_ONCE(ds_queue->split_queue_len);
+ return list_lru_shrink_count(&deferred_split_lru, sc);
}
static bool thp_underused(struct folio *folio)
@@ -4505,45 +4420,45 @@ static bool thp_underused(struct folio *folio)
return false;
}
+static enum lru_status deferred_split_isolate(struct list_head *item,
+ struct list_lru_one *lru,
+ void *cb_arg)
+{
+ struct folio *folio = container_of(item, struct folio, _deferred_list);
+ struct list_head *freeable = cb_arg;
+
+ if (folio_try_get(folio)) {
+ list_lru_isolate_move(lru, item, freeable);
+ return LRU_REMOVED;
+ }
+
+ /* We lost race with folio_put() */
+ list_lru_isolate(lru, item);
+ if (folio_test_partially_mapped(folio)) {
+ folio_clear_partially_mapped(folio);
+ mod_mthp_stat(folio_order(folio),
+ MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, -1);
+ }
+ return LRU_REMOVED;
+}
+
static unsigned long deferred_split_scan(struct shrinker *shrink,
struct shrink_control *sc)
{
- struct deferred_split *ds_queue;
- unsigned long flags;
+ LIST_HEAD(dispose);
struct folio *folio, *next;
- int split = 0, i;
- struct folio_batch fbatch;
+ int split = 0;
+ unsigned long isolated;
- folio_batch_init(&fbatch);
+ isolated = list_lru_shrink_walk_irq(&deferred_split_lru, sc,
+ deferred_split_isolate, &dispose);
-retry:
- ds_queue = split_queue_lock_irqsave(sc->nid, sc->memcg, &flags);
- /* Take pin on all head pages to avoid freeing them under us */
- list_for_each_entry_safe(folio, next, &ds_queue->split_queue,
- _deferred_list) {
- if (folio_try_get(folio)) {
- folio_batch_add(&fbatch, folio);
- } else if (folio_test_partially_mapped(folio)) {
- /* We lost race with folio_put() */
- folio_clear_partially_mapped(folio);
- mod_mthp_stat(folio_order(folio),
- MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, -1);
- }
- list_del_init(&folio->_deferred_list);
- ds_queue->split_queue_len--;
- if (!--sc->nr_to_scan)
- break;
- if (!folio_batch_space(&fbatch))
- break;
- }
- split_queue_unlock_irqrestore(ds_queue, flags);
-
- for (i = 0; i < folio_batch_count(&fbatch); i++) {
+ list_for_each_entry_safe(folio, next, &dispose, _deferred_list) {
bool did_split = false;
bool underused = false;
- struct deferred_split *fqueue;
- folio = fbatch.folios[i];
+ list_del_init(&folio->_deferred_list);
+
if (!folio_test_partially_mapped(folio)) {
/*
* See try_to_map_unused_to_zeropage(): we cannot
@@ -4572,63 +4487,23 @@ static unsigned long deferred_split_scan(struct shrinker *shrink,
* underused, then consider it used and don't add it back to
* split_queue.
*/
- if (did_split || !folio_test_partially_mapped(folio))
- continue;
+ if (!did_split && folio_test_partially_mapped(folio)) {
requeue:
- /*
- * Add back partially mapped folios, or underused folios that
- * we could not lock this round.
- */
- fqueue = folio_split_queue_lock_irqsave(folio, &flags);
- if (list_empty(&folio->_deferred_list)) {
- list_add_tail(&folio->_deferred_list, &fqueue->split_queue);
- fqueue->split_queue_len++;
+ rcu_read_lock();
+ list_lru_add_irq(&deferred_split_lru,
+ &folio->_deferred_list,
+ folio_nid(folio),
+ folio_memcg(folio));
+ rcu_read_unlock();
}
- split_queue_unlock_irqrestore(fqueue, flags);
- }
- folios_put(&fbatch);
-
- if (sc->nr_to_scan && !list_empty(&ds_queue->split_queue)) {
- cond_resched();
- goto retry;
+ folio_put(folio);
}
- /*
- * Stop shrinker if we didn't split any page, but the queue is empty.
- * This can happen if pages were freed under us.
- */
- if (!split && list_empty(&ds_queue->split_queue))
+ if (!split && !isolated)
return SHRINK_STOP;
return split;
}
-#ifdef CONFIG_MEMCG
-void reparent_deferred_split_queue(struct mem_cgroup *memcg)
-{
- struct mem_cgroup *parent = parent_mem_cgroup(memcg);
- struct deferred_split *ds_queue = &memcg->deferred_split_queue;
- struct deferred_split *parent_ds_queue = &parent->deferred_split_queue;
- int nid;
-
- spin_lock_irq(&ds_queue->split_queue_lock);
- spin_lock_nested(&parent_ds_queue->split_queue_lock, SINGLE_DEPTH_NESTING);
-
- if (!ds_queue->split_queue_len)
- goto unlock;
-
- list_splice_tail_init(&ds_queue->split_queue, &parent_ds_queue->split_queue);
- parent_ds_queue->split_queue_len += ds_queue->split_queue_len;
- ds_queue->split_queue_len = 0;
-
- for_each_node(nid)
- set_shrinker_bit(parent, nid, shrinker_id(deferred_split_shrinker));
-
-unlock:
- spin_unlock(&parent_ds_queue->split_queue_lock);
- spin_unlock_irq(&ds_queue->split_queue_lock);
-}
-#endif
-
#ifdef CONFIG_DEBUG_FS
static void split_huge_pages_all(void)
{
diff --git a/mm/internal.h b/mm/internal.h
index 09931b1e535f..e5bf549ad78e 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -861,7 +861,7 @@ static inline bool folio_unqueue_deferred_split(struct folio *folio)
/*
* At this point, there is no one trying to add the folio to
* deferred_list. If folio is not in deferred_list, it's safe
- * to check without acquiring the split_queue_lock.
+ * to check without acquiring the list_lru lock.
*/
if (data_race(list_empty(&folio->_deferred_list)))
return false;
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 1a25af3d6d0f..644de0410c12 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -1301,6 +1301,9 @@ static enum scan_result collapse_huge_page(struct mm_struct *mm, unsigned long s
if (result != SCAN_SUCCEED)
goto out_nolock;
+ if (folio_memcg_alloc_deferred(folio))
+ goto out_nolock;
+
mmap_read_lock(mm);
result = hugepage_vma_revalidate(mm, pmd_addr, /*expect_anon=*/ true,
&vma, cc, order);
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index aa7361b0d2da..fe896824078b 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -4065,11 +4065,6 @@ static struct mem_cgroup *mem_cgroup_alloc(struct mem_cgroup *parent)
for (i = 0; i < MEMCG_CGWB_FRN_CNT; i++)
memcg->cgwb_frn[i].done =
__WB_COMPLETION_INIT(&memcg_cgwb_frn_waitq);
-#endif
-#ifdef CONFIG_TRANSPARENT_HUGEPAGE
- spin_lock_init(&memcg->deferred_split_queue.split_queue_lock);
- INIT_LIST_HEAD(&memcg->deferred_split_queue.split_queue);
- memcg->deferred_split_queue.split_queue_len = 0;
#endif
lru_gen_init_memcg(memcg);
return memcg;
@@ -4221,11 +4216,10 @@ static void mem_cgroup_css_offline(struct cgroup_subsys_state *css)
zswap_memcg_offline_cleanup(memcg);
memcg_offline_kmem(memcg);
- reparent_deferred_split_queue(memcg);
/*
- * The reparenting of objcg must be after the reparenting of the
- * list_lru and deferred_split_queue above, which ensures that they will
- * not mistakenly get the parent list_lru and deferred_split_queue.
+ * The reparenting of objcg must be after the reparenting of
+ * the list_lru in memcg_offline_kmem(), which ensures that
+ * they will not mistakenly get the parent list_lru.
*/
memcg_reparent_objcgs(memcg);
reparent_shrinker_deferred(memcg);
diff --git a/mm/memory.c b/mm/memory.c
index 552fe26a042a..b97e3145982a 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -4759,6 +4759,10 @@ static struct folio *alloc_swap_folio(struct vm_fault *vmf)
folio_put(folio);
goto next;
}
+ if (order > 1 && folio_memcg_alloc_deferred(folio)) {
+ folio_put(folio);
+ goto fallback;
+ }
return folio;
next:
count_mthp_stat(order, MTHP_STAT_SWPIN_FALLBACK);
@@ -5279,6 +5283,10 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf)
folio_put(folio);
goto next;
}
+ if (order > 1 && folio_memcg_alloc_deferred(folio)) {
+ folio_put(folio);
+ goto fallback;
+ }
folio_throttle_swaprate(folio, gfp);
/*
* When a folio is not zeroed during allocation
diff --git a/mm/mm_init.c b/mm/mm_init.c
index db5568cf36e1..c0a7f1cf6fef 100644
--- a/mm/mm_init.c
+++ b/mm/mm_init.c
@@ -1373,19 +1373,6 @@ static void __init calculate_node_totalpages(struct pglist_data *pgdat,
pr_debug("On node %d totalpages: %lu\n", pgdat->node_id, realtotalpages);
}
-#ifdef CONFIG_TRANSPARENT_HUGEPAGE
-static void pgdat_init_split_queue(struct pglist_data *pgdat)
-{
- struct deferred_split *ds_queue = &pgdat->deferred_split_queue;
-
- spin_lock_init(&ds_queue->split_queue_lock);
- INIT_LIST_HEAD(&ds_queue->split_queue);
- ds_queue->split_queue_len = 0;
-}
-#else
-static void pgdat_init_split_queue(struct pglist_data *pgdat) {}
-#endif
-
#ifdef CONFIG_COMPACTION
static void pgdat_init_kcompactd(struct pglist_data *pgdat)
{
@@ -1401,8 +1388,6 @@ static void __meminit pgdat_init_internals(struct pglist_data *pgdat)
pgdat_resize_init(pgdat);
pgdat_kswapd_lock_init(pgdat);
-
- pgdat_init_split_queue(pgdat);
pgdat_init_kcompactd(pgdat);
init_waitqueue_head(&pgdat->kswapd_wait);
--
2.54.0
^ permalink raw reply related
* [PATCH v4 7/8] mm/memory: flatten folio allocation retry loops
From: Johannes Weiner @ 2026-05-21 15:02 UTC (permalink / raw)
To: Andrew Morton
Cc: David Hildenbrand, Lorenzo Stoakes, Shakeel Butt, Michal Hocko,
Dave Chinner, Roman Gushchin, Muchun Song, Qi Zheng, Yosry Ahmed,
Zi Yan, Liam R . Howlett, Usama Arif, Kiryl Shutsemau,
Vlastimil Babka, Kairui Song, Mikhail Zaslonko, Vasily Gorbik,
Baolin Wang, Barry Song, Dev Jain, Lance Yang, Nico Pache,
Ryan Roberts, cgroups, linux-mm, linux-kernel
In-Reply-To: <20260521150330.1955924-1-hannes@cmpxchg.org>
alloc_swap_folio() and alloc_anon_folio() use a top-level if (folio)
that buries the success path four levels deep. This makes for awkward
long lines and wrapping. The next patch will add more code here, so
flatten this now to keep things clean and simple.
alloc_anon_folio() already has a next label, use it for !folio. Add
the equivalent to alloc_swap_folio().
No functional change intended.
Suggested-by: Lorenzo Stoakes (Oracle) <ljs@kernel.org>
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
---
mm/memory.c | 44 +++++++++++++++++++++++---------------------
1 file changed, 23 insertions(+), 21 deletions(-)
diff --git a/mm/memory.c b/mm/memory.c
index 0c9d9c2cbf0e..552fe26a042a 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -4752,13 +4752,15 @@ static struct folio *alloc_swap_folio(struct vm_fault *vmf)
while (orders) {
addr = ALIGN_DOWN(vmf->address, PAGE_SIZE << order);
folio = vma_alloc_folio(gfp, order, vma, addr);
- if (folio) {
- if (!mem_cgroup_swapin_charge_folio(folio, vma->vm_mm,
- gfp, entry))
- return folio;
+ if (!folio)
+ goto next;
+ if (mem_cgroup_swapin_charge_folio(folio, vma->vm_mm, gfp, entry)) {
count_mthp_stat(order, MTHP_STAT_SWPIN_FALLBACK_CHARGE);
folio_put(folio);
+ goto next;
}
+ return folio;
+next:
count_mthp_stat(order, MTHP_STAT_SWPIN_FALLBACK);
order = next_order(&orders, order);
}
@@ -5270,24 +5272,24 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf)
while (orders) {
addr = ALIGN_DOWN(vmf->address, PAGE_SIZE << order);
folio = vma_alloc_folio(gfp, order, vma, addr);
- if (folio) {
- if (mem_cgroup_charge(folio, vma->vm_mm, gfp)) {
- count_mthp_stat(order, MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE);
- folio_put(folio);
- goto next;
- }
- folio_throttle_swaprate(folio, gfp);
- /*
- * When a folio is not zeroed during allocation
- * (__GFP_ZERO not used) or user folios require special
- * handling, folio_zero_user() is used to make sure
- * that the page corresponding to the faulting address
- * will be hot in the cache after zeroing.
- */
- if (user_alloc_needs_zeroing())
- folio_zero_user(folio, vmf->address);
- return folio;
+ if (!folio)
+ goto next;
+ if (mem_cgroup_charge(folio, vma->vm_mm, gfp)) {
+ count_mthp_stat(order, MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE);
+ folio_put(folio);
+ goto next;
}
+ folio_throttle_swaprate(folio, gfp);
+ /*
+ * When a folio is not zeroed during allocation
+ * (__GFP_ZERO not used) or user folios require special
+ * handling, folio_zero_user() is used to make sure
+ * that the page corresponding to the faulting address
+ * will be hot in the cache after zeroing.
+ */
+ if (user_alloc_needs_zeroing())
+ folio_zero_user(folio, vmf->address);
+ return folio;
next:
count_mthp_stat(order, MTHP_STAT_ANON_FAULT_FALLBACK);
order = next_order(&orders, order);
--
2.54.0
^ permalink raw reply related
* [PATCH v4 6/8] mm: list_lru: introduce folio_memcg_list_lru_alloc()
From: Johannes Weiner @ 2026-05-21 15:02 UTC (permalink / raw)
To: Andrew Morton
Cc: David Hildenbrand, Lorenzo Stoakes, Shakeel Butt, Michal Hocko,
Dave Chinner, Roman Gushchin, Muchun Song, Qi Zheng, Yosry Ahmed,
Zi Yan, Liam R . Howlett, Usama Arif, Kiryl Shutsemau,
Vlastimil Babka, Kairui Song, Mikhail Zaslonko, Vasily Gorbik,
Baolin Wang, Barry Song, Dev Jain, Lance Yang, Nico Pache,
Ryan Roberts, cgroups, linux-mm, linux-kernel
In-Reply-To: <20260521150330.1955924-1-hannes@cmpxchg.org>
memcg_list_lru_alloc() is called every time an object that may end up
on the list_lru is created. It needs to quickly check if the list_lru
heads for the memcg already exist, and allocate them when they don't.
Doing this with folio objects is tricky: folio_memcg() is not stable
and requires either RCU protection or pinning the cgroup. But it's
desirable to make the existence check lightweight under RCU, and only
pin the memcg when we need to allocate list_lru heads and may block.
In preparation for switching the THP shrinker to list_lru, add a
helper function for allocating list_lru heads coming from a folio.
Reviewed-by: David Hildenbrand (Arm) <david@kernel.org>
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Reviewed-by: Lorenzo Stoakes (Oracle) <ljs@kernel.org>
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
---
include/linux/list_lru.h | 27 +++++++++++++++++++++++++++
mm/list_lru.c | 39 ++++++++++++++++++++++++++++++++++-----
2 files changed, 61 insertions(+), 5 deletions(-)
diff --git a/include/linux/list_lru.h b/include/linux/list_lru.h
index c79ed378311f..733a262b91e5 100644
--- a/include/linux/list_lru.h
+++ b/include/linux/list_lru.h
@@ -81,6 +81,33 @@ static inline int list_lru_init_memcg_key(struct list_lru *lru, struct shrinker
int memcg_list_lru_alloc(struct mem_cgroup *memcg, struct list_lru *lru,
gfp_t gfp);
+
+#ifdef CONFIG_MEMCG
+/**
+ * folio_memcg_list_lru_alloc - allocate list_lru heads for shrinkable folio
+ * @folio: the newly allocated & charged folio
+ * @lru: the list_lru this might be queued on
+ * @gfp: gfp mask
+ *
+ * Allocate list_lru heads (per-memcg, per-node) needed to queue this
+ * particular folio down the line.
+ *
+ * This does memcg_list_lru_alloc(), but on the memcg that @folio is
+ * associated with. Handles folio_memcg() access rules in the fast
+ * path (list_lru heads allocated) and the allocation slowpath.
+ *
+ * Returns 0 on success, a negative error value otherwise.
+ */
+int folio_memcg_list_lru_alloc(struct folio *folio, struct list_lru *lru,
+ gfp_t gfp);
+#else
+static inline int folio_memcg_list_lru_alloc(struct folio *folio,
+ struct list_lru *lru, gfp_t gfp)
+{
+ return 0;
+}
+#endif
+
void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup *parent);
/**
diff --git a/mm/list_lru.c b/mm/list_lru.c
index df58226eea8c..4eeb4351c475 100644
--- a/mm/list_lru.c
+++ b/mm/list_lru.c
@@ -562,17 +562,14 @@ static inline bool memcg_list_lru_allocated(struct mem_cgroup *memcg,
return idx < 0 || xa_load(&lru->xa, idx);
}
-int memcg_list_lru_alloc(struct mem_cgroup *memcg, struct list_lru *lru,
- gfp_t gfp)
+static int __memcg_list_lru_alloc(struct mem_cgroup *memcg,
+ struct list_lru *lru, gfp_t gfp)
{
unsigned long flags;
struct list_lru_memcg *mlru = NULL;
struct mem_cgroup *pos, *parent;
XA_STATE(xas, &lru->xa, 0);
- if (!list_lru_memcg_aware(lru) || memcg_list_lru_allocated(memcg, lru))
- return 0;
-
gfp &= GFP_RECLAIM_MASK;
/*
* Because the list_lru can be reparented to the parent cgroup's
@@ -613,6 +610,38 @@ int memcg_list_lru_alloc(struct mem_cgroup *memcg, struct list_lru *lru,
return xas_error(&xas);
}
+
+int memcg_list_lru_alloc(struct mem_cgroup *memcg, struct list_lru *lru,
+ gfp_t gfp)
+{
+ if (!list_lru_memcg_aware(lru) || memcg_list_lru_allocated(memcg, lru))
+ return 0;
+ return __memcg_list_lru_alloc(memcg, lru, gfp);
+}
+
+int folio_memcg_list_lru_alloc(struct folio *folio, struct list_lru *lru,
+ gfp_t gfp)
+{
+ struct mem_cgroup *memcg;
+ int res;
+
+ if (!list_lru_memcg_aware(lru))
+ return 0;
+
+ /* Fast path when list_lru heads already exist */
+ rcu_read_lock();
+ memcg = folio_memcg(folio);
+ res = memcg_list_lru_allocated(memcg, lru);
+ rcu_read_unlock();
+ if (likely(res))
+ return 0;
+
+ /* Allocation may block, pin the memcg */
+ memcg = get_mem_cgroup_from_folio(folio);
+ res = __memcg_list_lru_alloc(memcg, lru, gfp);
+ mem_cgroup_put(memcg);
+ return res;
+}
#else
static inline void memcg_init_list_lru(struct list_lru *lru, bool memcg_aware)
{
--
2.54.0
^ permalink raw reply related
* [PATCH v4 5/8] mm: list_lru: introduce caller locking for additions and deletions
From: Johannes Weiner @ 2026-05-21 15:02 UTC (permalink / raw)
To: Andrew Morton
Cc: David Hildenbrand, Lorenzo Stoakes, Shakeel Butt, Michal Hocko,
Dave Chinner, Roman Gushchin, Muchun Song, Qi Zheng, Yosry Ahmed,
Zi Yan, Liam R . Howlett, Usama Arif, Kiryl Shutsemau,
Vlastimil Babka, Kairui Song, Mikhail Zaslonko, Vasily Gorbik,
Baolin Wang, Barry Song, Dev Jain, Lance Yang, Nico Pache,
Ryan Roberts, cgroups, linux-mm, linux-kernel
In-Reply-To: <20260521150330.1955924-1-hannes@cmpxchg.org>
Locking is currently internal to the list_lru API. However, a caller
might want to keep auxiliary state synchronized with the LRU state.
For example, the THP shrinker uses the lock of its custom LRU to keep
PG_partially_mapped and vmstats consistent.
To allow the THP shrinker to switch to list_lru, provide normal and
irqsafe locking primitives as well as caller-locked variants of the
addition and deletion functions.
Reviewed-by: David Hildenbrand (Arm) <david@kernel.org>
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Reviewed-by: Lorenzo Stoakes (Oracle) <ljs@kernel.org>
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
---
include/linux/list_lru.h | 41 ++++++++++++
mm/list_lru.c | 131 ++++++++++++++++++++++++++++++---------
2 files changed, 141 insertions(+), 31 deletions(-)
diff --git a/include/linux/list_lru.h b/include/linux/list_lru.h
index fe739d35a864..c79ed378311f 100644
--- a/include/linux/list_lru.h
+++ b/include/linux/list_lru.h
@@ -83,6 +83,44 @@ int memcg_list_lru_alloc(struct mem_cgroup *memcg, struct list_lru *lru,
gfp_t gfp);
void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup *parent);
+/**
+ * list_lru_lock: lock the sublist for the given node and memcg
+ * @lru: the lru pointer
+ * @nid: the node id of the sublist to lock.
+ * @memcg: the cgroup of the sublist to lock.
+ *
+ * Returns the locked list_lru_one sublist. The caller must call
+ * list_lru_unlock() when done.
+ *
+ * You must ensure that the memcg is not freed during this call (e.g., with
+ * rcu or by taking a css refcnt).
+ *
+ * Return: the locked list_lru_one, or NULL on failure
+ */
+struct list_lru_one *list_lru_lock(struct list_lru *lru, int nid,
+ struct mem_cgroup *memcg);
+
+/**
+ * list_lru_unlock: unlock a sublist locked by list_lru_lock()
+ * @l: the list_lru_one to unlock
+ */
+void list_lru_unlock(struct list_lru_one *l);
+
+struct list_lru_one *list_lru_lock_irq(struct list_lru *lru, int nid,
+ struct mem_cgroup *memcg);
+void list_lru_unlock_irq(struct list_lru_one *l);
+
+struct list_lru_one *list_lru_lock_irqsave(struct list_lru *lru, int nid,
+ struct mem_cgroup *memcg, unsigned long *irq_flags);
+void list_lru_unlock_irqrestore(struct list_lru_one *l,
+ unsigned long *irq_flags);
+
+/* Caller-locked variants, see list_lru_add() etc for documentation */
+bool __list_lru_add(struct list_lru *lru, struct list_lru_one *l,
+ struct list_head *item, int nid, struct mem_cgroup *memcg);
+bool __list_lru_del(struct list_lru *lru, struct list_lru_one *l,
+ struct list_head *item, int nid);
+
/**
* list_lru_add: add an element to the lru list's tail
* @lru: the lru pointer
@@ -115,6 +153,9 @@ void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup *paren
bool list_lru_add(struct list_lru *lru, struct list_head *item, int nid,
struct mem_cgroup *memcg);
+bool list_lru_add_irq(struct list_lru *lru, struct list_head *item, int nid,
+ struct mem_cgroup *memcg);
+
/**
* list_lru_add_obj: add an element to the lru list's tail
* @lru: the lru pointer
diff --git a/mm/list_lru.c b/mm/list_lru.c
index 65962dbf6dda..df58226eea8c 100644
--- a/mm/list_lru.c
+++ b/mm/list_lru.c
@@ -15,17 +15,23 @@
#include "slab.h"
#include "internal.h"
-static inline void lock_list_lru(struct list_lru_one *l, bool irq)
+static inline void lock_list_lru(struct list_lru_one *l, bool irq,
+ unsigned long *irq_flags)
{
- if (irq)
+ if (irq_flags)
+ spin_lock_irqsave(&l->lock, *irq_flags);
+ else if (irq)
spin_lock_irq(&l->lock);
else
spin_lock(&l->lock);
}
-static inline void unlock_list_lru(struct list_lru_one *l, bool irq_off)
+static inline void unlock_list_lru(struct list_lru_one *l, bool irq_off,
+ unsigned long *irq_flags)
{
- if (irq_off)
+ if (irq_flags)
+ spin_unlock_irqrestore(&l->lock, *irq_flags);
+ else if (irq_off)
spin_unlock_irq(&l->lock);
else
spin_unlock(&l->lock);
@@ -78,7 +84,7 @@ list_lru_from_memcg_idx(struct list_lru *lru, int nid, int idx)
static inline struct list_lru_one *
lock_list_lru_of_memcg(struct list_lru *lru, int nid, struct mem_cgroup *memcg,
- bool irq, bool skip_empty)
+ bool irq, unsigned long *irq_flags, bool skip_empty)
{
struct list_lru_one *l;
@@ -86,12 +92,12 @@ lock_list_lru_of_memcg(struct list_lru *lru, int nid, struct mem_cgroup *memcg,
again:
l = list_lru_from_memcg_idx(lru, nid, memcg_kmem_id(memcg));
if (likely(l)) {
- lock_list_lru(l, irq);
+ lock_list_lru(l, irq, irq_flags);
if (likely(READ_ONCE(l->nr_items) != LONG_MIN)) {
rcu_read_unlock();
return l;
}
- unlock_list_lru(l, irq);
+ unlock_list_lru(l, irq, irq_flags);
}
/*
* Caller may simply bail out if raced with reparenting or
@@ -132,38 +138,106 @@ list_lru_from_memcg_idx(struct list_lru *lru, int nid, int idx)
static inline struct list_lru_one *
lock_list_lru_of_memcg(struct list_lru *lru, int nid, struct mem_cgroup *memcg,
- bool irq, bool skip_empty)
+ bool irq, unsigned long *irq_flags, bool skip_empty)
{
struct list_lru_one *l = &lru->node[nid].lru;
- lock_list_lru(l, irq);
+ lock_list_lru(l, irq, irq_flags);
return l;
}
#endif /* CONFIG_MEMCG */
-/* The caller must ensure the memcg lifetime. */
-bool list_lru_add(struct list_lru *lru, struct list_head *item, int nid,
- struct mem_cgroup *memcg)
+struct list_lru_one *list_lru_lock(struct list_lru *lru, int nid,
+ struct mem_cgroup *memcg)
{
- struct list_lru_node *nlru = &lru->node[nid];
- struct list_lru_one *l;
+ return lock_list_lru_of_memcg(lru, nid, memcg, /*irq=*/false,
+ /*irq_flags=*/NULL, /*skip_empty=*/false);
+}
+
+void list_lru_unlock(struct list_lru_one *l)
+{
+ unlock_list_lru(l, /*irq_off=*/false, /*irq_flags=*/NULL);
+}
+
+struct list_lru_one *list_lru_lock_irq(struct list_lru *lru, int nid,
+ struct mem_cgroup *memcg)
+{
+ return lock_list_lru_of_memcg(lru, nid, memcg, /*irq=*/true,
+ /*irq_flags=*/NULL, /*skip_empty=*/false);
+}
+
+void list_lru_unlock_irq(struct list_lru_one *l)
+{
+ unlock_list_lru(l, /*irq_off=*/true, /*irq_flags=*/NULL);
+}
- l = lock_list_lru_of_memcg(lru, nid, memcg, false, false);
+struct list_lru_one *list_lru_lock_irqsave(struct list_lru *lru, int nid,
+ struct mem_cgroup *memcg,
+ unsigned long *flags)
+{
+ return lock_list_lru_of_memcg(lru, nid, memcg, /*irq=*/true,
+ /*irq_flags=*/flags, /*skip_empty=*/false);
+}
+
+void list_lru_unlock_irqrestore(struct list_lru_one *l, unsigned long *flags)
+{
+ unlock_list_lru(l, /*irq_off=*/true, /*irq_flags=*/flags);
+}
+
+bool __list_lru_add(struct list_lru *lru, struct list_lru_one *l,
+ struct list_head *item, int nid,
+ struct mem_cgroup *memcg)
+{
if (list_empty(item)) {
list_add_tail(item, &l->list);
/* Set shrinker bit if the first element was added */
if (!l->nr_items++)
set_shrinker_bit(memcg, nid, lru_shrinker_id(lru));
- unlock_list_lru(l, false);
- atomic_long_inc(&nlru->nr_items);
+ atomic_long_inc(&lru->node[nid].nr_items);
return true;
}
- unlock_list_lru(l, false);
return false;
}
EXPORT_SYMBOL_GPL(list_lru_add);
+bool __list_lru_del(struct list_lru *lru, struct list_lru_one *l,
+ struct list_head *item, int nid)
+{
+ if (!list_empty(item)) {
+ list_del_init(item);
+ l->nr_items--;
+ atomic_long_dec(&lru->node[nid].nr_items);
+ return true;
+ }
+ return false;
+}
+
+/* The caller must ensure the memcg lifetime. */
+bool list_lru_add(struct list_lru *lru, struct list_head *item, int nid,
+ struct mem_cgroup *memcg)
+{
+ struct list_lru_one *l;
+ bool ret;
+
+ l = list_lru_lock(lru, nid, memcg);
+ ret = __list_lru_add(lru, l, item, nid, memcg);
+ list_lru_unlock(l);
+ return ret;
+}
+
+bool list_lru_add_irq(struct list_lru *lru, struct list_head *item,
+ int nid, struct mem_cgroup *memcg)
+{
+ struct list_lru_one *l;
+ bool ret;
+
+ l = list_lru_lock_irq(lru, nid, memcg);
+ ret = __list_lru_add(lru, l, item, nid, memcg);
+ list_lru_unlock_irq(l);
+ return ret;
+}
+
bool list_lru_add_obj(struct list_lru *lru, struct list_head *item)
{
bool ret;
@@ -185,19 +259,13 @@ EXPORT_SYMBOL_GPL(list_lru_add_obj);
bool list_lru_del(struct list_lru *lru, struct list_head *item, int nid,
struct mem_cgroup *memcg)
{
- struct list_lru_node *nlru = &lru->node[nid];
struct list_lru_one *l;
+ bool ret;
- l = lock_list_lru_of_memcg(lru, nid, memcg, false, false);
- if (!list_empty(item)) {
- list_del_init(item);
- l->nr_items--;
- unlock_list_lru(l, false);
- atomic_long_dec(&nlru->nr_items);
- return true;
- }
- unlock_list_lru(l, false);
- return false;
+ l = list_lru_lock(lru, nid, memcg);
+ ret = __list_lru_del(lru, l, item, nid);
+ list_lru_unlock(l);
+ return ret;
}
bool list_lru_del_obj(struct list_lru *lru, struct list_head *item)
@@ -270,7 +338,8 @@ __list_lru_walk_one(struct list_lru *lru, int nid, struct mem_cgroup *memcg,
unsigned long isolated = 0;
restart:
- l = lock_list_lru_of_memcg(lru, nid, memcg, irq_off, true);
+ l = lock_list_lru_of_memcg(lru, nid, memcg, /*irq=*/irq_off,
+ /*irq_flags=*/NULL, /*skip_empty=*/true);
if (!l)
return isolated;
list_for_each_safe(item, n, &l->list) {
@@ -311,7 +380,7 @@ __list_lru_walk_one(struct list_lru *lru, int nid, struct mem_cgroup *memcg,
BUG();
}
}
- unlock_list_lru(l, irq_off);
+ unlock_list_lru(l, irq_off, NULL);
out:
return isolated;
}
--
2.54.0
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).