[PATCH v3] mm/shrinker: allocate shrinker_info_unit from a dedicated kmem_cache

[PATCH v3] mm/shrinker: allocate shrinker_info_unit from a dedicated kmem_cache

[Konstantin Khorenko]

shrinker_info_unit holds BITS_PER_LONG atomic_long_t of deferred work
plus a one-word bitmap, 520 bytes on 64-bit. kzalloc_node() takes it
from kmalloc-1k, so almost half of every unit is padding.

Allocate the units from a cache of their own, sized to the unit. The
cache is not cache-line aligned: the counters of a unit already share
cache lines between shrinkers, and aligning the unit would only keep
neighbouring units off each other's boundary line, at the cost of part
of the saving.

Suggested-by: Andrew Morton <akpm@linux-foundation.org>
Assisted-by: LLM
Signed-off-by: Konstantin Khorenko <khorenko@virtuozzo.com>
---
This is v3 of patch 1/2 of "mm/shrinker: make per-memcg shrinker units
cheaper", sent on its own as Andrew asked in the v2 thread.

Patch 2/2 of v1-v2, "mm/shrinker: allocate the per-memcg shrinker units
on demand", is not part of this posting. It is the change that makes a
memcg pay only for the shrinkers it uses, and it is discussed separately
in the v2 thread.

Tested on a 4-CPU, single-node VM with the test from the earlier cover
letters: 8000 tmpfs mounts, then 2000 empty memory cgroups. The units
move from kmalloc-1k to the new cache, 31 per order-2 slab, and the
unreclaimable slab per cgroup goes from 144 KB to 87 KB. Removing the
cgroups and dropping the caches returns the cache to its earlier size.

memory.reclaim on a cgroup still frees its objects on 140 ext4
superblocks, and cgroups making their first objects while mounts keep
raising shrinker_nr_max leave nothing in dmesg, also with
slub_debug=FZPU on the cache. selftests/cgroup test_memcontrol passes.
test_kmem_memcg_deletion also fails without the patch, by a similar
margin.

Changes in v3:
- Allocate the unit from a dedicated kmem_cache instead of shrinking it
  to 63 ids so that it fits kmalloc-512 (Andrew Morton, Muchun Song).
  New subject, since the unit keeps its size.
- Link to v2: https://patch.msgid.link/20261014-b4-prep-vstor-144715-size-shrinker_info_unit-to-a-power-of-two-v2-0-51af2aedbe6c@virtuozzo.com

Changes in v2:
- Patch 1: check the unit size with BUILD_BUG_ON(!is_power_of_2())
  in shrinker_unit_alloc() instead of an open-coded static_assert()
  in the header (Muchun Song).
- Link to v1: https://patch.msgid.link/20261012-b4-prep-vstor-144715-size-shrinker_info_unit-to-a-power-of-two-v1-0-5a7608393b7a@virtuozzo.com
---
 include/linux/memcontrol.h |  1 +
 mm/memcontrol.c            |  2 ++
 mm/shrinker.c              | 11 +++++++++--
 3 files changed, 12 insertions(+), 2 deletions(-)

diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index 7d1c0ce189a88..b5a89594c263c 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -1675,6 +1675,7 @@ static inline u64 mem_cgroup_get_socket_pressure(struct mem_cgroup *memcg)
 }
 #endif
 
+void shrinker_unit_cache_init(void);
 int alloc_shrinker_info(struct mem_cgroup *memcg);
 void free_shrinker_info(struct mem_cgroup *memcg);
 void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id);
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 856a7d07586cc..7a817181168ed 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -5717,6 +5717,8 @@ int __init mem_cgroup_init(void)
 	memcg_pn_cachep = KMEM_CACHE(mem_cgroup_per_node,
 				     SLAB_PANIC | SLAB_HWCACHE_ALIGN);
 
+	shrinker_unit_cache_init();
+
 	return 0;
 }
 
diff --git a/mm/shrinker.c b/mm/shrinker.c
index 7ec2a9704f6f2..4fb424e721bca 100644
--- a/mm/shrinker.c
+++ b/mm/shrinker.c
@@ -12,6 +12,12 @@ DEFINE_MUTEX(shrinker_mutex);
 
 #ifdef CONFIG_MEMCG
 static int shrinker_nr_max;
+static struct kmem_cache *shrinker_unit_cachep;
+
+void __init shrinker_unit_cache_init(void)
+{
+	shrinker_unit_cachep = KMEM_CACHE(shrinker_info_unit, SLAB_PANIC);
+}
 
 static inline int shrinker_unit_size(int nr_items)
 {
@@ -33,7 +39,7 @@ static inline void shrinker_unit_free(struct shrinker_info *info, int start)
 		if (!unit[i])
 			break;
 
-		kfree(unit[i]);
+		kmem_cache_free(shrinker_unit_cachep, unit[i]);
 		unit[i] = NULL;
 	}
 }
@@ -47,7 +53,8 @@ static inline int shrinker_unit_alloc(struct shrinker_info *new,
 	int i;
 
 	for (i = start; i < nr; i++) {
-		unit = kzalloc_node(sizeof(*unit), GFP_KERNEL, nid);
+		unit = kmem_cache_alloc_node(shrinker_unit_cachep,
+					     GFP_KERNEL | __GFP_ZERO, nid);
 		if (!unit) {
 			shrinker_unit_free(new, start);
 			return -ENOMEM;

---
base-commit: a5ebb76233b79db01e82e061173cb73c6d2b5c6b
change-id: 20261016-vstor-144715-shrinker_info_unit-kmem_cache-69ea3f78b635

Best regards,
--  
Konstantin Khorenko <khorenko@virtuozzo.com>

Re: [PATCH v3] mm/shrinker: allocate shrinker_info_unit from a dedicated kmem_cache

[Harry Yoo]

On Mon, Oct 19, 2026 at 06:11:44PM +0200, Konstantin Khorenko wrote:
> shrinker_info_unit holds BITS_PER_LONG atomic_long_t of deferred work
> plus a one-word bitmap, 520 bytes on 64-bit. kzalloc_node() takes it
> from kmalloc-1k, so almost half of every unit is padding.
> 
> Allocate the units from a cache of their own, sized to the unit. The
> cache is not cache-line aligned: the counters of a unit already share
> cache lines between shrinkers, and aligning the unit would only keep
> neighbouring units off each other's boundary line, at the cost of part
> of the saving.
> 
> Suggested-by: Andrew Morton <akpm@linux-foundation.org>
> Assisted-by: LLM
> Signed-off-by: Konstantin Khorenko <khorenko@virtuozzo.com>
> ---

Looks good to me,
Reviewed-by: Harry Yoo (Meta) <harry@kernel.org>

-- 
Cheers,
Harry / Hyeonggon