[PATCH 0/2] mm/shrinker: make per-memcg shrinker units cheaper

[PATCH 0/2] mm/shrinker: make per-memcg shrinker units cheaper

[Konstantin Khorenko]

Every memory cgroup allocates, per NUMA node, one shrinker_info_unit
per 64 memcg-aware shrinker ids up to shrinker_nr_max, whether or not it
ever charges an object to those shrinkers. Every superblock registers
such a shrinker, so the cost is

  memory cgroups x NUMA nodes x shrinker_nr_max / 64 x 1 KiB

because the unit is 520 bytes and comes from kmalloc-1k.

Page cache reclaim does not make this go away:

- Live cgroups keep their units until they are removed.
- The units are not charged to any memcg, so memory.max does not see
  them and a delegated subtree creates cgroups at the host's expense.
- A dying cgroup keeps its units until the last reference to it is
  dropped, for instance by cgroup writeback, a swap entry or a socket.
  Nothing reclaims it for the sake of the units, and MemAvailable
  counts them as unreclaimable.
- shrinker_nr_max never comes down: after a burst of mounts every new
  cgroup pays for the peak until reboot.
- Both factors grow with the number of workloads on the host.

Patch 1 lets a unit cover BITS_PER_LONG - 1 ids so it fills a 512-byte
slot exactly: half the memory, nothing else changes.

Patch 2 allocates a unit only when the cgroup gets a list on that
shrinker's list_lru, which is where map bits are set: a cgroup pays for
the shrinkers it holds objects of, not for every superblock on the host.
The units stay uncharged, but how many a cgroup makes the host allocate
is now bounded by the shrinkers it uses. The unit pointer array in
shrinker_info still grows with shrinker_nr_max, at 8 bytes per 63 ids
per node. Patch 2 relies on commit fafaeceb89a5 ("mm: switch deferred
split shrinker to list_lru") and is not meant for stable.

On a host running a few hundred containers (45000 memory cgroups, most
of them dying, shrinker_nr_max 52480, 4 nodes) the units took 145 GB of
377 GB.

Reproducer without containers: 48 CPUs, 4 nodes, v7.3-rc6, fresh boot
per kernel. 8000 tmpfs mounts raise shrinker_nr_max to 8064, then 2000
empty memory cgroups are created. Active objects added to the 512-byte
and 1-KiB kmalloc caches, and slab per cgroup:

                  objects added       slab per cgroup
  base            1011518 x 1024 B    548 KB
  + patch 1       1023787 x  512 B    289 KB
  + patches 1-2       136 x  512 B     42 KB

The 42 KB left is the same on all three: the memcg's own structures and
8 KB of shrinker_info pointer arrays at this shrinker_nr_max. A cgroup
that looks up names on two filesystems gets 8 units (4 KB) with both
patches, and so does each ancestor that had none. Writing a page of a
file in each cgroup and removing the cgroups keeps the same amounts
pinned by the 2000 dying cgroups until the cache is dropped.

Tested with selftests/cgroup test_kmem and test_memcontrol, a check that
memory.reclaim reaches 140 ext4 superblocks with consecutive shrinker
ids, and unit installs from 64 cgroups racing with 3000 mounts that keep
growing shrinker_nr_max. On a kernel with lockdep, KASAN and DEBUG_VM the
same runs leave nothing in dmesg. test_memcg_sock fails there now and
then, with and without the series: memory.stat's sock counter is not
flushed yet when the test compares it with memory.current.

---
Konstantin Khorenko (2):
      mm/shrinker: size shrinker_info_unit to a power of two
      mm/shrinker: allocate the per-memcg shrinker units on demand

 include/linux/memcontrol.h |   2 +
 include/linux/shrinker.h   |   9 ++-
 mm/list_lru.c              |   6 ++
 mm/shrinker.c              | 161 ++++++++++++++++++++++++++++++++++++---------
 4 files changed, 147 insertions(+), 31 deletions(-)
---
base-commit: a5ebb76233b79db01e82e061173cb73c6d2b5c6b
change-id: 20261011-b4-prep-vstor-144715-size-shrinker_info_unit-to-a-power-of-two-9f2e043163af

Best regards,
--  
Konstantin Khorenko <khorenko@virtuozzo.com>

[PATCH 1/2] mm/shrinker: size shrinker_info_unit to a power of two

[Konstantin Khorenko]

Every memory cgroup allocates, per NUMA node, one shrinker_info_unit
for every SHRINKER_UNIT_BITS shrinker ids up to shrinker_nr_max. The
unit holds BITS_PER_LONG atomic_long_t of deferred work, a power of
two, plus the one-word bitmap of shrinkers with objects charged to the
cgroup. That word takes the unit just past the power of two, to 520
bytes on 64-bit, so every unit comes from kmalloc-1k and almost half of
it is padding.

The units are allocated when the cgroup goes online and freed when it is
freed, so a dying cgroup keeps them until the last reference to it is
gone. Their number is the number of memory cgroups, times the NUMA
nodes, times shrinker_nr_max / SHRINKER_UNIT_BITS. shrinker_nr_max is
the peak number of memcg-aware shrinkers registered at once, and every
superblock registers one, so a host that mounts many filesystems and
runs many cgroups carries the padding as unreclaimable slab, and keeps
paying it for new cgroups after the mounts are gone.

Let a unit cover one id less than a word holds. The bitmap still fits
one word, and nr_deferred[] plus that word add up to BITS_PER_LONG
longs, so the unit fills a power-of-two kmalloc slot exactly and, on
64-bit, the memory taken by units halves, for about one unit more in
BITS_PER_LONG.
round_up() in expand_shrinker_info() needs a power-of-two step and
becomes roundup(); the other uses of SHRINKER_UNIT_BITS are divisions,
modulos and multiplications that do not care. Assert the unit size at
build time, since the layout depends on it.

Assisted-by: LLM
Signed-off-by: Konstantin Khorenko <khorenko@virtuozzo.com>
---
 include/linux/shrinker.h | 9 ++++++++-
 mm/shrinker.c            | 2 +-
 2 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/include/linux/shrinker.h b/include/linux/shrinker.h
index 1a00be90d93a9..c250ab68d346c 100644
--- a/include/linux/shrinker.h
+++ b/include/linux/shrinker.h
@@ -7,7 +7,11 @@
 #include <linux/refcount.h>
 #include <linux/completion.h>
 
-#define SHRINKER_UNIT_BITS	BITS_PER_LONG
+/*
+ * One id less than a word holds, so that nr_deferred[] and the map word add
+ * up to BITS_PER_LONG longs: a power of two, which fills a kmalloc slot.
+ */
+#define SHRINKER_UNIT_BITS	(BITS_PER_LONG - 1)
 
 /*
  * Bitmap and deferred work of shrinker::id corresponding to memcg-aware
@@ -18,6 +22,9 @@ struct shrinker_info_unit {
 	DECLARE_BITMAP(map, SHRINKER_UNIT_BITS);
 };
 
+static_assert(!(sizeof(struct shrinker_info_unit) &
+		(sizeof(struct shrinker_info_unit) - 1)));
+
 struct shrinker_info {
 	struct rcu_head rcu;
 	int map_nr_max;
diff --git a/mm/shrinker.c b/mm/shrinker.c
index 7ec2a9704f6f2..a9418072a509e 100644
--- a/mm/shrinker.c
+++ b/mm/shrinker.c
@@ -159,7 +159,7 @@ static int expand_one_shrinker_info(struct mem_cgroup *memcg, int new_size,
 static int expand_shrinker_info(int new_id)
 {
 	int ret = 0;
-	int new_nr_max = round_up(new_id + 1, SHRINKER_UNIT_BITS);
+	int new_nr_max = roundup(new_id + 1, SHRINKER_UNIT_BITS);
 	int new_size, old_size = 0;
 	struct mem_cgroup *memcg;
 

-- 
2.43.0

Re: [PATCH 1/2] mm/shrinker: size shrinker_info_unit to a power of two

[Qi Zheng]

On 10/13/26 3:02 AM, Konstantin Khorenko wrote:
> Every memory cgroup allocates, per NUMA node, one shrinker_info_unit
> for every SHRINKER_UNIT_BITS shrinker ids up to shrinker_nr_max. The
> unit holds BITS_PER_LONG atomic_long_t of deferred work, a power of
> two, plus the one-word bitmap of shrinkers with objects charged to the
> cgroup. That word takes the unit just past the power of two, to 520
> bytes on 64-bit, so every unit comes from kmalloc-1k and almost half of
> it is padding.
> 
> The units are allocated when the cgroup goes online and freed when it is
> freed, so a dying cgroup keeps them until the last reference to it is
> gone. Their number is the number of memory cgroups, times the NUMA
> nodes, times shrinker_nr_max / SHRINKER_UNIT_BITS. shrinker_nr_max is
> the peak number of memcg-aware shrinkers registered at once, and every
> superblock registers one, so a host that mounts many filesystems and
> runs many cgroups carries the padding as unreclaimable slab, and keeps
> paying it for new cgroups after the mounts are gone.
> 
> Let a unit cover one id less than a word holds. The bitmap still fits
> one word, and nr_deferred[] plus that word add up to BITS_PER_LONG
> longs, so the unit fills a power-of-two kmalloc slot exactly and, on
> 64-bit, the memory taken by units halves, for about one unit more in
> BITS_PER_LONG.
> round_up() in expand_shrinker_info() needs a power-of-two step and
> becomes roundup(); the other uses of SHRINKER_UNIT_BITS are divisions,
> modulos and multiplications that do not care. Assert the unit size at
> build time, since the layout depends on it.
> 
> Assisted-by: LLM
> Signed-off-by: Konstantin Khorenko <khorenko@virtuozzo.com>
> ---
>   include/linux/shrinker.h | 9 ++++++++-
>   mm/shrinker.c            | 2 +-
>   2 files changed, 9 insertions(+), 2 deletions(-)
> 
> diff --git a/include/linux/shrinker.h b/include/linux/shrinker.h
> index 1a00be90d93a9..c250ab68d346c 100644
> --- a/include/linux/shrinker.h
> +++ b/include/linux/shrinker.h
> @@ -7,7 +7,11 @@
>   #include <linux/refcount.h>
>   #include <linux/completion.h>
>   
> -#define SHRINKER_UNIT_BITS	BITS_PER_LONG
> +/*
> + * One id less than a word holds, so that nr_deferred[] and the map word add
> + * up to BITS_PER_LONG longs: a power of two, which fills a kmalloc slot.
> + */
> +#define SHRINKER_UNIT_BITS	(BITS_PER_LONG - 1)
>   
>   /*
>    * Bitmap and deferred work of shrinker::id corresponding to memcg-aware
> @@ -18,6 +22,9 @@ struct shrinker_info_unit {
>   	DECLARE_BITMAP(map, SHRINKER_UNIT_BITS);
>   };
>   
> +static_assert(!(sizeof(struct shrinker_info_unit) &
> +		(sizeof(struct shrinker_info_unit) - 1)));
> +
>   struct shrinker_info {
>   	struct rcu_head rcu;
>   	int map_nr_max;
> diff --git a/mm/shrinker.c b/mm/shrinker.c
> index 7ec2a9704f6f2..a9418072a509e 100644
> --- a/mm/shrinker.c
> +++ b/mm/shrinker.c
> @@ -159,7 +159,7 @@ static int expand_one_shrinker_info(struct mem_cgroup *memcg, int new_size,
>   static int expand_shrinker_info(int new_id)
>   {
>   	int ret = 0;
> -	int new_nr_max = round_up(new_id + 1, SHRINKER_UNIT_BITS);
> +	int new_nr_max = roundup(new_id + 1, SHRINKER_UNIT_BITS);
>   	int new_size, old_size = 0;
>   	struct mem_cgroup *memcg;
>   
> 

Acked-by: Qi Zheng <qi.zheng@linux.dev>

Thanks,
Qi

Re: [PATCH 1/2] mm/shrinker: size shrinker_info_unit to a power of two

[Konstantin Khorenko]

On 10/13/26 04:52, Qi Zheng wrote:
> On 10/13/26 3:02 AM, Konstantin Khorenko wrote:
>> Every memory cgroup allocates, per NUMA node, one shrinker_info_unit
>> for every SHRINKER_UNIT_BITS shrinker ids up to shrinker_nr_max. The
>> unit holds BITS_PER_LONG atomic_long_t of deferred work, a power of
>> two, plus the one-word bitmap of shrinkers with objects charged to the
>> cgroup. That word takes the unit just past the power of two, to 520
>> bytes on 64-bit, so every unit comes from kmalloc-1k and almost half of
>> it is padding.
>>
>> The units are allocated when the cgroup goes online and freed when it is
>> freed, so a dying cgroup keeps them until the last reference to it is
>> gone. Their number is the number of memory cgroups, times the NUMA
>> nodes, times shrinker_nr_max / SHRINKER_UNIT_BITS. shrinker_nr_max is
>> the peak number of memcg-aware shrinkers registered at once, and every
>> superblock registers one, so a host that mounts many filesystems and
>> runs many cgroups carries the padding as unreclaimable slab, and keeps
>> paying it for new cgroups after the mounts are gone.
>>
>> Let a unit cover one id less than a word holds. The bitmap still fits
>> one word, and nr_deferred[] plus that word add up to BITS_PER_LONG
>> longs, so the unit fills a power-of-two kmalloc slot exactly and, on
>> 64-bit, the memory taken by units halves, for about one unit more in
>> BITS_PER_LONG.
>> round_up() in expand_shrinker_info() needs a power-of-two step and
>> becomes roundup(); the other uses of SHRINKER_UNIT_BITS are divisions,
>> modulos and multiplications that do not care. Assert the unit size at
>> build time, since the layout depends on it.
>>
>> Assisted-by: LLM
>> Signed-off-by: Konstantin Khorenko <khorenko@virtuozzo.com>
>> ---
<skipped>

> 
> Acked-by: Qi Zheng <qi.zheng@linux.dev>

Qi, thank you for the review!

> 
> Thanks,
> Qi
> 
> 
> 

Re: [PATCH 1/2] mm/shrinker: size shrinker_info_unit to a power of two

[Muchun Song]

> On Oct 13, 2026, at 03:02, Konstantin Khorenko <khorenko@virtuozzo.com> wrote:
> 
> Every memory cgroup allocates, per NUMA node, one shrinker_info_unit
> for every SHRINKER_UNIT_BITS shrinker ids up to shrinker_nr_max. The
> unit holds BITS_PER_LONG atomic_long_t of deferred work, a power of
> two, plus the one-word bitmap of shrinkers with objects charged to the
> cgroup. That word takes the unit just past the power of two, to 520
> bytes on 64-bit, so every unit comes from kmalloc-1k and almost half of
> it is padding.
> 
> The units are allocated when the cgroup goes online and freed when it is
> freed, so a dying cgroup keeps them until the last reference to it is
> gone. Their number is the number of memory cgroups, times the NUMA
> nodes, times shrinker_nr_max / SHRINKER_UNIT_BITS. shrinker_nr_max is
> the peak number of memcg-aware shrinkers registered at once, and every
> superblock registers one, so a host that mounts many filesystems and
> runs many cgroups carries the padding as unreclaimable slab, and keeps
> paying it for new cgroups after the mounts are gone.
> 
> Let a unit cover one id less than a word holds. The bitmap still fits
> one word, and nr_deferred[] plus that word add up to BITS_PER_LONG
> longs, so the unit fills a power-of-two kmalloc slot exactly and, on
> 64-bit, the memory taken by units halves, for about one unit more in
> BITS_PER_LONG.
> round_up() in expand_shrinker_info() needs a power-of-two step and
> becomes roundup(); the other uses of SHRINKER_UNIT_BITS are divisions,
> modulos and multiplications that do not care. Assert the unit size at
> build time, since the layout depends on it.
> 
> Assisted-by: LLM
> Signed-off-by: Konstantin Khorenko <khorenko@virtuozzo.com>
> ---
> include/linux/shrinker.h | 9 ++++++++-
> mm/shrinker.c            | 2 +-
> 2 files changed, 9 insertions(+), 2 deletions(-)
> 
> diff --git a/include/linux/shrinker.h b/include/linux/shrinker.h
> index 1a00be90d93a9..c250ab68d346c 100644
> --- a/include/linux/shrinker.h
> +++ b/include/linux/shrinker.h
> @@ -7,7 +7,11 @@
> #include <linux/refcount.h>
> #include <linux/completion.h>
> 
> -#define SHRINKER_UNIT_BITS BITS_PER_LONG
> +/*
> + * One id less than a word holds, so that nr_deferred[] and the map word add
> + * up to BITS_PER_LONG longs: a power of two, which fills a kmalloc slot.
> + */
> +#define SHRINKER_UNIT_BITS (BITS_PER_LONG - 1)
> 
> /*
>  * Bitmap and deferred work of shrinker::id corresponding to memcg-aware
> @@ -18,6 +22,9 @@ struct shrinker_info_unit {
> DECLARE_BITMAP(map, SHRINKER_UNIT_BITS);
> };
> 
> +static_assert(!(sizeof(struct shrinker_info_unit) &
> + 		(sizeof(struct shrinker_info_unit) - 1)));

This assertion is not straightforward. I'd like to use is_power_of_two().
Like:

  BUILD_BUG_ON(!is_power_of_2(sizeof(struct shrinker_info_unit)));

Muchun,
Thanks.

> +
> struct shrinker_info {
> struct rcu_head rcu;
> int map_nr_max;
> diff --git a/mm/shrinker.c b/mm/shrinker.c
> index 7ec2a9704f6f2..a9418072a509e 100644
> --- a/mm/shrinker.c
> +++ b/mm/shrinker.c
> @@ -159,7 +159,7 @@ static int expand_one_shrinker_info(struct mem_cgroup *memcg, int new_size,
> static int expand_shrinker_info(int new_id)
> {
> 	int ret = 0;
> - 	int new_nr_max = round_up(new_id + 1, SHRINKER_UNIT_BITS);
> + 	int new_nr_max = roundup(new_id + 1, SHRINKER_UNIT_BITS);
> 	int new_size, old_size = 0;
> 	struct mem_cgroup *memcg;
> 
> 
> -- 
> 2.43.0
> 

Re: [PATCH 1/2] mm/shrinker: size shrinker_info_unit to a power of two

[Konstantin Khorenko]

On 10/13/26 08:18, Muchun Song wrote:
> 
> 
>> On Oct 13, 2026, at 03:02, Konstantin Khorenko <khorenko@virtuozzo.com> wrote:
>>
<skipped>

>> /*
>>  * Bitmap and deferred work of shrinker::id corresponding to memcg-aware
>> @@ -18,6 +22,9 @@ struct shrinker_info_unit {
>> DECLARE_BITMAP(map, SHRINKER_UNIT_BITS);
>> };
>>
>> +static_assert(!(sizeof(struct shrinker_info_unit) &
>> + 		(sizeof(struct shrinker_info_unit) - 1)));
> 
> This assertion is not straightforward. I'd like to use is_power_of_two().
> Like:
> 
>   BUILD_BUG_ON(!is_power_of_2(sizeof(struct shrinker_info_unit)));

No problem, i'll send a v2 with the modified assertion.
Muchun, thank you for the review!

--
Best regards,

Konstantin Khorenko,
Virtuozzo Linux Kernel Team

[PATCH 2/2] mm/shrinker: allocate the per-memcg shrinker units on demand

[Konstantin Khorenko]

A memory cgroup allocates a shrinker_info_unit for every
SHRINKER_UNIT_BITS ids up to shrinker_nr_max on every NUMA node when it
comes online, whether or not it will ever hold objects of those
shrinkers. Every superblock registers a memcg-aware shrinker, so on a
host with many mounts each new cgroup pays for all of them while its
tasks touch a few.

A map bit is only ever set on a memcg that holds a list_lru list of
that shrinker. So the point where a memcg gets such a list is the one
place a unit can be installed ahead of its first use, and that is where
it is installed.

The rule the code now relies on:

  A non-root memcg that has a list on a list_lru has the unit of that
  list's shrinker, on every node.

It holds because:

  - the unit is installed before the list is published;
  - a unit lives until the memcg is freed.

So set_shrinker_bit() always finds the unit. A reader that finds none
knows the memcg never held objects of that shrinker: there is nothing
to walk and no deferred work.

There is one case where a memcg has no list: with cgroup.memory=nokmem
a list_lru is never per-memcg, but a SHRINKER_MEMCG_AWARE shrinker that
also has SHRINKER_NONSLAB still gets an id. If such a shrinker keeps
lists of its own and sets map bits itself, it has to call
memcg_shrinker_unit_alloc() first. No shrinker in the tree does.

The install runs with the gfp of the list allocation, possibly atomic,
so it cannot take shrinker_mutex. A spinlock orders it against the info
replacement in expand_one_shrinker_info() instead, so a unit cannot be
installed into an info that is being copied out of.

The root memcg keeps all its units: global reclaim keeps the deferred
work of memcg-aware shrinkers there, and root has no list allocation to
install them from.

The units of a memcg now scale with the number of shrinkers it and its
descendants hold objects of, not with shrinker_nr_max.

Assisted-by: LLM
Signed-off-by: Konstantin Khorenko <khorenko@virtuozzo.com>
---
 include/linux/memcontrol.h |   2 +
 mm/list_lru.c              |   6 ++
 mm/shrinker.c              | 159 ++++++++++++++++++++++++++++++++++++---------
 3 files changed, 138 insertions(+), 29 deletions(-)

diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index 7d1c0ce189a88..a14f43c3a1b01 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -1676,6 +1676,8 @@ static inline u64 mem_cgroup_get_socket_pressure(struct mem_cgroup *memcg)
 #endif
 
 int alloc_shrinker_info(struct mem_cgroup *memcg);
+int memcg_shrinker_unit_alloc(struct mem_cgroup *memcg, int shrinker_id,
+			      gfp_t gfp);
 void free_shrinker_info(struct mem_cgroup *memcg);
 void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id);
 void reparent_shrinker_deferred(struct mem_cgroup *memcg);
diff --git a/mm/list_lru.c b/mm/list_lru.c
index 36662d02ff963..7247b7c081b4c 100644
--- a/mm/list_lru.c
+++ b/mm/list_lru.c
@@ -597,6 +597,12 @@ static int __memcg_list_lru_alloc(struct mem_cgroup *memcg,
 			parent = parent_mem_cgroup(pos);
 		}
 
+		if (!css_is_dying(&pos->css) &&
+		    memcg_shrinker_unit_alloc(pos, lru_shrinker_id(lru), gfp)) {
+			kfree(mlru);
+			return -ENOMEM;
+		}
+
 		if (!mlru) {
 			mlru = memcg_init_list_lru_one(lru, gfp);
 			if (!mlru)
diff --git a/mm/shrinker.c b/mm/shrinker.c
index a9418072a509e..79f48fb7fa261 100644
--- a/mm/shrinker.c
+++ b/mm/shrinker.c
@@ -13,6 +13,14 @@ DEFINE_MUTEX(shrinker_mutex);
 #ifdef CONFIG_MEMCG
 static int shrinker_nr_max;
 
+/*
+ * Orders installing a unit against the pointer copy in
+ * expand_one_shrinker_info(). Installs run from the list_lru allocation
+ * path with the caller's gfp and cannot wait for shrinker_mutex; they can
+ * nest in the i_pages lock, which is taken from interrupts.
+ */
+static DEFINE_SPINLOCK(shrinker_unit_lock);
+
 static inline int shrinker_unit_size(int nr_items)
 {
 	return (DIV_ROUND_UP(nr_items, SHRINKER_UNIT_BITS) * sizeof(struct shrinker_info_unit *));
@@ -30,9 +38,6 @@ static inline void shrinker_unit_free(struct shrinker_info *info, int start)
 	nr = DIV_ROUND_UP(info->map_nr_max, SHRINKER_UNIT_BITS);
 
 	for (i = start; i < nr; i++) {
-		if (!unit[i])
-			break;
-
 		kfree(unit[i]);
 		unit[i] = NULL;
 	}
@@ -96,7 +101,13 @@ int alloc_shrinker_info(struct mem_cgroup *memcg)
 		if (!info)
 			goto err;
 		info->map_nr_max = shrinker_nr_max;
-		if (shrinker_unit_alloc(info, NULL, nid)) {
+		/*
+		 * Global reclaim keeps the deferred work of memcg-aware
+		 * shrinkers in the root's units, and root has no list_lru
+		 * allocation to install them from.
+		 */
+		if (mem_cgroup_is_root(memcg) &&
+		    shrinker_unit_alloc(info, NULL, nid)) {
 			kvfree(info);
 			goto err;
 		}
@@ -124,6 +135,7 @@ static int expand_one_shrinker_info(struct mem_cgroup *memcg, int new_size,
 {
 	struct shrinker_info *new, *old;
 	struct mem_cgroup_per_node *pn;
+	unsigned long flags;
 	int nid;
 
 	for_each_node(nid) {
@@ -143,13 +155,20 @@ static int expand_one_shrinker_info(struct mem_cgroup *memcg, int new_size,
 
 		new->map_nr_max = new_nr_max;
 
-		memcpy(new->unit, old->unit, old_size);
-		if (shrinker_unit_alloc(new, old, nid)) {
+		if (mem_cgroup_is_root(memcg) &&
+		    shrinker_unit_alloc(new, old, nid)) {
 			kvfree(new);
 			return -ENOMEM;
 		}
 
+		/*
+		 * A unit installed meanwhile lands in old before the copy or
+		 * in new after the publish.
+		 */
+		spin_lock_irqsave(&shrinker_unit_lock, flags);
+		memcpy(new->unit, old->unit, old_size);
 		rcu_assign_pointer(pn->shrinker_info, new);
+		spin_unlock_irqrestore(&shrinker_unit_lock, flags);
 		kvfree_rcu(old, rcu);
 	}
 
@@ -204,21 +223,26 @@ static inline int calc_shrinker_id(int index, int offset)
 
 void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id)
 {
-	if (shrinker_id >= 0 && memcg && !mem_cgroup_is_root(memcg)) {
-		struct shrinker_info *info;
+	struct shrinker_info_unit *unit;
+	struct shrinker_info *info;
 
-		rcu_read_lock();
-		info = rcu_dereference(memcg->nodeinfo[nid]->shrinker_info);
-		if (!WARN_ON_ONCE(shrinker_id >= info->map_nr_max)) {
-			struct shrinker_info_unit *unit;
-
-			unit = info->unit[shrinker_id_to_index(shrinker_id)];
-			/* Pairs with smp mb in shrink_slab() */
-			smp_mb__before_atomic();
-			set_bit(shrinker_id_to_offset(shrinker_id), unit->map);
-		}
-		rcu_read_unlock();
-	}
+	if (shrinker_id < 0 || !memcg || mem_cgroup_is_root(memcg))
+		return;
+
+	rcu_read_lock();
+	info = rcu_dereference(memcg->nodeinfo[nid]->shrinker_info);
+	if (WARN_ON_ONCE(shrinker_id >= info->map_nr_max))
+		goto unlock;
+
+	unit = READ_ONCE(info->unit[shrinker_id_to_index(shrinker_id)]);
+	if (WARN_ON_ONCE(!unit))
+		goto unlock;
+
+	/* Pairs with smp mb in shrink_slab() */
+	smp_mb__before_atomic();
+	set_bit(shrinker_id_to_offset(shrinker_id), unit->map);
+unlock:
+	rcu_read_unlock();
 }
 
 static DEFINE_IDR(shrinker_idr);
@@ -263,14 +287,16 @@ static void shrinker_memcg_remove(struct shrinker *shrinker)
 static long xchg_nr_deferred_memcg(int nid, struct shrinker *shrinker,
 				   struct mem_cgroup *memcg)
 {
+	int offset = shrinker_id_to_offset(shrinker->id);
 	struct shrinker_info *info;
 	struct shrinker_info_unit *unit;
 	long nr_deferred;
 
 	rcu_read_lock();
 	info = rcu_dereference(memcg->nodeinfo[nid]->shrinker_info);
-	unit = info->unit[shrinker_id_to_index(shrinker->id)];
-	nr_deferred = atomic_long_xchg(&unit->nr_deferred[shrinker_id_to_offset(shrinker->id)], 0);
+	unit = READ_ONCE(info->unit[shrinker_id_to_index(shrinker->id)]);
+	/* No unit: this memcg never held objects of this shrinker. */
+	nr_deferred = unit ? atomic_long_xchg(&unit->nr_deferred[offset], 0) : 0;
 	rcu_read_unlock();
 
 	return nr_deferred;
@@ -279,20 +305,83 @@ static long xchg_nr_deferred_memcg(int nid, struct shrinker *shrinker,
 static long add_nr_deferred_memcg(long nr, int nid, struct shrinker *shrinker,
 				  struct mem_cgroup *memcg)
 {
+	int offset = shrinker_id_to_offset(shrinker->id);
 	struct shrinker_info *info;
 	struct shrinker_info_unit *unit;
 	long nr_deferred;
 
 	rcu_read_lock();
 	info = rcu_dereference(memcg->nodeinfo[nid]->shrinker_info);
-	unit = info->unit[shrinker_id_to_index(shrinker->id)];
-	nr_deferred =
-		atomic_long_add_return(nr, &unit->nr_deferred[shrinker_id_to_offset(shrinker->id)]);
+	unit = READ_ONCE(info->unit[shrinker_id_to_index(shrinker->id)]);
+	nr_deferred = unit ? atomic_long_add_return(nr, &unit->nr_deferred[offset]) : 0;
 	rcu_read_unlock();
 
 	return nr_deferred;
 }
 
+static int shrinker_unit_install(struct mem_cgroup *memcg, int nid, int index,
+				 gfp_t gfp)
+{
+	struct shrinker_info_unit *unit;
+	struct shrinker_info *info;
+	unsigned long flags;
+
+	rcu_read_lock();
+	info = rcu_dereference(memcg->nodeinfo[nid]->shrinker_info);
+	/* Online memcgs have an info, and ids are expanded into it first. */
+	if (WARN_ON_ONCE(!info ||
+			 index >= shrinker_id_to_index(info->map_nr_max)) ||
+	    READ_ONCE(info->unit[index])) {
+		rcu_read_unlock();
+		return 0;
+	}
+	rcu_read_unlock();
+
+	unit = kzalloc_node(sizeof(*unit), gfp, nid);
+	if (!unit)
+		return -ENOMEM;
+
+	spin_lock_irqsave(&shrinker_unit_lock, flags);
+	/* memcg is pinned: only expand replaces info, and it holds the lock */
+	info = rcu_dereference_protected(memcg->nodeinfo[nid]->shrinker_info,
+					 lockdep_is_held(&shrinker_unit_lock));
+	if (!info->unit[index]) {
+		/* Walkers read the pointer locklessly: publish a zeroed unit. */
+		smp_store_release(&info->unit[index], unit);
+		unit = NULL;
+	}
+	spin_unlock_irqrestore(&shrinker_unit_lock, flags);
+
+	kfree(unit);
+	return 0;
+}
+
+/*
+ * Called before @memcg gets a list on a list_lru of shrinker @shrinker_id,
+ * from the path that allocates the list. A map bit is only ever set on a
+ * memcg that has the list, so set_shrinker_bit() always finds the unit.
+ * A shrinker that sets map bits without a list_lru, such as a NONSLAB
+ * one under cgroup.memory=nokmem, must call this first.
+ */
+int memcg_shrinker_unit_alloc(struct mem_cgroup *memcg, int shrinker_id,
+			      gfp_t gfp)
+{
+	int nid, ret;
+
+	if (shrinker_id < 0 || mem_cgroup_is_root(memcg))
+		return 0;
+
+	for_each_node(nid) {
+		ret = shrinker_unit_install(memcg, nid,
+					    shrinker_id_to_index(shrinker_id),
+					    gfp);
+		if (ret)
+			return ret;
+	}
+
+	return 0;
+}
+
 void reparent_shrinker_deferred(struct mem_cgroup *memcg)
 {
 	int nid, index, offset;
@@ -307,8 +396,17 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg)
 		child_info = shrinker_info_protected(memcg, nid);
 		parent_info = shrinker_info_protected(parent, nid);
 		for (index = 0; index < shrinker_id_to_index(child_info->map_nr_max); index++) {
-			child_unit = child_info->unit[index];
-			parent_unit = parent_info->unit[index];
+			child_unit = READ_ONCE(child_info->unit[index]);
+			if (!child_unit)
+				continue;
+
+			if (!READ_ONCE(parent_info->unit[index]))
+				shrinker_unit_install(parent, nid, index, GFP_KERNEL);
+			parent_unit = READ_ONCE(parent_info->unit[index]);
+			/* Drop the debt rather than fail the cgroup teardown. */
+			if (!parent_unit)
+				continue;
+
 			for (offset = 0; offset < SHRINKER_UNIT_BITS; offset++) {
 				nr = atomic_long_read(&child_unit->nr_deferred[offset]);
 				atomic_long_add(nr, &parent_unit->nr_deferred[offset]);
@@ -529,10 +627,13 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid,
 	if (index < shrinker_id_to_index(info->map_nr_max)) {
 		struct shrinker_info_unit *unit;
 
-		unit = info->unit[index];
+		unit = READ_ONCE(info->unit[index]);
 
 		rcu_read_unlock();
 
+		if (!unit)
+			goto next;
+
 		for_each_set_bit(offset, unit->map, SHRINKER_UNIT_BITS) {
 			struct shrink_control sc = {
 				.gfp_mask = gfp_mask,
@@ -587,7 +688,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid,
 			freed += ret;
 			shrinker_put(shrinker);
 		}
-
+next:
 		index++;
 		goto again;
 	}

-- 
2.43.0