Re: [PATCH] memcg: multi-memcg percpu charge cache

Vlastimil Babka
2025-04-30T09:57:17Z

On 4/16/25 20:02, Shakeel Butt wrote:
> Memory cgroup accounting is expensive and to reduce the cost, the kernel
> maintains per-cpu charge cache for a single memcg. So, if a charge
> request comes for a different memcg, the kernel will flush the old
> memcg's charge cache and then charge the newer memcg a fixed amount (64
> pages), subtracts the charge request amount and stores the remaining in
> the per-cpu charge cache for the newer memcg.
>
> This mechanism is based on the assumption that the kernel, for locality,
> keep a process on a CPU for long period of time and most of the charge
> requests from that process will be served by that CPU's local charge
> cache.
>
> However this assumption breaks down for incoming network traffic in a
> multi-tenant machine. We are in the process of running multiple
> workloads on a single machine and if such workloads are network heavy,
> we are seeing very high network memory accounting cost. We have observed
> multiple CPUs spending almost 100% of their time in net_rx_action and
> almost all of that time is spent in memcg accounting of the network
> traffic.
>
> More precisely, net_rx_action is serving packets from multiple workloads
> and is observing/serving mix of packets of these workloads. The memcg
> switch of per-cpu cache is very expensive and we are observing a lot of
> memcg switches on the machine. Almost all the time is being spent on
> charging new memcg and flushing older memcg cache. So, definitely we
> need per-cpu cache that support multiple memcgs for this scenario.
>
> This patch implements a simple (and dumb) multiple memcg percpu charge
> cache. Actually we started with more sophisticated LRU based approach but
> the dumb one was always better than the sophisticated one by 1% to 3%,
> so going with the simple approach.
>
> Some of the design choices are:
>
> 1. Fit all caches memcgs in a single cacheline.
> 2. The cache array can be mix of empty slots or memcg charged slots, so
> the kernel has to traverse the full array.
> 3. The cache drain from the reclaim will drain all cached memcgs to keep
> things simple.
>
> To evaluate the impact of this optimization, on a 72 CPUs machine, we
> ran the following workload where each netperf client runs in a different
> cgroup. The next-20250415 kernel is used as base.
>
> $ netserver -6
> $ netperf -6 -H ::1 -l 60 -t TCP_SENDFILE -- -m 10K
>
> number of clients | Without patch | With patch
> 6 | 42584.1 Mbps | 48603.4 Mbps (14.13% improvement)
> 12 | 30617.1 Mbps | 47919.7 Mbps (56.51% improvement)
> 18 | 25305.2 Mbps | 45497.3 Mbps (79.79% improvement)
> 24 | 20104.1 Mbps | 37907.7 Mbps (88.55% improvement)
> 30 | 14702.4 Mbps | 30746.5 Mbps (109.12% improvement)
> 36 | 10801.5 Mbps | 26476.3 Mbps (145.11% improvement)
>
> The results show drastic improvement for network intensive workloads.
>
> Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>

Acked-by: Vlastimil Babka <vbabka@suse.cz>

See below

> ---
> mm/memcontrol.c | 128 ++++++++++++++++++++++++++++++++++--------------
> 1 file changed, 91 insertions(+), 37 deletions(-)
>
> diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> index 1ad326e871c1..0a02ba07561e 100644
> --- a/mm/memcontrol.c
> +++ b/mm/memcontrol.c
> @@ -1769,10 +1769,11 @@ void mem_cgroup_print_oom_group(struct mem_cgroup *memcg)
> pr_cont(" are going to be killed due to memory.oom.group set\n");
> }
>
> +#define NR_MEMCG_STOCK 7
> struct memcg_stock_pcp {
> local_trylock_t stock_lock;
> - struct mem_cgroup *cached; /* this never be root cgroup */
> - unsigned int nr_pages;
> + uint8_t nr_pages[NR_MEMCG_STOCK];
> + struct mem_cgroup *cached[NR_MEMCG_STOCK];

I have noticed memcg_stock is a DEFINE_PER_CPU and not
DEFINE_PER_CPU_ALIGNED so I think that the intended cacheline usage isn't
guaranteed now.

Actually tried compiling and got in objdump -t vmlinux:

ffffffff83a26e60 l O .data..percpu 0000000000000088 memcg_stock

AFAICS that's aligned to 32 bytes only (0x60 is 96) bytes, not 64.

changing to _ALIGNED gives me:

ffffffff83a2c5c0 l O .data..percpu 0000000000000088 memcg_stock

0xc0 is 192 so multiple of 64, so seems to work as intended and indeed
necessary. So you should change it too while adding the comment.

> struct obj_cgroup *cached_objcg;
> struct pglist_data *cached_pgdat;