Re: [PATCH v3 4/4] memcg: Uncharge pages outside memcg_stock.lock in refill_stock().

[Shakeel Butt <shakeel.butt@linux.dev>]

On Thu, Oct 29, 2026 at 07:44:15PM +0000, Kuniyuki Iwashima wrote:
> On host w/ AMD EPYC 9B45 128-Core Processor (Zen 5 Turin,
> 2 sockets, 512 vCPUs), tcp_rr w/ 20000 flows throughput often
> degrades by ~50%, 32M -> 17M~27M.
> 
> In the bad case, the number of involuntary context switches
> was extremely high by 12x (241733 -> 3093367).
> 
> In perf record, 96% samples had the same pattern, where the
> thread was interrupted during page_counter_uncharge() by softirq,
> and later ksoftirqd was woken up:
> 
>   # perf record -C 0-15 -e sched:sched_wakeup -g \
>       -o /tmp/sched_wakeup.data -- sleep 1
>   # perf report -i /tmp/sched_wakeup.data --stdio --no-children
>     96.36%  tcp_rr   [kernel.kallsyms]  [k] wake_up_process
>             |
>             ---wake_up_process
>                |
>                 --96.00%--irq_exit_rcu
>                           |
>                            --95.82%--sysvec_call_function_single
>                                      asm_sysvec_call_function_single
>                                      |
>                                       --95.58%--page_counter_uncharge
>                                                 mem_cgroup_sk_uncharge
>                                                 sock_rfree
>                                                 tcp_recvmsg
>                                                 __x64_sys_recvfrom
>                                                 do_syscall_64
>                                                 entry_SYSCALL_64_after_hwframe
> 
>   # perf script -i /tmp/sched_wakeup.data | grep -A 20 "comm=ksoftirqd"
>   tcp_rr  101952 [001]  9679.340985: sched:sched_wakeup: comm=ksoftirqd/1 pid=23 prio=120 target_cpu=001
>   	ffffffffb64cd0cc wake_up_process+0xe6c ([kernel.kallsyms])
>   	ffffffffb64cb034 irq_exit_rcu+0x264 ([kernel.kallsyms])
>   	ffffffffb68ad31c sysvec_call_function_single+0xcc ([kernel.kallsyms])
>   	ffffffffb560158a asm_sysvec_call_function_single+0x1a ([kernel.kallsyms])
>   	ffffffffb653c4ce page_counter_uncharge+0x2e ([kernel.kallsyms])
>   	ffffffffb6541b28 mem_cgroup_sk_uncharge+0x3c8 ([kernel.kallsyms])
>   	ffffffffb6561383 sock_rfree+0x83 ([kernel.kallsyms])
>   	ffffffffb6577faf tcp_recvmsg+0x2ff ([kernel.kallsyms])
>   	ffffffffb65604d9 __x64_sys_recvfrom+0x159 ([kernel.kallsyms])
> 
> The problem is that when stock_pages exceeds MEMCG_STOCK_HIGH
> or all slots in struct memcg_stock_pcp are full, refill_stock()
> calls memcg_uncharge() or drain_stock() while holding
> memcg_stock.lock.
> 
> Because local_trylock() only disables preemption and leaves
> interrupts and softirqs enabled, an RPS IPI arriving while
> refill_stock() stalls on page_counter_uncharge() invokes
> NET_RX_SOFTIRQ (process_backlog()) on the same CPU while
> memcg_stock.lock is held.
> 
> Every incoming packet in process_backlog() then fails
> local_trylock() in consume_stock() and falls back to
> try_charge_memcg() and refill_stock(), hammering the global
> page_counter atomic twice per packet, and slow processing
> finally wakes up ksoftirqd.
> 
> Let's move memcg_uncharge() and css_put() outside
> local_unlock(&memcg_stock.lock) in refill_stock().
> 
> With the change, the flakiness disappeared on the 512 vCPU platform.
> 
> In addition, on machine w/ AMD EPYC 7B12 64-Core Processor (Zen 2
> Rome, 2 sockets, 256 vCPU), tcp_rr w/ 10000 flows achieves 36% more
> throughput with lower latency.
> 
>   tcp_rr (10000 flows, 128 threads, 60s, average of 3 runs):
> 
>   Metric                     |       Baseline |    Patched |    Diff
>   ---------------------------+----------------+------------+--------
>   Throughput (RPS)           |     20,884,499 | 28,453,051 | +36.24%
>   Latency Mean (us)          |         478.46 |     351.07 | -26.63%
>   Latency p50 (us)           |         453.96 |     343.04 | -24.43%
>   Latency p90 (us)           |         669.00 |     479.57 | -28.32%
>   Latency p99 (us)           |         969.38 |     593.92 | -38.73%
> 
> Signed-off-by: Kuniyuki Iwashima <kuniyu@google.com>

Acked-by: Shakeel Butt <shakeel.butt@linux.dev>