[PATCH v2 0/6] memcg: Batch mod_memcg_state() for MEMCG_SOCK.

[Kuniyuki Iwashima <kuniyu@google.com>]

On our production fleet, we saw mem_cgroup_sk_charge() spent over
3x the cycles of try_charge_memcg() although the former should be
a thin wrapper of the latter.

Looking at perf during tcp_rr, most of the cycles were spent in
mod_memcg_state(MEMCG_SOCK), which incurred two heavy memory stalls
due to cache line misses. (see patch 4)

Also, tcp_rr w/ 20000 flows throughput often degrades by ~50% on
many core machine. (see patch 5)

This series adds nr_sock[] to struct memcg_stock_pcp to batch
mod_memcg_state() for MEMCG_SOCK and move heavy operations to
out of local_lock(&memcg_stock.lock).

With this series, tcp_rr w/ 10000 flows achieves 31% more
throughput with lower latency on machine w/ AMD EPYC 7B12 64-Core
Processor (Zen 2 Rome, 2 sockets, 256 vCPU)

  tcp_rr (10000 flows, 128 threads, 60s, average of 3 runs):

  Metric                     |       Baseline |        Patched |    Diff
  ---------------------------+----------------+----------------+--------
  Throughput (RPS)           |     20,884,499 |     27,466,490 | +31.52%
  Latency Mean (us)          |         478.46 |         363.71 | -23.98%
  Latency p50 (us)           |         453.96 |         356.68 | -21.43%
  Latency p90 (us)           |         669.00 |         479.56 | -28.32%
  Latency p99 (us)           |         969.38 |         617.80 | -36.27%


Changes:
  v2:
    * Rebased on mm.git/mm-new
    * Add patch 5 for flaky throughput issue
    * Patch 4:
      * Benchmark again after rebase and update numbers
      * Check in_nmi() in __refill_stock()

  v1: https://lore.kernel.org/linux-mm/20261016052811.3085635-1-kuniyu@google.com/


Kuniyuki Iwashima (6):
  memcg: Remove unnecessary READ_ONCE() under memcg_stock.lock.
  memcg: Reduce nesting in consume_stock() and refill_stock().
  memcg: Annotate slow paths in consume_stock() and refill_stock().
  memcg: Batch mod_memcg_state() for MEMCG_SOCK.
  memcg: Uncharge pages outside memcg_stock.lock in __refill_stock().
  selftests: cgroup: Deflake test_memcg_sock() in test_memcontrol.c.

 mm/memcontrol.c                               | 178 +++++++++++++-----
 .../selftests/cgroup/test_memcontrol.c        |   8 +
 2 files changed, 139 insertions(+), 47 deletions(-)

-- 
2.56.0.313.g2479322e68-goog