* [PATCH 1/3] selftests/bpf: add memcg_stat_reader BPF-vs-memory.stat benchmark
2026-07-04 4:56 [PATCH 0/3] selftests/bpf: compare BPF and memory.stat memcg stat readers Ziyang Men
@ 2026-07-04 4:56 ` Ziyang Men
2026-07-04 4:56 ` [PATCH 2/3] selftests/bpf: add memcg_stat_churn BPF-vs-memory.stat benchmark under churn Ziyang Men
` (2 subsequent siblings)
3 siblings, 0 replies; 9+ messages in thread
From: Ziyang Men @ 2026-07-04 4:56 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
Eduard Zingerman, Kumar Kartikeya Dwivedi, bpf
Cc: Martin KaFai Lau, Song Liu, Yonghong Song, Jiri Olsa,
Emil Tsalapatis, Shuah Khan, Roman Gushchin, kernel-team,
linux-mm, cgroups, linux-kselftest, linux-kernel, Ziyang Men,
Shakeel Butt
Add a test_progs selftest that reads memory-cgroup statistics for a whole
synthetic cgroup subtree two ways and compares their cost:
(A) the traditional path: open/read/parse memory.stat, memory.current
and memory.max for every cgroup from userspace;
(B) the BPF path: a single SEC("iter.s/cgroup") program walked over the
subtree in BPF_CGROUP_ITER_DESCENDANTS_PRE order that calls the
memcg kfuncs (bpf_get_mem_cgroup, bpf_mem_cgroup_flush_stats,
bpf_mem_cgroup_page_state, bpf_mem_cgroup_vm_events,
bpf_put_mem_cgroup) per cgroup and stashes the results in a hash map
keyed by cgroup id, drained once afterwards.
The traditional path reads the control files through a new read_cgroup_file()
helper added to cgroup_helpers (the read counterpart of write_cgroup_file),
instead of open-coding the cgroupfs path layout in the test.
The test builds a parameterized tree (fanout x depth) and charges
anonymous memory into the leaves from a helper child so the per-cgroup and
hierarchical stats are non-zero, then:
- asserts the BPF snapshot agrees with memory.stat for the anon counter
(rstat-flushed and deterministic) within a small tolerance, and that
the iterator visited every cgroup in the subtree;
- reports memory.current drift as informational only: it is a live
page_counter that both sides read identically, so a difference between
the one-shot BPF walk and the longer file-reading loop is time skew,
not a correctness issue;
- reports the average wall-clock cost of each path reading the full
~memory.stat field set.
Subtests small/medium/large vary the tree size; large_sparse charges only
a fraction of the leaves to exercise rstat's "flush is O(updated subtree)"
behaviour. The pass/fail result depends only on the correctness checks; the
timing table is an informational diagnostic captured like any other test
output, i.e. printed only under -v (or when the test fails), never on a
normal PASS.
The BPF field fold guards each counter with bpf_core_enum_value_exists()
so an enumerator missing from the running kernel's BTF is skipped rather than
poisoning the whole program load, keeping the test loadable across kernel
configs and versions. When the memcg kfuncs are unavailable (CONFIG_MEMCG=n)
the test skips cleanly instead of failing to load; the base selftest config
now selects CONFIG_MEMCG=y as well.
The file path cost is dominated by per-cgroup VFS open/read and string
parsing, while the BPF path avoids per-cgroup syscalls and string parsing, so
it is ~32-37x faster for a whole-tree scan. The file path cost tracks the
number of cgroups traversed rather than the amount of memory charged: large
and large_sparse read the same 1111 files in about the same time. The BPF path
is cheaper still when fewer leaves are charged (large_sparse), because the
rstat flush is O(updated subtree).
Sample output (v7.1 VM); times are us, average per full-tree pass reading the
full memory.stat field set; ro = bpf read()-only (no map drain):
==== memcg_stat_reader: small ====
tree: nodes=21 leaves=16 charged_leaves=16 fanout=4 depth=2 charge=256KB/leaf iters=200
file_avg=2777.9 bpf_avg=84.7 bpf_ro=30.4 speedup(file/bpf)=32.82x
==== memcg_stat_reader: medium ====
tree: nodes=111 leaves=100 charged_leaves=100 fanout=10 depth=2 charge=256KB/leaf iters=50
file_avg=14633.3 bpf_avg=428.5 bpf_ro=144.5 speedup(file/bpf)=34.15x
==== memcg_stat_reader: large ====
tree: nodes=1111 leaves=1000 charged_leaves=1000 fanout=10 depth=3 charge=256KB/leaf iters=10
file_avg=156774.0 bpf_avg=4245.0 bpf_ro=1416.1 speedup(file/bpf)=36.93x
==== memcg_stat_reader: large_sparse ====
tree: nodes=1111 leaves=1000 charged_leaves=125 fanout=10 depth=3 charge=256KB/leaf iters=10
file_avg=158895.6 bpf_avg=4307.8 bpf_ro=1452.4 speedup(file/bpf)=36.89x
This builds on the memcg BPF kfuncs and complements the existing
cgroup_iter_memcg selftest.
Suggested-by: Shakeel Butt <shakeel.butt@linux.dev>
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ziyang Men <ziyang.meme@gmail.com>
---
tools/testing/selftests/bpf/cgroup_helpers.c | 46 ++
tools/testing/selftests/bpf/cgroup_helpers.h | 2 +
tools/testing/selftests/bpf/config | 1 +
.../testing/selftests/bpf/memcg_stat_reader.h | 35 +
.../bpf/prog_tests/memcg_stat_reader.c | 617 ++++++++++++++++++
.../selftests/bpf/progs/memcg_stat_reader.c | 181 +++++
6 files changed, 882 insertions(+)
create mode 100644 tools/testing/selftests/bpf/memcg_stat_reader.h
create mode 100644 tools/testing/selftests/bpf/prog_tests/memcg_stat_reader.c
create mode 100644 tools/testing/selftests/bpf/progs/memcg_stat_reader.c
diff --git a/tools/testing/selftests/bpf/cgroup_helpers.c b/tools/testing/selftests/bpf/cgroup_helpers.c
index 45cd0b479fe3..fe8ec07c6100 100644
--- a/tools/testing/selftests/bpf/cgroup_helpers.c
+++ b/tools/testing/selftests/bpf/cgroup_helpers.c
@@ -188,6 +188,52 @@ int write_cgroup_file_parent(const char *relative_path, const char *file,
return __write_cgroup_file(cgroup_path, file, buf);
}
+static int __read_cgroup_file(const char *cgroup_path, const char *file,
+ char *buf, size_t buf_size)
+{
+ char file_path[PATH_MAX + 1];
+ ssize_t len;
+ int fd;
+
+ snprintf(file_path, sizeof(file_path), "%s/%s", cgroup_path, file);
+ fd = open(file_path, O_RDONLY);
+ if (fd < 0) {
+ log_err("Opening %s", file_path);
+ return 1;
+ }
+
+ len = read(fd, buf, buf_size - 1);
+ close(fd);
+ if (len < 0) {
+ log_err("Reading %s", file_path);
+ return 1;
+ }
+ buf[len] = '\0';
+ return 0;
+}
+
+/**
+ * read_cgroup_file() - Read from a cgroup file
+ * @relative_path: The cgroup path, relative to the workdir
+ * @file: The name of the file in cgroupfs to read from
+ * @buf: Buffer to read into; NUL-terminated on success
+ * @buf_size: Size of @buf; at most @buf_size - 1 bytes are read
+ *
+ * Read from a file in the given cgroup's directory. As with reading any
+ * cgroupfs control/stat file, @buf should be large enough to hold the whole
+ * value in a single read().
+ *
+ * If successful, 0 is returned.
+ */
+int read_cgroup_file(const char *relative_path, const char *file,
+ char *buf, size_t buf_size)
+{
+ char cgroup_path[PATH_MAX - 24];
+
+ format_cgroup_path(cgroup_path, relative_path);
+ return __read_cgroup_file(cgroup_path, file, buf, buf_size);
+}
+
/**
* setup_cgroup_environment() - Setup the cgroup environment
*
diff --git a/tools/testing/selftests/bpf/cgroup_helpers.h b/tools/testing/selftests/bpf/cgroup_helpers.h
index 3857304be874..1ed76dd3a1da 100644
--- a/tools/testing/selftests/bpf/cgroup_helpers.h
+++ b/tools/testing/selftests/bpf/cgroup_helpers.h
@@ -15,6 +15,8 @@ int write_cgroup_file(const char *relative_path, const char *file,
const char *buf);
int write_cgroup_file_parent(const char *relative_path, const char *file,
const char *buf);
+int read_cgroup_file(const char *relative_path, const char *file,
+ char *buf, size_t buf_size);
int cgroup_setup_and_join(const char *relative_path);
int get_root_cgroup(void);
int create_and_get_cgroup(const char *relative_path);
diff --git a/tools/testing/selftests/bpf/config b/tools/testing/selftests/bpf/config
index adb25146e88c..4e75b4ea8649 100644
--- a/tools/testing/selftests/bpf/config
+++ b/tools/testing/selftests/bpf/config
@@ -55,6 +55,7 @@ CONFIG_LIRC=y
CONFIG_LIVEPATCH=y
CONFIG_LWTUNNEL=y
CONFIG_LWTUNNEL_BPF=y
+CONFIG_MEMCG=y
CONFIG_MODULE_SIG=y
CONFIG_MODULE_SRCVERSION_ALL=y
CONFIG_MODULE_UNLOAD=y
diff --git a/tools/testing/selftests/bpf/memcg_stat_reader.h b/tools/testing/selftests/bpf/memcg_stat_reader.h
new file mode 100644
index 000000000000..72afebe95ccb
--- /dev/null
+++ b/tools/testing/selftests/bpf/memcg_stat_reader.h
@@ -0,0 +1,35 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */
+#ifndef __MEMCG_STAT_READER_H
+#define __MEMCG_STAT_READER_H
+
+/*
+ * One per-cgroup snapshot, produced by the BPF cgroup iterator and read back
+ * from a BPF hash map keyed by cgroup id. The "matched" subset is always
+ * populated so it can be compared field-by-field against what userspace parses
+ * out of memory.stat / memory.current / memory.max. The "full" fold is only
+ * populated when collect_full is set and exists to (a) force the extra kfunc
+ * reads to happen (so the full-vs-matched timing is honest) and (b) give a
+ * coarse, informational signal of how many fields the full path touched.
+ */
+struct memcg_stat_snapshot {
+ __u64 cgroup_id;
+
+ /* Matched subset. Page-state values are in bytes (already unit-scaled
+ * by the kernel), so they compare directly against memory.stat.
+ */
+ __u64 anon; /* NR_ANON_MAPPED, bytes */
+ __u64 file; /* NR_FILE_PAGES, bytes */
+ __u64 shmem; /* NR_SHMEM, bytes */
+ __u64 file_mapped; /* NR_FILE_MAPPED, bytes */
+ __u64 pgfault; /* PGFAULT, count */
+ __u64 usage_pages; /* page_counter memory.usage, in PAGES */
+ __u64 max_pages; /* page_counter memory.max, in PAGES */
+
+ /* Full-mode fold: sum and count of every field the full path read. */
+ __u64 full_sum;
+ __u32 full_fields;
+ __u32 pad;
+};
+
+#endif /* __MEMCG_STAT_READER_H */
diff --git a/tools/testing/selftests/bpf/prog_tests/memcg_stat_reader.c b/tools/testing/selftests/bpf/prog_tests/memcg_stat_reader.c
new file mode 100644
index 000000000000..b1e631b1520a
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/memcg_stat_reader.c
@@ -0,0 +1,617 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */
+
+/*
+ * memcg_stat_reader
+ * =================
+ * Read memory-cgroup statistics for a whole synthetic cgroup subtree TWO ways
+ * and compare them:
+ *
+ * (A) traditional: open+read+parse memory.stat / memory.current / memory.max
+ * for every cgroup, in userspace;
+ * (B) BPF: a single SEC("iter.s/cgroup") program walked over the subtree in
+ * DESCENDANTS_PRE order, calling the memcg kfuncs per cgroup and stashing
+ * the results in a hash map keyed by cgroup id, drained once afterwards.
+ *
+ * The test (a) asserts the BPF path agrees with the file path for a checked
+ * field subset (correctness) and (b) reports the wall-clock cost of each path
+ * reading the full ~memory.stat field set, across cgroup trees of increasing
+ * size and load.
+ *
+ * The pass/fail result depends only on the correctness checks; the timing table
+ * is an informational diagnostic captured like any other test output, i.e. shown
+ * only under -v (or when the test fails), never on a normal PASS.
+ */
+#include <test_progs.h>
+#include <bpf/libbpf.h>
+#include <bpf/btf.h>
+#include <stdlib.h>
+#include <string.h>
+#include <time.h>
+#include <unistd.h>
+#include <sys/mman.h>
+#include <sys/wait.h>
+#include "cgroup_helpers.h"
+#include "memcg_stat_reader.h"
+#include "memcg_stat_reader.skel.h"
+
+#define SUBTREE_ROOT "/mcg_stat"
+
+#define WARMUP_ITERS 3
+
+struct cg_node {
+ char rel[128];
+ __u64 id;
+ bool is_leaf;
+};
+
+/* Field subset the BPF prog reads matched against memory.stat by hand. */
+struct file_snap {
+ __u64 anon, file, shmem, file_mapped, pgfault;
+ __u64 current; /* memory.current, bytes */
+ __u64 max; /* memory.max, bytes (valid unless max_is_max) */
+ __u64 full_sum;
+ __u32 full_fields;
+ bool max_is_max;
+};
+
+struct timing {
+ double avg_us; /* average per full-tree pass */
+ double ro_avg_us; /* BPF read()-only average (no map drain); 0 for file */
+ int nodes_seen; /* entries produced (BPF) */
+ __u32 fields; /* fields/cgroup touched (informational) */
+};
+
+static volatile __u64 sink; /* keep the optimizer from eliding reads */
+static long page_size;
+
+static long long now_ns(void)
+{
+ struct timespec t;
+
+ clock_gettime(CLOCK_MONOTONIC, &t);
+ return (long long)t.tv_sec * 1000000000LL + t.tv_nsec;
+}
+
+/* ---- tree construction ------------------------------------------------- */
+
+static struct cg_node *nodes;
+static int n_nodes;
+static int n_leaves;
+
+static int add_node(const char *rel, bool is_leaf, int *keep_fd)
+{
+ int fd;
+
+ fd = create_and_get_cgroup(rel);
+ if (fd < 0)
+ return -1;
+ if (keep_fd)
+ *keep_fd = fd;
+ else
+ close(fd);
+
+ strncpy(nodes[n_nodes].rel, rel, sizeof(nodes[n_nodes].rel) - 1);
+ nodes[n_nodes].rel[sizeof(nodes[n_nodes].rel) - 1] = '\0';
+ nodes[n_nodes].id = get_cgroup_id(rel);
+ nodes[n_nodes].is_leaf = is_leaf;
+ if (is_leaf)
+ n_leaves++;
+ n_nodes++;
+ return 0;
+}
+
+/* Recursively create children of @rel. @rel must already exist and be recorded. */
+static int build_children(const char *rel, int fanout, int depth)
+{
+ /* size 128 should be enough for file path with max depth 3 is the test*/
+ char child[128];
+ int i;
+
+ if (depth == 0)
+ return 0;
+
+ /* Enable memory on this interior node so its children get memory. */
+ if (enable_controllers(rel, "memory"))
+ return -1;
+
+ for (i = 0; i < fanout; i++) {
+ snprintf(child, sizeof(child), "%s/c%d", rel, i);
+ if (add_node(child, depth == 1, NULL))
+ return -1;
+ if (build_children(child, fanout, depth - 1))
+ return -1;
+ }
+ return 0;
+}
+
+static size_t tree_capacity(int fanout, int depth)
+{
+ size_t total = 1, level = 1;
+ int d;
+
+ for (d = 0; d < depth; d++) {
+ level *= fanout;
+ total += level;
+ }
+ return total;
+}
+
+static int build_tree(int fanout, int depth, int *root_fd)
+{
+ n_nodes = 0;
+ n_leaves = 0;
+ nodes = calloc(tree_capacity(fanout, depth), sizeof(*nodes));
+ if (!nodes)
+ return -1;
+
+ /* special handle for the leaf (0 depth) */
+ if (add_node(SUBTREE_ROOT, depth == 0, root_fd))
+ return -1;
+ return build_children(SUBTREE_ROOT, fanout, depth);
+}
+
+/* ---- charging ---------------------------------------------------------- */
+
+/*
+ * A forked child walks the leaves, joining each and faulting in a private anon
+ * region so the charge lands on that leaf, then keeps every region mapped and
+ * blocks. Interior nodes accumulate the charge hierarchically. The child is
+ * left stopped (blocked on the control pipe) so the stats are static while the
+ * parent measures.
+ */
+static pid_t charger_pid = -1;
+static int charger_ctrl[2] = { -1, -1 };
+
+static int start_charger(size_t charge_bytes, int charge_fraction)
+{
+ int ready[2];
+ pid_t pid;
+ int i, mod;
+ char c;
+
+ if (!ASSERT_OK(pipe(ready), "pipe ready"))
+ return -1;
+ if (!ASSERT_OK(pipe(charger_ctrl), "pipe ctrl")) {
+ close(ready[0]);
+ close(ready[1]);
+ return -1;
+ }
+
+ pid = fork();
+ if (pid < 0) {
+ ASSERT_GE(pid, 0, "fork charger");
+ close(ready[0]);
+ close(ready[1]);
+ close(charger_ctrl[0]);
+ close(charger_ctrl[1]);
+ charger_ctrl[0] = charger_ctrl[1] = -1;
+ return -1;
+ }
+
+ if (pid == 0) {
+ /* child (assert only in the parent so it isn't printed twice) */
+ int leaf_idx = 0;
+
+ close(ready[0]);
+ close(charger_ctrl[1]);
+
+ mod = charge_fraction > 0 ? charge_fraction : 1;
+ for (i = 0; i < n_nodes; i++) {
+ void *p;
+
+ if (!nodes[i].is_leaf)
+ continue;
+ if ((leaf_idx++ % mod) != 0)
+ continue;
+ /*
+ * cgroup_helpers builds paths from getpid(); in this
+ * forked child that differs from the parent that built
+ * the tree, so use the _parent (getppid()) variant to
+ * resolve the leaf under the parent's work dir.
+ */
+ if (join_parent_cgroup(nodes[i].rel))
+ _exit(1);
+ p = mmap(NULL, charge_bytes, PROT_READ | PROT_WRITE,
+ MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
+ if (p == MAP_FAILED)
+ _exit(2);
+ memset(p, 1, charge_bytes);
+ /* keep p mapped so the charge persists */
+ }
+ /* signal ready, then block until the parent closes the pipe */
+ if (write(ready[1], "x", 1) != 1)
+ _exit(3);
+ while (read(charger_ctrl[0], &c, 1) > 0)
+ ;
+ _exit(0);
+ }
+
+ /* parent */
+ charger_pid = pid;
+ close(ready[1]);
+ close(charger_ctrl[0]);
+ charger_ctrl[0] = -1;
+
+ /* wait until the child has charged every leaf */
+ if (!ASSERT_EQ(read(ready[0], &c, 1), 1, "charger ready")) {
+ close(ready[0]);
+ return -1;
+ }
+ close(ready[0]);
+ return 0;
+}
+
+static void stop_charger(void)
+{
+ int status;
+
+ if (charger_ctrl[1] >= 0) {
+ close(charger_ctrl[1]); /* unblock the child -> it exits */
+ charger_ctrl[1] = -1;
+ }
+ if (charger_pid > 0) {
+ if (waitpid(charger_pid, &status, 0) == charger_pid &&
+ (!WIFEXITED(status) || WEXITSTATUS(status) != 0))
+ fprintf(stderr,
+ "charger child exited abnormally (status=0x%x)\n",
+ status);
+ charger_pid = -1;
+ }
+}
+
+/* ---- file (traditional) reader ----------------------------------------- */
+
+static void parse_stat(char *buf, struct file_snap *o)
+{
+ char *save, *line;
+
+ for (line = strtok_r(buf, "\n", &save); line;
+ line = strtok_r(NULL, "\n", &save)) {
+ unsigned long long val;
+ char name[64];
+
+ if (sscanf(line, "%63s %llu", name, &val) != 2)
+ continue;
+ o->full_sum += val;
+ o->full_fields++;
+ if (!strcmp(name, "anon"))
+ o->anon = val;
+ else if (!strcmp(name, "file"))
+ o->file = val;
+ else if (!strcmp(name, "shmem"))
+ o->shmem = val;
+ else if (!strcmp(name, "file_mapped"))
+ o->file_mapped = val;
+ else if (!strcmp(name, "pgfault"))
+ o->pgfault = val;
+ }
+}
+
+static int file_read_node(const char *rel, struct file_snap *o)
+{
+ char buf[8192];
+
+ memset(o, 0, sizeof(*o));
+
+ if (read_cgroup_file(rel, "memory.stat", buf, sizeof(buf)))
+ return -1;
+ parse_stat(buf, o);
+
+ if (!read_cgroup_file(rel, "memory.current", buf, sizeof(buf)))
+ o->current = strtoull(buf, NULL, 10);
+ if (!read_cgroup_file(rel, "memory.max", buf, sizeof(buf))) {
+ if (!strncmp(buf, "max", 3))
+ o->max_is_max = true;
+ else
+ o->max = strtoull(buf, NULL, 10);
+ }
+ return 0;
+}
+
+static void time_file(int iters, struct timing *res)
+{
+ long long total = 0;
+ struct file_snap s;
+ int it, i;
+
+ for (it = 0; it < WARMUP_ITERS; it++)
+ for (i = 0; i < n_nodes; i++)
+ file_read_node(nodes[i].rel, &s);
+
+ for (it = 0; it < iters; it++) {
+ long long t0 = now_ns();
+
+ for (i = 0; i < n_nodes; i++) {
+ file_read_node(nodes[i].rel, &s);
+ sink += s.anon + s.full_sum;
+ }
+ total += now_ns() - t0;
+ }
+ res->avg_us = (double)total / iters / 1000.0;
+ res->fields = s.full_fields;
+}
+
+/* ---- BPF reader -------------------------------------------------------- */
+
+static int bpf_walk_once(struct bpf_link *link)
+{
+ char buf[4096];
+ ssize_t r;
+ int fd;
+
+ fd = bpf_iter_create(bpf_link__fd(link));
+ if (fd < 0)
+ return -1;
+ while ((r = read(fd, buf, sizeof(buf))) > 0)
+ ;
+ close(fd);
+ return r == 0 ? 0 : -1;
+}
+
+static int drain_map(int mfd, struct memcg_stat_snapshot *out, int max)
+{
+ __u64 key = 0, next;
+ int n = 0, err;
+
+ err = bpf_map_get_next_key(mfd, NULL, &next);
+ while (err == 0) {
+ if (n < max && !bpf_map_lookup_elem(mfd, &next, &out[n])) {
+ sink += out[n].anon + out[n].full_sum;
+ n++;
+ }
+ key = next;
+ err = bpf_map_get_next_key(mfd, &key, &next);
+ }
+ return n;
+}
+
+static void time_bpf(struct bpf_link *link, struct memcg_stat_reader *skel,
+ int iters, struct timing *res)
+{
+ struct memcg_stat_snapshot *tmp;
+ long long total = 0, ro_total = 0;
+ int mfd = bpf_map__fd(skel->maps.results);
+ int it, got = 0;
+
+ tmp = calloc(n_nodes + 8, sizeof(*tmp));
+ if (!ASSERT_OK_PTR(tmp, "calloc tmp"))
+ return;
+
+ skel->bss->collect_full = 1;
+
+ for (it = 0; it < WARMUP_ITERS; it++) {
+ bpf_walk_once(link);
+ drain_map(mfd, tmp, n_nodes + 8);
+ }
+
+ for (it = 0; it < iters; it++) {
+ long long t0, t1, t2;
+
+ t0 = now_ns();
+ bpf_walk_once(link);
+ t1 = now_ns();
+ got = drain_map(mfd, tmp, n_nodes + 8);
+ t2 = now_ns();
+
+ total += t2 - t0;
+ ro_total += t1 - t0;
+ }
+
+ res->avg_us = (double)total / iters / 1000.0;
+ res->ro_avg_us = (double)ro_total / iters / 1000.0;
+ res->nodes_seen = got;
+ res->fields = tmp[0].full_fields;
+ free(tmp);
+}
+
+/* ---- correctness ------------------------------------------------------- */
+
+static void check_correctness(struct bpf_link *link,
+ struct memcg_stat_reader *skel)
+{
+ int mfd = bpf_map__fd(skel->maps.results);
+ __u64 total_anon = 0, worst_cur_drift = 0;
+ __u64 anon_tol = 4 * page_size;
+ int i, anon_mism = 0, missing = 0;
+
+ skel->bss->collect_full = 0;
+ if (!ASSERT_OK(bpf_walk_once(link), "bpf walk"))
+ return;
+
+ for (i = 0; i < n_nodes; i++) {
+ struct memcg_stat_snapshot b;
+ __u64 cur, drift;
+ struct file_snap f;
+
+ if (bpf_map_lookup_elem(mfd, &nodes[i].id, &b)) {
+ missing++;
+ continue;
+ }
+ if (file_read_node(nodes[i].rel, &f)) {
+ missing++;
+ continue;
+ }
+ total_anon += b.anon;
+
+ /*
+ * anon (NR_ANON_MAPPED) is rstat-flushed and, with the charger
+ * stopped, deterministic: BPF and memory.stat must agree. The
+ * tolerance is far tighter than a units error (bytes vs pages
+ * differ by PAGE_SIZE), so a wrong-unit/wrong-field bug trips it.
+ */
+ if ((b.anon > f.anon ? b.anon - f.anon : f.anon - b.anon) > anon_tol) {
+ anon_mism++;
+ if (anon_mism <= 5)
+ fprintf(stderr,
+ "anon mismatch %s: bpf=%llu file=%llu\n",
+ nodes[i].rel, b.anon, f.anon);
+ }
+
+ /*
+ * memory.current is the LIVE page_counter. Both sides read the
+ * same counter, but the BPF values are captured in one fast walk
+ * while the files are read across the whole (much longer) loop,
+ * so any difference is time skew on a moving counter, not a BPF
+ * bug -- track it as informational only.
+ */
+ cur = b.usage_pages * page_size;
+ drift = cur > f.current ? cur - f.current : f.current - cur;
+ if (drift > worst_cur_drift)
+ worst_cur_drift = drift;
+ }
+
+ ASSERT_EQ(missing, 0, "all cgroups present in map");
+ ASSERT_EQ(anon_mism, 0, "bpf vs file anon (rstat-flushed)");
+ ASSERT_GT(total_anon, 0, "tree charged some anon");
+ printf("max memory.current drift bpf-vs-file: %llu bytes (live counter, read across the walk window)\n",
+ worst_cur_drift);
+}
+
+/* ---- one case ---------------------------------------------------------- */
+
+struct testcase {
+ const char *name;
+ int fanout;
+ int depth;
+ size_t charge_bytes;
+ int charge_fraction; /* charge every Nth leaf; 1 = all */
+ int iters;
+};
+
+static void run_case(const struct testcase *tc)
+{
+ struct timing f = {}, b = {};
+ struct memcg_stat_reader *skel = NULL;
+ struct bpf_link *link = NULL;
+ int root_fd = -1;
+ int charged;
+
+ if (!ASSERT_OK(build_tree(tc->fanout, tc->depth, &root_fd), "build tree"))
+ goto out;
+
+ if (start_charger(tc->charge_bytes, tc->charge_fraction))
+ goto out;
+
+ skel = memcg_stat_reader__open();
+ if (!ASSERT_OK_PTR(skel, "skel open"))
+ goto out;
+ if (!ASSERT_OK(bpf_map__set_max_entries(skel->maps.results, n_nodes + 8),
+ "set max_entries"))
+ goto out;
+ if (!ASSERT_OK(memcg_stat_reader__load(skel), "skel load"))
+ goto out;
+
+ DECLARE_LIBBPF_OPTS(bpf_iter_attach_opts, opts);
+ union bpf_iter_link_info linfo = {};
+
+ linfo.cgroup.cgroup_fd = root_fd;
+ linfo.cgroup.order = BPF_CGROUP_ITER_DESCENDANTS_PRE;
+ opts.link_info = &linfo;
+ opts.link_info_len = sizeof(linfo);
+
+ link = bpf_program__attach_iter(skel->progs.cgroup_memcg_stat_reader,
+ &opts);
+ if (!ASSERT_OK_PTR(link, "attach iter"))
+ goto out;
+
+ check_correctness(link, skel);
+
+ time_file(tc->iters, &f);
+ time_bpf(link, skel, tc->iters, &b);
+
+ charged = tc->charge_fraction > 0 ?
+ (n_leaves + tc->charge_fraction - 1) / tc->charge_fraction :
+ n_leaves;
+
+ /*
+ * Informational timing diagnostic: captured like any test output, so it
+ * is shown under -v or on failure but not on a normal PASS. The pass/fail
+ * verdict is decided solely by the correctness checks, never by these
+ * numbers.
+ */
+ printf("\n==== memcg_stat_reader: %s ====\n", tc->name);
+ printf("tree: nodes=%d leaves=%d charged_leaves=%d fanout=%d depth=%d charge=%zuKB/leaf iters=%d\n",
+ n_nodes, n_leaves, charged, tc->fanout, tc->depth,
+ tc->charge_bytes >> 10, tc->iters);
+ printf("all times in us (average per full-tree pass, full memory.stat field set); ro = bpf read()-only (no map drain)\n");
+ printf("file_avg=%.1f bpf_avg=%.1f bpf_ro=%.1f speedup(file/bpf)=%.2fx\n",
+ f.avg_us, b.avg_us, b.ro_avg_us,
+ b.avg_us > 0 ? f.avg_us / b.avg_us : 0.0);
+ printf("per-cgroup: file avg=%.0f ns bpf avg=%.0f ns\n",
+ f.avg_us * 1000.0 / n_nodes, b.avg_us * 1000.0 / n_nodes);
+ printf("fields/cgroup: bpf=%u | file stat lines=%u\n", b.fields, f.fields);
+ printf("bpf entries produced: %d (expected %d)\n", b.nodes_seen, n_nodes);
+
+ ASSERT_EQ(b.nodes_seen, n_nodes, "bpf visited whole subtree");
+
+out:
+ bpf_link__destroy(link);
+ memcg_stat_reader__destroy(skel);
+ if (root_fd >= 0)
+ close(root_fd);
+ stop_charger(); /* reap charger so leaves become empty */
+
+ /*
+ * Remove the subtree in reverse creation order. Nodes are recorded in
+ * DFS pre-order (a parent precedes all its descendants), so iterating
+ * backwards removes every child before its parent.
+ */
+ if (nodes) {
+ int i;
+
+ for (i = n_nodes - 1; i >= 0; i--)
+ remove_cgroup(nodes[i].rel);
+ free(nodes);
+ nodes = NULL;
+ }
+}
+
+static const struct testcase cases[] = {
+ { "small", 4, 2, 256 << 10, 1, 200 },
+ { "medium", 10, 2, 256 << 10, 1, 50 },
+ { "large", 10, 3, 256 << 10, 1, 10 },
+ { "large_sparse", 10, 3, 256 << 10, 8, 10 },
+};
+
+/*
+ * The memcg kfuncs the BPF program relies on (bpf_get_mem_cgroup et al.) are
+ * built only with CONFIG_MEMCG (mm/bpf_memcontrol.c). On a kernel without it
+ * they are absent from vmlinux BTF and the program fails to load, so probe for
+ * one of them and skip cleanly rather than reporting a spurious failure.
+ */
+static bool memcg_kfuncs_available(void)
+{
+ struct btf *btf;
+ bool ok;
+
+ btf = btf__load_vmlinux_btf();
+ if (!btf)
+ return false;
+ ok = btf__find_by_name_kind(btf, "bpf_get_mem_cgroup", BTF_KIND_FUNC) > 0;
+ btf__free(btf);
+ return ok;
+}
+
+void test_memcg_stat_reader(void)
+{
+ int i;
+
+ if (!memcg_kfuncs_available()) {
+ test__skip();
+ return;
+ }
+
+ page_size = sysconf(_SC_PAGESIZE);
+
+ if (!ASSERT_OK(setup_cgroup_environment(), "setup cgroup env"))
+ return;
+
+ for (i = 0; i < ARRAY_SIZE(cases); i++) {
+ if (!test__start_subtest(cases[i].name))
+ continue;
+ run_case(&cases[i]);
+ }
+
+ cleanup_cgroup_environment();
+}
diff --git a/tools/testing/selftests/bpf/progs/memcg_stat_reader.c b/tools/testing/selftests/bpf/progs/memcg_stat_reader.c
new file mode 100644
index 000000000000..a2c1b1b48364
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/memcg_stat_reader.c
@@ -0,0 +1,181 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_core_read.h>
+#include "memcg_stat_reader.h"
+
+char _license[] SEC("license") = "GPL";
+
+/*
+ * Flipped by userspace between timed runs (a plain .bss global, writable at
+ * runtime through the skeleton mmap):
+ * 0 - collect only the matched subset (a handful of kfunc calls)
+ * 1 - additionally fold in the full memory.stat field set (many kfunc calls)
+ */
+int collect_full;
+
+/*
+ * Per-cgroup results, keyed by cgroup id. The BPF-side id (cgrp->kn->id)
+ * equals the userspace get_cgroup_id() value, so the test can correlate map
+ * entries back to the cgroups it created. max_entries is resized by userspace
+ * (bpf_map__set_max_entries) to the size of the subtree before load.
+ */
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __uint(max_entries, 1);
+ __type(key, __u64);
+ __type(value, struct memcg_stat_snapshot);
+} results SEC(".maps");
+
+/*
+ * Accumulate one page-state / vm-event read. Each enumerator is guarded by
+ * bpf_core_enum_value_exists(): the full field set below spans counters that are
+ * config- or version-gated (e.g. NR_SECONDARY_PAGETABLE, PGDEMOTE_KHUGEPAGED,
+ * MEMCG_PERCPU_B), so on a kernel whose BTF lacks one, the bpf_core_enum_value()
+ * relocation would otherwise poison the instruction and fail the *entire*
+ * program load. With the _exists guard the missing enumerator relocates to a
+ * compile-time-false branch that the verifier drops as dead code, so the fold is
+ * simply skipped and the rest of the program (including the matched/correctness
+ * path) still loads.
+ */
+#define FOLD_PS(ENUM, NAME) do { \
+ if (bpf_core_enum_value_exists(enum ENUM, NAME)) { \
+ __u64 __v = bpf_mem_cgroup_page_state(memcg, \
+ bpf_core_enum_value(enum ENUM, NAME)); \
+ if (__v != (__u64)-1) { \
+ sum += __v; \
+ nr++; \
+ } \
+ } \
+} while (0)
+
+#define FOLD_EV(NAME) do { \
+ if (bpf_core_enum_value_exists(enum vm_event_item, NAME)) { \
+ __u64 __v = bpf_mem_cgroup_vm_events(memcg, \
+ bpf_core_enum_value(enum vm_event_item, NAME)); \
+ if (__v != (__u64)-1) { \
+ sum += __v; \
+ nr++; \
+ } \
+ } \
+} while (0)
+
+/*
+ * Read a broad memory.stat field set so the timed "full" run pays the realistic
+ * per-field kfunc cost. Enumerators absent from the running kernel's BTF are
+ * skipped (see the _exists guard in FOLD_PS/FOLD_EV), so this stays loadable
+ * across kernel configs/versions. __always_inline so the acquired memcg
+ * reference stays in the main frame (no cross-subprog reference tracking); the
+ * runtime collect_full branch keeps it off the matched path.
+ */
+static __always_inline void collect_full_stats(struct mem_cgroup *memcg,
+ struct memcg_stat_snapshot *snap)
+{
+ __u64 sum = 0;
+ __u32 nr = 0;
+
+ /* node_stat_item: size + event counters that memory.stat prints */
+ FOLD_PS(node_stat_item, NR_ANON_MAPPED);
+ FOLD_PS(node_stat_item, NR_FILE_PAGES);
+ FOLD_PS(node_stat_item, NR_FILE_MAPPED);
+ FOLD_PS(node_stat_item, NR_FILE_DIRTY);
+ FOLD_PS(node_stat_item, NR_WRITEBACK);
+ FOLD_PS(node_stat_item, NR_SHMEM);
+ FOLD_PS(node_stat_item, NR_INACTIVE_ANON);
+ FOLD_PS(node_stat_item, NR_ACTIVE_ANON);
+ FOLD_PS(node_stat_item, NR_INACTIVE_FILE);
+ FOLD_PS(node_stat_item, NR_ACTIVE_FILE);
+ FOLD_PS(node_stat_item, NR_UNEVICTABLE);
+ FOLD_PS(node_stat_item, NR_SLAB_RECLAIMABLE_B);
+ FOLD_PS(node_stat_item, NR_SLAB_UNRECLAIMABLE_B);
+ FOLD_PS(node_stat_item, NR_KERNEL_STACK_KB);
+ FOLD_PS(node_stat_item, NR_PAGETABLE);
+ FOLD_PS(node_stat_item, NR_SECONDARY_PAGETABLE);
+ FOLD_PS(node_stat_item, NR_VMALLOC);
+ FOLD_PS(node_stat_item, WORKINGSET_REFAULT_ANON);
+ FOLD_PS(node_stat_item, WORKINGSET_REFAULT_FILE);
+ FOLD_PS(node_stat_item, WORKINGSET_ACTIVATE_ANON);
+ FOLD_PS(node_stat_item, WORKINGSET_ACTIVATE_FILE);
+ FOLD_PS(node_stat_item, WORKINGSET_RESTORE_ANON);
+ FOLD_PS(node_stat_item, WORKINGSET_RESTORE_FILE);
+ FOLD_PS(node_stat_item, WORKINGSET_NODERECLAIM);
+ FOLD_PS(node_stat_item, PGDEMOTE_KSWAPD);
+ FOLD_PS(node_stat_item, PGDEMOTE_DIRECT);
+ FOLD_PS(node_stat_item, PGDEMOTE_KHUGEPAGED);
+ FOLD_PS(node_stat_item, PGSTEAL_KSWAPD);
+ FOLD_PS(node_stat_item, PGSTEAL_DIRECT);
+ FOLD_PS(node_stat_item, PGSTEAL_KHUGEPAGED);
+ FOLD_PS(node_stat_item, PGSCAN_KSWAPD);
+ FOLD_PS(node_stat_item, PGSCAN_DIRECT);
+ FOLD_PS(node_stat_item, PGSCAN_KHUGEPAGED);
+ FOLD_PS(node_stat_item, PGREFILL);
+
+ /* memcg_stat_item: numbered past NR_VM_NODE_STAT_ITEMS */
+ FOLD_PS(memcg_stat_item, MEMCG_KMEM);
+ FOLD_PS(memcg_stat_item, MEMCG_SOCK);
+ FOLD_PS(memcg_stat_item, MEMCG_PERCPU_B);
+
+ /* vm_event_item: the raw-count tail of memory.stat */
+ FOLD_EV(PGFAULT);
+ FOLD_EV(PGMAJFAULT);
+ FOLD_EV(PGACTIVATE);
+ FOLD_EV(PGDEACTIVATE);
+ FOLD_EV(PGLAZYFREE);
+ FOLD_EV(PGLAZYFREED);
+
+ snap->full_sum = sum;
+ snap->full_fields = nr;
+}
+
+SEC("iter.s/cgroup")
+int cgroup_memcg_stat_reader(struct bpf_iter__cgroup *ctx)
+{
+ struct cgroup *cgrp = ctx->cgroup;
+ struct memcg_stat_snapshot snap = {};
+ struct cgroup_subsys_state *css;
+ struct mem_cgroup *memcg;
+ __u64 cg_id;
+
+ /*
+ * DESCENDANTS_PRE ends with a terminal element where cgroup == NULL.
+ * Return 0 (not 1) so the walk runs to completion.
+ */
+ if (!cgrp)
+ return 0;
+
+ css = &cgrp->self;
+ memcg = bpf_get_mem_cgroup(css);
+ if (!memcg)
+ return 0;
+
+ /* Bring this memcg's rstat up to date before reading it. */
+ bpf_mem_cgroup_flush_stats(memcg);
+
+ cg_id = BPF_CORE_READ(cgrp, kn, id);
+ snap.cgroup_id = cg_id;
+
+ /* Matched subset: always collected so correctness holds in both modes. */
+ snap.anon = bpf_mem_cgroup_page_state(memcg,
+ bpf_core_enum_value(enum node_stat_item, NR_ANON_MAPPED));
+ snap.file = bpf_mem_cgroup_page_state(memcg,
+ bpf_core_enum_value(enum node_stat_item, NR_FILE_PAGES));
+ snap.shmem = bpf_mem_cgroup_page_state(memcg,
+ bpf_core_enum_value(enum node_stat_item, NR_SHMEM));
+ snap.file_mapped = bpf_mem_cgroup_page_state(memcg,
+ bpf_core_enum_value(enum node_stat_item, NR_FILE_MAPPED));
+ snap.pgfault = bpf_mem_cgroup_vm_events(memcg,
+ bpf_core_enum_value(enum vm_event_item, PGFAULT));
+
+ /* page_counter fields need no kfunc; read them off the trusted ptr. */
+ snap.usage_pages = BPF_CORE_READ(memcg, memory.usage.counter);
+ snap.max_pages = BPF_CORE_READ(memcg, memory.max);
+
+ if (collect_full)
+ collect_full_stats(memcg, &snap);
+
+ bpf_map_update_elem(&results, &cg_id, &snap, BPF_ANY);
+
+ bpf_put_mem_cgroup(memcg);
+ return 0;
+}
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 9+ messages in thread* [PATCH 2/3] selftests/bpf: add memcg_stat_churn BPF-vs-memory.stat benchmark under churn
2026-07-04 4:56 [PATCH 0/3] selftests/bpf: compare BPF and memory.stat memcg stat readers Ziyang Men
2026-07-04 4:56 ` [PATCH 1/3] selftests/bpf: add memcg_stat_reader BPF-vs-memory.stat benchmark Ziyang Men
@ 2026-07-04 4:56 ` Ziyang Men
2026-07-04 5:39 ` bot+bpf-ci
2026-07-04 4:56 ` [PATCH 3/3] selftests/bpf: add memcg_stat_churn_percpu BPF-vs-memory.stat benchmark under cross-CPU churn Ziyang Men
2026-07-07 0:17 ` [PATCH 0/3] selftests/bpf: compare BPF and memory.stat memcg stat readers Eduard Zingerman
3 siblings, 1 reply; 9+ messages in thread
From: Ziyang Men @ 2026-07-04 4:56 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
Eduard Zingerman, Kumar Kartikeya Dwivedi, bpf
Cc: Martin KaFai Lau, Song Liu, Yonghong Song, Jiri Olsa,
Emil Tsalapatis, Shuah Khan, Roman Gushchin, kernel-team,
linux-mm, cgroups, linux-kselftest, linux-kernel, Ziyang Men,
Shakeel Butt
Add a test_progs selftest that complements memcg_stat_reader by
comparing the same two ways of reading memory-cgroup statistics -- the
traditional per-cgroup memory.stat parse and a single
SEC("iter.s/cgroup") BPF walk -- but while the tree is under continuous
allocation churn instead of frozen.
Where memcg_stat_reader charges a tree once and reads it quiescent,
memcg_stat_churn forks one "churner" process per selected leaf. Each
churner joins its leaf, pins a small resident anon set (so the tree
always carries some charge) and then loops mmap()+memset()+munmap() for
the whole measurement window, keeping every touched memcg's per-cpu
rstat perpetually dirty. While that runs, the parent repeatedly samples
both readers, averages the wall-clock cost of each and reports the
file/BPF speedup ratio.
Each churner is a CPU-busy process (a tight mmap()+memset()+munmap()
loop) and there is one per selected leaf, so the test is registered
serial (serial_test_) rather than parallel: under test_progs -j it must
not steal CPU from co-scheduled workers, and -- since the whole point is
to time the two readers -- its own measurements must not be perturbed by
them.
The BPF program, its per-cgroup hash map and the snapshot struct are
reused verbatim from memcg_stat_reader (progs/memcg_stat_reader.c and
memcg_stat_reader.h); only the userspace load model and sampling loop
are new, so there is no new BPF object and no Makefile change.
Both readers flush rstat through the same mem_cgroup_flush_stats() path
-- bpf_mem_cgroup_flush_stats() is a thin wrapper around it -- and
flushing the subtree root flushes the whole subtree, so each whole-tree
pass pays one real flush F plus N cheap threshold checks, common to both
paths. If the two readers are simply interleaved back to back, the much
shorter BPF pass freeloads on the flush just performed by the ~20x
longer file pass, so the measurement ends up depending on ordering
rather than work. To avoid that, the parent idles a fixed gap
(CHURN_GAP_US, 50 ms) before every timed read so the tree re-accumulates
a roughly fixed amount of churn first; the resulting flush is then paid
inside the timed region, giving every read approximately the same start
state. The file/BPF order is also alternated across samples. The gap is
effectively a staleness / poll-interval knob: a larger gap means a
larger flush that both paths pay, so the reported ratio, which is
(F + read_file) / (F + read_bpf), is correspondingly more conservative.
Because stats are a moving target under churn, the test does not do a
field-by-field BPF-vs-file equality check (that is memcg_stat_reader's
job). Pass/fail gates only on structural sanity: a walk on the freshly
loaded map must visit every cgroup (missing == 0), every timed walk must
complete, and the tree must carry some anon charge. The timing table and
the final "RATIO" line are informational diagnostics, captured like any
other test output, i.e. shown under -v or on failure, never on a normal
PASS.
Both subtests run on a large (1111-cgroup) tree, where the whole-tree read
cost is large enough to dominate the rstat flush and scheduler jitter, so the
reported ratios are reproducible run to run; on a small (tens of cgroups) tree
the sub-millisecond BPF read is swamped by that noise and the ratio bounces.
They differ only in churn density -- large_dense churns one leaf in four,
large_sparse one in eight -- which changes how much of the tree the shared
flush has to touch. Because the flush cost F is common to both readers, the
speedup tracks how much of the whole-tree read is flush versus per-cgroup read
work; the BPF path avoids the per-cgroup VFS open/read and memory.stat string
parsing, so it wins on the read side regardless.
Sample output (v7.1 VM, 60 CPUs); times are us, average per full-tree read
under churn reading the full memory.stat field set; ratio = file/bpf; ro = bpf
read()-only (no map drain):
==== memcg_stat_churn: large_dense ====
tree: nodes=1111 leaves=1000 churners=250 fanout=10 depth=3 region=256KB resident=128KB samples=8 gap=50ms
file_avg=323034.3 bpf_avg=14933.4 bpf_ro=6926.8 ratio(file/bpf)=21.63x
==== memcg_stat_churn: large_sparse ====
tree: nodes=1111 leaves=1000 churners=125 fanout=10 depth=3 region=256KB resident=128KB samples=8 gap=50ms
file_avg=347304.8 bpf_avg=13592.8 bpf_ro=6628.6 ratio(file/bpf)=25.55x
large_dense churns twice as many leaves as large_sparse (one in four rather
than one in eight), so its shared flush touches more of the tree and its
file/BPF ratio is a little lower (21.6x vs 25.5x); the BPF path stays well over
20x faster either way.
This builds on the memcg BPF kfuncs and complements the memcg_stat_reader
selftest added in the previous patch.
Suggested-by: Shakeel Butt <shakeel.butt@linux.dev>
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ziyang Men <ziyang.meme@gmail.com>
---
.../bpf/prog_tests/memcg_stat_churn.c | 716 ++++++++++++++++++
1 file changed, 716 insertions(+)
create mode 100644 tools/testing/selftests/bpf/prog_tests/memcg_stat_churn.c
diff --git a/tools/testing/selftests/bpf/prog_tests/memcg_stat_churn.c b/tools/testing/selftests/bpf/prog_tests/memcg_stat_churn.c
new file mode 100644
index 000000000000..3e386d0b4c03
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/memcg_stat_churn.c
@@ -0,0 +1,716 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+
+/*
+ * memcg_stat_churn
+ * ================
+ * A load variant of the memcg_stat_reader benchmark. Where memcg_stat_reader
+ * charges a quiescent tree once and then measures both readers against static
+ * stats, this test keeps the memory-cgroup rstat perpetually DIRTY while it
+ * measures:
+ *
+ * - Build a synthetic cgroup subtree (fanout x depth), same as the reader.
+ * - Fork one "churner" process per selected leaf. Each churner joins its
+ * leaf, pins a small resident anon set (so tree anon stays > 0), then loops
+ * mmap()+memset()+munmap() for the whole measurement window. The constant
+ * charge/uncharge traffic keeps every touched memcg's per-cpu stats dirty,
+ * so each reader pays a realistic flush/read cost instead of a warm no-op.
+ * - While the churn runs, the parent repeatedly SAMPLES both readers:
+ * (A) traditional: open/read/parse memory.stat (+current/+max) for every
+ * cgroup from userspace;
+ * (B) BPF: one SEC("iter.s/cgroup") walk over the subtree calling the
+ * memcg kfuncs into a hash map, drained once per sample.
+ * Before each timed read the parent idles for a fixed gap (untimed) so the
+ * tree re-accumulates a roughly fixed amount of dirty rstat; every read
+ * (file/BPF x matched/full) therefore starts from approximately the same
+ * state and pays its own rstat flush inside the timed region. The
+ * file-vs-BPF order is also alternated across samples so residual jitter
+ * doesn't systematically favour whichever reader runs first.
+ * - Times are averaged over all samples and the file/BPF speedup ratio is
+ * reported. The gap is the "staleness / poll-interval" knob: a larger gap
+ * means a larger flush that both paths pay, so the ratio is more
+ * conservative (see CHURN_GAP_US).
+ *
+ * The BPF program, its hash map and the snapshot struct are REUSED verbatim
+ * from memcg_stat_reader (progs/memcg_stat_reader.c + memcg_stat_reader.h); only
+ * the userspace load model and sampling loop are new here.
+ *
+ * Under churn the stats are a moving target, so this test does NOT do a
+ * field-by-field BPF-vs-file equality check (that is memcg_stat_reader's job).
+ * Pass/fail gates only on structural sanity -- the iterator visited every
+ * cgroup and the tree carries some anon charge. The timing table and final
+ * RATIO line are informational diagnostics, printed like any other test output
+ * (i.e. under -v or on failure, never on a normal PASS).
+ */
+#include <test_progs.h>
+#include <bpf/libbpf.h>
+#include <bpf/btf.h>
+#include <stdlib.h>
+#include <string.h>
+#include <time.h>
+#include <unistd.h>
+#include <sys/mman.h>
+#include <sys/wait.h>
+#include "cgroup_helpers.h"
+#include "memcg_stat_reader.h"
+#include "memcg_stat_reader.skel.h"
+
+#define SUBTREE_ROOT "/mcg_churn"
+
+#define WARMUP_ITERS 2
+
+struct cg_node {
+ char rel[128];
+ __u64 id;
+ bool is_leaf;
+};
+
+/* Field subset parsed from memory.stat (mirrors memcg_stat_reader). */
+struct file_snap {
+ __u64 anon, file, shmem, file_mapped, pgfault;
+ __u64 current; /* memory.current, bytes */
+ __u64 max; /* memory.max, bytes (valid unless max_is_max) */
+ __u64 full_sum;
+ __u32 full_fields;
+ bool max_is_max;
+};
+
+static volatile __u64 sink; /* keep the optimizer from eliding reads */
+
+static long long now_ns(void)
+{
+ struct timespec t;
+
+ clock_gettime(CLOCK_MONOTONIC, &t);
+ return (long long)t.tv_sec * 1000000000LL + t.tv_nsec;
+}
+
+/* ---- tree construction (same shape as memcg_stat_reader) --------------- */
+
+static struct cg_node *nodes;
+static int n_nodes;
+static int n_leaves;
+
+static int add_node(const char *rel, bool is_leaf, int *keep_fd)
+{
+ int fd;
+
+ fd = create_and_get_cgroup(rel);
+ if (fd < 0)
+ return -1;
+ if (keep_fd)
+ *keep_fd = fd;
+ else
+ close(fd);
+
+ strncpy(nodes[n_nodes].rel, rel, sizeof(nodes[n_nodes].rel) - 1);
+ nodes[n_nodes].rel[sizeof(nodes[n_nodes].rel) - 1] = '\0';
+ nodes[n_nodes].id = get_cgroup_id(rel);
+ nodes[n_nodes].is_leaf = is_leaf;
+ if (is_leaf)
+ n_leaves++;
+ n_nodes++;
+ return 0;
+}
+
+/* Recursively create children of @rel. @rel must already exist and be recorded. */
+static int build_children(const char *rel, int fanout, int depth)
+{
+ char child[128];
+ int i;
+
+ if (depth == 0)
+ return 0;
+
+ /* Enable memory on this interior node so its children get a memcg. */
+ if (enable_controllers(rel, "memory"))
+ return -1;
+
+ for (i = 0; i < fanout; i++) {
+ snprintf(child, sizeof(child), "%s/c%d", rel, i);
+ if (add_node(child, depth == 1, NULL))
+ return -1;
+ if (build_children(child, fanout, depth - 1))
+ return -1;
+ }
+ return 0;
+}
+
+static size_t tree_capacity(int fanout, int depth)
+{
+ size_t total = 1, level = 1;
+ int d;
+
+ for (d = 0; d < depth; d++) {
+ level *= fanout;
+ total += level;
+ }
+ return total;
+}
+
+static int build_tree(int fanout, int depth, int *root_fd)
+{
+ n_nodes = 0;
+ n_leaves = 0;
+ nodes = calloc(tree_capacity(fanout, depth), sizeof(*nodes));
+ if (!nodes)
+ return -1;
+
+ if (add_node(SUBTREE_ROOT, depth == 0, root_fd))
+ return -1;
+ return build_children(SUBTREE_ROOT, fanout, depth);
+}
+
+/* ---- churn load -------------------------------------------------------- */
+
+/*
+ * Shared control block, mmap'd MAP_SHARED before the forks so the parent can
+ * signal all churners to stop with a single write.
+ */
+struct churn_ctl {
+ volatile int stop;
+};
+
+static struct churn_ctl *ctl;
+static pid_t *churn_pids;
+static int n_churners;
+static int churn_ready[2] = { -1, -1 }; /* churner -> parent "ready" barrier */
+
+/*
+ * One churner process. Joins its leaf, pins a resident anon set so the tree
+ * always carries some charge, signals readiness, then continuously faults in
+ * and frees a private anon region until told to stop. Never returns.
+ */
+static void churner_child(const struct cg_node *leaf, size_t region_bytes,
+ size_t resident_bytes)
+{
+ void *resident;
+
+ close(churn_ready[0]);
+
+ /*
+ * cgroup_helpers builds paths from getpid(); in this forked child that
+ * differs from the parent that built the tree, so use the _parent
+ * (getppid()) variant to resolve the leaf under the parent's work dir.
+ */
+ if (join_parent_cgroup(leaf->rel))
+ _exit(1);
+
+ resident = mmap(NULL, resident_bytes, PROT_READ | PROT_WRITE,
+ MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
+ if (resident == MAP_FAILED)
+ _exit(2);
+ memset(resident, 1, resident_bytes); /* fault in, keep mapped */
+
+ if (write(churn_ready[1], "x", 1) != 1)
+ _exit(3);
+ close(churn_ready[1]); /* so a sibling's early death yields EOF, not a parent hang */
+
+ while (!ctl->stop) {
+ void *p = mmap(NULL, region_bytes, PROT_READ | PROT_WRITE,
+ MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
+
+ if (p == MAP_FAILED)
+ continue;
+ memset(p, 1, region_bytes); /* fault in -> anon charge */
+ munmap(p, region_bytes); /* free -> uncharge (keeps rstat dirty) */
+ }
+ munmap(resident, resident_bytes);
+ _exit(0);
+}
+
+/*
+ * Fork one churner per @charge_fraction-th leaf. Returns 0 once every churner
+ * has joined its leaf and pinned its resident set (so measurement starts under
+ * steady-state load). On failure the caller's cleanup path calls
+ * stop_churners() to reap whatever was started.
+ */
+static int start_churners(size_t region_bytes, size_t resident_bytes,
+ int charge_fraction)
+{
+ int mod = charge_fraction > 0 ? charge_fraction : 1;
+ int leaf_idx = 0;
+ int i;
+
+ ctl = mmap(NULL, sizeof(*ctl), PROT_READ | PROT_WRITE,
+ MAP_SHARED | MAP_ANONYMOUS, -1, 0);
+ if (!ASSERT_NEQ(ctl, MAP_FAILED, "mmap churn_ctl")) {
+ ctl = NULL;
+ return -1;
+ }
+ ctl->stop = 0;
+
+ if (!ASSERT_OK(pipe(churn_ready), "pipe churn_ready"))
+ return -1;
+
+ churn_pids = calloc(n_leaves, sizeof(*churn_pids));
+ if (!ASSERT_OK_PTR(churn_pids, "calloc churn_pids"))
+ return -1;
+
+ for (i = 0; i < n_nodes; i++) {
+ pid_t pid;
+
+ if (!nodes[i].is_leaf)
+ continue;
+ if ((leaf_idx++ % mod) != 0)
+ continue;
+
+ pid = fork();
+ if (pid < 0) {
+ ASSERT_GE(pid, 0, "fork churner");
+ return -1;
+ }
+ if (pid == 0)
+ churner_child(&nodes[i], region_bytes, resident_bytes);
+
+ churn_pids[n_churners++] = pid;
+ }
+
+ /* parent: this end is only for the children to signal on */
+ close(churn_ready[1]);
+ churn_ready[1] = -1;
+
+ /* wait until every churner has joined + pinned its resident set */
+ for (i = 0; i < n_churners; i++) {
+ char c;
+ ssize_t r = read(churn_ready[0], &c, 1);
+
+ if (r == 0)
+ fprintf(stderr,
+ "a churner exited before signaling ready (join_parent_cgroup/mmap failure?)\n");
+ if (!ASSERT_EQ(r, 1, "churner ready"))
+ return -1;
+ }
+ return 0;
+}
+
+static void stop_churners(void)
+{
+ int i, status;
+
+ if (ctl)
+ ctl->stop = 1; /* release all churn loops */
+
+ if (churn_ready[1] >= 0) {
+ close(churn_ready[1]);
+ churn_ready[1] = -1;
+ }
+ if (churn_ready[0] >= 0) {
+ close(churn_ready[0]);
+ churn_ready[0] = -1;
+ }
+
+ for (i = 0; i < n_churners; i++) {
+ if (!churn_pids || churn_pids[i] <= 0)
+ continue;
+ if (waitpid(churn_pids[i], &status, 0) == churn_pids[i] &&
+ (!WIFEXITED(status) || WEXITSTATUS(status) != 0))
+ fprintf(stderr,
+ "churner %d exited abnormally (status=0x%x)\n",
+ churn_pids[i], status);
+ }
+
+ free(churn_pids);
+ churn_pids = NULL;
+ n_churners = 0;
+
+ if (ctl) {
+ munmap((void *)ctl, sizeof(*ctl));
+ ctl = NULL;
+ }
+}
+
+/* ---- file (traditional) reader ----------------------------------------- */
+
+static void parse_stat(char *buf, struct file_snap *o)
+{
+ char *save, *line;
+
+ for (line = strtok_r(buf, "\n", &save); line;
+ line = strtok_r(NULL, "\n", &save)) {
+ unsigned long long val;
+ char name[64];
+
+ if (sscanf(line, "%63s %llu", name, &val) != 2)
+ continue;
+ o->full_sum += val;
+ o->full_fields++;
+ if (!strcmp(name, "anon"))
+ o->anon = val;
+ else if (!strcmp(name, "file"))
+ o->file = val;
+ else if (!strcmp(name, "shmem"))
+ o->shmem = val;
+ else if (!strcmp(name, "file_mapped"))
+ o->file_mapped = val;
+ else if (!strcmp(name, "pgfault"))
+ o->pgfault = val;
+ }
+}
+
+static int file_read_node(const char *rel, struct file_snap *o)
+{
+ char buf[8192];
+
+ memset(o, 0, sizeof(*o));
+
+ if (read_cgroup_file(rel, "memory.stat", buf, sizeof(buf)))
+ return -1;
+ parse_stat(buf, o);
+
+ if (!read_cgroup_file(rel, "memory.current", buf, sizeof(buf)))
+ o->current = strtoull(buf, NULL, 10);
+ if (!read_cgroup_file(rel, "memory.max", buf, sizeof(buf))) {
+ if (!strncmp(buf, "max", 3))
+ o->max_is_max = true;
+ else
+ o->max = strtoull(buf, NULL, 10);
+ }
+ return 0;
+}
+
+/*
+ * One timed traditional pass over the whole tree; returns nanoseconds.
+ * @gap_us idles (untimed) before the pass so the tree re-accumulates a roughly
+ * fixed amount of churn first; the resulting rstat flush is then paid inside the
+ * timed region, giving every read approximately the same start state.
+ */
+static long long file_pass(int gap_us)
+{
+ struct file_snap s;
+ long long t0;
+ int i;
+
+ if (gap_us)
+ usleep(gap_us);
+ t0 = now_ns();
+ for (i = 0; i < n_nodes; i++) {
+ file_read_node(nodes[i].rel, &s);
+ sink += s.anon + s.full_sum;
+ }
+ return now_ns() - t0;
+}
+
+/* ---- BPF reader -------------------------------------------------------- */
+
+static int bpf_walk_once(struct bpf_link *link)
+{
+ char buf[4096];
+ ssize_t r;
+ int fd;
+
+ fd = bpf_iter_create(bpf_link__fd(link));
+ if (fd < 0)
+ return -1;
+ while ((r = read(fd, buf, sizeof(buf))) > 0)
+ ;
+ close(fd);
+ return r == 0 ? 0 : -1;
+}
+
+static int drain_map(int mfd, struct memcg_stat_snapshot *out, int max)
+{
+ __u64 key = 0, next;
+ int n = 0, err;
+
+ err = bpf_map_get_next_key(mfd, NULL, &next);
+ while (err == 0) {
+ if (n < max && !bpf_map_lookup_elem(mfd, &next, &out[n])) {
+ sink += out[n].anon + out[n].full_sum;
+ n++;
+ }
+ key = next;
+ err = bpf_map_get_next_key(mfd, &key, &next);
+ }
+ return n;
+}
+
+/*
+ * One timed BPF pass: kernel walk (ro) + map drain into userspace. Returns the
+ * total nanoseconds; *ro_ns gets the walk-only time, *got the entries drained.
+ * @gap_us idles (untimed) before the walk, exactly as in file_pass(), so the
+ * per-node rstat flush the walk pays reflects the same accumulated churn.
+ */
+static long long bpf_pass(struct bpf_link *link, struct memcg_stat_reader *skel,
+ struct memcg_stat_snapshot *tmp,
+ long long *ro_ns, int *got, int *werr, int gap_us)
+{
+ int mfd = bpf_map__fd(skel->maps.results);
+ long long t0, t1, t2;
+ int err;
+
+ skel->bss->collect_full = 1;
+
+ if (gap_us)
+ usleep(gap_us);
+ t0 = now_ns();
+ err = bpf_walk_once(link);
+ t1 = now_ns();
+ *got = drain_map(mfd, tmp, n_nodes + 8);
+ t2 = now_ns();
+
+ if (werr)
+ *werr = err;
+ *ro_ns = t1 - t0;
+ return t2 - t0;
+}
+
+/* ---- structural sanity (no field-by-field check under churn) ------------ */
+
+static void check_structural(struct bpf_link *link,
+ struct memcg_stat_reader *skel)
+{
+ int mfd = bpf_map__fd(skel->maps.results);
+ __u64 total_anon = 0;
+ int i, missing = 0;
+
+ skel->bss->collect_full = 0;
+ if (!ASSERT_OK(bpf_walk_once(link), "bpf walk"))
+ return;
+
+ for (i = 0; i < n_nodes; i++) {
+ struct memcg_stat_snapshot b;
+
+ if (bpf_map_lookup_elem(mfd, &nodes[i].id, &b)) {
+ missing++;
+ continue;
+ }
+ total_anon += b.anon;
+ }
+
+ ASSERT_EQ(missing, 0, "all cgroups present in map");
+ /*
+ * The churners pin a resident anon set for the whole window, so with no
+ * swap and no ancestor memory.max forcing reclaim (the base selftest
+ * config sets neither), the tree always carries anon while churn runs.
+ */
+ ASSERT_GT(total_anon, 0, "tree carries anon under churn");
+}
+
+/* ---- one case ---------------------------------------------------------- */
+
+struct sample_acc {
+ long long file_ns;
+ long long bpf_ns, bpf_ro_ns;
+ int last_got;
+};
+
+struct testcase {
+ const char *name;
+ int fanout;
+ int depth;
+ int churn_fraction; /* one churner per Nth leaf; 1 = all */
+ size_t region_bytes; /* per-iteration churn region */
+ size_t resident_bytes; /* pinned resident set per churner */
+ int samples;
+ int gap_us; /* idle before EACH read: the "staleness" knob (see cases[]) */
+};
+
+static void run_case(const struct testcase *tc)
+{
+ struct memcg_stat_snapshot *tmp = NULL;
+ struct memcg_stat_reader *skel = NULL;
+ struct bpf_link *link = NULL;
+ struct sample_acc acc = {};
+ double f, b, bro;
+ int root_fd = -1;
+ int churners = 0;
+ int bad_walks = 0;
+ int s, w;
+
+ if (!ASSERT_OK(build_tree(tc->fanout, tc->depth, &root_fd), "build tree"))
+ goto out;
+
+ if (start_churners(tc->region_bytes, tc->resident_bytes,
+ tc->churn_fraction))
+ goto out;
+ churners = n_churners;
+
+ skel = memcg_stat_reader__open();
+ if (!ASSERT_OK_PTR(skel, "skel open"))
+ goto out;
+ if (!ASSERT_OK(bpf_map__set_max_entries(skel->maps.results, n_nodes + 8),
+ "set max_entries"))
+ goto out;
+ if (!ASSERT_OK(memcg_stat_reader__load(skel), "skel load"))
+ goto out;
+
+ DECLARE_LIBBPF_OPTS(bpf_iter_attach_opts, opts);
+ union bpf_iter_link_info linfo = {};
+
+ linfo.cgroup.cgroup_fd = root_fd;
+ linfo.cgroup.order = BPF_CGROUP_ITER_DESCENDANTS_PRE;
+ opts.link_info = &linfo;
+ opts.link_info_len = sizeof(linfo);
+
+ link = bpf_program__attach_iter(skel->progs.cgroup_memcg_stat_reader,
+ &opts);
+ if (!ASSERT_OK_PTR(link, "attach iter"))
+ goto out;
+
+ tmp = calloc(n_nodes + 8, sizeof(*tmp));
+ if (!ASSERT_OK_PTR(tmp, "calloc tmp"))
+ goto out;
+
+ /*
+ * Authoritative completeness/correctness gate: run once on the freshly
+ * loaded (still empty) map, so missing==0 proves this walk visited every
+ * cgroup. The map is not cleared between the later timed walks, so the
+ * end-of-loop count is only a weaker, informational cross-check.
+ */
+ check_structural(link, skel);
+
+ /* warm caches/vmstats for both paths symmetrically (same gap regime) */
+ for (w = 0; w < WARMUP_ITERS; w++) {
+ long long ro;
+ int got;
+
+ file_pass(tc->gap_us);
+ bpf_pass(link, skel, tmp, &ro, &got, NULL, tc->gap_us);
+ }
+
+ /*
+ * Timed samples. Every read idles tc->gap_us (untimed) first, so the
+ * tree re-accumulates a roughly fixed amount of churn and each read
+ * starts from approximately the same state, paying its own rstat flush
+ * inside the timed region. The file/bpf order is flipped on odd samples
+ * so any residual jitter doesn't systematically favour whichever reader
+ * runs first.
+ */
+ for (s = 0; s < tc->samples; s++) {
+ long long ro;
+ int got, werr;
+
+ if (s & 1) {
+ acc.bpf_ns += bpf_pass(link, skel, tmp, &ro, &got, &werr, tc->gap_us);
+ acc.bpf_ro_ns += ro;
+ acc.last_got = got;
+ bad_walks += !!werr;
+ acc.file_ns += file_pass(tc->gap_us);
+ } else {
+ acc.file_ns += file_pass(tc->gap_us);
+ acc.bpf_ns += bpf_pass(link, skel, tmp, &ro, &got, &werr, tc->gap_us);
+ acc.bpf_ro_ns += ro;
+ acc.last_got = got;
+ bad_walks += !!werr;
+ }
+ }
+
+ f = (double)acc.file_ns / tc->samples / 1000.0;
+ b = (double)acc.bpf_ns / tc->samples / 1000.0;
+ bro = (double)acc.bpf_ro_ns / tc->samples / 1000.0;
+
+ /*
+ * Informational timing diagnostic (captured like any test output: shown
+ * under -v or on failure, not on a normal PASS). The pass/fail verdict
+ * comes solely from the structural checks above.
+ */
+ printf("\n==== memcg_stat_churn: %s ====\n", tc->name);
+ printf("tree: nodes=%d leaves=%d churners=%d fanout=%d depth=%d region=%zuKB resident=%zuKB samples=%d gap=%dms\n",
+ n_nodes, n_leaves, churners, tc->fanout, tc->depth,
+ tc->region_bytes >> 10, tc->resident_bytes >> 10, tc->samples,
+ tc->gap_us / 1000);
+ printf("all times in us (average per full-tree read under churn, full memory.stat field set); ratio = file/bpf; ro = bpf read()-only (no map drain)\n");
+ printf("each read idles gap=%dms first so every read starts from ~the same accumulated churn; the rstat flush is counted in the read\n",
+ tc->gap_us / 1000);
+ printf("file_avg=%.1f bpf_avg=%.1f bpf_ro=%.1f ratio(file/bpf)=%.2fx\n",
+ f, b, bro, b > 0 ? f / b : 0.0);
+ printf("per-cgroup: file avg=%.0f ns bpf avg=%.0f ns\n",
+ f * 1000.0 / n_nodes, b * 1000.0 / n_nodes);
+ printf("bpf entries produced: %d (expected %d)\n", acc.last_got, n_nodes);
+ printf("RATIO (under churn): file/bpf = %.2fx\n", b > 0 ? f / b : 0.0);
+
+ ASSERT_EQ(bad_walks, 0, "all timed bpf walks completed");
+ ASSERT_EQ(acc.last_got, n_nodes, "bpf visited whole subtree under churn");
+
+out:
+ free(tmp);
+ bpf_link__destroy(link);
+ memcg_stat_reader__destroy(skel);
+ if (root_fd >= 0)
+ close(root_fd);
+ stop_churners(); /* reap churners so leaves become removable */
+
+ /*
+ * Remove the subtree in reverse creation order. Nodes are recorded in
+ * DFS pre-order (a parent precedes all its descendants), so iterating
+ * backwards removes every child before its parent.
+ */
+ if (nodes) {
+ int i;
+
+ for (i = n_nodes - 1; i >= 0; i--)
+ remove_cgroup(nodes[i].rel);
+ free(nodes);
+ nodes = NULL;
+ }
+}
+
+/*
+ * gap_us: idle time inserted (untimed) before every read so the tree
+ * re-accumulates a roughly fixed amount of dirty rstat first; the read then
+ * pays that flush inside its timed region. This gives all four reads
+ * (file/bpf x matched/full) approximately the same start state and folds the
+ * flush cost into the measured time. It is the "staleness / poll-interval"
+ * knob: larger gap -> larger common flush -> the file/bpf ratio compresses.
+ * Pick it past the point where the flush cost saturates; validate by checking
+ * that bpf matched <= bpf full is restored and that doubling it barely moves
+ * the numbers. 50 ms is a reasonable default here.
+ */
+#define CHURN_GAP_US (50 * 1000)
+
+static const struct testcase cases[] = {
+ /*
+ * Both cases use a large (1111-cgroup) tree, where the whole-tree read is
+ * big enough that its cost dominates the rstat flush and scheduler jitter,
+ * so the reported ratios are reproducible run to run; on a small (tens of
+ * cgroups) tree the sub-millisecond BPF read is swamped by that noise and
+ * the ratio bounces. They differ only in churn density -- large_dense
+ * churns one leaf in 4, large_sparse one in 8 -- which changes how much of
+ * the tree the shared flush has to touch. samples are kept even so the
+ * file/bpf order-alternation (s & 1) cancels residual first-mover bias.
+ */
+ /* name fan dep frac region resident samp gap */
+ { "large_dense", 10, 3, 4, 256 << 10, 128 << 10, 8, CHURN_GAP_US },
+ { "large_sparse", 10, 3, 8, 256 << 10, 128 << 10, 8, CHURN_GAP_US },
+};
+
+/*
+ * The memcg kfuncs the reused BPF program relies on (bpf_get_mem_cgroup et al.)
+ * are built only with CONFIG_MEMCG (mm/bpf_memcontrol.c). On a kernel without
+ * it they are absent from vmlinux BTF and the program fails to load, so probe
+ * for one and skip cleanly rather than reporting a spurious failure.
+ */
+static bool memcg_kfuncs_available(void)
+{
+ struct btf *btf;
+ bool ok;
+
+ btf = btf__load_vmlinux_btf();
+ if (!btf)
+ return false;
+ ok = btf__find_by_name_kind(btf, "bpf_get_mem_cgroup", BTF_KIND_FUNC) > 0;
+ btf__free(btf);
+ return ok;
+}
+
+void serial_test_memcg_stat_churn(void)
+{
+ int i;
+
+ if (!memcg_kfuncs_available()) {
+ test__skip();
+ return;
+ }
+
+ if (!ASSERT_OK(setup_cgroup_environment(), "setup cgroup env"))
+ return;
+
+ for (i = 0; i < ARRAY_SIZE(cases); i++) {
+ if (!test__start_subtest(cases[i].name))
+ continue;
+ run_case(&cases[i]);
+ }
+
+ cleanup_cgroup_environment();
+}
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 9+ messages in thread* Re: [PATCH 2/3] selftests/bpf: add memcg_stat_churn BPF-vs-memory.stat benchmark under churn
2026-07-04 4:56 ` [PATCH 2/3] selftests/bpf: add memcg_stat_churn BPF-vs-memory.stat benchmark under churn Ziyang Men
@ 2026-07-04 5:39 ` bot+bpf-ci
0 siblings, 0 replies; 9+ messages in thread
From: bot+bpf-ci @ 2026-07-04 5:39 UTC (permalink / raw)
To: ziyang.meme, ast, daniel, andrii, eddyz87, memxor, bpf
Cc: martin.lau, song, yonghong.song, jolsa, emil, shuah,
roman.gushchin, kernel-team, linux-mm, cgroups, linux-kselftest,
linux-kernel, ziyang.meme, shakeel.butt, ast, andrii, daniel,
martin.lau, eddyz87, yonghong.song, clm, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 5070 bytes --]
> diff --git a/tools/testing/selftests/bpf/prog_tests/memcg_stat_churn.c b/tools/testing/selftests/bpf/prog_tests/memcg_stat_churn.c
> new file mode 100644
> index 000000000000..3e386d0b4c03
> --- /dev/null
> +++ b/tools/testing/selftests/bpf/prog_tests/memcg_stat_churn.c
[ ... ]
> +static long long now_ns(void)
> +{
> + struct timespec t;
> +
> + clock_gettime(CLOCK_MONOTONIC, &t);
> + return (long long)t.tv_sec * 1000000000LL + t.tv_nsec;
> +}
> +
> +/* ---- tree construction (same shape as memcg_stat_reader) --------------- */
> +
> +static struct cg_node *nodes;
> +static int n_nodes;
> +static int n_leaves;
> +
> +static int add_node(const char *rel, bool is_leaf, int *keep_fd)
> +{
> + int fd;
> +
> + fd = create_and_get_cgroup(rel);
> + if (fd < 0)
> + return -1;
> + if (keep_fd)
> + *keep_fd = fd;
> + else
> + close(fd);
> +
> + strncpy(nodes[n_nodes].rel, rel, sizeof(nodes[n_nodes].rel) - 1);
> + nodes[n_nodes].rel[sizeof(nodes[n_nodes].rel) - 1] = '\0';
> + nodes[n_nodes].id = get_cgroup_id(rel);
> + nodes[n_nodes].is_leaf = is_leaf;
> + if (is_leaf)
> + n_leaves++;
> + n_nodes++;
> + return 0;
> +}
> +
> +/* Recursively create children of @rel. @rel must already exist and be recorded. */
> +static int build_children(const char *rel, int fanout, int depth)
> +{
> + char child[128];
> + int i;
> +
> + if (depth == 0)
> + return 0;
> +
> + /* Enable memory on this interior node so its children get a memcg. */
> + if (enable_controllers(rel, "memory"))
> + return -1;
> +
> + for (i = 0; i < fanout; i++) {
> + snprintf(child, sizeof(child), "%s/c%d", rel, i);
> + if (add_node(child, depth == 1, NULL))
> + return -1;
> + if (build_children(child, fanout, depth - 1))
> + return -1;
> + }
> + return 0;
> +}
> +
> +static size_t tree_capacity(int fanout, int depth)
> +{
> + size_t total = 1, level = 1;
> + int d;
> +
> + for (d = 0; d < depth; d++) {
> + level *= fanout;
> + total += level;
> + }
> + return total;
> +}
> +
> +static int build_tree(int fanout, int depth, int *root_fd)
> +{
> + n_nodes = 0;
> + n_leaves = 0;
> + nodes = calloc(tree_capacity(fanout, depth), sizeof(*nodes));
> + if (!nodes)
> + return -1;
> +
> + if (add_node(SUBTREE_ROOT, depth == 0, root_fd))
> + return -1;
> + return build_children(SUBTREE_ROOT, fanout, depth);
> +}
[ ... ]
> +static void parse_stat(char *buf, struct file_snap *o)
> +{
> + char *save, *line;
> +
> + for (line = strtok_r(buf, "\n", &save); line;
> + line = strtok_r(NULL, "\n", &save)) {
> + unsigned long long val;
> + char name[64];
> +
> + if (sscanf(line, "%63s %llu", name, &val) != 2)
> + continue;
> + o->full_sum += val;
> + o->full_fields++;
> + if (!strcmp(name, "anon"))
> + o->anon = val;
> + else if (!strcmp(name, "file"))
> + o->file = val;
> + else if (!strcmp(name, "shmem"))
> + o->shmem = val;
> + else if (!strcmp(name, "file_mapped"))
> + o->file_mapped = val;
> + else if (!strcmp(name, "pgfault"))
> + o->pgfault = val;
> + }
> +}
> +
> +static int file_read_node(const char *rel, struct file_snap *o)
> +{
> + char buf[8192];
> +
> + memset(o, 0, sizeof(*o));
> +
> + if (read_cgroup_file(rel, "memory.stat", buf, sizeof(buf)))
> + return -1;
> + parse_stat(buf, o);
> +
> + if (!read_cgroup_file(rel, "memory.current", buf, sizeof(buf)))
> + o->current = strtoull(buf, NULL, 10);
> + if (!read_cgroup_file(rel, "memory.max", buf, sizeof(buf))) {
> + if (!strncmp(buf, "max", 3))
> + o->max_is_max = true;
> + else
> + o->max = strtoull(buf, NULL, 10);
> + }
> + return 0;
> +}
[ ... ]
> +static int drain_map(int mfd, struct memcg_stat_snapshot *out, int max)
> +{
> + __u64 key = 0, next;
> + int n = 0, err;
> +
> + err = bpf_map_get_next_key(mfd, NULL, &next);
> + while (err == 0) {
> + if (n < max && !bpf_map_lookup_elem(mfd, &next, &out[n])) {
> + sink += out[n].anon + out[n].full_sum;
> + n++;
> + }
> + key = next;
> + err = bpf_map_get_next_key(mfd, &key, &next);
> + }
> + return n;
> +}
[ ... ]
> +static bool memcg_kfuncs_available(void)
> +{
> + struct btf *btf;
> + bool ok;
> +
> + btf = btf__load_vmlinux_btf();
> + if (!btf)
> + return false;
> + ok = btf__find_by_name_kind(btf, "bpf_get_mem_cgroup", BTF_KIND_FUNC) > 0;
> + btf__free(btf);
> + return ok;
> +}
This isn't a bug, but several of these helpers appear nearly identical
to the ones in prog_tests/memcg_stat_reader.c -- tree_capacity(),
build_tree(), add_node(), build_children(), drain_map(), bpf_walk_once(),
file_read_node(), parse_stat(), now_ns(), and memcg_kfuncs_available().
The commit message notes the BPF program, map, and snapshot struct are
reused, but the userspace tree-construction and file-reader helpers are
duplicated rather than shared. Could the common pieces move into a small
shared header so the two tests don't drift apart over time?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/28695985027
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH 3/3] selftests/bpf: add memcg_stat_churn_percpu BPF-vs-memory.stat benchmark under cross-CPU churn
2026-07-04 4:56 [PATCH 0/3] selftests/bpf: compare BPF and memory.stat memcg stat readers Ziyang Men
2026-07-04 4:56 ` [PATCH 1/3] selftests/bpf: add memcg_stat_reader BPF-vs-memory.stat benchmark Ziyang Men
2026-07-04 4:56 ` [PATCH 2/3] selftests/bpf: add memcg_stat_churn BPF-vs-memory.stat benchmark under churn Ziyang Men
@ 2026-07-04 4:56 ` Ziyang Men
2026-07-04 5:58 ` bot+bpf-ci
2026-07-07 0:17 ` [PATCH 0/3] selftests/bpf: compare BPF and memory.stat memcg stat readers Eduard Zingerman
3 siblings, 1 reply; 9+ messages in thread
From: Ziyang Men @ 2026-07-04 4:56 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
Eduard Zingerman, Kumar Kartikeya Dwivedi, bpf
Cc: Martin KaFai Lau, Song Liu, Yonghong Song, Jiri Olsa,
Emil Tsalapatis, Shuah Khan, Roman Gushchin, kernel-team,
linux-mm, cgroups, linux-kselftest, linux-kernel, Ziyang Men,
Shakeel Butt
Add a test_progs selftest that extends memcg_stat_churn by making the
per-cgroup cross-CPU rstat flush fan-out an explicit, swept variable, and
compares the same two ways of reading memory-cgroup statistics under it --
the traditional per-cgroup memory.stat parse and a single
SEC("iter.s/cgroup") BPF walk.
Both readers flush rstat through the same mem_cgroup_flush_stats() path, and
the cost of that flush grows with the number of (cgroup, cpu) pairs that have
pending updates. Where memcg_stat_churn dirties each cgroup on essentially
one CPU (a read flushes one per-cpu tree per cgroup), this test makes K -- the
number of CPUs a cgroup is dirtied on -- a first-class knob: a read of a
cgroup dirtied on K CPUs must visit K per-cpu trees for it.
Load model. It forks one "churner" process per hot leaf and, rather than
pinning each to a single CPU, has each churner migrate its own affinity
round-robin across K CPUs, doing one mmap()+memset()+munmap() on each. The
charge/uncharge happens on whatever CPU the task currently runs on, so cycling
the affinity queues that leaf's rstat dirty on all K CPUs; those per-cpu
entries persist until flushed, so between two reads the leaf ends up dirty on
all K CPUs. The BPF program, its hash map and the snapshot struct are REUSED
verbatim from memcg_stat_reader (progs/memcg_stat_reader.c +
memcg_stat_reader.h); only the userspace load model and sampling loop are new,
so there is no new BPF object and no Makefile change.
Two things keep the measurement honest under this load. One CPU is reserved
for the reader and the parent (which does the timed reads) is pinned to it, so
the timed reads run on a churn-free CPU and are not preempted by the churn --
measuring the reader on a contended CPU swamps a whole-tree walk with
scheduler latency rather than read work. And before every timed read the
parent does an untimed whole-subtree flush (settle_flush()) and then idles a
fixed gap, so each read re-accumulates exactly gap-worth of churn and pays its
own flush inside the timed region regardless of read ordering; the file/BPF
order is also alternated across the (even) sample count.
Because stats are a moving target under churn, the test does not do a
field-by-field BPF-vs-file equality check (that is memcg_stat_reader's job).
Pass/fail gates only on structural sanity: a walk on the freshly loaded map
must visit every cgroup (missing == 0), every timed walk must complete, and
the tree must carry some anon charge. The timing table and RATIO line are
informational diagnostics, captured like any other test output (shown under
-v or on failure, never on a normal PASS).
The narrow/wide/widest subtests run on a large (1111-cgroup) tree and churn a
fixed set of 64 hot leaves, sweeping only K. The large tree keeps the
whole-tree read cost large enough to dominate the rstat flush and scheduler
jitter, so the reported ratios are reproducible; on a small (tens of cgroups)
tree the sub-millisecond BPF read is swamped by that noise and the ratio
bounces run to run.
The file path cost is dominated by per-cgroup VFS open/read and string parsing,
so the flush is a small fraction of it and it stays roughly flat in K. The BPF
path avoids the per-cgroup syscalls and parsing, so the flush is a larger
fraction of it and it grows with K; the file/BPF ratio therefore compresses as
K grows, most visibly at the largest fan-out, but BPF stays many times faster
throughout.
Sample output (v7.1 VM, 60 CPUs); times are us, average per full-tree read
under churn reading the full memory.stat field set; ratio = file/bpf; ro = bpf
read()-only (no map drain):
==== memcg_stat_churn_percpu: narrow ====
tree: nodes=1111 leaves=1000 hot_leaves=64 cpus_per_leaf=1 dirty_pairs=64 cpus=60 reserved=1 churners=64 ...
file_avg=358204.7 bpf_avg=15889.0 bpf_ro=7864.0 ratio(file/bpf)=22.54x
==== memcg_stat_churn_percpu: wide ====
tree: nodes=1111 leaves=1000 hot_leaves=64 cpus_per_leaf=8 dirty_pairs=512 cpus=60 reserved=1 churners=64 ...
file_avg=246735.7 bpf_avg=10719.8 bpf_ro=5719.0 ratio(file/bpf)=23.02x
==== memcg_stat_churn_percpu: widest ====
tree: nodes=1111 leaves=1000 hot_leaves=64 cpus_per_leaf=59 dirty_pairs=3776 cpus=60 reserved=1 churners=64 ...
file_avg=260315.6 bpf_avg=18734.4 bpf_ro=14129.8 ratio(file/bpf)=13.90x
The file/BPF ratio holds around 22-23x through K=8 and compresses to ~14x at
K=59, where each of the 64 hot cgroups is dirty on all 59 churner CPUs (3776
per-cpu trees flushed in a single read). The K=1 vs K=8 step is within
run-to-run noise: on a 1111-cgroup read the flush of 64 vs 512 per-cpu trees is
a small fraction of the walk, so the fan-out only bites clearly at the largest K.
This builds on the memcg BPF kfuncs and completes the memcg_stat_reader /
memcg_stat_churn selftest family.
Suggested-by: Shakeel Butt <shakeel.butt@linux.dev>
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ziyang Men <ziyang.meme@gmail.com>
---
.../bpf/prog_tests/memcg_stat_churn_percpu.c | 902 ++++++++++++++++++
1 file changed, 902 insertions(+)
create mode 100644 tools/testing/selftests/bpf/prog_tests/memcg_stat_churn_percpu.c
diff --git a/tools/testing/selftests/bpf/prog_tests/memcg_stat_churn_percpu.c b/tools/testing/selftests/bpf/prog_tests/memcg_stat_churn_percpu.c
new file mode 100644
index 000000000000..16c3f261c878
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/memcg_stat_churn_percpu.c
@@ -0,0 +1,902 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+
+/*
+ * memcg_stat_churn_percpu
+ * =======================
+ * A CPU-spread variant of memcg_stat_churn. It measures the same two whole-tree
+ * memcg-stat readers -- the traditional per-cgroup memory.stat parse and a single
+ * SEC("iter.s/cgroup") BPF walk -- under continuous allocation churn, but this
+ * time each churned cgroup is deliberately dirtied on MANY CPUs at once:
+ *
+ * - Build a synthetic cgroup subtree (fanout x depth), same as the reader.
+ * - Reserve one CPU for the reader and pin the reader (this parent) to it, so
+ * the timed reads run on a churn-free CPU and are not preempted by the load
+ * (measuring on a contended CPU swamps a short walk with scheduler latency).
+ * - Fork one "churner" process per hot leaf (tc->churn_leaves; 0 = all leaves,
+ * spread evenly across the tree). Each churner joins its leaf and then, in a
+ * loop, migrates its own affinity round-robin across K CPUs
+ * (tc->cpus_per_leaf; 0 = all churner CPUs) doing one mmap()+memset()+
+ * munmap() on each. The charge/uncharge happens on whatever CPU the task is
+ * currently running on, so cycling the affinity queues this leaf's rstat
+ * dirty on all K CPUs; those per-cpu entries persist until flushed, so
+ * between two reads the leaf ends up dirty on all K CPUs.
+ * - While the churn runs, the parent repeatedly SAMPLES both readers exactly as
+ * in memcg_stat_churn: settle_flush() then an untimed gap before each read so
+ * each read starts from exactly gap-worth of churn and pays its own flush
+ * inside the timed region; the file/BPF order is alternated.
+ *
+ * Why this matters: both readers flush rstat through the same
+ * mem_cgroup_flush_stats() path, and the cost of that flush grows with the
+ * number of (cgroup, cpu) pairs that have pending updates. Where memcg_stat_churn
+ * dirties each cgroup on essentially one CPU (a read flushes one per-cpu tree per
+ * cgroup), this test makes K a first-class knob: a read of a cgroup dirtied on K
+ * CPUs must visit K per-cpu trees. Sweeping K (see the narrow/wide/widest cases)
+ * drives the shared flush cost F up and compresses the file/BPF ratio, isolating
+ * the effect of per-cgroup cross-CPU fan-out. Because the reader runs on a
+ * reserved, churn-free CPU, the flush cost -- not scheduler jitter -- is what the
+ * timed reads capture; the ratio is robust because the flush hits both readers
+ * equally.
+ *
+ * The BPF program, its hash map and the snapshot struct are REUSED verbatim from
+ * memcg_stat_reader (progs/memcg_stat_reader.c + memcg_stat_reader.h); only the
+ * userspace load model (CPU-pinned churners) and sampling loop are new here.
+ *
+ * Under churn the stats are a moving target, so this test does NOT do a
+ * field-by-field BPF-vs-file equality check (that is memcg_stat_reader's job).
+ * Pass/fail gates only on structural sanity -- the iterator visited every cgroup
+ * and the tree carries some anon charge. The timing table and final RATIO line
+ * are informational diagnostics, printed like any other test output (i.e. under
+ * -v or on failure, never on a normal PASS).
+ */
+#ifndef _GNU_SOURCE
+#define _GNU_SOURCE /* sched_setaffinity(), CPU_SET() (lib.mk also -D's it) */
+#endif
+#include <test_progs.h>
+#include <bpf/libbpf.h>
+#include <bpf/btf.h>
+#include <sched.h>
+#include <stdlib.h>
+#include <string.h>
+#include <time.h>
+#include <unistd.h>
+#include <sys/mman.h>
+#include <sys/wait.h>
+#include "cgroup_helpers.h"
+#include "memcg_stat_reader.h"
+#include "memcg_stat_reader.skel.h"
+
+#define SUBTREE_ROOT "/mcg_pcpu"
+
+#define WARMUP_ITERS 2
+
+struct cg_node {
+ char rel[128];
+ __u64 id;
+ bool is_leaf;
+};
+
+/* Field subset parsed from memory.stat (mirrors memcg_stat_reader). */
+struct file_snap {
+ __u64 anon, file, shmem, file_mapped, pgfault;
+ __u64 current; /* memory.current, bytes */
+ __u64 max; /* memory.max, bytes (valid unless max_is_max) */
+ __u64 full_sum;
+ __u32 full_fields;
+ bool max_is_max;
+};
+
+static volatile __u64 sink; /* keep the optimizer from eliding reads */
+
+static long long now_ns(void)
+{
+ struct timespec t;
+
+ clock_gettime(CLOCK_MONOTONIC, &t);
+ return (long long)t.tv_sec * 1000000000LL + t.tv_nsec;
+}
+
+/* ---- allowed CPU set --------------------------------------------------- */
+
+static int *cpu_list; /* ids of the CPUs this process may run on */
+static int n_cpu; /* number of such CPUs */
+static int n_reserved; /* CPUs held out for the reader (0 or 1) */
+static int reader_cpu = -1; /* CPU the reader is pinned to, or -1 */
+static cpu_set_t orig_affinity; /* parent's affinity, restored on exit */
+
+/*
+ * Collect the CPUs the test is allowed to run on (respecting any cpuset the
+ * harness put us in). We spawn one pinned churner per usable CPU minus the one
+ * reserved for the reader. Bounded by CPU_SETSIZE (1024); machines wider than
+ * that would use only the first CPU_SETSIZE CPUs, which is fine for a diagnostic.
+ */
+static int collect_cpus(void)
+{
+ cpu_set_t set;
+ int i, want, n = 0;
+
+ CPU_ZERO(&set);
+ if (sched_getaffinity(0, sizeof(set), &set))
+ return -1;
+ orig_affinity = set; /* restored in test teardown */
+ want = CPU_COUNT(&set);
+ if (want <= 0)
+ return -1;
+ cpu_list = calloc(want, sizeof(*cpu_list));
+ if (!cpu_list)
+ return -1;
+ for (i = 0; i < CPU_SETSIZE && n < want; i++)
+ if (CPU_ISSET(i, &set))
+ cpu_list[n++] = i;
+ n_cpu = n;
+ return 0;
+}
+
+/* ---- tree construction (same shape as memcg_stat_reader) --------------- */
+
+static struct cg_node *nodes;
+static int n_nodes;
+static int n_leaves;
+
+static int add_node(const char *rel, bool is_leaf, int *keep_fd)
+{
+ int fd;
+
+ fd = create_and_get_cgroup(rel);
+ if (fd < 0)
+ return -1;
+ if (keep_fd)
+ *keep_fd = fd;
+ else
+ close(fd);
+
+ strncpy(nodes[n_nodes].rel, rel, sizeof(nodes[n_nodes].rel) - 1);
+ nodes[n_nodes].rel[sizeof(nodes[n_nodes].rel) - 1] = '\0';
+ nodes[n_nodes].id = get_cgroup_id(rel);
+ nodes[n_nodes].is_leaf = is_leaf;
+ if (is_leaf)
+ n_leaves++;
+ n_nodes++;
+ return 0;
+}
+
+/* Recursively create children of @rel. @rel must already exist and be recorded. */
+static int build_children(const char *rel, int fanout, int depth)
+{
+ char child[128];
+ int i;
+
+ if (depth == 0)
+ return 0;
+
+ /* Enable memory on this interior node so its children get a memcg. */
+ if (enable_controllers(rel, "memory"))
+ return -1;
+
+ for (i = 0; i < fanout; i++) {
+ snprintf(child, sizeof(child), "%s/c%d", rel, i);
+ if (add_node(child, depth == 1, NULL))
+ return -1;
+ if (build_children(child, fanout, depth - 1))
+ return -1;
+ }
+ return 0;
+}
+
+static size_t tree_capacity(int fanout, int depth)
+{
+ size_t total = 1, level = 1;
+ int d;
+
+ for (d = 0; d < depth; d++) {
+ level *= fanout;
+ total += level;
+ }
+ return total;
+}
+
+static int build_tree(int fanout, int depth, int *root_fd)
+{
+ n_nodes = 0;
+ n_leaves = 0;
+ nodes = calloc(tree_capacity(fanout, depth), sizeof(*nodes));
+ if (!nodes)
+ return -1;
+
+ if (add_node(SUBTREE_ROOT, depth == 0, root_fd))
+ return -1;
+ return build_children(SUBTREE_ROOT, fanout, depth);
+}
+
+/* ---- churn load (migrating churners, K CPUs per hot cgroup) ------------- */
+
+/*
+ * Shared control block, mmap'd MAP_SHARED before the forks so the parent can
+ * signal all churners to stop with a single write.
+ */
+struct churn_ctl {
+ volatile int stop;
+};
+
+static struct churn_ctl *ctl;
+static pid_t *churn_pids;
+static int n_churners;
+static int n_hot_leaves; /* distinct cgroups (leaves) churned */
+static int n_cpus_per_leaf; /* K: CPUs each hot cgroup is dirtied on */
+static int churn_ready[2] = { -1, -1 }; /* churner -> parent "ready" barrier */
+
+/* Pin the calling task to a single CPU. */
+static int pin_cpu(int cpu)
+{
+ cpu_set_t set;
+
+ CPU_ZERO(&set);
+ CPU_SET(cpu, &set);
+ return sched_setaffinity(0, sizeof(set), &set);
+}
+
+/*
+ * One churner process, dedicated to a single hot leaf but spread over K CPUs.
+ * It joins its leaf, pins a resident anon set so the tree always carries some
+ * charge, signals readiness, then loops: migrate to the next of its K CPUs and
+ * do one mmap()+memset()+munmap() there. The charge/uncharge happens on
+ * whatever CPU the task currently runs on, so cycling the affinity queues this
+ * leaf's rstat dirty on all K CPUs; those per-cpu entries persist until flushed,
+ * so between two reads the leaf ends up dirty on all K CPUs and a reader's flush
+ * of the subtree must visit K per-cpu trees for this one cgroup. Never returns.
+ *
+ * @base is this churner's starting index into the churner CPU pool
+ * (cpu_list[n_reserved ..]); its K CPUs are (base + 0..K-1) mod pool size.
+ */
+static void churner_child(const struct cg_node *leaf, int base, int k,
+ size_t region_bytes, size_t resident_bytes)
+{
+ int c_pool = n_cpu - n_reserved;
+ void *resident;
+ int j = 0;
+
+ close(churn_ready[0]);
+
+ /*
+ * Move onto our first CPU before charging. Children inherit the reader's
+ * reserved-CPU affinity from the parent, so without this the resident set
+ * would be charged on the reader's CPU.
+ */
+ if (pin_cpu(cpu_list[n_reserved + base % c_pool]))
+ _exit(4);
+
+ /*
+ * cgroup_helpers builds paths from getpid(); in this forked child that
+ * differs from the parent that built the tree, so use the _parent
+ * (getppid()) variant to resolve the leaf under the parent's work dir.
+ */
+ if (join_parent_cgroup(leaf->rel))
+ _exit(1);
+
+ resident = mmap(NULL, resident_bytes, PROT_READ | PROT_WRITE,
+ MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
+ if (resident == MAP_FAILED)
+ _exit(2);
+ memset(resident, 1, resident_bytes); /* fault in, keep mapped */
+
+ if (write(churn_ready[1], "x", 1) != 1)
+ _exit(3);
+ close(churn_ready[1]); /* so a sibling's early death yields EOF, not a parent hang */
+
+ while (!ctl->stop) {
+ void *p;
+
+ /* migrate to the next of our K CPUs, then dirty the leaf there */
+ pin_cpu(cpu_list[n_reserved + (base + j) % c_pool]);
+ if (++j == k)
+ j = 0;
+
+ p = mmap(NULL, region_bytes, PROT_READ | PROT_WRITE,
+ MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
+ if (p == MAP_FAILED)
+ continue;
+ memset(p, 1, region_bytes); /* fault in -> anon charge */
+ munmap(p, region_bytes); /* free -> uncharge (keeps rstat dirty) */
+ }
+ munmap(resident, resident_bytes);
+ _exit(0);
+}
+
+/*
+ * Fork one migrating churner per hot leaf (H = @churn_leaves, 0 = all leaves),
+ * each spread over K CPUs (@cpus_per_leaf, 0 or > pool => all churner CPUs).
+ * Hot leaves are chosen evenly across the tree. Returns 0 once every churner
+ * has joined its leaf and pinned its resident set (so measurement starts under
+ * steady-state load). On failure the caller's cleanup path calls
+ * stop_churners() to reap whatever was started.
+ */
+static int start_churners(size_t region_bytes, size_t resident_bytes,
+ int churn_leaves, int cpus_per_leaf)
+{
+ int *leaf_idx = NULL, *pool = NULL;
+ int n_leaf_idx = 0, pool_n = 0;
+ int c_pool = n_cpu - n_reserved; /* CPUs available to churners */
+ int k_eff, i, h, ret = -1;
+
+ /* K: CPUs each hot cgroup is dirtied on (0 or too big => all churner CPUs) */
+ k_eff = (cpus_per_leaf > 0 && cpus_per_leaf < c_pool) ? cpus_per_leaf
+ : c_pool;
+ n_cpus_per_leaf = k_eff;
+
+ /* gather all leaf node indices in creation order */
+ leaf_idx = calloc(n_leaves, sizeof(*leaf_idx));
+ if (!ASSERT_OK_PTR(leaf_idx, "calloc leaf_idx"))
+ return -1;
+ for (i = 0; i < n_nodes; i++)
+ if (nodes[i].is_leaf)
+ leaf_idx[n_leaf_idx++] = i;
+
+ /*
+ * H hot cgroups (one migrating churner each), spread evenly across all
+ * leaves; churn_leaves <= 0 or >= n_leaves => every leaf is hot.
+ */
+ pool_n = (churn_leaves > 0 && churn_leaves < n_leaf_idx) ? churn_leaves
+ : n_leaf_idx;
+ pool = calloc(pool_n, sizeof(*pool));
+ if (!ASSERT_OK_PTR(pool, "calloc pool")) {
+ free(leaf_idx);
+ return -1;
+ }
+ for (i = 0; i < pool_n; i++)
+ pool[i] = leaf_idx[(int)((long long)i * n_leaf_idx / pool_n)];
+ free(leaf_idx);
+ n_hot_leaves = pool_n;
+
+ ctl = mmap(NULL, sizeof(*ctl), PROT_READ | PROT_WRITE,
+ MAP_SHARED | MAP_ANONYMOUS, -1, 0);
+ if (!ASSERT_NEQ(ctl, MAP_FAILED, "mmap churn_ctl")) {
+ ctl = NULL;
+ goto out;
+ }
+ ctl->stop = 0;
+
+ if (!ASSERT_OK(pipe(churn_ready), "pipe churn_ready"))
+ goto out;
+
+ churn_pids = calloc(pool_n, sizeof(*churn_pids));
+ if (!ASSERT_OK_PTR(churn_pids, "calloc churn_pids"))
+ goto out;
+
+ /*
+ * One migrating churner per hot leaf. Churner h spans the K CPUs
+ * (h*K + 0..K-1) mod c_pool of the churner pool (cpu_list[n_reserved ..]),
+ * so different churners start on different CPUs and cpu_list[0] (the
+ * reader's reserved CPU) is never used by a churner.
+ */
+ for (h = 0; h < pool_n; h++) {
+ const struct cg_node *leaf = &nodes[pool[h]];
+ pid_t pid = fork();
+
+ if (pid < 0) {
+ ASSERT_GE(pid, 0, "fork churner");
+ goto out;
+ }
+ if (pid == 0)
+ churner_child(leaf, h * k_eff, k_eff,
+ region_bytes, resident_bytes);
+
+ churn_pids[n_churners++] = pid;
+ }
+
+ /* parent: this end is only for the children to signal on */
+ close(churn_ready[1]);
+ churn_ready[1] = -1;
+
+ /* wait until every churner has joined + pinned its resident set */
+ for (h = 0; h < n_churners; h++) {
+ char c;
+ ssize_t r = read(churn_ready[0], &c, 1);
+
+ if (r == 0)
+ fprintf(stderr,
+ "a churner exited before signaling ready (affinity/join/mmap failure?)\n");
+ if (!ASSERT_EQ(r, 1, "churner ready"))
+ goto out;
+ }
+ ret = 0;
+out:
+ free(pool);
+ return ret;
+}
+
+static void stop_churners(void)
+{
+ int i, status;
+
+ if (ctl)
+ ctl->stop = 1; /* release all churn loops */
+
+ if (churn_ready[1] >= 0) {
+ close(churn_ready[1]);
+ churn_ready[1] = -1;
+ }
+ if (churn_ready[0] >= 0) {
+ close(churn_ready[0]);
+ churn_ready[0] = -1;
+ }
+
+ for (i = 0; i < n_churners; i++) {
+ if (!churn_pids || churn_pids[i] <= 0)
+ continue;
+ if (waitpid(churn_pids[i], &status, 0) == churn_pids[i] &&
+ (!WIFEXITED(status) || WEXITSTATUS(status) != 0))
+ fprintf(stderr,
+ "churner %d exited abnormally (status=0x%x)\n",
+ churn_pids[i], status);
+ }
+
+ free(churn_pids);
+ churn_pids = NULL;
+ n_churners = 0;
+
+ if (ctl) {
+ munmap((void *)ctl, sizeof(*ctl));
+ ctl = NULL;
+ }
+}
+
+/* ---- file (traditional) reader ----------------------------------------- */
+
+static void parse_stat(char *buf, struct file_snap *o)
+{
+ char *save, *line;
+
+ for (line = strtok_r(buf, "\n", &save); line;
+ line = strtok_r(NULL, "\n", &save)) {
+ unsigned long long val;
+ char name[64];
+
+ if (sscanf(line, "%63s %llu", name, &val) != 2)
+ continue;
+ o->full_sum += val;
+ o->full_fields++;
+ if (!strcmp(name, "anon"))
+ o->anon = val;
+ else if (!strcmp(name, "file"))
+ o->file = val;
+ else if (!strcmp(name, "shmem"))
+ o->shmem = val;
+ else if (!strcmp(name, "file_mapped"))
+ o->file_mapped = val;
+ else if (!strcmp(name, "pgfault"))
+ o->pgfault = val;
+ }
+}
+
+static int file_read_node(const char *rel, struct file_snap *o)
+{
+ char buf[8192];
+
+ memset(o, 0, sizeof(*o));
+
+ if (read_cgroup_file(rel, "memory.stat", buf, sizeof(buf)))
+ return -1;
+ parse_stat(buf, o);
+
+ if (!read_cgroup_file(rel, "memory.current", buf, sizeof(buf)))
+ o->current = strtoull(buf, NULL, 10);
+ if (!read_cgroup_file(rel, "memory.max", buf, sizeof(buf))) {
+ if (!strncmp(buf, "max", 3))
+ o->max_is_max = true;
+ else
+ o->max = strtoull(buf, NULL, 10);
+ }
+ return 0;
+}
+
+/*
+ * Untimed whole-subtree flush used to normalise the pre-read state. Reading the
+ * subtree root's memory.stat flushes the entire subtree's rstat, so the
+ * usleep(gap) that follows re-accumulates *exactly* gap-worth of churn no matter
+ * what the previous timed read was. Without this reset the accumulation window
+ * would be (previous_read_duration + gap), and since a file pass is ~20x longer
+ * than a BPF walk that made the flush a BPF read pays depend on ordering -- a
+ * ~15% window asymmetry, enough to invert bpf_matched vs bpf_full on
+ * flush-dominated cases (e.g. "hot": few cgroups churned from many CPUs).
+ * nodes[0] is SUBTREE_ROOT (added first in build_tree), whose memcg covers the
+ * whole tree, so one read here flushes every node the timed reads care about.
+ */
+static void settle_flush(void)
+{
+ char buf[8192];
+
+ if (nodes && n_nodes > 0)
+ read_cgroup_file(nodes[0].rel, "memory.stat", buf, sizeof(buf));
+}
+
+/*
+ * One timed traditional pass over the whole tree; returns nanoseconds. A
+ * settle_flush() then an untimed @gap_us idle precede the pass so the tree
+ * re-accumulates exactly gap-worth of churn first; the resulting rstat flush is
+ * then paid inside the timed region, giving every read the same start state.
+ */
+static long long file_pass(int gap_us)
+{
+ struct file_snap s;
+ long long t0;
+ int i;
+
+ settle_flush();
+ if (gap_us)
+ usleep(gap_us);
+ t0 = now_ns();
+ for (i = 0; i < n_nodes; i++) {
+ file_read_node(nodes[i].rel, &s);
+ sink += s.anon + s.full_sum;
+ }
+ return now_ns() - t0;
+}
+
+/* ---- BPF reader -------------------------------------------------------- */
+
+static int bpf_walk_once(struct bpf_link *link)
+{
+ char buf[4096];
+ ssize_t r;
+ int fd;
+
+ fd = bpf_iter_create(bpf_link__fd(link));
+ if (fd < 0)
+ return -1;
+ while ((r = read(fd, buf, sizeof(buf))) > 0)
+ ;
+ close(fd);
+ return r == 0 ? 0 : -1;
+}
+
+static int drain_map(int mfd, struct memcg_stat_snapshot *out, int max)
+{
+ __u64 key = 0, next;
+ int n = 0, err;
+
+ err = bpf_map_get_next_key(mfd, NULL, &next);
+ while (err == 0) {
+ if (n < max && !bpf_map_lookup_elem(mfd, &next, &out[n])) {
+ sink += out[n].anon + out[n].full_sum;
+ n++;
+ }
+ key = next;
+ err = bpf_map_get_next_key(mfd, &key, &next);
+ }
+ return n;
+}
+
+/*
+ * One timed BPF pass: kernel walk (ro) + map drain into userspace. Returns the
+ * total nanoseconds; *ro_ns gets the walk-only time, *got the entries drained.
+ * A settle_flush() then an untimed @gap_us idle precede the walk, exactly as in
+ * file_pass(), so the rstat flush the walk pays reflects the same gap-worth of
+ * accumulated churn regardless of read ordering.
+ */
+static long long bpf_pass(struct bpf_link *link, struct memcg_stat_reader *skel,
+ struct memcg_stat_snapshot *tmp,
+ long long *ro_ns, int *got, int *werr, int gap_us)
+{
+ int mfd = bpf_map__fd(skel->maps.results);
+ long long t0, t1, t2;
+ int err;
+
+ skel->bss->collect_full = 1;
+
+ settle_flush();
+ if (gap_us)
+ usleep(gap_us);
+ t0 = now_ns();
+ err = bpf_walk_once(link);
+ t1 = now_ns();
+ *got = drain_map(mfd, tmp, n_nodes + 8);
+ t2 = now_ns();
+
+ if (werr)
+ *werr = err;
+ *ro_ns = t1 - t0;
+ return t2 - t0;
+}
+
+/* ---- structural sanity (no field-by-field check under churn) ------------ */
+
+static void check_structural(struct bpf_link *link,
+ struct memcg_stat_reader *skel)
+{
+ int mfd = bpf_map__fd(skel->maps.results);
+ __u64 total_anon = 0;
+ int i, missing = 0;
+
+ skel->bss->collect_full = 0;
+ if (!ASSERT_OK(bpf_walk_once(link), "bpf walk"))
+ return;
+
+ for (i = 0; i < n_nodes; i++) {
+ struct memcg_stat_snapshot b;
+
+ if (bpf_map_lookup_elem(mfd, &nodes[i].id, &b)) {
+ missing++;
+ continue;
+ }
+ total_anon += b.anon;
+ }
+
+ ASSERT_EQ(missing, 0, "all cgroups present in map");
+ /*
+ * The churners pin a resident anon set for the whole window, so with no
+ * swap and no ancestor memory.max forcing reclaim (the base selftest
+ * config sets neither), the tree always carries anon while churn runs.
+ */
+ ASSERT_GT(total_anon, 0, "tree carries anon under churn");
+}
+
+/* ---- one case ---------------------------------------------------------- */
+
+struct sample_acc {
+ long long file_ns;
+ long long bpf_ns, bpf_ro_ns;
+ int last_got;
+};
+
+struct testcase {
+ const char *name;
+ int fanout;
+ int depth;
+ int churn_leaves; /* H: # hot cgroups (one migrating churner each); 0 = all leaves */
+ int cpus_per_leaf; /* K: CPUs each hot cgroup is dirtied on; 0 = all churner CPUs */
+ size_t region_bytes; /* per-iteration churn region */
+ size_t resident_bytes; /* pinned resident set per churner */
+ int samples;
+ int gap_us; /* idle before EACH read: the "staleness" knob (see cases[]) */
+};
+
+static void run_case(const struct testcase *tc)
+{
+ struct memcg_stat_snapshot *tmp = NULL;
+ struct memcg_stat_reader *skel = NULL;
+ struct bpf_link *link = NULL;
+ struct sample_acc acc = {};
+ double f, b, bro;
+ int root_fd = -1;
+ int churners = 0;
+ int bad_walks = 0;
+ int s, w;
+
+ if (!ASSERT_OK(build_tree(tc->fanout, tc->depth, &root_fd), "build tree"))
+ goto out;
+
+ if (start_churners(tc->region_bytes, tc->resident_bytes,
+ tc->churn_leaves, tc->cpus_per_leaf))
+ goto out;
+ churners = n_churners;
+
+ skel = memcg_stat_reader__open();
+ if (!ASSERT_OK_PTR(skel, "skel open"))
+ goto out;
+ if (!ASSERT_OK(bpf_map__set_max_entries(skel->maps.results, n_nodes + 8),
+ "set max_entries"))
+ goto out;
+ if (!ASSERT_OK(memcg_stat_reader__load(skel), "skel load"))
+ goto out;
+
+ DECLARE_LIBBPF_OPTS(bpf_iter_attach_opts, opts);
+ union bpf_iter_link_info linfo = {};
+
+ linfo.cgroup.cgroup_fd = root_fd;
+ linfo.cgroup.order = BPF_CGROUP_ITER_DESCENDANTS_PRE;
+ opts.link_info = &linfo;
+ opts.link_info_len = sizeof(linfo);
+
+ link = bpf_program__attach_iter(skel->progs.cgroup_memcg_stat_reader,
+ &opts);
+ if (!ASSERT_OK_PTR(link, "attach iter"))
+ goto out;
+
+ tmp = calloc(n_nodes + 8, sizeof(*tmp));
+ if (!ASSERT_OK_PTR(tmp, "calloc tmp"))
+ goto out;
+
+ /*
+ * Authoritative completeness/correctness gate: run once on the freshly
+ * loaded (still empty) map, so missing==0 proves this walk visited every
+ * cgroup. The map is not cleared between the later timed walks, so the
+ * end-of-loop count is only a weaker, informational cross-check.
+ */
+ check_structural(link, skel);
+
+ /* warm caches/vmstats for both paths symmetrically (same gap regime) */
+ for (w = 0; w < WARMUP_ITERS; w++) {
+ long long ro;
+ int got;
+
+ file_pass(tc->gap_us);
+ bpf_pass(link, skel, tmp, &ro, &got, NULL, tc->gap_us);
+ }
+
+ /*
+ * Timed samples. Every read settle_flush()es then idles tc->gap_us
+ * (untimed) first, so the tree re-accumulates exactly gap-worth of churn
+ * and each read pays its own rstat flush inside the timed region. The
+ * file/bpf order is flipped on odd samples so any residual jitter doesn't
+ * systematically favour whichever reader runs first.
+ */
+ for (s = 0; s < tc->samples; s++) {
+ long long ro;
+ int got, werr;
+
+ if (s & 1) {
+ acc.bpf_ns += bpf_pass(link, skel, tmp, &ro, &got, &werr, tc->gap_us);
+ acc.bpf_ro_ns += ro;
+ acc.last_got = got;
+ bad_walks += !!werr;
+ acc.file_ns += file_pass(tc->gap_us);
+ } else {
+ acc.file_ns += file_pass(tc->gap_us);
+ acc.bpf_ns += bpf_pass(link, skel, tmp, &ro, &got, &werr, tc->gap_us);
+ acc.bpf_ro_ns += ro;
+ acc.last_got = got;
+ bad_walks += !!werr;
+ }
+ }
+
+ f = (double)acc.file_ns / tc->samples / 1000.0;
+ b = (double)acc.bpf_ns / tc->samples / 1000.0;
+ bro = (double)acc.bpf_ro_ns / tc->samples / 1000.0;
+
+ /*
+ * Informational timing diagnostic (captured like any test output: shown
+ * under -v or on failure, not on a normal PASS). The pass/fail verdict
+ * comes solely from the structural checks above.
+ */
+ printf("\n==== memcg_stat_churn_percpu: %s ====\n", tc->name);
+ printf("tree: nodes=%d leaves=%d hot_leaves=%d cpus_per_leaf=%d dirty_pairs=%d cpus=%d reserved=%d churners=%d fanout=%d depth=%d region=%zuKB resident=%zuKB samples=%d gap=%dms\n",
+ n_nodes, n_leaves, n_hot_leaves, n_cpus_per_leaf,
+ n_hot_leaves * n_cpus_per_leaf, n_cpu, n_reserved, churners,
+ tc->fanout, tc->depth, tc->region_bytes >> 10,
+ tc->resident_bytes >> 10, tc->samples, tc->gap_us / 1000);
+ printf("each hot cgroup churned across %d CPUs (migrating churner) so a reader flush visits ~%d per-cpu trees per hot cgroup; reader pinned to reserved CPU %d\n",
+ n_cpus_per_leaf, n_cpus_per_leaf, reader_cpu);
+ printf("all times in us (average per full-tree read under churn, full memory.stat field set); ratio = file/bpf; ro = bpf read()-only (no map drain)\n");
+ printf("each read flushes then idles gap=%dms so every read starts from exactly gap-worth of churn; the rstat flush is counted in the read\n",
+ tc->gap_us / 1000);
+ printf("file_avg=%.1f bpf_avg=%.1f bpf_ro=%.1f ratio(file/bpf)=%.2fx\n",
+ f, b, bro, b > 0 ? f / b : 0.0);
+ printf("per-cgroup: file avg=%.0f ns bpf avg=%.0f ns\n",
+ f * 1000.0 / n_nodes, b * 1000.0 / n_nodes);
+ printf("bpf entries produced: %d (expected %d)\n", acc.last_got, n_nodes);
+ printf("RATIO (%d CPUs/cgroup): file/bpf = %.2fx\n",
+ n_cpus_per_leaf, b > 0 ? f / b : 0.0);
+
+ ASSERT_EQ(bad_walks, 0, "all timed bpf walks completed");
+ ASSERT_EQ(acc.last_got, n_nodes, "bpf visited whole subtree under churn");
+
+out:
+ free(tmp);
+ bpf_link__destroy(link);
+ memcg_stat_reader__destroy(skel);
+ if (root_fd >= 0)
+ close(root_fd);
+ stop_churners(); /* reap churners so leaves become removable */
+
+ /*
+ * Remove the subtree in reverse creation order. Nodes are recorded in
+ * DFS pre-order (a parent precedes all its descendants), so iterating
+ * backwards removes every child before its parent.
+ */
+ if (nodes) {
+ int i;
+
+ for (i = n_nodes - 1; i >= 0; i--)
+ remove_cgroup(nodes[i].rel);
+ free(nodes);
+ nodes = NULL;
+ }
+}
+
+/*
+ * gap_us: idle time inserted (untimed) before every read so the tree
+ * re-accumulates a roughly fixed amount of dirty rstat first; the read then
+ * pays that flush inside its timed region. This gives all four reads
+ * (file/bpf x matched/full) approximately the same start state and folds the
+ * flush cost into the measured time. It is the "staleness / poll-interval"
+ * knob: larger gap -> larger common flush -> the file/bpf ratio compresses.
+ * See memcg_stat_churn for the full rationale; 50 ms is a reasonable default.
+ */
+#define CHURN_GAP_US (50 * 1000)
+
+static const struct testcase cases[] = {
+ /*
+ * The narrow/wide/widest trio runs on a large (1111-cgroup) tree and
+ * churns a fixed set of 64 hot leaves, sweeping only K, the number of CPUs
+ * each hot cgroup is dirtied on, to isolate per-cgroup cross-CPU flush
+ * fan-out:
+ *
+ * narrow - K=1 : each hot cgroup dirty on 1 CPU (64 x 1 dirty pairs).
+ * wide - K=8 : each hot cgroup dirty on 8 CPUs (64 x 8 dirty pairs).
+ * widest - K=all: each hot cgroup dirty on every churner CPU (64 x cpus).
+ *
+ * The hot-cgroup count (64) is identical across all three, so only the
+ * per-cgroup CPU fan-out changes. As K grows the shared rstat flush F grows
+ * (more per-cpu trees to visit), so both readers' cost rises and the
+ * file/bpf ratio compresses toward 1; widest is the most conservative
+ * regime. The large tree keeps the whole-tree read cost dominant over the
+ * flush/scheduler jitter, so the ratios are reproducible (a small tree makes
+ * the sub-millisecond BPF read too noisy to compare). samples are kept even
+ * so the file/bpf order-alternation (s & 1) cancels first-mover bias; widest
+ * gets more samples as its bigger flush has more variance.
+ */
+ /* name fan dep H K region resident samp gap */
+ { "narrow", 10, 3, 64, 1, 256 << 10, 128 << 10, 8, CHURN_GAP_US },
+ { "wide", 10, 3, 64, 8, 256 << 10, 128 << 10, 8, CHURN_GAP_US },
+ { "widest", 10, 3, 64, 0, 256 << 10, 128 << 10, 10, CHURN_GAP_US },
+};
+
+/*
+ * The memcg kfuncs the reused BPF program relies on (bpf_get_mem_cgroup et al.)
+ * are built only with CONFIG_MEMCG (mm/bpf_memcontrol.c). On a kernel without
+ * it they are absent from vmlinux BTF and the program fails to load, so probe
+ * for one and skip cleanly rather than reporting a spurious failure.
+ */
+static bool memcg_kfuncs_available(void)
+{
+ struct btf *btf;
+ bool ok;
+
+ btf = btf__load_vmlinux_btf();
+ if (!btf)
+ return false;
+ ok = btf__find_by_name_kind(btf, "bpf_get_mem_cgroup", BTF_KIND_FUNC) > 0;
+ btf__free(btf);
+ return ok;
+}
+
+/*
+ * Reserve one CPU for the reader (this parent) and pin the parent to it, so its
+ * timed reads run on a CPU that carries no churner. Needs at least 2 CPUs; on a
+ * single-CPU host we skip reserving and the one churner shares the CPU with the
+ * reader (noisier, but 1-CPU hosts are not the target). Best-effort: if the
+ * parent cannot be pinned we carry on without a reservation.
+ */
+static void reserve_reader_cpu(void)
+{
+ if (n_cpu < 2)
+ return;
+ if (pin_cpu(cpu_list[0]))
+ return;
+ n_reserved = 1;
+ reader_cpu = cpu_list[0];
+}
+
+void serial_test_memcg_stat_churn_percpu(void)
+{
+ int i;
+
+ if (!memcg_kfuncs_available()) {
+ test__skip();
+ return;
+ }
+
+ if (!ASSERT_OK(collect_cpus(), "collect cpus"))
+ return;
+
+ reserve_reader_cpu();
+
+ if (!ASSERT_OK(setup_cgroup_environment(), "setup cgroup env"))
+ goto restore;
+
+ for (i = 0; i < ARRAY_SIZE(cases); i++) {
+ if (!test__start_subtest(cases[i].name))
+ continue;
+ run_case(&cases[i]);
+ }
+
+ cleanup_cgroup_environment();
+restore:
+ /* undo reserve_reader_cpu() so later test_progs tests keep full affinity */
+ sched_setaffinity(0, sizeof(orig_affinity), &orig_affinity);
+ free(cpu_list);
+ cpu_list = NULL;
+ n_cpu = 0;
+ n_reserved = 0;
+ reader_cpu = -1;
+}
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 9+ messages in thread* Re: [PATCH 3/3] selftests/bpf: add memcg_stat_churn_percpu BPF-vs-memory.stat benchmark under cross-CPU churn
2026-07-04 4:56 ` [PATCH 3/3] selftests/bpf: add memcg_stat_churn_percpu BPF-vs-memory.stat benchmark under cross-CPU churn Ziyang Men
@ 2026-07-04 5:58 ` bot+bpf-ci
0 siblings, 0 replies; 9+ messages in thread
From: bot+bpf-ci @ 2026-07-04 5:58 UTC (permalink / raw)
To: ziyang.meme, ast, daniel, andrii, eddyz87, memxor, bpf
Cc: martin.lau, song, yonghong.song, jolsa, emil, shuah,
roman.gushchin, kernel-team, linux-mm, cgroups, linux-kselftest,
linux-kernel, ziyang.meme, shakeel.butt, ast, andrii, daniel,
martin.lau, eddyz87, yonghong.song, clm, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 1520 bytes --]
> diff --git a/tools/testing/selftests/bpf/prog_tests/memcg_stat_churn_percpu.c b/tools/testing/selftests/bpf/prog_tests/memcg_stat_churn_percpu.c
> new file mode 100644
[ ... ]
> +static void settle_flush(void)
> +{
The settle_flush() comment explains the reset with:
> + * ~15% window asymmetry, enough to invert bpf_matched vs bpf_full on
> + * flush-dominated cases (e.g. "hot": few cgroups churned from many CPUs).
This isn't a bug, but the cases in this file are narrow/wide/widest, and
there is no "hot" case here (or in the sibling tests). The timed reads also
always run with collect_full = 1, so there is no bpf_matched-vs-bpf_full
comparison in this test. Does this example still describe this file, or was
it carried over from another test?
[ ... ]
> +#define CHURN_GAP_US (50 * 1000)
The gap comment just above this says:
> + * pays that flush inside its timed region. This gives all four reads
> + * (file/bpf x matched/full) approximately the same start state and folds the
This isn't a bug, but the sampling loop does one file_pass() and one
bpf_pass() per sample (bpf_pass always sets collect_full = 1), so there are
two reads per sample rather than the "four reads (file/bpf x matched/full)"
described here. Should this be reworded for this test?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/28695985027
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH 0/3] selftests/bpf: compare BPF and memory.stat memcg stat readers
2026-07-04 4:56 [PATCH 0/3] selftests/bpf: compare BPF and memory.stat memcg stat readers Ziyang Men
` (2 preceding siblings ...)
2026-07-04 4:56 ` [PATCH 3/3] selftests/bpf: add memcg_stat_churn_percpu BPF-vs-memory.stat benchmark under cross-CPU churn Ziyang Men
@ 2026-07-07 0:17 ` Eduard Zingerman
2026-07-07 1:50 ` Shakeel Butt
3 siblings, 1 reply; 9+ messages in thread
From: Eduard Zingerman @ 2026-07-07 0:17 UTC (permalink / raw)
To: Ziyang Men, Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
Kumar Kartikeya Dwivedi, bpf, shakeel.butt
Cc: Martin KaFai Lau, Song Liu, Yonghong Song, Jiri Olsa,
Emil Tsalapatis, Shuah Khan, Roman Gushchin, kernel-team,
linux-mm, cgroups, linux-kselftest, linux-kernel, Shakeel Butt
On Fri, 2026-07-03 at 21:56 -0700, Ziyang Men wrote:
[...]
Hi Ziyang,
I'm a bit hesitant adding 2.5K lines of code to the BPF selftests,
as this code would need to be (a) maintained, (b) run at each CI invocation.
Hence, the tests added need to be relevant for the BPF sub-system.
Regarding the benchmarking part, as you state yourself:
> In my testing (a 60-CPU VM) the BPF path is roughly an order of magnitude
> faster than the per-cgroup memory.stat parse for a whole-tree scan, mainly
> because it avoids the per-cgroup open/read and string parsing.
With this, I think the benchmarking code can be dropped altogether.
Next, the three memcg_stat_{reader,churn,churn_percpu}.c files share a
lot of utility code almost verbatim (e.g. tree definition/construction).
Such duplication should be avoided.
Finally, from the BPF point of view the test exercises the following functionality:
- kfuncs:
- bpf_mem_cgroup_page_state
- bpf_mem_cgroup_vm_events
- bpf_put_mem_cgroup
- bpf_get_mem_cgroup
- main iterator logic.
All kfuncs but bpf_get_mem_cgroup() are thin wrappers around mm/memcontrol.c code,
all kfuncs including the bpf_get_mem_cgroup() are already exercised in the selftests.
The iterator logic itself is covered by 8 sub-tests in the prog_tests/cgroup_iter.c.
Hence two questions:
- What do these new tests add in terms of tests coverage?
- Why do BPF selftests need to exercise the churn and churn_percpu scenarios?
Shakeel, could you please comment as well?
Thanks,
Eduard.
[...]
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: [PATCH 0/3] selftests/bpf: compare BPF and memory.stat memcg stat readers
2026-07-07 0:17 ` [PATCH 0/3] selftests/bpf: compare BPF and memory.stat memcg stat readers Eduard Zingerman
@ 2026-07-07 1:50 ` Shakeel Butt
2026-07-07 9:21 ` Eduard Zingerman
0 siblings, 1 reply; 9+ messages in thread
From: Shakeel Butt @ 2026-07-07 1:50 UTC (permalink / raw)
To: Eduard Zingerman
Cc: Ziyang Men, Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
Kumar Kartikeya Dwivedi, bpf, Martin KaFai Lau, Song Liu,
Yonghong Song, Jiri Olsa, Emil Tsalapatis, Shuah Khan,
Roman Gushchin, kernel-team, linux-mm, cgroups, linux-kselftest,
linux-kernel
On Mon, Jul 06, 2026 at 05:17:50PM -0700, Eduard Zingerman wrote:
> On Fri, 2026-07-03 at 21:56 -0700, Ziyang Men wrote:
>
> [...]
>
> Hi Ziyang,
>
> I'm a bit hesitant adding 2.5K lines of code to the BPF selftests,
> as this code would need to be (a) maintained, (b) run at each CI invocation.
> Hence, the tests added need to be relevant for the BPF sub-system.
>
> Regarding the benchmarking part, as you state yourself:
>
> > In my testing (a 60-CPU VM) the BPF path is roughly an order of magnitude
> > faster than the per-cgroup memory.stat parse for a whole-tree scan, mainly
> > because it avoids the per-cgroup open/read and string parsing.
>
> With this, I think the benchmarking code can be dropped altogether.
>
> Next, the three memcg_stat_{reader,churn,churn_percpu}.c files share a
> lot of utility code almost verbatim (e.g. tree definition/construction).
> Such duplication should be avoided.
>
> Finally, from the BPF point of view the test exercises the following functionality:
> - kfuncs:
> - bpf_mem_cgroup_page_state
> - bpf_mem_cgroup_vm_events
> - bpf_put_mem_cgroup
> - bpf_get_mem_cgroup
> - main iterator logic.
>
> All kfuncs but bpf_get_mem_cgroup() are thin wrappers around mm/memcontrol.c code,
> all kfuncs including the bpf_get_mem_cgroup() are already exercised in the selftests.
> The iterator logic itself is covered by 8 sub-tests in the prog_tests/cgroup_iter.c.
> Hence two questions:
> - What do these new tests add in terms of tests coverage?
> - Why do BPF selftests need to exercise the churn and churn_percpu scenarios?
>
> Shakeel, could you please comment as well?
Hi Eduard,
Thanks a lot for taking a look. The main motivation I had behind requesting
Ziyang to send this series (beside making him learn the tooling and process of
sending patches to lkml) was to have a reference implementation and performance
comparison for BPF based cgroup/memcg stats collection.
However you have correctly pointed out that selftests might not be the right
place for such kind of code as selftests are more focused on functional tests
and run by a lot of CIs while this is a performance benchmarking code.
I am wondering if there is a place for this benchmarking code in kernel under
tools folder but archiving it on lkml might be good enough and should be easily
searchable. Anyways thanks again for your time.
thanks,
Shakeel
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: [PATCH 0/3] selftests/bpf: compare BPF and memory.stat memcg stat readers
2026-07-07 1:50 ` Shakeel Butt
@ 2026-07-07 9:21 ` Eduard Zingerman
0 siblings, 0 replies; 9+ messages in thread
From: Eduard Zingerman @ 2026-07-07 9:21 UTC (permalink / raw)
To: Shakeel Butt
Cc: Ziyang Men, Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
Kumar Kartikeya Dwivedi, bpf, Martin KaFai Lau, Song Liu,
Yonghong Song, Jiri Olsa, Emil Tsalapatis, Shuah Khan,
Roman Gushchin, kernel-team, linux-mm, cgroups, linux-kselftest,
linux-kernel
On Mon, 2026-07-06 at 18:50 -0700, Shakeel Butt wrote:
> On Mon, Jul 06, 2026 at 05:17:50PM -0700, Eduard Zingerman wrote:
> > On Fri, 2026-07-03 at 21:56 -0700, Ziyang Men wrote:
> >
> > [...]
> >
> > Hi Ziyang,
> >
> > I'm a bit hesitant adding 2.5K lines of code to the BPF selftests,
> > as this code would need to be (a) maintained, (b) run at each CI invocation.
> > Hence, the tests added need to be relevant for the BPF sub-system.
> >
> > Regarding the benchmarking part, as you state yourself:
> >
> > > In my testing (a 60-CPU VM) the BPF path is roughly an order of magnitude
> > > faster than the per-cgroup memory.stat parse for a whole-tree scan, mainly
> > > because it avoids the per-cgroup open/read and string parsing.
> >
> > With this, I think the benchmarking code can be dropped altogether.
> >
> > Next, the three memcg_stat_{reader,churn,churn_percpu}.c files share a
> > lot of utility code almost verbatim (e.g. tree definition/construction).
> > Such duplication should be avoided.
> >
> > Finally, from the BPF point of view the test exercises the following functionality:
> > - kfuncs:
> > - bpf_mem_cgroup_page_state
> > - bpf_mem_cgroup_vm_events
> > - bpf_put_mem_cgroup
> > - bpf_get_mem_cgroup
> > - main iterator logic.
> >
> > All kfuncs but bpf_get_mem_cgroup() are thin wrappers around mm/memcontrol.c code,
> > all kfuncs including the bpf_get_mem_cgroup() are already exercised in the selftests.
> > The iterator logic itself is covered by 8 sub-tests in the prog_tests/cgroup_iter.c.
> > Hence two questions:
> > - What do these new tests add in terms of tests coverage?
> > - Why do BPF selftests need to exercise the churn and churn_percpu scenarios?
> >
> > Shakeel, could you please comment as well?
>
> Hi Eduard,
>
> Thanks a lot for taking a look. The main motivation I had behind requesting
> Ziyang to send this series (beside making him learn the tooling and process of
> sending patches to lkml) was to have a reference implementation and performance
> comparison for BPF based cgroup/memcg stats collection.
>
> However you have correctly pointed out that selftests might not be the right
> place for such kind of code as selftests are more focused on functional tests
> and run by a lot of CIs while this is a performance benchmarking code.
>
> I am wondering if there is a place for this benchmarking code in kernel under
> tools folder but archiving it on lkml might be good enough and should be easily
> searchable. Anyways thanks again for your time.
Hi Shakeel,
We do have bpf benchmarks in the kernel tree, the entry point is
tools/testing/selftests/bpf/bench.c. These are supposed to be
performance measurements and are executed manually from time to time
(quite rarely, as far as I understand), not by CI.
However, if I understand Ziyang's assessment correctly, this code is
not really a performance test, but kind of a load test.
Thanks,
Eduard
^ permalink raw reply [flat|nested] 9+ messages in thread