[PATCH v3 0/4] cgroup: expose cpu.stat and io.stat to BPF

[PATCH v3 0/4] cgroup: expose cpu.stat and io.stat to BPF

[Ziyang Men]

Collecting cgroup statistics is expensive: the existing method is to
open and parse a cgroup file for every cgroup of interest. The memory
controller already has an efficient alternative through BPF; this series
extends that model to the CPU and block I/O controllers.

Patch 1 adds the CPU kfuncs, and patch 2 adds their selftest. Patch 3
adds the blkcg kfuncs, and patch 4 adds their selftest.

This v3 combines the two previously posted v2 series:
CPU v2: https://lore.kernel.org/all/20260818002450.3071325-1-ziyang.meme@gmail.com/
blkcg v2: https://lore.kernel.org/all/20260817214205.723267-1-ziyang.meme@gmail.com/

===
Changes since CPU v2:
- Rename bpf_cpu.c to bpf_cgroup.c and use unlikely() and container_of()
  for the task_group conversion.
- Remove bpf_css_flush_rstat() and register the existing
  css_rstat_flush() in the common kfunc set.
- Remove the SLEEPABLE mark bpf_cgroup_base_stat().

Changes since blkcg v2:
- Add bpf_cgroup_css() and bpf_css_release() to keep a controller css
  alive across the sleepable rstat flush.
- Add an RCU-protected checked css-to-blkcg conversion and make the blkg
  iterator take the typed blkcg pointer.
- Use the existing css_rstat_flush() and remove bpf_blkcg_flush_stats().
- Do not expose root io.stat values.

===
Changes since CPU v1:
- Make the rstat flush generic, move it to rstat.c, and make it take a css.
- Expose cgroup_base_stat directly instead of repacking its CPU times.
- Replace the throttled-time aggregation kfunc with a checked RCU cast in
  the CPU cgroup BPF code. The cast preserves the verifier type needed for
  the BPF program to perform the per-CPU sum itself.
- Remove the scheduler changes.

Built with LLVM. Pass test on v7.2-rc5.

Ziyang Men (4):
  cgroup: add BPF kfuncs to read a cpu cgroup's stats
  selftests/bpf: add cgroup_iter_cpu test for cpu cgroup kfuncs
  block: add BPF kfuncs to read blkcg io.stat
  selftests/bpf: add test for blkcg io.stat BPF kfuncs

 MAINTAINERS                                   |   1 +
 block/Makefile                                |   3 +
 block/bpf_blkcg.c                             | 138 ++++++++++
 kernel/cgroup/Makefile                        |   2 +
 kernel/cgroup/bpf_cgroup.c                    | 100 +++++++
 kernel/cgroup/rstat.c                         |  54 +++-
 tools/testing/selftests/bpf/cgroup_iter_cpu.h |  22 ++
 tools/testing/selftests/bpf/cgroup_iter_io.h  |  17 ++
 tools/testing/selftests/bpf/config            |   4 +
 .../bpf/prog_tests/cgroup_iter_cpu.c          | 259 ++++++++++++++++++
 .../selftests/bpf/prog_tests/cgroup_iter_io.c | 254 +++++++++++++++++
 .../selftests/bpf/progs/cgroup_iter_cpu.c     | 113 ++++++++
 .../selftests/bpf/progs/cgroup_iter_io.c      |  74 +++++
 13 files changed, 1037 insertions(+), 4 deletions(-)
 create mode 100644 block/bpf_blkcg.c
 create mode 100644 kernel/cgroup/bpf_cgroup.c
 create mode 100644 tools/testing/selftests/bpf/cgroup_iter_cpu.h
 create mode 100644 tools/testing/selftests/bpf/cgroup_iter_io.h
 create mode 100644 tools/testing/selftests/bpf/prog_tests/cgroup_iter_cpu.c
 create mode 100644 tools/testing/selftests/bpf/prog_tests/cgroup_iter_io.c
 create mode 100644 tools/testing/selftests/bpf/progs/cgroup_iter_cpu.c
 create mode 100644 tools/testing/selftests/bpf/progs/cgroup_iter_io.c


base-commit: 3b5f4b83c4abc0c9b0a7b9e2b44e816611b7f2ec
-- 
2.53.0-Meta

[PATCH v3 1/4] cgroup: add BPF kfuncs to read a cpu cgroup's stats

[Ziyang Men]

Collecting cgroup statistics is expensive because the existing method
opens and parses a cgroup file. memcg already provides an efficient BPF
interface; extend that model to the CPU controller.

Register css_rstat_flush() as a common kfunc and add
bpf_cgroup_base_stat(). The latter returns cgroup_base_stat after the
same cputime adjustment used by cpu.stat.

The BPF program reads the plain CFS bandwidth counters directly. Add
bpf_css_to_task_group() to check the controller and give the verifier a
typed task_group pointer for bpf_per_cpu_ptr().

css_rstat_flush() may reschedule and requires a sleepable program.
bpf_cgroup_base_stat() only takes locks and is not marked sleepable, but
those locks are not NMI-safe.

Suggested-by: Shakeel Butt <shakeel.butt@linux.dev>
Suggested-by: Tejun Heo <tj@kernel.org>
Assisted-by: Claude:claude-opus-5
Signed-off-by: Ziyang Men <ziyang.meme@gmail.com>
---
 kernel/cgroup/Makefile     |  2 ++
 kernel/cgroup/bpf_cgroup.c | 58 ++++++++++++++++++++++++++++++++++++++
 kernel/cgroup/rstat.c      | 54 ++++++++++++++++++++++++++++++++---
 3 files changed, 110 insertions(+), 4 deletions(-)
 create mode 100644 kernel/cgroup/bpf_cgroup.c

diff --git a/kernel/cgroup/Makefile b/kernel/cgroup/Makefile
index ede31601a363..29f29228865b 100644
--- a/kernel/cgroup/Makefile
+++ b/kernel/cgroup/Makefile
@@ -1,6 +1,8 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-y := cgroup.o rstat.o namespace.o cgroup-v1.o freezer.o
 
+obj-$(CONFIG_BPF_SYSCALL) += bpf_cgroup.o
+
 obj-$(CONFIG_CGROUP_FREEZER) += legacy_freezer.o
 obj-$(CONFIG_CGROUP_PIDS) += pids.o
 obj-$(CONFIG_CGROUP_RDMA) += rdma.o
diff --git a/kernel/cgroup/bpf_cgroup.c b/kernel/cgroup/bpf_cgroup.c
new file mode 100644
index 000000000000..cd28c838dc7b
--- /dev/null
+++ b/kernel/cgroup/bpf_cgroup.c
@@ -0,0 +1,58 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Cgroup BPF kfuncs
+ *
+ * Author: Ziyang Men <ziyang.meme@gmail.com>
+ */
+
+#include <linux/bpf.h>
+#include <linux/btf_ids.h>
+#include <linux/cgroup.h>
+
+#include "../sched/sched.h"
+
+#ifdef CONFIG_CGROUP_SCHED
+__bpf_kfunc_start_defs();
+
+/**
+ * bpf_css_to_task_group - Cast a CPU controller css to its task group
+ * @css: CPU controller css
+ *
+ * Must be called under RCU. The kfunc gives BPF a typed task_group pointer.
+ *
+ * Return: The task group, or NULL if @css belongs to another controller.
+ */
+__bpf_kfunc struct task_group *
+bpf_css_to_task_group(struct cgroup_subsys_state *css)
+{
+	if (unlikely(css->ss != &cpu_cgrp_subsys))
+		return NULL;
+
+	return container_of(css, struct task_group, css);
+}
+
+__bpf_kfunc_end_defs();
+
+BTF_KFUNCS_START(bpf_cpu_cgroup_kfunc_ids)
+BTF_ID_FLAGS(func, bpf_css_to_task_group,
+	     KF_RCU | KF_RCU_PROTECTED | KF_RET_NULL)
+BTF_KFUNCS_END(bpf_cpu_cgroup_kfunc_ids)
+
+static const struct btf_kfunc_id_set bpf_cpu_cgroup_kfunc_set = {
+	.owner		= THIS_MODULE,
+	.set		= &bpf_cpu_cgroup_kfunc_ids,
+};
+
+static int __init bpf_cpu_cgroup_kfunc_init(void)
+{
+	int err;
+
+	err = register_btf_kfunc_id_set(BPF_PROG_TYPE_UNSPEC,
+					&bpf_cpu_cgroup_kfunc_set);
+	if (err)
+		pr_warn("error while registering cpu cgroup kfuncs: %d\n", err);
+
+	return err;
+}
+late_initcall(bpf_cpu_cgroup_kfunc_init);
+#endif /* CONFIG_CGROUP_SCHED */
diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c
index de816a43db9f..5db72504a8a5 100644
--- a/kernel/cgroup/rstat.c
+++ b/kernel/cgroup/rstat.c
@@ -752,10 +752,49 @@ void cgroup_base_stat_cputime_show(struct seq_file *seq)
 	cgroup_force_idle_show(seq, &bstat);
 }
 
-/* Add bpf kfuncs for css_rstat_updated() and css_rstat_flush() */
+#ifdef CONFIG_BPF_SYSCALL
+
+__bpf_kfunc_start_defs();
+
+/**
+ * bpf_cgroup_base_stat - Read a cgroup's base statistics
+ * @cgrp: cgroup to read from
+ * @out: zero-initialized output in nanoseconds
+ *
+ * CPU time is adjusted as for cpu.stat.
+ */
+__bpf_kfunc void bpf_cgroup_base_stat(struct cgroup *cgrp,
+				      struct cgroup_base_stat *out)
+{
+	if (cgroup_parent(cgrp)) {
+		__css_rstat_lock(&cgrp->self, -1);
+		*out = cgrp->bstat;
+		cputime_adjust(&cgrp->bstat.cputime, &cgrp->prev_cputime,
+			       &out->cputime.utime, &out->cputime.stime);
+		__css_rstat_unlock(&cgrp->self, -1);
+	} else {
+		root_cgroup_cputime(out);
+	}
+}
+
+__bpf_kfunc_end_defs();
+
+BTF_KFUNCS_START(bpf_rstat_common_kfunc_ids)
+BTF_ID_FLAGS(func, css_rstat_flush, KF_SLEEPABLE)
+/* The reader does not sleep, but its locks are not NMI-safe. */
+BTF_ID_FLAGS(func, bpf_cgroup_base_stat)
+BTF_KFUNCS_END(bpf_rstat_common_kfunc_ids)
+
+static const struct btf_kfunc_id_set bpf_rstat_common_kfunc_set = {
+	.owner		= THIS_MODULE,
+	.set		= &bpf_rstat_common_kfunc_ids,
+};
+
+#endif /* CONFIG_BPF_SYSCALL */
+
+/* Add a bpf kfunc for css_rstat_updated(). */
 BTF_KFUNCS_START(bpf_rstat_kfunc_ids)
 BTF_ID_FLAGS(func, css_rstat_updated)
-BTF_ID_FLAGS(func, css_rstat_flush, KF_SLEEPABLE)
 BTF_KFUNCS_END(bpf_rstat_kfunc_ids)
 
 static const struct btf_kfunc_id_set bpf_rstat_kfunc_set = {
@@ -765,7 +804,14 @@ static const struct btf_kfunc_id_set bpf_rstat_kfunc_set = {
 
 static int __init bpf_rstat_kfunc_init(void)
 {
-	return register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACING,
-					 &bpf_rstat_kfunc_set);
+	int ret;
+
+	ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACING,
+					&bpf_rstat_kfunc_set);
+#ifdef CONFIG_BPF_SYSCALL
+	ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_UNSPEC,
+					       &bpf_rstat_common_kfunc_set);
+#endif
+	return ret;
 }
 late_initcall(bpf_rstat_kfunc_init);
-- 
2.53.0-Meta

[PATCH v3 2/4] selftests/bpf: add cgroup_iter_cpu test for cpu cgroup kfuncs

[Ziyang Men]

Add cgroup_iter_cpu, a selftest for the CPU controller BPF kfuncs.

The userspace side runs a CPU hog in a test cgroup with cpu.max settled
then:

  - checks the CPU-time and throttling counters are nonzero,
  - compares whether all values the program read are same as those
    reading from cgroup file.

Enable CONFIG_CGROUP_SCHED, CONFIG_FAIR_GROUP_SCHED and
CONFIG_CFS_BANDWIDTH in the selftest config.

Tested on VM with v7.2-rc5.

Suggested-by: Shakeel Butt <shakeel.butt@linux.dev>
Assisted-by: Claude:claude-opus-5
Signed-off-by: Ziyang Men <ziyang.meme@gmail.com>
---
 tools/testing/selftests/bpf/cgroup_iter_cpu.h |  22 ++
 tools/testing/selftests/bpf/config            |   3 +
 .../bpf/prog_tests/cgroup_iter_cpu.c          | 259 ++++++++++++++++++
 .../selftests/bpf/progs/cgroup_iter_cpu.c     | 113 ++++++++
 4 files changed, 397 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/cgroup_iter_cpu.h
 create mode 100644 tools/testing/selftests/bpf/prog_tests/cgroup_iter_cpu.c
 create mode 100644 tools/testing/selftests/bpf/progs/cgroup_iter_cpu.c

diff --git a/tools/testing/selftests/bpf/cgroup_iter_cpu.h b/tools/testing/selftests/bpf/cgroup_iter_cpu.h
new file mode 100644
index 000000000000..74599a5c0e4d
--- /dev/null
+++ b/tools/testing/selftests/bpf/cgroup_iter_cpu.h
@@ -0,0 +1,22 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */
+#ifndef __CGROUP_ITER_CPU_H
+#define __CGROUP_ITER_CPU_H
+
+struct cpu_query {
+	/* base cpu time, from cpu.stat */
+	__u64 usage_usec;
+	__u64 user_usec;
+	__u64 system_usec;
+	__u64 nice_usec;
+	__u64 forceidle_usec;
+	/* CFS bandwidth throttling, from cpu.stat and cpu.stat.local */
+	__u64 nr_periods;
+	__u64 nr_throttled;
+	__u64 throttled_usec;
+	__u64 nr_bursts;
+	__u64 burst_usec;
+	__u64 throttled_self_usec;
+};
+
+#endif /* __CGROUP_ITER_CPU_H */
diff --git a/tools/testing/selftests/bpf/config b/tools/testing/selftests/bpf/config
index ea7044f30adc..482b40dde2f9 100644
--- a/tools/testing/selftests/bpf/config
+++ b/tools/testing/selftests/bpf/config
@@ -11,6 +11,9 @@ CONFIG_BPF_STREAM_PARSER=y
 CONFIG_BPF_SYSCALL=y
 # CONFIG_BPF_UNPRIV_DEFAULT_OFF is not set
 CONFIG_CGROUP_BPF=y
+CONFIG_CGROUP_SCHED=y
+CONFIG_FAIR_GROUP_SCHED=y
+CONFIG_CFS_BANDWIDTH=y
 CONFIG_CRYPTO_HMAC=y
 CONFIG_CRYPTO_SHA256=y
 CONFIG_CRYPTO_USER_API=y
diff --git a/tools/testing/selftests/bpf/prog_tests/cgroup_iter_cpu.c b/tools/testing/selftests/bpf/prog_tests/cgroup_iter_cpu.c
new file mode 100644
index 000000000000..cd7e92ababfb
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/cgroup_iter_cpu.c
@@ -0,0 +1,259 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */
+#include <test_progs.h>
+#include <bpf/libbpf.h>
+#include <fcntl.h>
+#include <signal.h>
+#include <sys/prctl.h>
+#include <sys/wait.h>
+#include <unistd.h>
+#include "cgroup_helpers.h"
+#include "cgroup_iter_cpu.h"
+#include "cgroup_iter_cpu.skel.h"
+
+static int read_stats(struct bpf_link *link)
+{
+	int fd, ret = 0;
+	ssize_t bytes;
+
+	fd = bpf_iter_create(bpf_link__fd(link));
+	if (!ASSERT_OK_FD(fd, "bpf_iter_create"))
+		return 1;
+
+	bytes = read(fd, NULL, 0);
+	if (!ASSERT_EQ(bytes, 0, "read fd"))
+		ret = 1;
+
+	close(fd);
+	return ret;
+}
+
+/* Read cgroup file @name into @buf. */
+static int read_cgroup_file(int cgroup_fd, const char *name, char *buf,
+			    size_t size)
+{
+	ssize_t n;
+	int fd;
+
+	fd = openat(cgroup_fd, name, O_RDONLY);
+	if (fd < 0)
+		return -1;
+	n = read(fd, buf, size - 1);
+	close(fd);
+	if (n <= 0)
+		return -1;
+	buf[n] = '\0';
+	return 0;
+}
+
+/* Parse the "cpu.stat" file into @out. */
+static int parse_cpu_stat(int cgroup_fd, struct cpu_query *out)
+{
+	char buf[4096], *line, *sp;
+	unsigned long long v;
+
+	if (read_cgroup_file(cgroup_fd, "cpu.stat", buf, sizeof(buf)))
+		return -1;
+
+	for (line = strtok_r(buf, "\n", &sp); line;
+	     line = strtok_r(NULL, "\n", &sp)) {
+		if (sscanf(line, "usage_usec %llu", &v) == 1)
+			out->usage_usec = v;
+		else if (sscanf(line, "user_usec %llu", &v) == 1)
+			out->user_usec = v;
+		else if (sscanf(line, "system_usec %llu", &v) == 1)
+			out->system_usec = v;
+		else if (sscanf(line, "nice_usec %llu", &v) == 1)
+			out->nice_usec = v;
+		else if (sscanf(line, "core_sched.force_idle_usec %llu", &v) == 1)
+			out->forceidle_usec = v;
+		else if (sscanf(line, "nr_periods %llu", &v) == 1)
+			out->nr_periods = v;
+		else if (sscanf(line, "nr_throttled %llu", &v) == 1)
+			out->nr_throttled = v;
+		else if (sscanf(line, "throttled_usec %llu", &v) == 1)
+			out->throttled_usec = v;
+		else if (sscanf(line, "nr_bursts %llu", &v) == 1)
+			out->nr_bursts = v;
+		else if (sscanf(line, "burst_usec %llu", &v) == 1)
+			out->burst_usec = v;
+	}
+	return 0;
+}
+
+/*
+ * Parse the "cpu.stat.local" file into @out.
+ */
+static int parse_cpu_stat_local(int cgroup_fd, struct cpu_query *out)
+{
+	unsigned long long v;
+	char buf[256];
+
+	if (read_cgroup_file(cgroup_fd, "cpu.stat.local", buf, sizeof(buf)))
+		return -1;
+	if (sscanf(buf, "throttled_usec %llu", &v) != 1)
+		return -1;
+	out->throttled_self_usec = v;
+	return 0;
+}
+
+/* Read file value the bpf program reads. */
+static int parse_stats(int cgroup_fd, struct cpu_query *out, bool have_bw)
+{
+	if (parse_cpu_stat(cgroup_fd, out))
+		return -1;
+	if (have_bw && parse_cpu_stat_local(cgroup_fd, out))
+		return -1;
+	return 0;
+}
+
+/*
+ * Check whether this kernel accounts CFS bandwidth.
+ */
+static bool cgroup_has_bw_stat(int cgroup_fd)
+{
+	char buf[4096];
+
+	if (read_cgroup_file(cgroup_fd, "cpu.stat", buf, sizeof(buf)))
+		return false;
+	return strstr(buf, "nr_periods ");
+}
+
+/* Fork a child that spins in the current cgroup, kill it if the test exits. */
+static pid_t spawn_cpu_hog(void)
+{
+	pid_t pid = fork();
+
+	if (pid == 0) {
+		prctl(PR_SET_PDEATHSIG, SIGKILL);
+		while (1)
+			;
+	}
+	return pid;
+}
+
+void test_cgroup_iter_cpu(void)
+{
+	char *cgroup_rel_path = "/cgroup_iter_cpu_test";
+	struct cgroup_iter_cpu *skel;
+	struct cpu_query *q;
+	struct bpf_link *link;
+	bool wrote_max, have_bw;
+	int cgroup_fd;
+	pid_t hog;
+
+	cgroup_fd = cgroup_setup_and_join(cgroup_rel_path);
+	if (!ASSERT_OK_FD(cgroup_fd, "cgroup_setup_and_join"))
+		return;
+
+	wrote_max = !write_cgroup_file(cgroup_rel_path, "cpu.max", "10000 100000");
+
+	skel = cgroup_iter_cpu__open_and_load();
+	if (!ASSERT_OK_PTR(skel, "cgroup_iter_cpu__open_and_load"))
+		goto cleanup_cgroup_fd;
+
+	DECLARE_LIBBPF_OPTS(bpf_iter_attach_opts, opts);
+	union bpf_iter_link_info linfo = {
+		.cgroup.cgroup_fd = cgroup_fd,
+		.cgroup.order = BPF_CGROUP_ITER_SELF_ONLY,
+	};
+	opts.link_info = &linfo;
+	opts.link_info_len = sizeof(linfo);
+
+	link = bpf_program__attach_iter(skel->progs.cgroup_cpu_query, &opts);
+	if (!ASSERT_OK_PTR(link, "bpf_program__attach_iter"))
+		goto cleanup_skel;
+
+	q = &skel->data_query->cpu_query;
+
+	hog = spawn_cpu_hog();
+	if (!ASSERT_GT(hog, 0, "spawn_cpu_hog"))
+		goto cleanup_link;
+
+	sleep(1);
+
+	/* Run the bpf program before anything here reads cpu.stat. */
+	if (!ASSERT_OK(read_stats(link), "read stats"))
+		goto cleanup_hog;
+
+	have_bw = wrote_max && cgroup_has_bw_stat(cgroup_fd);
+
+	if (test__start_subtest("cgroup_iter_cpu__cputime")) {
+		ASSERT_GT(q->usage_usec, 0, "usage_usec");
+		ASSERT_GT(q->user_usec + q->system_usec, 0, "user+system_usec");
+	}
+	if (test__start_subtest("cgroup_iter_cpu__throttling")) {
+		if (!have_bw) {
+			test__skip();
+		} else {
+			ASSERT_GT(q->nr_periods, 0, "nr_periods");
+			ASSERT_GT(q->nr_throttled, 0, "nr_throttled");
+			ASSERT_GT(q->throttled_usec, 0, "throttled_usec");
+			ASSERT_GT(q->throttled_self_usec, 0, "throttled_self_usec");
+		}
+	}
+
+	/*
+	 * cpu.stat cputime grows on every tick a task in the cgroup runs, so
+	 * stop them all before comparing
+	 */
+	if (test__start_subtest("cgroup_iter_cpu__match")) {
+		struct cpu_query filev = {};
+		int i, stable = 0;
+
+		kill(hog, SIGSTOP);
+		waitpid(hog, NULL, WUNTRACED);
+		if (!ASSERT_OK(join_root_cgroup(), "join_root_cgroup"))
+			goto cleanup_hog;
+
+		/*
+		 * The period timer keeps adding to nr_periods for a while
+		 * after the hog stops
+		 */
+		for (i = 0; i < 20; i++) {
+			struct cpu_query before = {}, after = {};
+
+			if (!ASSERT_OK(parse_stats(cgroup_fd, &before, have_bw), "cpu.stat") ||
+			    !ASSERT_OK(read_stats(link), "read stats") ||
+			    !ASSERT_OK(parse_stats(cgroup_fd, &after, have_bw), "cpu.stat"))
+				goto cleanup_hog;
+
+			if (!memcmp(&before, &after, sizeof(before))) {
+				filev = before;
+				stable = 1;
+				break;
+			}
+			usleep(100000);
+		}
+
+		if (!ASSERT_TRUE(stable, "cpu.stat stable"))
+			goto cleanup_hog;
+
+		ASSERT_EQ(q->usage_usec, filev.usage_usec, "usage_usec");
+		ASSERT_EQ(q->user_usec, filev.user_usec, "user_usec");
+		ASSERT_EQ(q->system_usec, filev.system_usec, "system_usec");
+		ASSERT_EQ(q->nice_usec, filev.nice_usec, "nice_usec");
+		ASSERT_EQ(q->forceidle_usec, filev.forceidle_usec, "forceidle_usec");
+
+		if (have_bw) {
+			ASSERT_EQ(q->nr_periods, filev.nr_periods, "nr_periods");
+			ASSERT_EQ(q->nr_throttled, filev.nr_throttled, "nr_throttled");
+			ASSERT_EQ(q->throttled_usec, filev.throttled_usec, "throttled_usec");
+			ASSERT_EQ(q->nr_bursts, filev.nr_bursts, "nr_bursts");
+			ASSERT_EQ(q->burst_usec, filev.burst_usec, "burst_usec");
+			ASSERT_EQ(q->throttled_self_usec, filev.throttled_self_usec,
+				  "throttled_self_usec");
+		}
+	}
+
+cleanup_hog:
+	kill(hog, SIGKILL);
+	waitpid(hog, NULL, 0);
+cleanup_link:
+	bpf_link__destroy(link);
+cleanup_skel:
+	cgroup_iter_cpu__destroy(skel);
+cleanup_cgroup_fd:
+	close(cgroup_fd);
+	cleanup_cgroup_environment();
+}
diff --git a/tools/testing/selftests/bpf/progs/cgroup_iter_cpu.c b/tools/testing/selftests/bpf/progs/cgroup_iter_cpu.c
new file mode 100644
index 000000000000..0e916234e7eb
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/cgroup_iter_cpu.c
@@ -0,0 +1,113 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_core_read.h>
+#include "cgroup_iter_cpu.h"
+
+char _license[] SEC("license") = "GPL";
+
+struct cpu_query cpu_query SEC(".data.query");
+
+extern const void __cpu_possible_mask __ksym;
+
+struct cgroup_base_stat___local {
+	struct task_cputime cputime;
+	__u64 forceidle_sum;
+	__u64 ntime;
+} __attribute__((preserve_access_index));
+
+static __always_inline __u64 read_throttled_self(struct task_group *tg, __u32 cpu)
+{
+	struct cfs_rq *cfs_rq;
+
+	cfs_rq = bpf_per_cpu_ptr(tg->cfs_rq, cpu);
+	if (!cfs_rq)
+		return 0;
+
+	return BPF_CORE_READ(cfs_rq, throttled_clock_self_time);
+}
+
+SEC("iter.s/cgroup")
+int cgroup_cpu_query(struct bpf_iter__cgroup *ctx)
+{
+	struct cgroup_base_stat___local bstat = {};
+	struct cgroup *cgrp = ctx->cgroup;
+	struct cgroup_subsys_state *css;
+	struct task_group *tg;
+	__u64 throttled_self = 0;
+	int ssid;
+
+	if (!cgrp)
+		return 1;
+
+	css_rstat_flush(&cgrp->self);
+	bpf_cgroup_base_stat(cgrp, (struct cgroup_base_stat *)&bstat);
+
+	cpu_query.usage_usec = bstat.cputime.sum_exec_runtime / 1000;
+	cpu_query.user_usec = bstat.cputime.utime / 1000;
+	cpu_query.system_usec = bstat.cputime.stime / 1000;
+	cpu_query.nice_usec = bstat.ntime / 1000;
+	cpu_query.forceidle_usec = 0;
+	if (bpf_core_field_exists(bstat.forceidle_sum))
+		cpu_query.forceidle_usec = bstat.forceidle_sum / 1000;
+
+	bpf_rcu_read_lock();
+	if (!bpf_core_enum_value_exists(enum cgroup_subsys_id, cpu_cgrp_id) ||
+	    !bpf_ksym_exists(bpf_css_to_task_group))
+		goto unlock;
+
+	ssid = bpf_core_enum_value(enum cgroup_subsys_id, cpu_cgrp_id);
+	css = cgrp->subsys[ssid];
+	if (!css)
+		goto unlock;
+
+	tg = bpf_css_to_task_group(css);
+	if (tg && bpf_core_field_exists(tg->cfs_bandwidth.nr_periods)) {
+		cpu_query.nr_periods =
+			(__u32)BPF_CORE_READ(tg, cfs_bandwidth.nr_periods);
+		cpu_query.nr_throttled =
+			(__u32)BPF_CORE_READ(tg, cfs_bandwidth.nr_throttled);
+		cpu_query.throttled_usec =
+			BPF_CORE_READ(tg, cfs_bandwidth.throttled_time) / 1000;
+		cpu_query.nr_bursts =
+			(__u32)BPF_CORE_READ(tg, cfs_bandwidth.nr_burst);
+		cpu_query.burst_usec =
+			BPF_CORE_READ(tg, cfs_bandwidth.burst_time) / 1000;
+	}
+
+	if (tg && bpf_core_field_exists(tg->cfs_rq) &&
+	    bpf_core_field_exists(struct cfs_rq, throttled_clock_self_time)) {
+		__u32 mask_bytes = bpf_core_type_size(struct cpumask);
+		__u32 full_words = mask_bytes / sizeof(__u64);
+		int *cpu;
+
+		if (full_words)
+			bpf_for_each(bits, cpu,
+				     (const __u64 *)&__cpu_possible_mask,
+				     full_words)
+				throttled_self += read_throttled_self(tg, *cpu);
+
+		if (mask_bytes & (sizeof(__u64) - 1)) {
+			__u32 tail = 0;
+			const void *src = (const char *)&__cpu_possible_mask +
+					  full_words * sizeof(__u64);
+			int bit;
+
+			if (!bpf_probe_read_kernel(&tail, sizeof(tail), src))
+				bpf_for(bit, 0, 32)
+					if (tail & (1U << bit)) {
+						__u32 tail_cpu = full_words * 64 + bit;
+
+						throttled_self +=
+							read_throttled_self(tg, tail_cpu);
+					}
+		}
+	}
+
+unlock:
+	bpf_rcu_read_unlock();
+	cpu_query.throttled_self_usec = throttled_self / 1000;
+
+	return 0;
+}
-- 
2.53.0-Meta

[PATCH v3 3/4] block: add BPF kfuncs to read blkcg io.stat

[Ziyang Men]

Collecting cgroup statistics is expensive because the existing method
opens and parses a cgroup file for every cgroup. memcg already provides
an efficient BPF interface; extend that model to the block controller.

Add bpf_cgroup_css() and bpf_css_release() to acquire a controller's
css from a cgroup. The reference keeps the css alive across the
sleepable css_rstat_flush().

Add bpf_css_to_blkcg() as a checked RCU-protected css-to-blkcg
conversion and an open-coded iterator for the per-device blkgs.

Suggested-by: Shakeel Butt <shakeel.butt@linux.dev>
Suggested-by: Tejun Heo <tj@kernel.org>
Assisted-by: Claude:claude-opus-5
Signed-off-by: Ziyang Men <ziyang.meme@gmail.com>
---
 MAINTAINERS                |   1 +
 block/Makefile             |   3 +
 block/bpf_blkcg.c          | 138 +++++++++++++++++++++++++++++++++++++
 kernel/cgroup/bpf_cgroup.c |  62 ++++++++++++++---
 4 files changed, 194 insertions(+), 10 deletions(-)
 create mode 100644 block/bpf_blkcg.c

diff --git a/MAINTAINERS b/MAINTAINERS
index 2f9472c1a090..87c56e955577 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -6617,6 +6617,7 @@ F:	block/blk-cgroup.c
 F:	block/blk-iocost.c
 F:	block/blk-iolatency.c
 F:	block/blk-throttle.c
+F:	block/bpf_blkcg.c
 F:	include/linux/blk-cgroup.h
 
 CONTROL GROUP - CPUSET
diff --git a/block/Makefile b/block/Makefile
index e7bd320e3d69..572e49988c8e 100644
--- a/block/Makefile
+++ b/block/Makefile
@@ -17,6 +17,9 @@ obj-$(CONFIG_BLK_ERROR_INJECTION) += error-injection.o
 obj-$(CONFIG_BLK_DEV_BSG_COMMON) += bsg.o
 obj-$(CONFIG_BLK_DEV_BSGLIB)	+= bsg-lib.o
 obj-$(CONFIG_BLK_CGROUP)	+= blk-cgroup.o
+ifdef CONFIG_BPF_SYSCALL
+obj-$(CONFIG_BLK_CGROUP)	+= bpf_blkcg.o
+endif
 obj-$(CONFIG_BLK_CGROUP_RWSTAT)	+= blk-cgroup-rwstat.o
 obj-$(CONFIG_BLK_CGROUP_FC_APPID) += blk-cgroup-fc-appid.o
 obj-$(CONFIG_BLK_DEV_THROTTLING)	+= blk-throttle.o
diff --git a/block/bpf_blkcg.c b/block/bpf_blkcg.c
new file mode 100644
index 000000000000..d8ab8006bc57
--- /dev/null
+++ b/block/bpf_blkcg.c
@@ -0,0 +1,138 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * Block I/O Controller-related BPF kfuncs and auxiliary code
+ */
+
+#include "blk-cgroup.h"
+
+#include <linux/bpf.h>
+#include <linux/btf_ids.h>
+#include <linux/rculist.h>
+
+__bpf_kfunc_start_defs();
+
+/**
+ * bpf_css_to_blkcg - Cast an io controller css to its block cgroup
+ * @css: io controller css
+ *
+ * Must be called under RCU.
+ *
+ * Return: The block cgroup, or NULL if @css belongs to another controller.
+ */
+__bpf_kfunc struct blkcg *
+bpf_css_to_blkcg(struct cgroup_subsys_state *css)
+{
+	if (unlikely(css->ss != &io_cgrp_subsys))
+		return NULL;
+
+	return css_to_blkcg(css);
+}
+
+struct bpf_iter_blkg {
+	__u64 __opaque[2];
+} __aligned(8);
+
+struct bpf_iter_blkg_kern {
+	struct blkcg *blkcg;
+	struct blkcg_gq *pos;
+} __aligned(8);
+
+/**
+ * bpf_iter_blkg_new - Start iterating a block cgroup's per-device blkgs
+ * @it: iterator to initialize
+ * @blkcg: block cgroup to iterate
+ *
+ * Each blkg holds one device's io.stat counters. Offline blkgs are skipped.
+ * A blkg without a disk can be returned. Root blkgs do not contain the
+ * system-wide statistics shown by root io.stat. Must run under RCU.
+ *
+ * Return: 0 on success.
+ */
+__bpf_kfunc int bpf_iter_blkg_new(struct bpf_iter_blkg *it,
+				  struct blkcg *blkcg)
+{
+	struct bpf_iter_blkg_kern *kit = (void *)it;
+
+	BUILD_BUG_ON(sizeof(struct bpf_iter_blkg_kern) > sizeof(struct bpf_iter_blkg));
+	BUILD_BUG_ON(__alignof__(struct bpf_iter_blkg_kern) !=
+		     __alignof__(struct bpf_iter_blkg));
+
+	kit->pos = NULL;
+	kit->blkcg = blkcg;
+	return 0;
+}
+
+/**
+ * bpf_iter_blkg_next - Return the next online blkg of the iterated block cgroup
+ * @it: iterator
+ *
+ * Return: the next online blkg, or NULL when the walk is done.
+ */
+__bpf_kfunc struct blkcg_gq *bpf_iter_blkg_next(struct bpf_iter_blkg *it)
+{
+	struct bpf_iter_blkg_kern *kit = (void *)it;
+	struct blkcg_gq *blkg = kit->pos;
+	struct hlist_node *node;
+
+	if (!kit->blkcg)
+		return NULL;
+
+	if (!blkg)
+		node = rcu_dereference(hlist_first_rcu(&kit->blkcg->blkg_list));
+	else
+		node = rcu_dereference(hlist_next_rcu(&blkg->blkcg_node));
+
+	/* Skip offline blkgs, matching io.stat. */
+	while (node) {
+		blkg = hlist_entry(node, struct blkcg_gq, blkcg_node);
+		/* A race only changes whether this blkg is returned. */
+		if (data_race(blkg->online)) {
+			kit->pos = blkg;
+			return blkg;
+		}
+		node = rcu_dereference(hlist_next_rcu(&blkg->blkcg_node));
+	}
+
+	/* The iterator must keep returning NULL after completion. */
+	kit->pos = NULL;
+	kit->blkcg = NULL;
+	return NULL;
+}
+
+/**
+ * bpf_iter_blkg_destroy - Tear down a blkg iterator
+ * @it: iterator
+ */
+__bpf_kfunc void bpf_iter_blkg_destroy(struct bpf_iter_blkg *it)
+{
+}
+
+__bpf_kfunc_end_defs();
+
+BTF_KFUNCS_START(bpf_blkcg_kfuncs)
+BTF_ID_FLAGS(func, bpf_css_to_blkcg,
+	     KF_RCU | KF_RCU_PROTECTED | KF_RET_NULL)
+
+BTF_ID_FLAGS(func, bpf_iter_blkg_new,
+	     KF_ITER_NEW | KF_RCU | KF_RCU_PROTECTED)
+BTF_ID_FLAGS(func, bpf_iter_blkg_next, KF_ITER_NEXT | KF_RET_NULL)
+BTF_ID_FLAGS(func, bpf_iter_blkg_destroy, KF_ITER_DESTROY)
+BTF_KFUNCS_END(bpf_blkcg_kfuncs)
+
+static const struct btf_kfunc_id_set bpf_blkcg_kfunc_set = {
+	.owner		= THIS_MODULE,
+	.set		= &bpf_blkcg_kfuncs,
+};
+
+static int __init bpf_blkcg_init(void)
+{
+	int err;
+
+	err = register_btf_kfunc_id_set(BPF_PROG_TYPE_UNSPEC,
+					&bpf_blkcg_kfunc_set);
+	if (err)
+		pr_warn("error while registering bpf blkcg kfuncs: %d\n", err);
+
+	return err;
+}
+late_initcall(bpf_blkcg_init);
diff --git a/kernel/cgroup/bpf_cgroup.c b/kernel/cgroup/bpf_cgroup.c
index cd28c838dc7b..e253633e8278 100644
--- a/kernel/cgroup/bpf_cgroup.c
+++ b/kernel/cgroup/bpf_cgroup.c
@@ -8,12 +8,50 @@
 #include <linux/bpf.h>
 #include <linux/btf_ids.h>
 #include <linux/cgroup.h>
+#include <linux/rcupdate.h>
 
+#ifdef CONFIG_CGROUP_SCHED
 #include "../sched/sched.h"
+#endif
 
-#ifdef CONFIG_CGROUP_SCHED
 __bpf_kfunc_start_defs();
 
+/**
+ * bpf_cgroup_css - Get a reference to one controller's css
+ * @cgrp: cgroup to look in
+ * @ssid: controller ID
+ *
+ * The returned css must be released with bpf_css_release().
+ *
+ * Return: The referenced css, or NULL.
+ */
+__bpf_kfunc struct cgroup_subsys_state *
+bpf_cgroup_css(struct cgroup *cgrp, int ssid)
+{
+	struct cgroup_subsys_state *css;
+
+	if (unlikely(ssid < 0 || ssid >= CGROUP_SUBSYS_COUNT))
+		return NULL;
+
+	rcu_read_lock();
+	css = rcu_dereference(cgrp->subsys[ssid]);
+	if (css && !css_tryget(css))
+		css = NULL;
+	rcu_read_unlock();
+
+	return css;
+}
+
+/**
+ * bpf_css_release - Release a css reference
+ * @css: css to release
+ */
+__bpf_kfunc void bpf_css_release(struct cgroup_subsys_state *css)
+{
+	css_put(css);
+}
+
+#ifdef CONFIG_CGROUP_SCHED
 /**
  * bpf_css_to_task_group - Cast a CPU controller css to its task group
  * @css: CPU controller css
@@ -30,29 +68,33 @@ bpf_css_to_task_group(struct cgroup_subsys_state *css)
 
 	return container_of(css, struct task_group, css);
 }
+#endif /* CONFIG_CGROUP_SCHED */
 
 __bpf_kfunc_end_defs();
 
-BTF_KFUNCS_START(bpf_cpu_cgroup_kfunc_ids)
+BTF_KFUNCS_START(bpf_cgroup_kfunc_ids)
+BTF_ID_FLAGS(func, bpf_cgroup_css, KF_ACQUIRE | KF_RCU | KF_RET_NULL)
+BTF_ID_FLAGS(func, bpf_css_release, KF_RELEASE)
+#ifdef CONFIG_CGROUP_SCHED
 BTF_ID_FLAGS(func, bpf_css_to_task_group,
 	     KF_RCU | KF_RCU_PROTECTED | KF_RET_NULL)
-BTF_KFUNCS_END(bpf_cpu_cgroup_kfunc_ids)
+#endif
+BTF_KFUNCS_END(bpf_cgroup_kfunc_ids)
 
-static const struct btf_kfunc_id_set bpf_cpu_cgroup_kfunc_set = {
+static const struct btf_kfunc_id_set bpf_cgroup_kfunc_set = {
 	.owner		= THIS_MODULE,
-	.set		= &bpf_cpu_cgroup_kfunc_ids,
+	.set		= &bpf_cgroup_kfunc_ids,
 };
 
-static int __init bpf_cpu_cgroup_kfunc_init(void)
+static int __init bpf_cgroup_kfunc_init(void)
 {
 	int err;
 
 	err = register_btf_kfunc_id_set(BPF_PROG_TYPE_UNSPEC,
-					&bpf_cpu_cgroup_kfunc_set);
+					&bpf_cgroup_kfunc_set);
 	if (err)
-		pr_warn("error while registering cpu cgroup kfuncs: %d\n", err);
+		pr_warn("error while registering cgroup kfuncs: %d\n", err);
 
 	return err;
 }
-late_initcall(bpf_cpu_cgroup_kfunc_init);
-#endif /* CONFIG_CGROUP_SCHED */
+late_initcall(bpf_cgroup_kfunc_init);
-- 
2.53.0-Meta

[PATCH v3 4/4] selftests/bpf: add test for blkcg io.stat BPF kfuncs

[Ziyang Men]

Add cgroup_iter_io to test the blkcg io.stat BPF kfuncs. The BPF
program acquires the I/O css, flushes its statistics, converts it to a
typed blkcg under RCU, walks the blkgs, and releases the css reference.

The test performs O_DIRECT I/O on a private loop device. It checks the
device ID and counters against io.stat.

Assisted-by: Claude:claude-opus-5
Signed-off-by: Ziyang Men <ziyang.meme@gmail.com>
---
 tools/testing/selftests/bpf/cgroup_iter_io.h  |  17 ++
 tools/testing/selftests/bpf/config            |   1 +
 .../selftests/bpf/prog_tests/cgroup_iter_io.c | 254 ++++++++++++++++++
 .../selftests/bpf/progs/cgroup_iter_io.c      |  74 +++++
 4 files changed, 346 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/cgroup_iter_io.h
 create mode 100644 tools/testing/selftests/bpf/prog_tests/cgroup_iter_io.c
 create mode 100644 tools/testing/selftests/bpf/progs/cgroup_iter_io.c

diff --git a/tools/testing/selftests/bpf/cgroup_iter_io.h b/tools/testing/selftests/bpf/cgroup_iter_io.h
new file mode 100644
index 000000000000..f4bbaaccdf71
--- /dev/null
+++ b/tools/testing/selftests/bpf/cgroup_iter_io.h
@@ -0,0 +1,17 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */
+#ifndef __CGROUP_ITER_IO_H
+#define __CGROUP_ITER_IO_H
+
+struct io_query {
+	/* one device's io.stat counters */
+	__u64 rbytes;
+	__u64 wbytes;
+	__u64 rios;
+	__u64 wios;
+	__u64 dbytes;
+	__u64 dios;
+	__u64 dev;	/* dev_t of the device the counters belong to */
+};
+
+#endif /* __CGROUP_ITER_IO_H */
diff --git a/tools/testing/selftests/bpf/config b/tools/testing/selftests/bpf/config
index 482b40dde2f9..0e10d625de6e 100644
--- a/tools/testing/selftests/bpf/config
+++ b/tools/testing/selftests/bpf/config
@@ -1,3 +1,4 @@
+CONFIG_BLK_CGROUP=y
 CONFIG_BLK_DEV_LOOP=y
 CONFIG_BOOTPARAM_HARDLOCKUP_PANIC=y
 CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC=1
diff --git a/tools/testing/selftests/bpf/prog_tests/cgroup_iter_io.c b/tools/testing/selftests/bpf/prog_tests/cgroup_iter_io.c
new file mode 100644
index 000000000000..2267780912d8
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/cgroup_iter_io.c
@@ -0,0 +1,254 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */
+#define _GNU_SOURCE
+#include <test_progs.h>
+#include <bpf/libbpf.h>
+#include <fcntl.h>
+#include <linux/loop.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/ioctl.h>
+#include <sys/stat.h>
+#include <sys/sysmacros.h>
+#include <unistd.h>
+#include "cgroup_helpers.h"
+#include "cgroup_iter_io.h"
+#include "cgroup_iter_io.skel.h"
+
+#define IO_SIZE (4 * 1024 * 1024)
+
+static int read_stats(struct bpf_link *link)
+{
+	int fd, ret = 0;
+	ssize_t bytes;
+
+	fd = bpf_iter_create(bpf_link__fd(link));
+	if (!ASSERT_OK_FD(fd, "bpf_iter_create"))
+		return 1;
+
+	/* Results land in skel->data_query; the read itself returns no data. */
+	bytes = read(fd, NULL, 0);
+	if (!ASSERT_EQ(bytes, 0, "read fd"))
+		ret = 1;
+
+	close(fd);
+	return ret;
+}
+
+/* Set up a loop device for cgroup-charged I/O. */
+static int loop_setup(char *loop_path, size_t sz, int *ctl_fd, int *loop_fd,
+		      int *back_fd)
+{
+	char back_path[] = "/tmp/cgroup_iter_io.XXXXXX";
+	int nr;
+
+	*ctl_fd = *loop_fd = *back_fd = -1;
+
+	*ctl_fd = open("/dev/loop-control", O_RDWR | O_CLOEXEC);
+	if (*ctl_fd < 0)
+		return -1;
+
+	nr = ioctl(*ctl_fd, LOOP_CTL_GET_FREE);
+	if (nr < 0)
+		goto err;
+	snprintf(loop_path, sz, "/dev/loop%d", nr);
+
+	*back_fd = mkstemp(back_path);
+	if (*back_fd < 0)
+		goto err;
+	unlink(back_path);
+	if (ftruncate(*back_fd, (off_t)IO_SIZE * 4))
+		goto err;
+
+	*loop_fd = open(loop_path, O_RDWR | O_CLOEXEC);
+	if (*loop_fd < 0)
+		goto err;
+	if (ioctl(*loop_fd, LOOP_SET_FD, *back_fd))
+		goto err;
+
+	return 0;
+err:
+	if (*loop_fd >= 0)
+		close(*loop_fd);
+	if (*back_fd >= 0)
+		close(*back_fd);
+	close(*ctl_fd);
+	*ctl_fd = *loop_fd = *back_fd = -1;
+	return -1;
+}
+
+static void loop_teardown(const char *loop_path, int ctl_fd, int loop_fd,
+			  int back_fd)
+{
+	int nr = -1;
+
+	if (loop_fd >= 0) {
+		ioctl(loop_fd, LOOP_CLR_FD, 0);
+		close(loop_fd);
+	}
+	if (back_fd >= 0)
+		close(back_fd);
+	if (ctl_fd >= 0) {
+		if (sscanf(loop_path, "/dev/loop%d", &nr) == 1 && nr >= 0)
+			ioctl(ctl_fd, LOOP_CTL_REMOVE, nr);
+		close(ctl_fd);
+	}
+}
+
+/* O_DIRECT keeps I/O charged to the current cgroup. */
+static int do_direct_io(const char *loop_path)
+{
+	void *buf;
+	int fd, ret = -1;
+
+	fd = open(loop_path, O_RDWR | O_DIRECT | O_CLOEXEC);
+	if (fd < 0)
+		return -1;
+	if (posix_memalign(&buf, 4096, IO_SIZE))
+		goto out_fd;
+	memset(buf, 0xab, IO_SIZE);
+
+	if (pwrite(fd, buf, IO_SIZE, 0) != IO_SIZE)
+		goto out_buf;
+	fsync(fd);
+	if (pread(fd, buf, IO_SIZE, 0) != IO_SIZE)
+		goto out_buf;
+	ret = 0;
+out_buf:
+	free(buf);
+out_fd:
+	close(fd);
+	return ret;
+}
+
+/* Read @dev's io.stat counters. @dev uses kernel dev_t encoding. */
+static int parse_io_stat(int cgroup_fd, __u64 dev, struct io_query *out)
+{
+	unsigned int want_maj = dev >> 20, want_min = dev & ((1U << 20) - 1);
+	char buf[4096], *line, *saveptr;
+	int fd, n, ret = -1;
+
+	fd = openat(cgroup_fd, "io.stat", O_RDONLY);
+	if (fd < 0)
+		return -1;
+	n = read(fd, buf, sizeof(buf) - 1);
+	close(fd);
+	if (n <= 0)
+		return -1;
+	buf[n] = '\0';
+
+	for (line = strtok_r(buf, "\n", &saveptr); line;
+	     line = strtok_r(NULL, "\n", &saveptr)) {
+		unsigned long long rb = 0, wb = 0, ri = 0, wi = 0, db = 0, di = 0;
+		unsigned int maj, min;
+
+		/* Only the device id is required; missing counters stay zero. */
+		if (sscanf(line,
+			   "%u:%u rbytes=%llu wbytes=%llu rios=%llu wios=%llu dbytes=%llu dios=%llu",
+			   &maj, &min, &rb, &wb, &ri, &wi, &db, &di) < 2)
+			continue;
+		if (maj != want_maj || min != want_min)
+			continue;
+
+		out->rbytes = rb;
+		out->wbytes = wb;
+		out->rios = ri;
+		out->wios = wi;
+		out->dbytes = db;
+		out->dios = di;
+		ret = 0;
+		break;
+	}
+	return ret;
+}
+
+void test_cgroup_iter_io(void)
+{
+	char *cgroup_rel_path = "/cgroup_iter_io_test";
+	int ctl_fd = -1, loop_fd = -1, back_fd = -1;
+	struct cgroup_iter_io *skel = NULL;
+	struct bpf_link *link = NULL;
+	char loop_path[64];
+	struct io_query *q;
+	int cgroup_fd;
+
+	cgroup_fd = cgroup_setup_and_join(cgroup_rel_path);
+	if (!ASSERT_OK_FD(cgroup_fd, "cgroup_setup_and_join"))
+		return;
+
+	if (loop_setup(loop_path, sizeof(loop_path), &ctl_fd, &loop_fd, &back_fd)) {
+		test__skip();	/* needs root + CONFIG_BLK_DEV_LOOP */
+		goto cleanup_cgroup_fd;
+	}
+
+	skel = cgroup_iter_io__open_and_load();
+	if (!ASSERT_OK_PTR(skel, "cgroup_iter_io__open_and_load"))
+		goto cleanup_loop;
+
+	/* Convert glibc st_rdev to kernel dev_t format. */
+	{
+		struct stat lst;
+
+		if (!ASSERT_OK(fstat(loop_fd, &lst), "fstat loop"))
+			goto cleanup_skel;
+		skel->data_query->target_dev =
+			((__u64)major(lst.st_rdev) << 20) | minor(lst.st_rdev);
+	}
+
+	DECLARE_LIBBPF_OPTS(bpf_iter_attach_opts, opts);
+	union bpf_iter_link_info linfo = {
+		.cgroup.cgroup_fd = cgroup_fd,
+		.cgroup.order = BPF_CGROUP_ITER_SELF_ONLY,
+	};
+	opts.link_info = &linfo;
+	opts.link_info_len = sizeof(linfo);
+
+	link = bpf_program__attach_iter(skel->progs.cgroup_io_query, &opts);
+	if (!ASSERT_OK_PTR(link, "bpf_program__attach_iter"))
+		goto cleanup_skel;
+
+	/* This process is in the test cgroup, so the loop I/O is charged here. */
+	if (!ASSERT_OK(do_direct_io(loop_path), "do_direct_io"))
+		goto cleanup_link;
+
+	if (!ASSERT_OK(read_stats(link), "read stats"))
+		goto cleanup_link;
+
+	q = &skel->data_query->io_query;
+	if (test__start_subtest("cgroup_iter_io__write")) {
+		ASSERT_GT(q->wbytes, 0, "wbytes");
+		ASSERT_GT(q->wios, 0, "wios");
+	}
+	if (test__start_subtest("cgroup_iter_io__read")) {
+		ASSERT_GT(q->rbytes, 0, "rbytes");
+		ASSERT_GT(q->rios, 0, "rios");
+	}
+	if (test__start_subtest("cgroup_iter_io__dev"))
+		ASSERT_GT(q->dev, 0, "dev");
+
+	/* Compare with io.stat without I/O between the reads. */
+	if (test__start_subtest("cgroup_iter_io__match")) {
+		struct io_query filev = {};
+
+		if (ASSERT_OK(read_stats(link), "read stats") &&
+		    ASSERT_OK(parse_io_stat(cgroup_fd, q->dev, &filev),
+			      "parse io.stat")) {
+			ASSERT_EQ(q->rbytes, filev.rbytes, "rbytes");
+			ASSERT_EQ(q->wbytes, filev.wbytes, "wbytes");
+			ASSERT_EQ(q->rios, filev.rios, "rios");
+			ASSERT_EQ(q->wios, filev.wios, "wios");
+			ASSERT_EQ(q->dbytes, filev.dbytes, "dbytes");
+			ASSERT_EQ(q->dios, filev.dios, "dios");
+		}
+	}
+
+cleanup_link:
+	bpf_link__destroy(link);
+cleanup_skel:
+	cgroup_iter_io__destroy(skel);
+cleanup_loop:
+	loop_teardown(loop_path, ctl_fd, loop_fd, back_fd);
+cleanup_cgroup_fd:
+	close(cgroup_fd);
+	cleanup_cgroup_environment();
+}
diff --git a/tools/testing/selftests/bpf/progs/cgroup_iter_io.c b/tools/testing/selftests/bpf/progs/cgroup_iter_io.c
new file mode 100644
index 000000000000..0e9c9cd6e33a
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/cgroup_iter_io.c
@@ -0,0 +1,74 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_core_read.h>
+#include "bpf_experimental.h"
+#include "cgroup_iter_io.h"
+
+char _license[] SEC("license") = "GPL";
+
+struct io_query io_query SEC(".data.query");
+
+/* Device selected by userspace in kernel dev_t format. */
+__u64 target_dev SEC(".data.query");
+
+/* Keep inline: RCU and open-coded iterators cannot cross a BPF call. */
+static __always_inline int read_target_dev(struct cgroup *cgrp,
+					   struct io_query *out)
+{
+	struct cgroup_subsys_state *css;
+	struct blkcg *blkcg;
+	struct blkcg_gq *pos;
+	__u64 dev;
+	int ssid;
+
+	ssid = bpf_core_enum_value(enum cgroup_subsys_id, io_cgrp_id);
+	css = bpf_cgroup_css(cgrp, ssid);
+	if (!css)
+		return 0;
+
+	css_rstat_flush(css);
+
+	bpf_rcu_read_lock();
+
+	blkcg = bpf_css_to_blkcg(css);
+	if (!blkcg) {
+		bpf_rcu_read_unlock();
+		bpf_css_release(css);
+		return 0;
+	}
+
+	bpf_for_each(blkg, pos, blkcg) {
+		dev = BPF_CORE_READ(pos, q, disk, part0, bd_dev);
+		if (dev != target_dev)
+			continue;
+
+		out->dev = dev;
+		out->rbytes = BPF_CORE_READ(pos, iostat.cur.bytes[BLKG_IOSTAT_READ]);
+		out->wbytes = BPF_CORE_READ(pos, iostat.cur.bytes[BLKG_IOSTAT_WRITE]);
+		out->rios = BPF_CORE_READ(pos, iostat.cur.ios[BLKG_IOSTAT_READ]);
+		out->wios = BPF_CORE_READ(pos, iostat.cur.ios[BLKG_IOSTAT_WRITE]);
+		out->dbytes = BPF_CORE_READ(pos, iostat.cur.bytes[BLKG_IOSTAT_DISCARD]);
+		out->dios = BPF_CORE_READ(pos, iostat.cur.ios[BLKG_IOSTAT_DISCARD]);
+		break;
+	}
+	bpf_rcu_read_unlock();
+	bpf_css_release(css);
+	return 1;
+}
+
+SEC("iter.s/cgroup")
+int cgroup_io_query(struct bpf_iter__cgroup *ctx)
+{
+	struct cgroup *cgrp = ctx->cgroup;
+
+	if (!cgrp)
+		return 1;
+
+	/* Start fresh so a device that is not found stays all-zero. */
+	__builtin_memset(&io_query, 0, sizeof(io_query));
+
+	read_target_dev(cgrp, &io_query);
+	return 0;
+}
-- 
2.53.0-Meta

Re: [PATCH v3 3/4] block: add BPF kfuncs to read blkcg io.stat

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 3582 bytes --]

> diff --git a/kernel/cgroup/bpf_cgroup.c b/kernel/cgroup/bpf_cgroup.c
> index cd28c838dc7b..e253633e8278 100644
> --- a/kernel/cgroup/bpf_cgroup.c
> +++ b/kernel/cgroup/bpf_cgroup.c
> @@ -8,12 +8,50 @@
>  #include <linux/bpf.h>
>  #include <linux/btf_ids.h>
>  #include <linux/cgroup.h>
> +#include <linux/rcupdate.h>
>
> +#ifdef CONFIG_CGROUP_SCHED
>  #include "../sched/sched.h"
> +#endif
>
> -#ifdef CONFIG_CGROUP_SCHED
>  __bpf_kfunc_start_defs();
>
> +/**
> + * bpf_cgroup_css - Get a reference to one controller's css
> + * @cgrp: cgroup to look in
> + * @ssid: controller ID
> + *
> + * The returned css must be released with bpf_css_release().
> + *
> + * Return: The referenced css, or NULL.
> + */
> +__bpf_kfunc struct cgroup_subsys_state *
> +bpf_cgroup_css(struct cgroup *cgrp, int ssid)
> +{
> +	struct cgroup_subsys_state *css;
> +
> +	if (unlikely(ssid < 0 || ssid >= CGROUP_SUBSYS_COUNT))
> +		return NULL;
> +
> +	rcu_read_lock();
> +	css = rcu_dereference(cgrp->subsys[ssid]);
> +	if (css && !css_tryget(css))
> +		css = NULL;
> +	rcu_read_unlock();
> +
> +	return css;
> +}
> +
> +/**
> + * bpf_css_release - Release a css reference
> + * @css: css to release
> + */
> +__bpf_kfunc void bpf_css_release(struct cgroup_subsys_state *css)
> +{
> +	css_put(css);
> +}
> +
> +#ifdef CONFIG_CGROUP_SCHED

[ ... ]

> @@ -30,29 +68,33 @@ bpf_css_to_task_group(struct cgroup_subsys_state *css)
>
>  	return container_of(css, struct task_group, css);
>  }
> +#endif /* CONFIG_CGROUP_SCHED */
>
>  __bpf_kfunc_end_defs();
>
> -BTF_KFUNCS_START(bpf_cpu_cgroup_kfunc_ids)
> +BTF_KFUNCS_START(bpf_cgroup_kfunc_ids)
> +BTF_ID_FLAGS(func, bpf_cgroup_css, KF_ACQUIRE | KF_RCU | KF_RET_NULL)
> +BTF_ID_FLAGS(func, bpf_css_release, KF_RELEASE)
> +#ifdef CONFIG_CGROUP_SCHED
>  BTF_ID_FLAGS(func, bpf_css_to_task_group,
>  	     KF_RCU | KF_RCU_PROTECTED | KF_RET_NULL)
> -BTF_KFUNCS_END(bpf_cpu_cgroup_kfunc_ids)
> +#endif
> +BTF_KFUNCS_END(bpf_cgroup_kfunc_ids)
>
> -static const struct btf_kfunc_id_set bpf_cpu_cgroup_kfunc_set = {
> +static const struct btf_kfunc_id_set bpf_cgroup_kfunc_set = {
>  	.owner		= THIS_MODULE,
> -	.set		= &bpf_cpu_cgroup_kfunc_ids,
> +	.set		= &bpf_cgroup_kfunc_ids,
>  };
>
> -static int __init bpf_cpu_cgroup_kfunc_init(void)
> +static int __init bpf_cgroup_kfunc_init(void)
>  {
>  	int err;
>
>  	err = register_btf_kfunc_id_set(BPF_PROG_TYPE_UNSPEC,
> -					&bpf_cpu_cgroup_kfunc_set);
> +					&bpf_cgroup_kfunc_set);
>  	if (err)
> -		pr_warn("error while registering cpu cgroup kfuncs: %d\n", err);
> +		pr_warn("error while registering cgroup kfuncs: %d\n", err);
>
>  	return err;
>  }
> -late_initcall(bpf_cpu_cgroup_kfunc_init);
> -#endif /* CONFIG_CGROUP_SCHED */
> +late_initcall(bpf_cgroup_kfunc_init);

This isn't a bug, but the patch adds two controller-agnostic kfuncs
(bpf_cgroup_css and bpf_css_release) in kernel/cgroup/bpf_cgroup.c, along
with a three-way rename (bpf_cpu_cgroup_kfunc_ids to bpf_cgroup_kfunc_ids)
and an ifdef restructure of the whole file.

The immediately preceding commit in this series (382c3bd4 "cgroup: add BPF
kfuncs to read a cpu cgroup's stats") already touched this same file under
a "cgroup:" prefix.

Would it be worth splitting the generic css acquire/release and the rename
into a separate "cgroup:" patch, so this one stays scoped to the block
changes?


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/32419919743

Re: [PATCH v3 2/4] selftests/bpf: add cgroup_iter_cpu test for cpu cgroup kfuncs

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 2671 bytes --]

> diff --git a/tools/testing/selftests/bpf/config b/tools/testing/selftests/bpf/config
> index ea7044f30adc..482b40dde2f9 100644
> --- a/tools/testing/selftests/bpf/config
> +++ b/tools/testing/selftests/bpf/config
> @@ -11,6 +11,9 @@ CONFIG_BPF_STREAM_PARSER=y
>  CONFIG_BPF_SYSCALL=y
>  # CONFIG_BPF_UNPRIV_DEFAULT_OFF is not set
>  CONFIG_CGROUP_BPF=y
> +CONFIG_CGROUP_SCHED=y
> +CONFIG_FAIR_GROUP_SCHED=y
> +CONFIG_CFS_BANDWIDTH=y
>  CONFIG_CRYPTO_HMAC=y
>  CONFIG_CRYPTO_SHA256=y
>  CONFIG_CRYPTO_USER_API=y

This isn't a bug, but CONFIG_CGROUP_SCHED already appears in all five
config.<arch> fragments (config.x86_64:30, config.aarch64:26,
config.s390x:19, config.ppc64el:17, config.riscv64:17), and
CONFIG_CFS_BANDWIDTH already appears in config.x86_64:24.

Now that they are in the common config, should the per-arch copies be
removed so each option lives in one place?

> diff --git a/tools/testing/selftests/bpf/prog_tests/cgroup_iter_cpu.c b/tools/testing/selftests/bpf/prog_tests/cgroup_iter_cpu.c
> new file mode 100644
> index 000000000000..cd7e92ababfb
> --- /dev/null
> +++ b/tools/testing/selftests/bpf/prog_tests/cgroup_iter_cpu.c

[ ... ]

> @@ -0,0 +1,259 @@
> +void test_cgroup_iter_cpu(void)
> +{

[ ... ]

> +	/*
> +	 * cpu.stat cputime grows on every tick a task in the cgroup runs, so
> +	 * stop them all before comparing
> +	 */
> +	if (test__start_subtest("cgroup_iter_cpu__match")) {
> +		struct cpu_query filev = {};
> +		int i, stable = 0;
> +
> +		kill(hog, SIGSTOP);
> +		waitpid(hog, NULL, WUNTRACED);
> +		if (!ASSERT_OK(join_root_cgroup(), "join_root_cgroup"))
> +			goto cleanup_hog;
> +
> +		/*
> +		 * The period timer keeps adding to nr_periods for a while
> +		 * after the hog stops
> +		 */
> +		for (i = 0; i < 20; i++) {
> +			struct cpu_query before = {}, after = {};
> +
> +			if (!ASSERT_OK(parse_stats(cgroup_fd, &before, have_bw), "cpu.stat") ||
> +			    !ASSERT_OK(read_stats(link), "read stats") ||
> +			    !ASSERT_OK(parse_stats(cgroup_fd, &after, have_bw), "cpu.stat"))
> +				goto cleanup_hog;
> +
> +			if (!memcmp(&before, &after, sizeof(before))) {
> +				filev = before;
> +				stable = 1;
> +				break;
> +			}
> +			usleep(100000);
> +		}

This isn't a bug, but could the 20 iterations and usleep(100000) get names
the way cgroup_iter_memcg.c names MEMCG_STAT_RETRIES /
MEMCG_STAT_RETRY_DELAY_US, so the relationship to the 100ms cpu.max period
is visible?

[ ... ]


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/32419919743