[PATCH 0/2] block: expose blkcg io.stat to BPF

[PATCH 0/2] block: expose blkcg io.stat to BPF

[Ziyang Men]

This series exposes the block I/O controller's per-device statistics
(io.stat) to BPF via read-only kfuncs, mirroring the kfuncs for the memory
controller in mm/bpf_memcontrol.c.

Motivation:
Collecting cgroup statistics is expensive: the existing method is to open
and parse a cgroup file for every cgroup of interest. memcg already has an
efficient alternative through BPF; this series extends that idea to block.

The main difference from the memory controller is that blkcg keeps one
blkg (one io.stat line) per block device, so the readers operate on a
blkg and the iterator yields them under RCU. Reads take the same
u64_stats seqlock as the io.stat file, so the kfuncs add no fast-path
cost.

The root cgroup needs its own path. It is not accounted through rstat at
all, so bpf_blkcg_flush_stats() branches the way blkcg_print_stat() does
and refills the root's per-device aggregates from the disks' own
statistics. That is the only reason the series touches blk-cgroup.c and
blk-cgroup.h: blkcg_fill_root_iostats() is no longer static. Nothing on
the I/O path changes.

Patch 1 adds the kfuncs (block/bpf_blkcg.c); patch 2 adds a test_progs
selftest that drives direct I/O to a loop device and checks 1) that the
measured values are non-zero (as cgroup_iter_memcg does), 2) that each
kfunc value exactly matches the cgroup's io.stat file for that device,
and 3) that the same device read through the root block cgroup is at or
above what the test cgroup was charged.

Tested on v7.2-rc5 with the series applied: all five cgroup_iter_io
subtests pass (write/read/dev/match/root).

Ziyang Men (2):
  block: add BPF kfuncs to read blkcg io.stat
  selftests/bpf: add test for blkcg io.stat BPF kfuncs

 MAINTAINERS                                   |   1 +
 block/Makefile                                |   3 +
 block/blk-cgroup.c                            |   2 +-
 block/blk-cgroup.h                            |   1 +
 block/bpf_blkcg.c                             | 315 ++++++++++++++++++
 tools/testing/selftests/bpf/cgroup_iter_io.h  |  17 +
 tools/testing/selftests/bpf/config            |   1 +
 .../selftests/bpf/prog_tests/cgroup_iter_io.c | 310 +++++++++++++++++
 .../selftests/bpf/progs/cgroup_iter_io.c      | 107 ++++++
 9 files changed, 756 insertions(+), 1 deletion(-)
 create mode 100644 block/bpf_blkcg.c
 create mode 100644 tools/testing/selftests/bpf/cgroup_iter_io.h
 create mode 100644 tools/testing/selftests/bpf/prog_tests/cgroup_iter_io.c
 create mode 100644 tools/testing/selftests/bpf/progs/cgroup_iter_io.c

-- 
2.53.0-Meta

[PATCH 1/2] block: add BPF kfuncs to read blkcg io.stat

[Ziyang Men]

Expose the block I/O controller's per-device statistics to BPF,
mirroring the memory controller kfuncs in mm/bpf_memcontrol.c.

A BPF program gets a blkcg from a cgroup's css with bpf_get_blkcg()
(or bpf_get_root_blkcg() for the root), flushes the stats with
bpf_blkcg_flush_stats(), then walks the cgroup's per-device blkgs with
the bpf_iter_blkg open-coded iterator and reads each device's counters
with bpf_blkg_iostat_bytes() and bpf_blkg_iostat_ios().  bpf_blkg_dev()
returns the device id for labelling.  The reference is released with
bpf_put_blkcg().

Unlike the memory controller, blkcg keeps one blkg (and one io.stat
line) per block device, so the reader kfuncs take a blkg and the
iterator yields them under RCU.  The counters are read under the same
u64_stats seqlock the io.stat file uses, so the kfuncs add no fast-path
cost: accounting stays in the per-cpu blkg iostat and is only folded on
flush.

bpf_blkcg_flush_stats() branches the way blkcg_print_stat() does.  A
non-root cgroup is flushed through rstat.  The root cgroup is not
accounted through rstat at all - blkcg_rstat_flush() returns early for
it and __blkcg_rstat_flush() stops propagating one level short - so its
per-device aggregates are refilled from the disks' own statistics
instead, by blkcg_fill_root_iostats(), which is no longer static for
that reason.  Without this a program reading the root cgroup would see
zeroes.  Those numbers cover every cgroup's I/O, exactly as the root
io.stat file reports them.

Two details are worth calling out:

bpf_iter_blkg_next() forgets the list head once the walk ends, not just
the position.  process_iter_next_call() in the verifier requires an
iterator to keep returning NULL once it has returned it, and stops
checking the loop for termination at that point; restarting the walk
would let such a loop spin forever.

The counter readers give up instead of retrying when called from NMI on
32-bit.  There the u64_stats read is a real seqcount loop, every writer
of blkg->iostat keeps interrupts off, and a perf event program can call
these kfuncs from NMI, where the loop would never end.  On 64-bit the
loop compiles away.

Signed-off-by: Ziyang Men <ziyang.meme@gmail.com>
---
 MAINTAINERS        |   1 +
 block/Makefile     |   3 +
 block/blk-cgroup.c |   2 +-
 block/blk-cgroup.h |   1 +
 block/bpf_blkcg.c  | 315 +++++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 321 insertions(+), 1 deletion(-)
 create mode 100644 block/bpf_blkcg.c

diff --git a/MAINTAINERS b/MAINTAINERS
index 2f9472c1a090..87c56e955577 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -6617,6 +6617,7 @@ F:	block/blk-cgroup.c
 F:	block/blk-iocost.c
 F:	block/blk-iolatency.c
 F:	block/blk-throttle.c
+F:	block/bpf_blkcg.c
 F:	include/linux/blk-cgroup.h
 
 CONTROL GROUP - CPUSET
diff --git a/block/Makefile b/block/Makefile
index e7bd320e3d69..572e49988c8e 100644
--- a/block/Makefile
+++ b/block/Makefile
@@ -17,6 +17,9 @@ obj-$(CONFIG_BLK_ERROR_INJECTION) += error-injection.o
 obj-$(CONFIG_BLK_DEV_BSG_COMMON) += bsg.o
 obj-$(CONFIG_BLK_DEV_BSGLIB)	+= bsg-lib.o
 obj-$(CONFIG_BLK_CGROUP)	+= blk-cgroup.o
+ifdef CONFIG_BPF_SYSCALL
+obj-$(CONFIG_BLK_CGROUP)	+= bpf_blkcg.o
+endif
 obj-$(CONFIG_BLK_CGROUP_RWSTAT)	+= blk-cgroup-rwstat.o
 obj-$(CONFIG_BLK_CGROUP_FC_APPID) += blk-cgroup-fc-appid.o
 obj-$(CONFIG_BLK_DEV_THROTTLING)	+= blk-throttle.o
diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c
index d9676126c5b5..8d538ad4e861 100644
--- a/block/blk-cgroup.c
+++ b/block/blk-cgroup.c
@@ -1086,7 +1086,7 @@ static void blkcg_rstat_flush(struct cgroup_subsys_state *css, int cpu)
  * flushing the root cgroup's stats by explicitly filling in the iostat
  * with disk level statistics.
  */
-static void blkcg_fill_root_iostats(void)
+void blkcg_fill_root_iostats(void)
 {
 	struct class_dev_iter iter;
 	struct device *dev;
diff --git a/block/blk-cgroup.h b/block/blk-cgroup.h
index 615390f751aa..8c9c2a1adfaa 100644
--- a/block/blk-cgroup.h
+++ b/block/blk-cgroup.h
@@ -205,6 +205,7 @@ void blkcg_deactivate_policy(struct gendisk *disk,
 			     const struct blkcg_policy *pol);
 
 const char *blkg_dev_name(struct blkcg_gq *blkg);
+void blkcg_fill_root_iostats(void);
 void blkcg_print_blkgs(struct seq_file *sf, struct blkcg *blkcg,
 		       u64 (*prfill)(struct seq_file *,
 				     struct blkg_policy_data *, int),
diff --git a/block/bpf_blkcg.c b/block/bpf_blkcg.c
new file mode 100644
index 000000000000..48a86f07e198
--- /dev/null
+++ b/block/bpf_blkcg.c
@@ -0,0 +1,315 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * Block I/O Controller-related BPF kfuncs and auxiliary code.
+ *
+ * These let a BPF program read a cgroup's io.stat counters. A program turns a
+ * cgroup's css into a struct blkcg with bpf_get_blkcg(), flushes the stats with
+ * bpf_blkcg_flush_stats(), then walks the cgroup's per-device blkgs with the
+ * bpf_iter_blkg open-coded iterator, reading each device's counters with
+ * bpf_blkg_iostat_bytes()/bpf_blkg_iostat_ios(). It mirrors the memory
+ * controller kfuncs in mm/bpf_memcontrol.c, but adds a per-device dimension:
+ * unlike memcg, blkcg keeps one blkg (and one io.stat line) per block device.
+ *
+ * This file lives in block/ because the blkcg/blkg struct layouts are private
+ * to block/blk-cgroup.h.
+ */
+
+#include "blk-cgroup.h"
+
+#include <linux/bpf.h>
+#include <linux/btf_ids.h>
+#include <linux/preempt.h>
+#include <linux/rculist.h>
+
+__bpf_kfunc_start_defs();
+
+/**
+ * bpf_get_root_blkcg - Returns a pointer to the root block cgroup
+ *
+ * The function has KF_ACQUIRE semantics, even though the root block cgroup is
+ * never destroyed and doesn't require reference counting. It's safe to pass it
+ * to bpf_put_blkcg().
+ *
+ * Note that the root cgroup is special: its counters are the disks' own
+ * statistics, so they cover every cgroup's I/O rather than only the root's.
+ * This matches what the root io.stat file prints.
+ *
+ * Return: A pointer to the root block cgroup.
+ */
+__bpf_kfunc struct blkcg *bpf_get_root_blkcg(void)
+{
+	/* css_get() is not needed */
+	return &blkcg_root;
+}
+
+/**
+ * bpf_get_blkcg - Get a reference to a block cgroup
+ * @css: pointer to the css structure
+ *
+ * It's fine to pass a css which belongs to any cgroup controller,
+ * e.g. unified hierarchy's main css.
+ *
+ * Implements KF_ACQUIRE semantics.
+ *
+ * Return: A pointer to a blkcg structure after bumping the corresponding css's
+ * reference counter, or NULL if the io controller is not enabled on the cgroup.
+ */
+__bpf_kfunc struct blkcg *bpf_get_blkcg(struct cgroup_subsys_state *css)
+{
+	struct blkcg *blkcg = NULL;
+
+	if (css->ss == &io_cgrp_subsys)
+		return css_tryget(css) ? css_to_blkcg(css) : NULL;
+
+	/*
+	 * Some other controller's css, or the cgroup's own one. Look up the io
+	 * controller's css; rcu keeps it alive between the load and the tryget.
+	 * Acquire and release rcu on one straight path, so that block/'s lock
+	 * context analysis can follow it.
+	 */
+	rcu_read_lock();
+	css = rcu_dereference_raw(css->cgroup->subsys[io_cgrp_id]);
+	if (css && css_tryget(css))
+		blkcg = css_to_blkcg(css);
+	rcu_read_unlock();
+
+	return blkcg;
+}
+
+/**
+ * bpf_put_blkcg - Put a reference to a block cgroup
+ * @blkcg: block cgroup to release
+ *
+ * Releases a previously acquired blkcg reference.
+ * Implements KF_RELEASE semantics.
+ */
+__bpf_kfunc void bpf_put_blkcg(struct blkcg *blkcg)
+{
+	css_put(&blkcg->css);
+}
+
+/**
+ * bpf_blkcg_flush_stats - Flush a block cgroup's io statistics
+ * @blkcg: block cgroup
+ *
+ * Call this before reading counters for up-to-date values. Sleepable.
+ *
+ * It does what reading the io.stat file does, which differs by cgroup. For a
+ * non-root cgroup it folds the per-cpu deltas into the per-device aggregates
+ * and up the cgroup tree. The root cgroup is not accounted through rstat at
+ * all, so for it the per-device aggregates are refilled from the disks'
+ * own statistics, which count every cgroup's I/O.
+ *
+ * The root branch is not self-limiting the way the rstat one is: it rereads
+ * every disk on every call, while a second rstat flush finds nothing left to
+ * fold. The numbers it produces are the same for every cgroup, so read them
+ * once rather than once per cgroup of a walk.
+ */
+__bpf_kfunc void bpf_blkcg_flush_stats(struct blkcg *blkcg)
+{
+	if (!blkcg->css.parent)
+		blkcg_fill_root_iostats();
+	else
+		css_rstat_flush(&blkcg->css);
+}
+
+struct bpf_iter_blkg {
+	__u64 __opaque[2];
+} __attribute__((aligned(8)));
+
+struct bpf_iter_blkg_kern {
+	struct blkcg *blkcg;
+	struct blkcg_gq *pos;
+} __attribute__((aligned(8)));
+
+/**
+ * bpf_iter_blkg_new - Start iterating a block cgroup's per-device blkgs
+ * @it: iterator to initialize
+ * @blkcg: block cgroup whose devices to walk
+ *
+ * Each yielded blkg holds one block device's counters, the same ones behind a
+ * per-device line of the io.stat file. Offline blkgs are skipped, as the file
+ * skips them. One case differs: the file prints no line for a blkg whose disk
+ * is gone, while the walk still yields it, and bpf_blkg_dev() returns 0 for
+ * it. Must be used inside an RCU read section.
+ *
+ * Return: 0 on success.
+ */
+__bpf_kfunc int bpf_iter_blkg_new(struct bpf_iter_blkg *it, struct blkcg *blkcg)
+{
+	struct bpf_iter_blkg_kern *kit = (void *)it;
+
+	BUILD_BUG_ON(sizeof(struct bpf_iter_blkg_kern) > sizeof(struct bpf_iter_blkg));
+	BUILD_BUG_ON(__alignof__(struct bpf_iter_blkg_kern) !=
+		     __alignof__(struct bpf_iter_blkg));
+
+	kit->blkcg = blkcg;
+	kit->pos = NULL;
+	return 0;
+}
+
+/**
+ * bpf_iter_blkg_next - Return the next online blkg of the iterated block cgroup
+ * @it: iterator
+ *
+ * Return: the next online blkg, or NULL when the walk is done.
+ */
+__bpf_kfunc struct blkcg_gq *bpf_iter_blkg_next(struct bpf_iter_blkg *it)
+{
+	struct bpf_iter_blkg_kern *kit = (void *)it;
+	struct blkcg_gq *blkg = kit->pos;
+	struct hlist_node *node;
+
+	/* Cleared once the walk is done, see below. */
+	if (!kit->blkcg)
+		return NULL;
+
+	if (!blkg)
+		node = rcu_dereference(hlist_first_rcu(&kit->blkcg->blkg_list));
+	else
+		node = rcu_dereference(hlist_next_rcu(&blkg->blkcg_node));
+
+	/* Skip offline blkgs, matching the io.stat file. */
+	while (node) {
+		blkg = hlist_entry(node, struct blkcg_gq, blkcg_node);
+		if (blkg->online) {
+			kit->pos = blkg;
+			return blkg;
+		}
+		node = rcu_dereference(hlist_next_rcu(&blkg->blkcg_node));
+	}
+
+	/*
+	 * Forget the list head as well. The verifier assumes that an iterator
+	 * which returned NULL keeps returning NULL, and stops checking the
+	 * loop for termination once it has; starting the walk over would let
+	 * such a loop spin forever.
+	 */
+	kit->pos = NULL;
+	kit->blkcg = NULL;
+	return NULL;
+}
+
+/**
+ * bpf_iter_blkg_destroy - Tear down a blkg iterator
+ * @it: iterator
+ */
+__bpf_kfunc void bpf_iter_blkg_destroy(struct bpf_iter_blkg *it)
+{
+}
+
+/*
+ * Read one counter out of @blkg's flushed io.stat aggregate. @counters is one
+ * of the two arrays in blkg->iostat.cur; both are guarded by that struct's
+ * seqlock, the one the io.stat file uses. Returns (u64)-1 if the counter
+ * cannot be read.
+ */
+static u64 blkg_iostat_read(struct blkcg_gq *blkg, const u64 *counters,
+			    enum blkg_iostat_type rw)
+{
+	struct blkg_iostat_set *bis = &blkg->iostat;
+	unsigned int seq;
+	u64 val;
+
+	if ((unsigned int)rw >= BLKG_IOSTAT_NR)
+		return (u64)-1;
+
+	/*
+	 * On 32-bit the loop below really is a seqcount retry loop. Every
+	 * writer of blkg->iostat keeps interrupts off, so only an NMI can land
+	 * inside an update, and then the loop would never end. These kfuncs
+	 * are reachable from a perf event program, which does run in NMI, so
+	 * give up rather than spin. On 64-bit the loop compiles away.
+	 */
+	if (BITS_PER_LONG == 32 && in_nmi())
+		return (u64)-1;
+
+	do {
+		seq = u64_stats_fetch_begin(&bis->sync);
+		val = counters[rw];
+	} while (u64_stats_fetch_retry(&bis->sync, seq));
+
+	return val;
+}
+
+/**
+ * bpf_blkg_iostat_bytes - Read a device's io.stat byte counter
+ * @blkg: block group (one device of a block cgroup)
+ * @rw: which counter (BLKG_IOSTAT_READ / _WRITE / _DISCARD)
+ *
+ * Reads the flushed aggregate, so call bpf_blkcg_flush_stats() first for
+ * up-to-date values. The read uses the u64_stats seqlock, like the io.stat
+ * file.
+ *
+ * Return: the number of bytes, or (u64)-1 if @rw is out of range or the
+ * counter cannot be read.
+ */
+__bpf_kfunc u64 bpf_blkg_iostat_bytes(struct blkcg_gq *blkg,
+				      enum blkg_iostat_type rw)
+{
+	return blkg_iostat_read(blkg, blkg->iostat.cur.bytes, rw);
+}
+
+/**
+ * bpf_blkg_iostat_ios - Read a device's io.stat I/O count
+ * @blkg: block group (one device of a block cgroup)
+ * @rw: which counter (BLKG_IOSTAT_READ / _WRITE / _DISCARD)
+ *
+ * Return: the number of I/Os, or (u64)-1 if @rw is out of range or the
+ * counter cannot be read.
+ */
+__bpf_kfunc u64 bpf_blkg_iostat_ios(struct blkcg_gq *blkg,
+				    enum blkg_iostat_type rw)
+{
+	return blkg_iostat_read(blkg, blkg->iostat.cur.ios, rw);
+}
+
+/**
+ * bpf_blkg_dev - Return a blkg's device id
+ * @blkg: block group
+ *
+ * Return: the device's dev_t (use MAJOR()/MINOR() to split), or 0 if the blkg
+ * has no disk.
+ */
+__bpf_kfunc u64 bpf_blkg_dev(struct blkcg_gq *blkg)
+{
+	if (!blkg->q || !blkg->q->disk)
+		return 0;
+
+	return blkg->q->disk->part0->bd_dev;
+}
+
+__bpf_kfunc_end_defs();
+
+BTF_KFUNCS_START(bpf_blkcg_kfuncs)
+BTF_ID_FLAGS(func, bpf_get_root_blkcg, KF_ACQUIRE | KF_RET_NULL)
+BTF_ID_FLAGS(func, bpf_get_blkcg, KF_ACQUIRE | KF_RET_NULL | KF_RCU)
+BTF_ID_FLAGS(func, bpf_put_blkcg, KF_RELEASE)
+BTF_ID_FLAGS(func, bpf_blkcg_flush_stats, KF_SLEEPABLE)
+
+BTF_ID_FLAGS(func, bpf_iter_blkg_new, KF_ITER_NEW | KF_RCU_PROTECTED)
+BTF_ID_FLAGS(func, bpf_iter_blkg_next, KF_ITER_NEXT | KF_RET_NULL)
+BTF_ID_FLAGS(func, bpf_iter_blkg_destroy, KF_ITER_DESTROY)
+
+BTF_ID_FLAGS(func, bpf_blkg_iostat_bytes, KF_RCU)
+BTF_ID_FLAGS(func, bpf_blkg_iostat_ios, KF_RCU)
+BTF_ID_FLAGS(func, bpf_blkg_dev, KF_RCU)
+BTF_KFUNCS_END(bpf_blkcg_kfuncs)
+
+static const struct btf_kfunc_id_set bpf_blkcg_kfunc_set = {
+	.owner		= THIS_MODULE,
+	.set		= &bpf_blkcg_kfuncs,
+};
+
+static int __init bpf_blkcg_init(void)
+{
+	int err;
+
+	err = register_btf_kfunc_id_set(BPF_PROG_TYPE_UNSPEC,
+					&bpf_blkcg_kfunc_set);
+	if (err)
+		pr_warn("error while registering bpf blkcg kfuncs: %d\n", err);
+
+	return err;
+}
+late_initcall(bpf_blkcg_init);
-- 
2.53.0-Meta

[PATCH 2/2] selftests/bpf: add test for blkcg io.stat BPF kfuncs

[Ziyang Men]

Add cgroup_iter_io, a test_progs test for the block I/O controller BPF
kfuncs.  A SEC("iter.s/cgroup") program acquires the cgroup's blkcg,
flushes stats, iterates its blkgs and reads the io.stat counters for a
target device.

The userspace side attaches a loop device, generates O_DIRECT read and
write I/O charged to a test cgroup, and then:

  - checks the write and read byte/io counters are nonzero,
  - checks the reported device id,
  - compares every kfunc-read value against the cgroup's io.stat file
    for the same device and requires an exact match,
  - reads the same device through bpf_get_root_blkcg() and checks the
    root counters are at or above the test cgroup's.

The measured device is pinned to the loop device, which has no
asynchronous writeback, so the kfunc snapshot and the io.stat file
snapshot are identical rather than merely close.  The root cgroup's
numbers for a device come from the disk itself and so cover every
cgroup's I/O to it, which is why the root check is "at or above" rather
than an exact match.

CONFIG_BLK_CGROUP is added to the test config; CONFIG_BLK_DEV_LOOP is
already present.

Signed-off-by: Ziyang Men <ziyang.meme@gmail.com>
---
 tools/testing/selftests/bpf/cgroup_iter_io.h  |  17 +
 tools/testing/selftests/bpf/config            |   1 +
 .../selftests/bpf/prog_tests/cgroup_iter_io.c | 310 ++++++++++++++++++
 .../selftests/bpf/progs/cgroup_iter_io.c      | 107 ++++++
 4 files changed, 435 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/cgroup_iter_io.h
 create mode 100644 tools/testing/selftests/bpf/prog_tests/cgroup_iter_io.c
 create mode 100644 tools/testing/selftests/bpf/progs/cgroup_iter_io.c

diff --git a/tools/testing/selftests/bpf/cgroup_iter_io.h b/tools/testing/selftests/bpf/cgroup_iter_io.h
new file mode 100644
index 000000000000..f4bbaaccdf71
--- /dev/null
+++ b/tools/testing/selftests/bpf/cgroup_iter_io.h
@@ -0,0 +1,17 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */
+#ifndef __CGROUP_ITER_IO_H
+#define __CGROUP_ITER_IO_H
+
+struct io_query {
+	/* one device's io.stat counters */
+	__u64 rbytes;
+	__u64 wbytes;
+	__u64 rios;
+	__u64 wios;
+	__u64 dbytes;
+	__u64 dios;
+	__u64 dev;	/* dev_t of the device the counters belong to */
+};
+
+#endif /* __CGROUP_ITER_IO_H */
diff --git a/tools/testing/selftests/bpf/config b/tools/testing/selftests/bpf/config
index ea7044f30adc..270e6bf9194d 100644
--- a/tools/testing/selftests/bpf/config
+++ b/tools/testing/selftests/bpf/config
@@ -1,3 +1,4 @@
+CONFIG_BLK_CGROUP=y
 CONFIG_BLK_DEV_LOOP=y
 CONFIG_BOOTPARAM_HARDLOCKUP_PANIC=y
 CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC=1
diff --git a/tools/testing/selftests/bpf/prog_tests/cgroup_iter_io.c b/tools/testing/selftests/bpf/prog_tests/cgroup_iter_io.c
new file mode 100644
index 000000000000..32cda8243318
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/cgroup_iter_io.c
@@ -0,0 +1,310 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */
+#define _GNU_SOURCE
+#include <test_progs.h>
+#include <bpf/libbpf.h>
+#include <fcntl.h>
+#include <linux/loop.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/ioctl.h>
+#include <sys/stat.h>
+#include <sys/sysmacros.h>
+#include <unistd.h>
+#include "cgroup_helpers.h"
+#include "cgroup_iter_io.h"
+#include "cgroup_iter_io.skel.h"
+
+#define IO_SIZE (4 * 1024 * 1024)
+
+static int read_stats(struct bpf_link *link)
+{
+	int fd, ret = 0;
+	ssize_t bytes;
+
+	fd = bpf_iter_create(bpf_link__fd(link));
+	if (!ASSERT_OK_FD(fd, "bpf_iter_create"))
+		return 1;
+
+	/* Results land in skel->data_query; the read itself returns no data. */
+	bytes = read(fd, NULL, 0);
+	if (!ASSERT_EQ(bytes, 0, "read fd"))
+		ret = 1;
+
+	close(fd);
+	return ret;
+}
+
+/*
+ * Attach a loop device to an anonymous temp file so we have a real block
+ * device to generate cgroup-charged I/O against. Returns 0 on success, or -1
+ * if loop devices are unavailable (non-root / no CONFIG_BLK_DEV_LOOP) so the
+ * caller can skip.
+ */
+static int loop_setup(char *loop_path, size_t sz, int *ctl_fd, int *loop_fd,
+		      int *back_fd)
+{
+	char back_path[] = "/tmp/cgroup_iter_io.XXXXXX";
+	int nr;
+
+	*ctl_fd = *loop_fd = *back_fd = -1;
+
+	*ctl_fd = open("/dev/loop-control", O_RDWR | O_CLOEXEC);
+	if (*ctl_fd < 0)
+		return -1;
+
+	nr = ioctl(*ctl_fd, LOOP_CTL_GET_FREE);
+	if (nr < 0)
+		goto err;
+	snprintf(loop_path, sz, "/dev/loop%d", nr);
+
+	*back_fd = mkstemp(back_path);
+	if (*back_fd < 0)
+		goto err;
+	unlink(back_path);
+	if (ftruncate(*back_fd, (off_t)IO_SIZE * 4))
+		goto err;
+
+	*loop_fd = open(loop_path, O_RDWR | O_CLOEXEC);
+	if (*loop_fd < 0)
+		goto err;
+	if (ioctl(*loop_fd, LOOP_SET_FD, *back_fd))
+		goto err;
+
+	return 0;
+err:
+	if (*loop_fd >= 0)
+		close(*loop_fd);
+	if (*back_fd >= 0)
+		close(*back_fd);
+	close(*ctl_fd);
+	*ctl_fd = *loop_fd = *back_fd = -1;
+	return -1;
+}
+
+static void loop_teardown(const char *loop_path, int ctl_fd, int loop_fd,
+			  int back_fd)
+{
+	int nr = -1;
+
+	if (loop_fd >= 0) {
+		ioctl(loop_fd, LOOP_CLR_FD, 0);
+		close(loop_fd);
+	}
+	if (back_fd >= 0)
+		close(back_fd);
+	if (ctl_fd >= 0) {
+		if (sscanf(loop_path, "/dev/loop%d", &nr) == 1 && nr >= 0)
+			ioctl(ctl_fd, LOOP_CTL_REMOVE, nr);
+		close(ctl_fd);
+	}
+}
+
+/* O_DIRECT I/O to the loop device, charged to the current cgroup. */
+static int do_direct_io(const char *loop_path)
+{
+	void *buf;
+	int fd, ret = -1;
+
+	fd = open(loop_path, O_RDWR | O_DIRECT | O_CLOEXEC);
+	if (fd < 0)
+		return -1;
+	if (posix_memalign(&buf, 4096, IO_SIZE))
+		goto out_fd;
+	memset(buf, 0xab, IO_SIZE);
+
+	if (pwrite(fd, buf, IO_SIZE, 0) != IO_SIZE)
+		goto out_buf;
+	fsync(fd);
+	if (pread(fd, buf, IO_SIZE, 0) != IO_SIZE)
+		goto out_buf;
+	ret = 0;
+out_buf:
+	free(buf);
+out_fd:
+	close(fd);
+	return ret;
+}
+
+/*
+ * Parse the io.stat line for device @dev out of the cgroup's io.stat file and
+ * fill @out. @dev is a kernel dev_t (as returned by bpf_blkg_dev), whose
+ * major:minor split matches how io.stat prints the device. Returns 0 if the
+ * device's line was found.
+ */
+static int parse_io_stat(int cgroup_fd, __u64 dev, struct io_query *out)
+{
+	unsigned int want_maj = dev >> 20, want_min = dev & ((1U << 20) - 1);
+	char buf[4096], *line, *saveptr;
+	int fd, n, ret = -1;
+
+	fd = openat(cgroup_fd, "io.stat", O_RDONLY);
+	if (fd < 0)
+		return -1;
+	n = read(fd, buf, sizeof(buf) - 1);
+	close(fd);
+	if (n <= 0)
+		return -1;
+	buf[n] = '\0';
+
+	for (line = strtok_r(buf, "\n", &saveptr); line;
+	     line = strtok_r(NULL, "\n", &saveptr)) {
+		unsigned long long rb = 0, wb = 0, ri = 0, wi = 0, db = 0, di = 0;
+		unsigned int maj, min;
+
+		/*
+		 * The "maj:min" token is always present; the field block is
+		 * optional (the kernel omits it for a device with no read/write
+		 * I/O), so a match of >= 2 is enough and absent fields stay 0.
+		 */
+		if (sscanf(line,
+			   "%u:%u rbytes=%llu wbytes=%llu rios=%llu wios=%llu dbytes=%llu dios=%llu",
+			   &maj, &min, &rb, &wb, &ri, &wi, &db, &di) < 2)
+			continue;
+		if (maj != want_maj || min != want_min)
+			continue;
+
+		out->rbytes = rb;
+		out->wbytes = wb;
+		out->rios = ri;
+		out->wios = wi;
+		out->dbytes = db;
+		out->dios = di;
+		ret = 0;
+		break;
+	}
+	return ret;
+}
+
+void test_cgroup_iter_io(void)
+{
+	char *cgroup_rel_path = "/cgroup_iter_io_test";
+	int ctl_fd = -1, loop_fd = -1, back_fd = -1;
+	struct cgroup_iter_io *skel = NULL;
+	struct bpf_link *link = NULL;
+	char loop_path[64];
+	struct io_query *q;
+	int cgroup_fd;
+
+	cgroup_fd = cgroup_setup_and_join(cgroup_rel_path);
+	if (!ASSERT_OK_FD(cgroup_fd, "cgroup_setup_and_join"))
+		return;
+
+	if (loop_setup(loop_path, sizeof(loop_path), &ctl_fd, &loop_fd, &back_fd)) {
+		test__skip();	/* needs root + CONFIG_BLK_DEV_LOOP */
+		goto cleanup_cgroup_fd;
+	}
+
+	skel = cgroup_iter_io__open_and_load();
+	if (!ASSERT_OK_PTR(skel, "cgroup_iter_io__open_and_load"))
+		goto cleanup_loop;
+
+	/*
+	 * Pin the read to the loop device so the measured device is stable and
+	 * quiesced. Convert the glibc-encoded st_rdev to the kernel dev_t
+	 * encoding (major << 20 | minor) that bpf_blkg_dev returns.
+	 */
+	{
+		struct stat lst;
+
+		if (!ASSERT_OK(fstat(loop_fd, &lst), "fstat loop"))
+			goto cleanup_skel;
+		skel->data_query->target_dev =
+			((__u64)major(lst.st_rdev) << 20) | minor(lst.st_rdev);
+	}
+
+	DECLARE_LIBBPF_OPTS(bpf_iter_attach_opts, opts);
+	union bpf_iter_link_info linfo = {
+		.cgroup.cgroup_fd = cgroup_fd,
+		.cgroup.order = BPF_CGROUP_ITER_SELF_ONLY,
+	};
+	opts.link_info = &linfo;
+	opts.link_info_len = sizeof(linfo);
+
+	link = bpf_program__attach_iter(skel->progs.cgroup_io_query, &opts);
+	if (!ASSERT_OK_PTR(link, "bpf_program__attach_iter"))
+		goto cleanup_skel;
+
+	/* This process is in the test cgroup, so the loop I/O is charged here. */
+	if (!ASSERT_OK(do_direct_io(loop_path), "do_direct_io"))
+		goto cleanup_link;
+
+	if (!ASSERT_OK(read_stats(link), "read stats"))
+		goto cleanup_link;
+
+	/*
+	 * Weak check: we did I/O, so the numbers must be non-zero. Follows the
+	 * pattern in cgroup_iter_memcg.
+	 */
+	q = &skel->data_query->io_query;
+	if (test__start_subtest("cgroup_iter_io__write")) {
+		ASSERT_GT(q->wbytes, 0, "wbytes");
+		ASSERT_GT(q->wios, 0, "wios");
+	}
+	if (test__start_subtest("cgroup_iter_io__read")) {
+		ASSERT_GT(q->rbytes, 0, "rbytes");
+		ASSERT_GT(q->rios, 0, "rios");
+	}
+	if (test__start_subtest("cgroup_iter_io__dev"))
+		ASSERT_GT(q->dev, 0, "dev");
+
+	/*
+	 * Stronger check: the kfunc-read values must equal what the io.stat
+	 * file reports for the same device. Refresh via the prog, then read
+	 * the file with no I/O in between, so both flushed snapshots match
+	 * exactly.
+	 */
+	if (test__start_subtest("cgroup_iter_io__match")) {
+		struct io_query filev = {};
+
+		if (ASSERT_OK(read_stats(link), "read stats") &&
+		    ASSERT_OK(parse_io_stat(cgroup_fd, q->dev, &filev),
+			      "parse io.stat")) {
+			ASSERT_EQ(q->rbytes, filev.rbytes, "rbytes");
+			ASSERT_EQ(q->wbytes, filev.wbytes, "wbytes");
+			ASSERT_EQ(q->rios, filev.rios, "rios");
+			ASSERT_EQ(q->wios, filev.wios, "wios");
+			ASSERT_EQ(q->dbytes, filev.dbytes, "dbytes");
+			ASSERT_EQ(q->dios, filev.dios, "dios");
+		}
+	}
+
+	/*
+	 * Separate program for the root block cgroup. Its counters do not come
+	 * from rstat, they are refilled from the disks themselves, so this
+	 * covers the other half of bpf_blkcg_flush_stats(). They cover every
+	 * cgroup's I/O to the loop device, and only this test touches it, so
+	 * they must be at or above what the test cgroup was charged.
+	 */
+	if (test__start_subtest("cgroup_iter_io__root")) {
+		struct bpf_link *root_link;
+		struct io_query *r;
+
+		skel->data_query->got_root_blkcg = 0;
+		root_link = bpf_program__attach_iter(skel->progs.cgroup_root_blkcg_query,
+						     &opts);
+		if (ASSERT_OK_PTR(root_link, "attach root iter")) {
+			if (ASSERT_OK(read_stats(root_link), "read root stats")) {
+				r = &skel->data_query->root_query;
+				ASSERT_EQ(skel->data_query->got_root_blkcg, 1,
+					  "got_root_blkcg");
+				ASSERT_EQ(r->dev, q->dev, "root dev");
+				ASSERT_GE(r->wbytes, q->wbytes, "root wbytes");
+				ASSERT_GE(r->wios, q->wios, "root wios");
+				ASSERT_GE(r->rbytes, q->rbytes, "root rbytes");
+				ASSERT_GE(r->rios, q->rios, "root rios");
+			}
+			bpf_link__destroy(root_link);
+		}
+	}
+
+cleanup_link:
+	bpf_link__destroy(link);
+cleanup_skel:
+	cgroup_iter_io__destroy(skel);
+cleanup_loop:
+	loop_teardown(loop_path, ctl_fd, loop_fd, back_fd);
+cleanup_cgroup_fd:
+	close(cgroup_fd);
+	cleanup_cgroup_environment();
+}
diff --git a/tools/testing/selftests/bpf/progs/cgroup_iter_io.c b/tools/testing/selftests/bpf/progs/cgroup_iter_io.c
new file mode 100644
index 000000000000..b839def94508
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/cgroup_iter_io.c
@@ -0,0 +1,107 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_core_read.h>
+#include "bpf_experimental.h"
+#include "cgroup_iter_io.h"
+
+char _license[] SEC("license") = "GPL";
+
+/* The counters of the device named by target_dev are stored here. */
+struct io_query io_query SEC(".data.query");
+
+/* The same device's counters read through the root block cgroup. */
+struct io_query root_query SEC(".data.query");
+
+/* Set to 1 by cgroup_root_blkcg_query when bpf_get_root_blkcg() succeeds. */
+__u64 got_root_blkcg SEC(".data.query");
+
+/* Device to read, set by userspace (kernel dev_t). Pinning the device keeps
+ * the read deterministic and lets the value be compared to io.stat exactly.
+ */
+__u64 target_dev SEC(".data.query");
+
+/*
+ * Flush @blkcg and copy the target device's counters into @out. Reading only
+ * the one pinned device keeps the result deterministic: that device is
+ * quiesced, so its counters match io.stat exactly, while picking "any device
+ * with I/O" would race with backing-store writeback.
+ */
+static __always_inline void read_target_dev(struct blkcg *blkcg,
+					    struct io_query *out)
+{
+	struct blkcg_gq *pos;
+
+	/* io.stat needs a flush before it can be read (sleepable). */
+	bpf_blkcg_flush_stats(blkcg);
+
+	/* The per-device blkg walk needs an RCU section. */
+	bpf_rcu_read_lock();
+	bpf_for_each(blkg, pos, blkcg) {
+		if (bpf_blkg_dev(pos) != target_dev)
+			continue;
+
+		out->dev = bpf_blkg_dev(pos);
+		out->rbytes = bpf_blkg_iostat_bytes(pos, BLKG_IOSTAT_READ);
+		out->wbytes = bpf_blkg_iostat_bytes(pos, BLKG_IOSTAT_WRITE);
+		out->rios = bpf_blkg_iostat_ios(pos, BLKG_IOSTAT_READ);
+		out->wios = bpf_blkg_iostat_ios(pos, BLKG_IOSTAT_WRITE);
+		out->dbytes = bpf_blkg_iostat_bytes(pos, BLKG_IOSTAT_DISCARD);
+		out->dios = bpf_blkg_iostat_ios(pos, BLKG_IOSTAT_DISCARD);
+		break;
+	}
+	bpf_rcu_read_unlock();
+}
+
+SEC("iter.s/cgroup")
+int cgroup_io_query(struct bpf_iter__cgroup *ctx)
+{
+	struct cgroup *cgrp = ctx->cgroup;
+	struct blkcg *blkcg;
+
+	/* The last iteration has a NULL cgroup, skip it. */
+	if (!cgrp)
+		return 1;
+
+	/* Start fresh so a device that is not found stays all-zero. */
+	__builtin_memset(&io_query, 0, sizeof(io_query));
+
+	blkcg = bpf_get_blkcg(&cgrp->self);
+	if (!blkcg)
+		return 0;
+
+	read_target_dev(blkcg, &io_query);
+
+	bpf_put_blkcg(blkcg);
+	return 0;
+}
+
+SEC("iter.s/cgroup")
+int cgroup_root_blkcg_query(struct bpf_iter__cgroup *ctx)
+{
+	struct cgroup *cgrp = ctx->cgroup;
+	struct blkcg *blkcg;
+
+	/* The last iteration has a NULL cgroup, skip it. */
+	if (!cgrp)
+		return 1;
+
+	__builtin_memset(&root_query, 0, sizeof(root_query));
+
+	blkcg = bpf_get_root_blkcg();
+	if (!blkcg)
+		return 0;
+
+	/*
+	 * The root cgroup takes its numbers from the disks themselves rather
+	 * than from rstat, so this also covers the root side of
+	 * bpf_blkcg_flush_stats(). The counters cover every cgroup's I/O, so
+	 * they can only be at or above what this test's own cgroup did.
+	 */
+	read_target_dev(blkcg, &root_query);
+
+	got_root_blkcg = 1;
+	bpf_put_blkcg(blkcg);
+	return 0;
+}
-- 
2.53.0-Meta

Re: [PATCH 2/2] selftests/bpf: add test for blkcg io.stat BPF kfuncs

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] Multi-line comment violates the BPF subsystem comment style guidelines.
--

--- Patch [2]: [PATCH 2/2] selftests/bpf: add test for blkcg io.stat BPF kfuncs ---
commit 0012ecaba2bee2bf79e38a8039b4f591d01a624b
Author: Ziyang Men <ziyang.meme@gmail.com>

selftests/bpf: add test for blkcg io.stat BPF kfuncs

This commit adds a cgroup_iter_io selftest for the block I/O controller
BPF kfuncs. It uses a BPF iterator program to flush stats and read io.stat
counters for a target device, comparing the results against the cgroup's
io.stat file. It also validates the root cgroup's block stats.

> diff --git a/tools/testing/selftests/bpf/progs/cgroup_iter_io.c b/tools/testing/selftests/bpf/progs/cgroup_iter_io.c
> new file mode 100644
> index 0000000000000..b839def945086
> --- /dev/null
> +++ b/tools/testing/selftests/bpf/progs/cgroup_iter_io.c

[ ... ]

> +/* Set to 1 by cgroup_root_blkcg_query when bpf_get_root_blkcg() succeeds. */
> +__u64 got_root_blkcg SEC(".data.query");
> +
> +/* Device to read, set by userspace (kernel dev_t). Pinning the device keeps
> + * the read deterministic and lets the value be compared to io.stat exactly.
> + */
> +__u64 target_dev SEC(".data.query");

[Severity: Low]
This isn't a bug, but does this multi-line comment follow the preferred
BPF subsystem formatting rules?

The guidelines for files under the BPF subsystem require the opening /*
to be placed on its own line before the text begins for multi-line comments.

[ ... ]

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260807193732.4073299-1-ziyang.meme@gmail.com?part=2

Re: [PATCH 1/2] block: add BPF kfuncs to read blkcg io.stat

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

New issues:
- [High] A Time-Of-Check to Time-Of-Use (TOCTOU) race condition in `bpf_blkg_dev` can lead to a NULL pointer dereference when accessing `disk->part0`.

Pre-existing issues:
- [High] Concurrent execution of `blkcg_fill_root_iostats()` lacks mutual exclusion, corrupting the `u64_stats_sync` seqcount and potentially deadlocking readers.
--

--- Patch [1]: [PATCH 1/2] block: add BPF kfuncs to read blkcg io.stat ---
commit 5b3bee962ae05470b0da306e0224ebb6eec2d0ce
Author: Ziyang Men <ziyang.meme@gmail.com>

block: add BPF kfuncs to read blkcg io.stat

This exposes the block I/O controller's per-device statistics to BPF,
mirroring the memory controller kfuncs. It allows BPF programs to read a
cgroup's io.stat counters and flush the stats through new BPF functions.
The patch also modifies blkcg_fill_root_iostats() to be accessible externally.

> diff --git a/block/bpf_blkcg.c b/block/bpf_blkcg.c
> new file mode 100644
> index 0000000000000..48a86f07e198a
> --- /dev/null
> +++ b/block/bpf_blkcg.c
[ ... ]
> @@ -0,0 +1,315 @@
[ ... ]
> +__bpf_kfunc void bpf_blkcg_flush_stats(struct blkcg *blkcg)
> +{
> +	if (!blkcg->css.parent)
> +		blkcg_fill_root_iostats();

[Severity: High]
This is a pre-existing issue, but can this lead to a corrupted sequence
counter and deadlocked readers? 

Multiple BPF programs can call bpf_blkcg_flush_stats() concurrently, which
calls blkcg_fill_root_iostats(). Inside blkcg_fill_root_iostats(), the
update is performed with:

    flags = u64_stats_update_begin_irqsave(&blkg->iostat.sync);
    blkg_iostat_set(&blkg->iostat.cur, &tmp);
    u64_stats_update_end_irqrestore(&blkg->iostat.sync, flags);

Since u64_stats_update_begin_irqsave() only protects against local interrupts,
multiple CPUs executing this concurrently lack an external lock to serialize
writers. Does this allow concurrent writers to corrupt the underlying
seqcount (e.g., leaving it permanently odd), causing subsequent readers
to spin infinitely in u64_stats_fetch_retry()?

> +	else
> +		css_rstat_flush(&blkcg->css);
> +}
[ ... ]
> +__bpf_kfunc u64 bpf_blkg_dev(struct blkcg_gq *blkg)
> +{
> +	if (!blkg->q || !blkg->q->disk)
> +		return 0;
> +
> +	return blkg->q->disk->part0->bd_dev;

[Severity: High]
Can the compiler double-fetch blkg->q->disk here and cause a NULL pointer
dereference?

If a BPF program calls bpf_blkg_dev() while the block device is concurrently
being removed, disk_release() can set disk->queue->disk = NULL synchronously.

Without READ_ONCE(), the compiler is permitted to fetch blkg->q->disk for
the NULL check, and then fetch it again for the return statement. If the
second fetch sees the concurrently-written NULL, it will dereference
NULL->part0 and crash the kernel. Is a local variable paired with
READ_ONCE() needed here?

> +}

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260807193732.4073299-1-ziyang.meme@gmail.com?part=1