[RFC/PATCH bpf-next 0/3] bpf: Add slab iterator and kfunc (v1)

[RFC/PATCH bpf-next 0/3] bpf: Add slab iterator and kfunc (v1)

[Namhyung Kim]

Hello,

I'm proposing a new iterator and a kfunc for the slab memory allocator
to get information of each kmem_cache like in /proc/slabinfo or
/sys/kernel/slab.  Maybe I need to call it kmem_cache iter but slab
was short and easier to call. :)

My use case is `perf lock contention` tool which shows contended locks
but many of them are not global locks and don't have symbols.  If it
can tranlate the address of the lock in a slab object to the name of
the slab, it'd be much more useful.

I'm not aware of type information in slab yet, but I was told there's
a work to associate BTF ID with it.  It'd be definitely helpful to my
use case.  Probably we need another kfunc to get the start address of
the object or the offset in the object from an address if the type
info is available.  But I want to start with a simple thing first.

The slab_iter iterates kmem_cache objects under slab_mutex and will be
useful for userspace to prepare some work for specific slabs like
setting up filters in advance.  And the bpf_get_slab_cache() kfunc
will return a pointer to a slab from the address of a lock.  And the
test code is to read from the iterator and make sure it finds a slab
cache of the task_struct for the current task.

The code is available at 'bpf/slab-iter-v1' branch in
https://git.kernel.org/pub/scm/linux/kernel/git/namhyung/linux-perf.git

Thanks,
Namhyung


Namhyung Kim (3):
  bpf: Add slab iterator
  mm/bpf: Add bpf_get_slab_cache() kfunc
  selftests/bpf: Add a test for slab_iter

 include/linux/btf_ids.h                       |   1 +
 kernel/bpf/Makefile                           |   1 +
 kernel/bpf/helpers.c                          |   1 +
 kernel/bpf/slab_iter.c                        | 131 ++++++++++++++++++
 mm/slab_common.c                              |  14 ++
 .../selftests/bpf/prog_tests/slab_iter.c      |  64 +++++++++
 tools/testing/selftests/bpf/progs/bpf_iter.h  |   7 +
 tools/testing/selftests/bpf/progs/slab_iter.c |  62 +++++++++
 8 files changed, 281 insertions(+)
 create mode 100644 kernel/bpf/slab_iter.c
 create mode 100644 tools/testing/selftests/bpf/prog_tests/slab_iter.c
 create mode 100644 tools/testing/selftests/bpf/progs/slab_iter.c

-- 
2.46.0.792.g87dc391469-goog

[RFC/PATCH bpf-next 1/3] bpf: Add slab iterator

[Namhyung Kim]

The new "slab" iterator will traverse the list of slab caches
(kmem_cache) and call attached BPF programs for each entry.  It should
check the argument (ctx.s) if it's NULL before using it.

The iteration will be done with slab_mutex held but it'd break and
return to user if the BPF program emits data to seq buffer more than
the buffer size given by the user.  IOW the whole iteration would be
protected by the slab_mutex as long as it won't emit anything.

It includes the internal "mm/slab.h" header to access kmem_cache,
slab_caches and slab_mutex.  Hope it's ok to mm folks.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 include/linux/btf_ids.h |   1 +
 kernel/bpf/Makefile     |   1 +
 kernel/bpf/slab_iter.c  | 131 ++++++++++++++++++++++++++++++++++++++++
 3 files changed, 133 insertions(+)
 create mode 100644 kernel/bpf/slab_iter.c

diff --git a/include/linux/btf_ids.h b/include/linux/btf_ids.h
index c0e3e1426a82f5c4..1474ab7f44a9cff6 100644
--- a/include/linux/btf_ids.h
+++ b/include/linux/btf_ids.h
@@ -283,5 +283,6 @@ extern u32 btf_tracing_ids[];
 extern u32 bpf_cgroup_btf_id[];
 extern u32 bpf_local_storage_map_btf_id[];
 extern u32 btf_bpf_map_id[];
+extern u32 bpf_slab_btf_id[];
 
 #endif
diff --git a/kernel/bpf/Makefile b/kernel/bpf/Makefile
index 9b9c151b5c826b31..e18b09069349e1e9 100644
--- a/kernel/bpf/Makefile
+++ b/kernel/bpf/Makefile
@@ -52,3 +52,4 @@ obj-$(CONFIG_BPF_PRELOAD) += preload/
 obj-$(CONFIG_BPF_SYSCALL) += relo_core.o
 obj-$(CONFIG_BPF_SYSCALL) += btf_iter.o
 obj-$(CONFIG_BPF_SYSCALL) += btf_relocate.o
+obj-$(CONFIG_BPF_SYSCALL) += slab_iter.o
diff --git a/kernel/bpf/slab_iter.c b/kernel/bpf/slab_iter.c
new file mode 100644
index 0000000000000000..bf1e50bd7497220e
--- /dev/null
+++ b/kernel/bpf/slab_iter.c
@@ -0,0 +1,131 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright (c) 2024 Google */
+#include <linux/bpf.h>
+#include <linux/btf_ids.h>
+#include <linux/slab.h>
+#include <linux/kernel.h>
+#include <linux/seq_file.h>
+
+#include "../../mm/slab.h" /* kmem_cache, slab_caches and slab_mutex */
+
+struct bpf_iter__slab {
+	__bpf_md_ptr(struct bpf_iter_meta *, meta);
+	__bpf_md_ptr(struct kmem_cache *, s);
+};
+
+static void *slab_iter_seq_start(struct seq_file *seq, loff_t *pos)
+{
+	loff_t cnt = 0;
+	struct kmem_cache *s = NULL;
+
+	mutex_lock(&slab_mutex);
+
+	/*
+	 * Find an entry at the given position in the slab_caches list instead
+	 * of keeping a reference (of the last visited entry, if any) out of
+	 * slab_mutex. It might miss something if one is deleted in the middle
+	 * while it releases the lock.  But it should be rare and there's not
+	 * much we can do about it.
+	 */
+	list_for_each_entry(s, &slab_caches, list) {
+		if (cnt == *pos)
+			break;
+
+		cnt++;
+	}
+
+	if (cnt != *pos)
+		return NULL;
+
+	++*pos;
+	return s;
+}
+
+static void slab_iter_seq_stop(struct seq_file *seq, void *v)
+{
+	struct bpf_iter_meta meta;
+	struct bpf_iter__slab ctx = {
+		.meta = &meta,
+		.s = v,
+	};
+	struct bpf_prog *prog;
+
+	meta.seq = seq;
+	prog = bpf_iter_get_info(&meta, true);
+	if (prog)
+		bpf_iter_run_prog(prog, &ctx);
+
+	mutex_unlock(&slab_mutex);
+}
+
+static void *slab_iter_seq_next(struct seq_file *seq, void *v, loff_t *pos)
+{
+	struct kmem_cache *s = v;
+
+	++*pos;
+
+	if (list_last_entry(&slab_caches, struct kmem_cache, list) == s)
+		return NULL;
+
+	return list_next_entry(s, list);
+}
+
+static int slab_iter_seq_show(struct seq_file *seq, void *v)
+{
+	struct bpf_iter_meta meta;
+	struct bpf_iter__slab ctx = {
+		.meta = &meta,
+		.s = v,
+	};
+	struct bpf_prog *prog;
+	int ret = 0;
+
+	meta.seq = seq;
+	prog = bpf_iter_get_info(&meta, false);
+	if (prog)
+		ret = bpf_iter_run_prog(prog, &ctx);
+
+	return ret;
+}
+
+static const struct seq_operations slab_iter_seq_ops = {
+	.start  = slab_iter_seq_start,
+	.next   = slab_iter_seq_next,
+	.stop   = slab_iter_seq_stop,
+	.show   = slab_iter_seq_show,
+};
+
+BTF_ID_LIST_GLOBAL_SINGLE(bpf_slab_btf_id, struct, kmem_cache)
+
+static const struct bpf_iter_seq_info slab_iter_seq_info = {
+	.seq_ops		= &slab_iter_seq_ops,
+};
+
+static void bpf_iter_slab_show_fdinfo(const struct bpf_iter_aux_info *aux,
+				      struct seq_file *seq)
+{
+	seq_puts(seq, "slab iter\n");
+}
+
+DEFINE_BPF_ITER_FUNC(slab, struct bpf_iter_meta *meta,
+		     struct kmem_cache *s)
+
+static struct bpf_iter_reg bpf_slab_reg_info = {
+	.target			= "slab",
+	.feature		= BPF_ITER_RESCHED,
+	.show_fdinfo		= bpf_iter_slab_show_fdinfo,
+	.ctx_arg_info_size	= 1,
+	.ctx_arg_info		= {
+		{ offsetof(struct bpf_iter__slab, s),
+		  PTR_TO_BTF_ID_OR_NULL | PTR_TRUSTED },
+	},
+	.seq_info		= &slab_iter_seq_info,
+};
+
+static int __init bpf_slab_iter_init(void)
+{
+	bpf_slab_reg_info.ctx_arg_info[0].btf_id = bpf_slab_btf_id[0];
+	return bpf_iter_reg_target(&bpf_slab_reg_info);
+}
+
+late_initcall(bpf_slab_iter_init);
-- 
2.46.0.792.g87dc391469-goog

[RFC/PATCH bpf-next 2/3] mm/bpf: Add bpf_get_slab_cache() kfunc

[Namhyung Kim]

The bpf_get_slab_cache() is to get a slab cache information from a
virtual address like virt_to_cache().  If the address is a pointer
to a slab object, it'd return a valid kmem_cache pointer, otherwise
NULL is returned.

It doesn't grab a reference count of the kmem_cache so the caller is
responsible to manage the access.  The intended use case for now is to
symbolize locks in slab objects from the lock contention tracepoints.

Suggested-by: Vlastimil Babka <vbabka@suse.cz>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 kernel/bpf/helpers.c |  1 +
 mm/slab_common.c     | 14 ++++++++++++++
 2 files changed, 15 insertions(+)

diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c
index 1a43d06eab286c26..03db007a247175c4 100644
--- a/kernel/bpf/helpers.c
+++ b/kernel/bpf/helpers.c
@@ -3090,6 +3090,7 @@ BTF_ID_FLAGS(func, bpf_iter_bits_new, KF_ITER_NEW)
 BTF_ID_FLAGS(func, bpf_iter_bits_next, KF_ITER_NEXT | KF_RET_NULL)
 BTF_ID_FLAGS(func, bpf_iter_bits_destroy, KF_ITER_DESTROY)
 BTF_ID_FLAGS(func, bpf_copy_from_user_str, KF_SLEEPABLE)
+BTF_ID_FLAGS(func, bpf_get_slab_cache, KF_RET_NULL)
 BTF_KFUNCS_END(common_btf_ids)
 
 static const struct btf_kfunc_id_set common_kfunc_set = {
diff --git a/mm/slab_common.c b/mm/slab_common.c
index 7443244656150325..a87adcf182f49fc4 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -1322,6 +1322,20 @@ size_t ksize(const void *objp)
 }
 EXPORT_SYMBOL(ksize);
 
+#ifdef CONFIG_BPF_SYSCALL
+__bpf_kfunc_start_defs();
+
+__bpf_kfunc struct kmem_cache *bpf_get_slab_cache(u64 addr)
+{
+	struct slab *slab;
+
+	slab = virt_to_slab((void *)(long)addr);
+	return slab ? slab->slab_cache : NULL;
+}
+
+__bpf_kfunc_end_defs();
+#endif /* CONFIG_BPF_SYSCALL */
+
 /* Tracepoints definitions. */
 EXPORT_TRACEPOINT_SYMBOL(kmalloc);
 EXPORT_TRACEPOINT_SYMBOL(kmem_cache_alloc);
-- 
2.46.0.792.g87dc391469-goog

[RFC/PATCH bpf-next 3/3] selftests/bpf: Add a test for slab_iter

[Namhyung Kim]

The test traverses all slab caches using the slab_iter and check if
current task's pointer is from "task_struct" slab cache.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 .../selftests/bpf/prog_tests/slab_iter.c      | 64 +++++++++++++++++++
 tools/testing/selftests/bpf/progs/bpf_iter.h  |  7 ++
 tools/testing/selftests/bpf/progs/slab_iter.c | 62 ++++++++++++++++++
 3 files changed, 133 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/slab_iter.c
 create mode 100644 tools/testing/selftests/bpf/progs/slab_iter.c

diff --git a/tools/testing/selftests/bpf/prog_tests/slab_iter.c b/tools/testing/selftests/bpf/prog_tests/slab_iter.c
new file mode 100644
index 0000000000000000..e461f6e703d67dc8
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/slab_iter.c
@@ -0,0 +1,64 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2024 Google */
+
+#include <test_progs.h>
+#include <bpf/libbpf.h>
+#include <bpf/btf.h>
+#include "slab_iter.skel.h"
+
+static void test_slab_iter_check_task(struct slab_iter *skel)
+{
+	LIBBPF_OPTS(bpf_test_run_opts, opts,
+		.flags = BPF_F_TEST_RUN_ON_CPU,
+	);
+	int prog_fd = bpf_program__fd(skel->progs.check_task_struct);
+
+	/* get task_struct and check it if's from a slab cache */
+	bpf_prog_test_run_opts(prog_fd, &opts);
+
+	/* the BPF program should set 'found' variable */
+	ASSERT_EQ(skel->bss->found, 1, "found task_struct");
+}
+
+void test_slab_iter(void)
+{
+	DECLARE_LIBBPF_OPTS(bpf_iter_attach_opts, opts);
+	struct slab_iter *skel = NULL;
+	union bpf_iter_link_info linfo = {};
+	struct bpf_link *link;
+	char buf[1024];
+	int iter_fd;
+
+	skel = slab_iter__open_and_load();
+	if (!ASSERT_OK_PTR(skel, "slab_iter__open_and_load"))
+		return;
+
+	opts.link_info = &linfo;
+	opts.link_info_len = sizeof(linfo);
+
+	link = bpf_program__attach_iter(skel->progs.slab_info_collector, &opts);
+	if (!ASSERT_OK_PTR(link, "attach_iter"))
+		goto destroy;
+
+	iter_fd = bpf_iter_create(bpf_link__fd(link));
+	if (!ASSERT_GE(iter_fd, 0, "iter_create"))
+		goto free_link;
+
+	memset(buf, 0, sizeof(buf));
+	while (read(iter_fd, buf, sizeof(buf) > 0)) {
+		/* read out all contents */
+		printf("%s", buf);
+	}
+
+	/* next reads should return 0 */
+	ASSERT_EQ(read(iter_fd, buf, sizeof(buf)), 0, "read");
+
+	test_slab_iter_check_task(skel);
+
+	close(iter_fd);
+
+free_link:
+	bpf_link__destroy(link);
+destroy:
+	slab_iter__destroy(skel);
+}
diff --git a/tools/testing/selftests/bpf/progs/bpf_iter.h b/tools/testing/selftests/bpf/progs/bpf_iter.h
index c41ee80533ca219a..33ea7181e1f03560 100644
--- a/tools/testing/selftests/bpf/progs/bpf_iter.h
+++ b/tools/testing/selftests/bpf/progs/bpf_iter.h
@@ -24,6 +24,7 @@
 #define BTF_F_PTR_RAW BTF_F_PTR_RAW___not_used
 #define BTF_F_ZERO BTF_F_ZERO___not_used
 #define bpf_iter__ksym bpf_iter__ksym___not_used
+#define bpf_iter__slab bpf_iter__slab___not_used
 #include "vmlinux.h"
 #undef bpf_iter_meta
 #undef bpf_iter__bpf_map
@@ -48,6 +49,7 @@
 #undef BTF_F_PTR_RAW
 #undef BTF_F_ZERO
 #undef bpf_iter__ksym
+#undef bpf_iter__slab
 
 struct bpf_iter_meta {
 	struct seq_file *seq;
@@ -165,3 +167,8 @@ struct bpf_iter__ksym {
 	struct bpf_iter_meta *meta;
 	struct kallsym_iter *ksym;
 };
+
+struct bpf_iter__slab {
+	struct bpf_iter_meta *meta;
+	struct kmem_cache *s;
+} __attribute__((preserve_access_index));
diff --git a/tools/testing/selftests/bpf/progs/slab_iter.c b/tools/testing/selftests/bpf/progs/slab_iter.c
new file mode 100644
index 0000000000000000..f806365506851774
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/slab_iter.c
@@ -0,0 +1,62 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2024 Google */
+
+#include "bpf_iter.h"
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+
+char _license[] SEC("license") = "GPL";
+
+#define SLAB_NAME_MAX  256
+
+struct {
+	__uint(type, BPF_MAP_TYPE_HASH);
+	__uint(key_size, sizeof(void *));
+	__uint(value_size, SLAB_NAME_MAX);
+	__uint(max_entries, 1024);
+} slab_hash SEC(".maps");
+
+extern struct kmem_cache *bpf_get_slab_cache(__u64 addr) __ksym;
+
+/* result, will be checked by userspace */
+int found;
+
+SEC("iter/slab")
+int slab_info_collector(struct bpf_iter__slab *ctx)
+{
+	struct seq_file *seq = ctx->meta->seq;
+	struct kmem_cache *s = ctx->s;
+
+	if (s) {
+		char name[SLAB_NAME_MAX];
+
+		/*
+		 * To make sure if the slab_iter implements the seq interface
+		 * properly and it's also useful for debugging.
+		 */
+		BPF_SEQ_PRINTF(seq, "%s: %u\n", s->name, s->object_size);
+
+		bpf_probe_read_kernel_str(name, sizeof(name), s->name);
+		bpf_map_update_elem(&slab_hash, &s, name, BPF_NOEXIST);
+	}
+
+	return 0;
+}
+
+SEC("raw_tp/bpf_test_finish")
+int BPF_PROG(check_task_struct)
+{
+	__u64 curr = bpf_get_current_task();
+	struct kmem_cache *s;
+	char *name;
+
+	s = bpf_get_slab_cache(curr);
+	if (s == NULL)
+		return 0;
+
+	name = bpf_map_lookup_elem(&slab_hash, &s);
+	if (name && !bpf_strncmp(name, 11, "task_struct"))
+		found = 1;
+
+	return 0;
+}
-- 
2.46.0.792.g87dc391469-goog

Re: [RFC/PATCH bpf-next 0/3] bpf: Add slab iterator and kfunc (v1)

[Roman Gushchin]

On Wed, Sep 25, 2024 at 03:30:20PM -0700, Namhyung Kim wrote:
> Hello,
> 
> I'm proposing a new iterator and a kfunc for the slab memory allocator
> to get information of each kmem_cache like in /proc/slabinfo or
> /sys/kernel/slab.

Hello, Namhyung!

I personally like the idea very much. With a growing number of kmem_caches
/proc/slabinfo getting close to it's limit, so having a more flexible
interface makes a lot of sense.

> Maybe I need to call it kmem_cache iter but slab
> was short and easier to call. :)

I'd personally prefer kmem_cache or slab_cache, just in case somebody later
would propose an iterator over individual slab objects within a kmem_cache.

Acked-by: Roman Gushchin <roman.gushchin@linux.dev> (mm/*)

Thanks!

Re: [RFC/PATCH bpf-next 0/3] bpf: Add slab iterator and kfunc (v1)

[Namhyung Kim]

On Thu, Sep 26, 2024 at 12:16:15AM +0000, Roman Gushchin wrote:
> On Wed, Sep 25, 2024 at 03:30:20PM -0700, Namhyung Kim wrote:
> > Hello,
> > 
> > I'm proposing a new iterator and a kfunc for the slab memory allocator
> > to get information of each kmem_cache like in /proc/slabinfo or
> > /sys/kernel/slab.
> 
> Hello, Namhyung!

Hello Roman!

> 
> I personally like the idea very much. With a growing number of kmem_caches
> /proc/slabinfo getting close to it's limit, so having a more flexible
> interface makes a lot of sense.
> 
> > Maybe I need to call it kmem_cache iter but slab
> > was short and easier to call. :)
> 
> I'd personally prefer kmem_cache or slab_cache, just in case somebody later
> would propose an iterator over individual slab objects within a kmem_cache.

I think we can add a parameter to limit or extend the functionality
like task iter and cgroup iter.  But I'm not sure we need to use a
different name for that.  Anyway I'm ok to rename it kmem_cache iter.

> 
> Acked-by: Roman Gushchin <roman.gushchin@linux.dev> (mm/*)

Thanks for your review!
Namhyung

Re: [RFC/PATCH bpf-next 0/3] bpf: Add slab iterator and kfunc (v1)

[Vlastimil Babka]

On 9/26/24 02:16, Roman Gushchin wrote:
> On Wed, Sep 25, 2024 at 03:30:20PM -0700, Namhyung Kim wrote:
>> Hello,
>> 
>> I'm proposing a new iterator and a kfunc for the slab memory allocator
>> to get information of each kmem_cache like in /proc/slabinfo or
>> /sys/kernel/slab.
> 
> Hello, Namhyung!
> 
> I personally like the idea very much. With a growing number of kmem_caches
> /proc/slabinfo getting close to it's limit, so having a more flexible
> interface makes a lot of sense.
> 
>> Maybe I need to call it kmem_cache iter but slab
>> was short and easier to call. :)
> 
> I'd personally prefer kmem_cache or slab_cache, just in case somebody later
> would propose an iterator over individual slab objects within a kmem_cache.

Yeah it would be better. Even though I doubt an iterator of slabs or objects
would be really feasible.

> Acked-by: Roman Gushchin <roman.gushchin@linux.dev> (mm/*)

Acked-by: Vlastimil Babka <vbabka@suse.cz> #mm/slab

> 
> Thanks!