* Re: [PATCH] tracing: ring-buffer: Fix to check event length before using
From: Masami Hiramatsu @ 2026-02-18 8:13 UTC (permalink / raw)
To: Masami Hiramatsu (Google)
Cc: Steven Rostedt, Mathieu Desnoyers, linux-kernel,
linux-trace-kernel
In-Reply-To: <177123421541.142205.9414352170164678966.stgit@devnote2>
I found another problem. Let me make a series for fixing issues.
Thanks,
On Mon, 16 Feb 2026 18:30:15 +0900
"Masami Hiramatsu (Google)" <mhiramat@kernel.org> wrote:
> From: Masami Hiramatsu (Google) <mhiramat@kernel.org>
>
> Check the event length before adding it for accessing next index in
> rb_read_data_buffer(). Since this function is used for validating
> possibly broken ring buffers, the length of the event could be broken.
> In that case, the new event (e + len) can point a wrong address.
> To avoid invalid memory access at boot, check whether the length of
> each event is in the possible range before using it.
>
> Fixes: 5f3b6e839f3c ("ring-buffer: Validate boot range memory events")
> Cc: stable@vger.kernel.org
> Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
> ---
> kernel/trace/ring_buffer.c | 6 +++++-
> 1 file changed, 5 insertions(+), 1 deletion(-)
>
> diff --git a/kernel/trace/ring_buffer.c b/kernel/trace/ring_buffer.c
> index 630221b00838..1ef17d6fd824 100644
> --- a/kernel/trace/ring_buffer.c
> +++ b/kernel/trace/ring_buffer.c
> @@ -1848,6 +1848,7 @@ static int rb_read_data_buffer(struct buffer_data_page *dpage, int tail, int cpu
> struct ring_buffer_event *event;
> u64 ts, delta;
> int events = 0;
> + int len;
> int e;
>
> *delta_ptr = 0;
> @@ -1855,9 +1856,12 @@ static int rb_read_data_buffer(struct buffer_data_page *dpage, int tail, int cpu
>
> ts = dpage->time_stamp;
>
> - for (e = 0; e < tail; e += rb_event_length(event)) {
> + for (e = 0; e < tail; e += len) {
>
> event = (struct ring_buffer_event *)(dpage->data + e);
> + len = rb_event_length(event);
> + if (len <= 0 || len > tail - e)
> + return -1;
>
> switch (event->type_len) {
>
>
--
Masami Hiramatsu (Google) <mhiramat@kernel.org>
^ permalink raw reply
* Re: [RFC PATCH v2 09/37] KVM: guest_memfd: Add support for KVM_SET_MEMORY_ATTRIBUTES2
From: Sean Christopherson @ 2026-02-17 23:04 UTC (permalink / raw)
To: Ackerley Tng
Cc: kvm, linux-doc, linux-kernel, linux-kselftest, linux-trace-kernel,
x86, aik, andrew.jones, binbin.wu, bp, brauner, chao.p.peng,
chao.p.peng, chenhuacai, corbet, dave.hansen, david, hpa,
ira.weiny, jgg, jmattson, jroedel, jthoughton, maobibo,
mathieu.desnoyers, maz, mhiramat, michael.roth, mingo, mlevitsk,
oupton, pankaj.gupta, pbonzini, prsampat, qperret, ricarkol,
rick.p.edgecombe, rientjes, rostedt, shivankg, shuah,
steven.price, tabba, tglx, vannapurve, vbabka, willy, wyihan,
yan.y.zhao
In-Reply-To: <CAEvNRgFMNywpDRr+WeNsVj=MnsbhZp9H3j0QRDo_eOP+kGCNJw@mail.gmail.com>
On Sat, Feb 14, 2026, Ackerley Tng wrote:
> Sean is against deferring whether to preserve memory to the underlying
> hardware because that is letting (effectively) micro-architectural
> behavior to define KVM's ABI. KVM's uAPI cannot let behavior be
> undefined, or be based on vendor, and maybe even on firmware version.
Concretely, we'll end up with a giant mess if TDX and/or SNP adds support for
preserving contents on conversion.
^ permalink raw reply
* Re: [RFC bpf-next 12/12] selftests/bpf: Add tracing multi benchmark test
From: Jiri Olsa @ 2026-02-17 22:06 UTC (permalink / raw)
To: bot+bpf-ci
Cc: ast, daniel, andrii, bpf, linux-trace-kernel, kafai, eddyz87,
songliubraving, yhs, menglong8.dong, rostedt, martin.lau,
yonghong.song, clm, ihor.solodrai
In-Reply-To: <9de646e4e4d2fda0d4f0543b422439f7c1fcc121d4db2e38a28bf3a454b96d98@mail.kernel.org>
On Tue, Feb 03, 2026 at 10:13:57AM +0000, bot+bpf-ci@kernel.org wrote:
> > diff --git a/tools/testing/selftests/bpf/prog_tests/tracing_multi.c b/tools/testing/selftests/bpf/prog_tests/tracing_multi.c
> > index 3ccf0d4ed..575454e31 100644
> > --- a/tools/testing/selftests/bpf/prog_tests/tracing_multi.c
> > +++ b/tools/testing/selftests/bpf/prog_tests/tracing_multi.c
> > @@ -6,6 +6,9 @@
> > #include <bpf/btf.h>
> > #include <linux/btf.h>
> > #include <search.h>
> > +#include <bpf/btf.h>
> > +#include <linux/btf.h>
> > +#include <search.h>
>
> These three includes are duplicates of the lines immediately above them.
> Were they added by mistake?
>
> [ ... ]
>
> > + t = btf__type_by_id(btf, proto->type);
> > + while (t && btf_type_is_modifier(t))
> > + t = btf__type_by_id(btf, t->type);
>
> The while line uses spaces for indentation instead of a tab.
yes, will fix, thnx
jirka
^ permalink raw reply
* Re: [RFC bpf-next 10/12] selftests/bpf: Add fentry tracing multi func test
From: Jiri Olsa @ 2026-02-17 22:06 UTC (permalink / raw)
To: bot+bpf-ci
Cc: ast, daniel, andrii, bpf, linux-trace-kernel, kafai, eddyz87,
songliubraving, yhs, menglong8.dong, rostedt, martin.lau,
yonghong.song, clm, ihor.solodrai
In-Reply-To: <cd5126aaea4095fde41096caf553d7ab7c42efc42ee2bf8a5897be48a0e66a41@mail.kernel.org>
On Tue, Feb 03, 2026 at 10:13:55AM +0000, bot+bpf-ci@kernel.org wrote:
> > diff --git a/tools/testing/selftests/bpf/progs/tracing_multi_check.c b/tools/testing/selftests/bpf/progs/tracing_multi_check.c
> > new file mode 100644
> > index 000000000..e5efa9884
> > --- /dev/null
> > +++ b/tools/testing/selftests/bpf/progs/tracing_multi_check.c
>
> [ ... ]
>
> > + } else if (ip == &bpf_fentry_test3) {
> > + char a, b;
> ^
>
> The commit message states "We need to cast to real arguments types in
> multi_arg_check, because the checked value can be shorter than u64."
>
> Should variable b be declared as int instead of char?
>
> The kernel function signature is:
>
> noinline int bpf_fentry_test3(char a, int b, u64 c)
>
> The second parameter is int, not char. While the test passes with the
> current test value of 5, this appears inconsistent with the stated goal
> of matching argument types.
yep, will fix
>
> > + __u64 c;
>
> [ ... ]
>
> > + } else if (ip == &bpf_fentry_test6) {
>
> [ ... ]
>
> > + e = (void*) value;;
> > + if (bpf_get_func_arg(ctx, 5, &value))
> > + return;
> > + f = value;;
> ^^
>
> This isn't a bug, but there are extra semicolons on these two lines.
ok, thnx
jirka
^ permalink raw reply
* Re: [RFC bpf-next 09/12] libbpf: Add support to create tracing multi link
From: Jiri Olsa @ 2026-02-17 22:06 UTC (permalink / raw)
To: Andrii Nakryiko
Cc: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko, bpf,
linux-trace-kernel, Martin KaFai Lau, Eduard Zingerman, Song Liu,
Yonghong Song, Menglong Dong, Steven Rostedt
In-Reply-To: <CAEf4BzYhP8CdS_Xpw1iEvGNxYGnf3Zq5zkb-HEZKvx5cxEJBqw@mail.gmail.com>
On Wed, Feb 04, 2026 at 11:05:16AM -0800, Andrii Nakryiko wrote:
> On Tue, Feb 3, 2026 at 1:40 AM Jiri Olsa <jolsa@kernel.org> wrote:
> >
> > Adding new interface function to attach programs with tracing
> > multi link:
> >
> > bpf_program__attach_tracing_multi(const struct bpf_program *prog,
> > const char *pattern,
> > const struct bpf_tracing_multi_opts *opts);
> >
> > The program is attach to functions specified by pattern or by
> > btf IDs specified in bpf_tracing_multi_opts object.
> >
> > Adding support for new sections to attach programs with above
> > functions:
> >
> > fentry.multi/pattern
> > fexit.multi/pattern
> >
> > Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> > ---
> > tools/lib/bpf/bpf.c | 7 ++++
> > tools/lib/bpf/bpf.h | 4 ++
> > tools/lib/bpf/libbpf.c | 87 ++++++++++++++++++++++++++++++++++++++++
> > tools/lib/bpf/libbpf.h | 14 +++++++
> > tools/lib/bpf/libbpf.map | 1 +
> > 5 files changed, 113 insertions(+)
>
> [...]
>
> > static const char * const map_type_name[] = {
> > @@ -9814,6 +9817,7 @@ static int attach_kprobe_session(const struct bpf_program *prog, long cookie, st
> > static int attach_uprobe_multi(const struct bpf_program *prog, long cookie, struct bpf_link **link);
> > static int attach_lsm(const struct bpf_program *prog, long cookie, struct bpf_link **link);
> > static int attach_iter(const struct bpf_program *prog, long cookie, struct bpf_link **link);
> > +static int attach_tracing_multi(const struct bpf_program *prog, long cookie, struct bpf_link **link);
> >
> > static const struct bpf_sec_def section_defs[] = {
> > SEC_DEF("socket", SOCKET_FILTER, 0, SEC_NONE),
> > @@ -9862,6 +9866,8 @@ static const struct bpf_sec_def section_defs[] = {
> > SEC_DEF("fexit.s+", TRACING, BPF_TRACE_FEXIT, SEC_ATTACH_BTF | SEC_SLEEPABLE, attach_trace),
> > SEC_DEF("fsession+", TRACING, BPF_TRACE_FSESSION, SEC_ATTACH_BTF, attach_trace),
> > SEC_DEF("fsession.s+", TRACING, BPF_TRACE_FSESSION, SEC_ATTACH_BTF | SEC_SLEEPABLE, attach_trace),
> > + SEC_DEF("fentry.multi+", TRACING, BPF_TRACE_FENTRY_MULTI, 0, attach_tracing_multi),
> > + SEC_DEF("fexit.multi+", TRACING, BPF_TRACE_FEXIT_MULTI, 0, attach_tracing_multi),
> > SEC_DEF("freplace+", EXT, 0, SEC_ATTACH_BTF, attach_trace),
> > SEC_DEF("lsm+", LSM, BPF_LSM_MAC, SEC_ATTACH_BTF, attach_lsm),
> > SEC_DEF("lsm.s+", LSM, BPF_LSM_MAC, SEC_ATTACH_BTF | SEC_SLEEPABLE, attach_lsm),
> > @@ -12237,6 +12243,87 @@ static int attach_uprobe_multi(const struct bpf_program *prog, long cookie, stru
> > return ret;
> > }
> >
> > +struct bpf_link *
> > +bpf_program__attach_tracing_multi(const struct bpf_program *prog, const char *pattern,
> > + const struct bpf_tracing_multi_opts *opts)
> > +{
> > + LIBBPF_OPTS(bpf_link_create_opts, lopts);
> > + __u32 *btf_ids, cnt, *free_ids = NULL;
> > + int prog_fd, link_fd, err;
> > + struct bpf_link *link;
> > +
> > + btf_ids = OPTS_GET(opts, btf_ids, false);
> > + cnt = OPTS_GET(opts, cnt, false);
> > +
> > + if (!pattern && !btf_ids && !cnt)
>
> let's check that either both btf_ids and cnt are specified or none
>
> then we can check that either pattern or btf_ids are specified
>
> still two checks, but will capture all the bad cases
ok
>
> > + return libbpf_err_ptr(-EINVAL);
> > + if (pattern && (btf_ids || cnt))
> > + return libbpf_err_ptr(-EINVAL);
> > +
>
> [...]
>
> > struct bpf_uprobe_opts {
> > /* size of this struct, for forward/backward compatibility */
> > size_t sz;
> > diff --git a/tools/lib/bpf/libbpf.map b/tools/lib/bpf/libbpf.map
> > index d18fbcea7578..a3ffb21270e9 100644
> > --- a/tools/lib/bpf/libbpf.map
> > +++ b/tools/lib/bpf/libbpf.map
> > @@ -358,6 +358,7 @@ LIBBPF_1.0.0 {
> > bpf_program__attach_ksyscall;
> > bpf_program__autoattach;
> > bpf_program__set_autoattach;
> > + bpf_program__attach_tracing_multi;
>
> stuck in the past? ;) we are in 1.7 cycle
a bit :) will fix, thnx
jirka
^ permalink raw reply
* Re: [RFC bpf-next 09/12] libbpf: Add support to create tracing multi link
From: Jiri Olsa @ 2026-02-17 22:05 UTC (permalink / raw)
To: bot+bpf-ci
Cc: ast, daniel, andrii, bpf, linux-trace-kernel, kafai, eddyz87,
songliubraving, yhs, menglong8.dong, rostedt, martin.lau,
yonghong.song, clm, ihor.solodrai
In-Reply-To: <9c8e4f31ee1d79f8ae145ec1c4d388f0829a77f1a75e8ec80e1d4bcf00bde751@mail.kernel.org>
On Tue, Feb 03, 2026 at 10:14:03AM +0000, bot+bpf-ci@kernel.org wrote:
> > diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
> > index 0c8bf0b5c..a16243300 100644
> > --- a/tools/lib/bpf/libbpf.c
> > +++ b/tools/lib/bpf/libbpf.c
>
> [ ... ]
>
> > +struct bpf_link *
> > +bpf_program__attach_tracing_multi(const struct bpf_program *prog, const char *pattern,
> > + const struct bpf_tracing_multi_opts *opts)
> > +{
> > + LIBBPF_OPTS(bpf_link_create_opts, lopts);
> > + __u32 *btf_ids, cnt, *free_ids = NULL;
> > + int prog_fd, link_fd, err;
> > + struct bpf_link *link;
> > +
> > + btf_ids = OPTS_GET(opts, btf_ids, false);
> > + cnt = OPTS_GET(opts, cnt, false);
> > +
> > + if (!pattern && !btf_ids && !cnt)
> > + return libbpf_err_ptr(-EINVAL);
> > + if (pattern && (btf_ids || cnt))
> > + return libbpf_err_ptr(-EINVAL);
> > +
> > + if (pattern) {
> > + err = bpf_object__load_vmlinux_btf(prog->obj, true);
> > + if (err)
> > + return libbpf_err_ptr(err);
> > +
> > + cnt = btf__find_by_glob_kind(prog->obj->btf_vmlinux, BTF_KIND_FUNC,
> > + pattern, NULL, &btf_ids);
> > + if (cnt <= 0)
> > + return libbpf_err_ptr(-EINVAL);
> > + free_ids = btf_ids;
> > + }
> > +
> > + lopts.tracing_multi.btf_ids = btf_ids;
> > + lopts.tracing_multi.btf_ids_cnt = cnt;
> > +
> > + link = calloc(1, sizeof(*link));
> > + if (!link)
> > + return libbpf_err_ptr(-ENOMEM);
>
> If calloc fails here and pattern was provided, btf__find_by_glob_kind() will
> have allocated btf_ids (tracked by free_ids). Does this leak free_ids? The
> error label below frees it properly, but this return bypasses that path.
>
> Something like:
>
> if (!link) {
> free(free_ids);
> return libbpf_err_ptr(-ENOMEM);
> }
>
> or using goto error would fix this.
yep, will fix
>
> > + link->detach = &bpf_link__detach_fd;
> > +
> > + prog_fd = bpf_program__fd(prog);
> > + link_fd = bpf_link_create(prog_fd, 0, prog->expected_attach_type, &lopts);
> > + if (link_fd < 0) {
> > + err = -errno;
> > + pr_warn("prog '%s': failed to attach: %s\n", prog->name, errstr(err));
> > + goto error;
> > + }
> > + link->fd = link_fd;
> > + free(free_ids);
> > + return link;
> > +error:
> > + free(link);
> > + free(free_ids);
> > + return libbpf_err_ptr(err);
> > +}
> > +
> > +static int attach_tracing_multi(const struct bpf_program *prog, long cookie, struct bpf_link **link)
> > +{
> > + const char *spec;
> > + char *pattern;
> > + bool is_fexit;
> > + int n;
> > +
> > + /* no auto-attach for SEC("fentry.multi") and SEC("fexit.multi") */
> > + if (strcmp(prog->sec_name, "fentry.multi") == 0 ||
> > + strcmp(prog->sec_name, "fexit.multi") == 0)
> > + return 0;
> > +
> > + is_fexit = str_has_pfx(prog->sec_name, "fexit.multi/");
> > + if (is_fexit)
> > + spec = prog->sec_name + sizeof("fexit.multi/") - 1;
> > + else
> > + spec = prog->sec_name + sizeof("fentry.multi/") - 1;
> > +
> > + n = sscanf(spec, "%m[a-zA-Z0-9_.*?]", &pattern);
> > + if (n < 1) {
> > + pr_warn("tracing multi pattern is invalid: %s\n", pattern);
>
> When sscanf fails to match (n < 1), pattern is never assigned because %m only
> allocates on successful match. Should this print spec instead of the
> uninitialized pattern? Compare with attach_kprobe_multi() which uses:
>
> pr_warn("kprobe multi pattern is invalid: %s\n", spec);
yes
>
> > + return -EINVAL;
> > + }
> > +
> > + *link = bpf_program__attach_tracing_multi(prog, pattern, NULL);
> > + return libbpf_get_error(*link);
>
> Is pattern being leaked here? sscanf with %m allocates the string, but there
> is no free(pattern) before returning. Compare with attach_kprobe_multi():
>
> *link = bpf_program__attach_kprobe_multi_opts(prog, pattern, &opts);
> free(pattern);
> return libbpf_get_error(*link);
yep, will fix, thnx
jirka
^ permalink raw reply
* Re: [RFC bpf-next 07/12] bpf: Add support to create tracing multi link
From: Jiri Olsa @ 2026-02-17 22:05 UTC (permalink / raw)
To: bot+bpf-ci
Cc: ast, daniel, andrii, bpf, linux-trace-kernel, kafai, eddyz87,
songliubraving, yhs, menglong8.dong, rostedt, martin.lau,
yonghong.song, clm, ihor.solodrai
In-Reply-To: <1aa999b2eb50ae6a0a80a5ef21c5f8c22bcb2605e971652c6f202a292b996032@mail.kernel.org>
On Tue, Feb 03, 2026 at 10:13:58AM +0000, bot+bpf-ci@kernel.org wrote:
> > diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
> > index 68600972a..010785246 100644
> > --- a/include/uapi/linux/bpf.h
> > +++ b/include/uapi/linux/bpf.h
>
> [ ... ]
>
> > @@ -1863,6 +1864,10 @@ union bpf_attr {
> > };
> > __u64 expected_revision;
> > } cgroup;
> > + struct {
> > + __aligned_u64 btf_ids; /* addresses to attach */
> > + __u32 btf_ids_cnt; /* addresses count */
> > + } tracing_multi;
>
> The comments say "addresses to attach" and "addresses count", but the field is
> named btf_ids and contains BTF type IDs, not addresses. Should these comments
> say "BTF IDs to attach" and "BTF IDs count" instead?
yep, wrong comments, will fix
>
> > diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c
> > index f7baeb827..82e625aa0 100644
> > --- a/kernel/trace/bpf_trace.c
> > +++ b/kernel/trace/bpf_trace.c
>
> [ ... ]
>
> > +static void bpf_tracing_multi_link_show_fdinfo(const struct bpf_link *link,
> > + struct seq_file *seq)
> > +{
> > + struct bpf_tracing_multi_link *tr_link =
> > + container_of(link, struct bpf_tracing_multi_link, link);
> > +
> > + seq_printf(seq, "attach_type:\t%d\n", tr_link->attach_type);
> > +}
> > +
> > +static int bpf_tracing_multi_link_fill_link_info(const struct bpf_link *link,
> > + struct bpf_link_info *info)
> > +{
> > + struct bpf_tracing_multi_link *tr_link =
> > + container_of(link, struct bpf_tracing_multi_link, link);
> > +
> > + info->tracing.attach_type = tr_link->attach_type;
> > + return 0;
> > +}
>
> Both functions above use tr_link->attach_type, but looking at
> bpf_tracing_multi_attach() below, this field is never initialized:
>
> > +int bpf_tracing_multi_attach(struct bpf_prog *prog, const union bpf_attr *attr)
> > +{
>
> [ ... ]
>
> > + link = kzalloc(struct_size(link, nodes, cnt), GFP_KERNEL);
> > + if (!link) {
> > + err = -ENOMEM;
> > + goto error;
> > + }
> > +
> > + link->nodes_cnt = cnt;
> > +
> > + bpf_link_init(&link->link, BPF_LINK_TYPE_TRACING_MULTI,
> > + &bpf_tracing_multi_link_lops, prog, prog->expected_attach_type);
>
> bpf_link_init() sets link->link.attach_type (inside the embedded struct bpf_link),
> but struct bpf_tracing_multi_link has a separate attach_type field that remains
> zero-initialized from kzalloc.
>
> The show_fdinfo and fill_link_info callbacks read tr_link->attach_type which
> will always be 0 (BPF_CGROUP_INET_INGRESS) instead of the actual attach type
> (BPF_TRACE_FENTRY_MULTI or BPF_TRACE_FEXIT_MULTI).
>
> Looking at the existing bpf_tracing_link_show_fdinfo() and
> bpf_tracing_link_fill_link_info() in kernel/bpf/syscall.c, they use
> link->attach_type (the inner struct's field). Should tr_link->attach_type
> either be removed and link->attach_type used instead, or should it be
> explicitly initialized to prog->expected_attach_type?
right, I'll add linkinfo/fdinfo in separate change, thnx
jirka
^ permalink raw reply
* Re: [RFC bpf-next 06/12] bpf: Add bpf_trampoline_multi_attach/detach functions
From: Jiri Olsa @ 2026-02-17 22:05 UTC (permalink / raw)
To: bot+bpf-ci
Cc: ast, daniel, andrii, bpf, linux-trace-kernel, kafai, eddyz87,
songliubraving, yhs, menglong8.dong, rostedt, martin.lau,
yonghong.song, clm, ihor.solodrai
In-Reply-To: <bfbbb9ef3aa022d56e5777597b48b596c4a959e606fcbc9d63f4ee1e6c07941f@mail.kernel.org>
On Tue, Feb 03, 2026 at 10:14:02AM +0000, bot+bpf-ci@kernel.org wrote:
> > diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c
> > index 2be2f1d0b..b76bb5450 100644
> > --- a/kernel/bpf/trampoline.c
> > +++ b/kernel/bpf/trampoline.c
>
> [ ... ]
>
> > +int bpf_trampoline_multi_attach(struct bpf_prog *prog, u32 *ids,
> > + struct bpf_tracing_multi_link *link)
> > +{
> > + struct bpf_attach_target_info tgt_info = {};
> > + struct bpf_tracing_multi_node *mnode;
> > + int j, i, err, cnt = link->nodes_cnt;
> > + struct fentry_multi_data data = {};
> > + struct bpf_trampoline *tr;
> > + u64 key;
> > +
> > + data.reg = alloc_ftrace_hash(FTRACE_HASH_DEFAULT_BITS);
> > + if (!data.reg)
> > + return -ENOMEM;
> > +
> > + data.modify = alloc_ftrace_hash(FTRACE_HASH_DEFAULT_BITS);
> > + if (!data.modify) {
> > + free_ftrace_hash(data.reg);
> > + return -ENOMEM;
> > + }
> > +
> > + for (i = 0; i < cnt; i++) {
> > + mnode = &link->nodes[i];
> > + err = bpf_check_attach_target(NULL, prog, NULL, ids[i], &tgt_info);
> > + if (err)
> > + goto rollback_put;
> > +
> > + key = bpf_trampoline_compute_key(NULL, prog->aux->attach_btf, ids[i]);
> > +
> > + tr = bpf_trampoline_get(key, &tgt_info);
> > + if (!tr)
> > + goto rollback_put;
>
> When bpf_trampoline_get() returns NULL here, the code jumps to
> rollback_put without setting err. If bpf_check_attach_target() succeeded
> on this iteration (setting err to 0), the function will return 0 instead
> of an error code.
>
> Should this be something like:
>
> tr = bpf_trampoline_get(key, &tgt_info);
> if (!tr) {
> err = -ENOMEM;
> goto rollback_put;
> }
>
yep, we should, will add, thnx
jirka
> > +
> > + mnode->trampoline = tr;
> > + mnode->node.prog = prog;
> > + }
>
> [ ... ]
>
> > +rollback_put:
> > + for (j = 0; j < i; j++) {
> > + mnode = &link->nodes[j];
> > + bpf_trampoline_put(mnode->trampoline);
> > + }
> > +
> > + free_fentry_multi_data(&data);
> > + return err;
> > +}
>
>
> ---
> AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
> See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
>
> CI run summary: https://github.com/kernel-patches/bpf/actions/runs/21625440628
>
> AI-authorship-score: low
> AI-authorship-explanation: The code follows established kernel patterns and contains a human-typical oversight error (missing error assignment); the commit message has a minor copy-paste error typical of human editing.
> issues-found: 1
> issue-severity-score: low
> issue-severity-explanation: Missing error code when bpf_trampoline_get fails causes silent failure under memory pressure, but does not cause crashes or data corruption.
^ permalink raw reply
* Re: [RFC bpf-next 05/12] bpf: Add multi tracing attach types
From: Jiri Olsa @ 2026-02-17 22:05 UTC (permalink / raw)
To: bot+bpf-ci
Cc: ast, daniel, andrii, bpf, linux-trace-kernel, kafai, eddyz87,
songliubraving, yhs, menglong8.dong, rostedt, martin.lau,
yonghong.song, clm, ihor.solodrai
In-Reply-To: <9df0acf7860b7e4a90b3e93e2d6924e9bad35fe307bef0bf9b50f8afb5cccacc@mail.kernel.org>
On Tue, Feb 03, 2026 at 10:13:53AM +0000, bot+bpf-ci@kernel.org wrote:
> > diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> > index 6b62b6d57..fb52ba2f7 100644
> > --- a/kernel/bpf/verifier.c
> > +++ b/kernel/bpf/verifier.c
>
> [ ... ]
>
> > @@ -25069,7 +25074,8 @@ static int check_attach_btf_id(struct bpf_verifier_env *env)
> > return 0;
> > } else if (prog->expected_attach_type == BPF_TRACE_ITER) {
> > return bpf_iter_prog_supported(prog);
> > - }
> > + } else if (is_tracing_multi(prog->expected_attach_type))
> > + return prog->type == BPF_PROG_TYPE_TRACING ? 0 : -EINVAL;
>
> This isn't a bug, but should this branch have braces for consistency with the
> preceding branches in the same if/else chain? The BPF_TRACE_RAW_TP and
> BPF_TRACE_ITER branches both use braces.
right, will add it, thnx
jirka
>
>
> ---
> AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
> See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
>
> CI run summary: https://github.com/kernel-patches/bpf/actions/runs/21625440628
>
> AI-authorship-score: low
> AI-authorship-explanation: The commit message contains natural colloquial language and personal attribution typical of human kernel developers, and the code follows established BPF patterns with typical human oversights in style consistency.
> issues-found: 1
> issue-severity-score: low
> issue-severity-explanation: Minor coding style inconsistency with missing braces in an else-if branch that does not affect functionality.
^ permalink raw reply
* [PATCH v17 1/3] lib: Introduce hierarchical per-cpu counters
From: Mathieu Desnoyers @ 2026-02-17 16:10 UTC (permalink / raw)
To: Andrew Morton
Cc: linux-kernel, Mathieu Desnoyers, Paul E. McKenney, Steven Rostedt,
Masami Hiramatsu, Dennis Zhou, Tejun Heo, Christoph Lameter,
Martin Liu, David Rientjes, christian.koenig, Shakeel Butt,
SeongJae Park, Michal Hocko, Johannes Weiner, Sweet Tea Dorminy,
Lorenzo Stoakes, Liam R . Howlett, Mike Rapoport,
Suren Baghdasaryan, Vlastimil Babka, Christian Brauner, Wei Yang,
David Hildenbrand, Miaohe Lin, Al Viro, linux-mm,
linux-trace-kernel, Yu Zhao, Roman Gushchin, Mateusz Guzik,
Matthew Wilcox, Baolin Wang, Aboorva Devarajan
In-Reply-To: <20260217161006.1105611-1-mathieu.desnoyers@efficios.com>
* Motivation
The purpose of this hierarchical split-counter scheme is to:
- Minimize contention when incrementing and decrementing counters,
- Provide fast access to a sum approximation,
- Provide a sum approximation with an acceptable accuracy level when
scaling to many-core systems.
- Provide approximate and precise comparison of two counters, and
between a counter and a value.
- Provide possible precise sum ranges for a given sum approximation.
Its goals are twofold:
- Improve the accuracy of the approximated RSS counter values returned
by proc interfaces [1],
- Reduce the latency of the OOM killer on large many-core systems.
* Design
The hierarchical per-CPU counters propagate a sum approximation through
a N-way tree. When reaching the batch size, the carry is propagated
through a binary tree which consists of logN(nr_cpu_ids) levels. The
batch size for each level is twice the batch size of the prior level.
Example propagation diagram with 8 cpus through a binary tree:
Level 0: 0 1 2 3 4 5 6 7
| / | / | / | /
| / | / | / | /
| / | / | / | /
Level 1: 0 1 2 3
| / | /
| / | /
| / | /
Level 2: 0 1
| /
| /
| /
Level 3: 0
For a binary tree, the maximum inaccuracy is bound by:
batch_size * log2(nr_cpu_ids) * nr_cpu_ids
which evolves with O(n*log(n)) as the number of CPUs increases.
For a N-way tree, the maximum inaccuracy can be pre-calculated
based on the the N-arity of each level and the batch size.
* Memory Use
The most important parts in terms of memory use are the per-cpu counters
and the tree items which propagate the carry.
In the proposed implementation, the per-cpu counters are allocated
within per-cpu data structures, so they end up using:
nr_possible_cpus * sizeof(unsigned long)
This is in addition to the tree items. The size of those items is
defined by the per_nr_cpu_order_config table "nr_items" field.
Each item is aligned on cacheline size (typically 64 bytes) to minimize
false sharing.
Here is the footprint for a few nr_cpu_ids on a 64-bit arch:
nr_cpu_ids percpu counters (bytes) nr_items items size (bytes) total (bytes)
2 16 1 64 80
4 32 3 192 224
8 64 7 448 512
64 512 21 1344 1856
128 1024 21 1344 2368
256 2048 37 2368 4416
512 4096 73 4672 8768
There are of course various trade offs we can make here. We can:
* Increase the n-arity of the intermediate items to shrink the nr_items
required for a given nr_cpus. This will increase contention of carry
propagation across more cores.
* Remove cacheline alignment of intermediate tree items. This will
shrink the memory needed for tree items, but will increase false
sharing.
* Represent intermediate tree items on a byte rather than long.
This further reduces the memory required for intermediate tree
items, but further increases false sharing.
* Represent per-cpu counters on bytes rather than long. This makes
the "sum" operation trickier, because it needs to iterate on the
intermediate carry propagation nodes as well and synchronize with
ongoing "tree add" operations. It further reduces memory use.
* Implement a custom strided allocator for intermediate items carry
propagation bytes. This shares cachelines across different tree
instances, keeping good locality. This ensures that all accesses
from a given location in the machine topology touch the same
cacheline for the various tree instances. This adds complexity,
but provides compactness as well as minimal false-sharing.
Compared to this, the upstream percpu counters use a 32-bit integer per-cpu
(4 bytes), and accumulate within a 64-bit global value.
So there is an extra memory footprint added by the current hpcc
implementation, but if it's an issue we have various options to consider
to reduce its footprint.
Link: https://lore.kernel.org/lkml/20250331223516.7810-2-sweettea-kernel@dorminy.me/ # [1]
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: "Paul E. McKenney" <paulmck@kernel.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Dennis Zhou <dennis@kernel.org>
Cc: Tejun Heo <tj@kernel.org>
Cc: Christoph Lameter <cl@linux.com>
Cc: Martin Liu <liumartin@google.com>
Cc: David Rientjes <rientjes@google.com>
Cc: christian.koenig@amd.com
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: SeongJae Park <sj@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Sweet Tea Dorminy <sweettea-kernel@dorminy.me>
Cc: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
Cc: "Liam R . Howlett" <liam.howlett@oracle.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Cc: Christian Brauner <brauner@kernel.org>
Cc: Wei Yang <richard.weiyang@gmail.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: Miaohe Lin <linmiaohe@huawei.com>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Cc: linux-mm@kvack.org
Cc: linux-trace-kernel@vger.kernel.org
Cc: Yu Zhao <yuzhao@google.com>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Mateusz Guzik <mjguzik@gmail.com>
Cc: Matthew Wilcox <willy@infradead.org>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Aboorva Devarajan <aboorvad@linux.ibm.com>
---
Changes since v16:
- Only perform atomic increments of intermediate tree nodes when
bits which are significant for carry propagation are being changed.
Changes since v15:
- Rebase on v2 of "mm: Fix OOM killer inaccuracy on large many-core systems".
- Update the commit message to explain the two goals of hpcc:
provide a more accurate approximation, and reduce OOM killer latency.
- Change percpu_counter_tree_approximate_accuracy_range to increment the
under/over parameters rather than set them. This requires callers to
initialize them to 0, but facilitates scenarios where accuracy needs to
be summed over many counters.
- Clarify comments above percpu_counter_tree_approximate_accuracy_range.
Changes since v14:
- Add Documentation/core-api/percpu-counter-tree.rst.
- Make percpu_counter_tree_approximate_accuracy_range static inline.
Changes since v12:
- Use atomic_long_set for percpu_counter_tree_set in UP build.
- percpu_counter_tree_precise_sum returns long type.
Changes since v11:
- Reduce level0 per-cpu memory allocation to the size required by those
percpu counters.
- Use unsigned long type rather than unsigned int.
- Introduce functions to return the min/max range of possible
precise sums.
Changes since v10:
- Fold "mm: Take into account hierarchical percpu tree items for static
mm_struct definitions".
Changes since v9:
- Introduce kerneldoc documentation.
- Document structure fields.
- Remove inline from percpu_counter_tree_add().
- Reject batch_size==1 which makes no sense.
- Fix copy-paste bug in percpu_counter_tree_precise_compare()
(wrong inequality comparison).
- Rename "inaccuracy" to "accuracy", which makes it easier to
document.
- Track accuracy limits more precisely. In two's complement
signed integers, the range before a n-bit underflow is one
unit larger than the range before a n-bit overflow. This sums
for all the counters within the tree. Therefore the "under"
vs "over" accuracy is not symmetrical.
Changes since v8:
- Remove migrate guard from the fast path. It does not
matter through which path the carry is propagated up
the tree.
- Rebase on top of v6.18-rc6.
- Introduce percpu_counter_tree_init_many and
percpu_counter_tree_destroy_many APIs.
- Move tree items allocation to the caller.
- Introduce percpu_counter_tree_items_size().
- Move percpu_counter_tree_subsystem_init() call before mm_core_init()
so percpu_counter_tree_items_size() is initialized before it is used.
Changes since v7:
- Explicitly initialize the subsystem from start_kernel() right
after mm_core_init() so it is up and running before the creation of
the first mm at boot.
- Remove the module.h include which is not needed with the explicit
initialization.
- Only consider levels>0 items for order={0,1} nr_items. No
functional change except to allocate only the amount of memory
which is strictly needed.
- Introduce positive precise sum API to handle a scenario where an
unlucky precise sum iteration would hit negative counter values
concurrently with counter updates.
Changes since v5:
- Introduce percpu_counter_tree_approximate_sum_positive.
- Introduce !CONFIG_SMP static inlines for UP build.
- Remove percpu_counter_tree_set_bias from the public API and make it
static.
Changes since v3:
- Add gfp flags to init function.
Changes since v2:
- Introduce N-way tree to reduce tree depth on larger systems.
Changes since v1:
- Remove percpu_counter_tree_precise_sum_unbiased from public header,
make this function static,
- Introduce precise and approximate comparisons between two counters,
- Reorder the struct percpu_counter_tree fields,
- Introduce approx_sum field, which points to the approximate sum
for the percpu_counter_tree_approximate_sum() fast path.
---
.../core-api/percpu-counter-tree.rst | 75 ++
include/linux/mm_types.h | 6 +-
include/linux/percpu_counter_tree.h | 367 ++++++++++
init/main.c | 2 +
lib/Makefile | 1 +
lib/percpu_counter_tree.c | 690 ++++++++++++++++++
6 files changed, 1138 insertions(+), 3 deletions(-)
create mode 100644 Documentation/core-api/percpu-counter-tree.rst
create mode 100644 include/linux/percpu_counter_tree.h
create mode 100644 lib/percpu_counter_tree.c
diff --git a/Documentation/core-api/percpu-counter-tree.rst b/Documentation/core-api/percpu-counter-tree.rst
new file mode 100644
index 000000000000..196da056e7b4
--- /dev/null
+++ b/Documentation/core-api/percpu-counter-tree.rst
@@ -0,0 +1,75 @@
+========================================
+The Hierarchical Per-CPU Counters (HPCC)
+========================================
+
+:Author: Mathieu Desnoyers
+
+Introduction
+============
+
+Counters come in many varieties, each with their own trade offs:
+
+ * A global atomic counter provides a fast read access to the current
+ sum, at the expense of cache-line bouncing on updates. This leads to
+ poor performance of frequent updates from various cores on large SMP
+ systems.
+
+ * A per-cpu split counter provides fast updates to per-cpu counters,
+ at the expense of a slower aggregation (sum). The sum operation needs
+ to iterate over all per-cpu counters to calculate the current total.
+
+The hierarchical per-cpu counters attempt to provide the best of both
+worlds (fast updates, and fast sum) by relaxing requirements on the sum
+accuracy. It allows quickly querying an approximated sum value, along
+with the possible min/max ranges of the associated precise sum. The
+exact precise sum can still be calculated with an iteration on all
+per-cpu counter, but the availability of an approximated sum value with
+possible precise sum min/max ranges allows eliminating candidates which
+are certainly outside of a known target range without the overhead of
+precise sums.
+
+Overview
+========
+
+The herarchical per-cpu counters are organized as a tree with the tree
+root at the bottom (last level) and the first level of the tree
+consisting of per-cpu counters.
+
+The intermediate tree levels contain carry propagation counters. When
+reaching a threshold (batch size), the carry is propagated down the
+tree.
+
+This allows reading an approximated value at the root, which has a
+bounded accuracy (minimum/maximum possible precise sum range) determined
+by the tree topology.
+
+Use Cases
+=========
+
+Use cases HPCC is meant to handle invove tracking resources which are
+used across many CPUs to quickly sum as feedback for decision making to
+apply throttling, quota limits, sort tasks, and perform memory or task
+migration decisions. When considering approximated sums within the
+accuracy range of the decision threshold, the user can either:
+
+ * Be conservative and fast: Consider that the sum has reached the
+ limit as soon as the given limit is within the approximation range.
+
+ * Be aggressive and fast: Consider that the sum is over the
+ limit only when the approximation range is over the given limit.
+
+ * Be precise and slow: Do a precise comparison with the limit, which
+ requires a precise sum when the limit is within the approximated
+ range.
+
+One use-case for these hierarchical counters is to implement a two-pass
+algorithm to speed up sorting picking a maximum/minimunm sum value from
+a set. A first pass compares the approximated values, and then a second
+pass only needs the precise sum for counter trees which are within the
+possible precise sum range of the counter tree chosen by the first pass.
+
+Functions and structures
+========================
+
+.. kernel-doc:: include/linux/percpu_counter_tree.h
+.. kernel-doc:: lib/percpu_counter_tree.c
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index 8731606d8d36..21e6b7814fef 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -1413,9 +1413,9 @@ static inline void __mm_flags_set_mask_bits_word(struct mm_struct *mm,
MT_FLAGS_USE_RCU)
extern struct mm_struct init_mm;
-#define MM_STRUCT_FLEXIBLE_ARRAY_INIT \
-{ \
- [0 ... sizeof(cpumask_t) + MM_CID_STATIC_SIZE - 1] = 0 \
+#define MM_STRUCT_FLEXIBLE_ARRAY_INIT \
+{ \
+ [0 ... sizeof(cpumask_t) + MM_CID_STATIC_SIZE + PERCPU_COUNTER_TREE_ITEMS_STATIC_SIZE - 1] = 0 \
}
/* Pointer magic because the dynamic array size confuses some compilers. */
diff --git a/include/linux/percpu_counter_tree.h b/include/linux/percpu_counter_tree.h
new file mode 100644
index 000000000000..828c763edd4a
--- /dev/null
+++ b/include/linux/percpu_counter_tree.h
@@ -0,0 +1,367 @@
+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */
+/* SPDX-FileCopyrightText: 2025 Mathieu Desnoyers <mathieu.desnoyers@efficios.com> */
+
+#ifndef _PERCPU_COUNTER_TREE_H
+#define _PERCPU_COUNTER_TREE_H
+
+#include <linux/preempt.h>
+#include <linux/atomic.h>
+#include <linux/percpu.h>
+
+#ifdef CONFIG_SMP
+
+#if NR_CPUS == (1U << 0)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 0
+#elif NR_CPUS <= (1U << 1)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 1
+#elif NR_CPUS <= (1U << 2)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 3
+#elif NR_CPUS <= (1U << 3)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 7
+#elif NR_CPUS <= (1U << 4)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 7
+#elif NR_CPUS <= (1U << 5)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 11
+#elif NR_CPUS <= (1U << 6)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 21
+#elif NR_CPUS <= (1U << 7)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 21
+#elif NR_CPUS <= (1U << 8)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 37
+#elif NR_CPUS <= (1U << 9)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 73
+#elif NR_CPUS <= (1U << 10)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 149
+#elif NR_CPUS <= (1U << 11)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 293
+#elif NR_CPUS <= (1U << 12)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 585
+#elif NR_CPUS <= (1U << 13)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 1173
+#elif NR_CPUS <= (1U << 14)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 2341
+#elif NR_CPUS <= (1U << 15)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 4681
+#elif NR_CPUS <= (1U << 16)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 4681
+#elif NR_CPUS <= (1U << 17)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 8777
+#elif NR_CPUS <= (1U << 18)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 17481
+#elif NR_CPUS <= (1U << 19)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 34953
+#elif NR_CPUS <= (1U << 20)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 69905
+#else
+# error "Unsupported number of CPUs."
+#endif
+
+struct percpu_counter_tree_level_item {
+ atomic_long_t count; /*
+ * Count the number of carry for this tree item.
+ * The carry counter is kept at the order of the
+ * carry accounted for at this tree level.
+ */
+} ____cacheline_aligned_in_smp;
+
+#define PERCPU_COUNTER_TREE_ITEMS_STATIC_SIZE \
+ (PERCPU_COUNTER_TREE_STATIC_NR_ITEMS * sizeof(struct percpu_counter_tree_level_item))
+
+struct percpu_counter_tree {
+ /* Fast-path fields. */
+ unsigned long __percpu *level0; /* Pointer to per-CPU split counters (tree level 0). */
+ unsigned long level0_bit_mask; /* Bit mask to apply to detect carry propagation from tree level 0. */
+ union {
+ unsigned long *i; /* Approximate sum for single-CPU topology. */
+ atomic_long_t *a; /* Approximate sum for SMP topology. */
+ } approx_sum;
+ long bias; /* Bias to apply to counter precise and approximate values. */
+
+ /* Slow-path fields. */
+ struct percpu_counter_tree_level_item *items; /* Array of tree items for levels 1 to N. */
+ unsigned long batch_size; /*
+ * The batch size is the increment step at level 0 which
+ * triggers a carry propagation. The batch size is required
+ * to be greater than 1, and a power of 2.
+ */
+ /*
+ * The tree approximate sum is guaranteed to be within this accuracy range:
+ * (precise_sum - approx_accuracy_range.under) <= approx_sum <= (precise_sum + approx_accuracy_range.over).
+ * This accuracy is derived from the hardware topology and the tree batch_size.
+ * The "under" accuracy is larger than the "over" accuracy because the negative range of a
+ * two's complement signed integer is one unit larger than the positive range. This delta
+ * is summed for each tree item, which leads to a significantly larger "under" accuracy range
+ * compared to the "over" accuracy range.
+ */
+ struct {
+ unsigned long under;
+ unsigned long over;
+ } approx_accuracy_range;
+};
+
+size_t percpu_counter_tree_items_size(void);
+int percpu_counter_tree_init_many(struct percpu_counter_tree *counters, struct percpu_counter_tree_level_item *items,
+ unsigned int nr_counters, unsigned long batch_size, gfp_t gfp_flags);
+int percpu_counter_tree_init(struct percpu_counter_tree *counter, struct percpu_counter_tree_level_item *items,
+ unsigned long batch_size, gfp_t gfp_flags);
+void percpu_counter_tree_destroy_many(struct percpu_counter_tree *counter, unsigned int nr_counters);
+void percpu_counter_tree_destroy(struct percpu_counter_tree *counter);
+void percpu_counter_tree_add(struct percpu_counter_tree *counter, long inc);
+long percpu_counter_tree_precise_sum(struct percpu_counter_tree *counter);
+int percpu_counter_tree_approximate_compare(struct percpu_counter_tree *a, struct percpu_counter_tree *b);
+int percpu_counter_tree_approximate_compare_value(struct percpu_counter_tree *counter, long v);
+int percpu_counter_tree_precise_compare(struct percpu_counter_tree *a, struct percpu_counter_tree *b);
+int percpu_counter_tree_precise_compare_value(struct percpu_counter_tree *counter, long v);
+void percpu_counter_tree_set(struct percpu_counter_tree *counter, long v);
+int percpu_counter_tree_subsystem_init(void);
+
+/**
+ * percpu_counter_tree_approximate_sum() - Return approximate counter sum.
+ * @counter: The counter to sum.
+ *
+ * Querying the approximate sum is fast, but it is only accurate within
+ * the bounds delimited by percpu_counter_tree_approximate_accuracy_range().
+ * This is meant to be used when speed is preferred over accuracy.
+ *
+ * Return: The current approximate counter sum.
+ */
+static inline
+long percpu_counter_tree_approximate_sum(struct percpu_counter_tree *counter)
+{
+ unsigned long v;
+
+ if (!counter->level0_bit_mask)
+ v = READ_ONCE(*counter->approx_sum.i);
+ else
+ v = atomic_long_read(counter->approx_sum.a);
+ return (long) (v + (unsigned long)READ_ONCE(counter->bias));
+}
+
+/**
+ * percpu_counter_tree_approximate_accuracy_range - Query the accuracy range for a counter tree.
+ * @counter: Counter to query.
+ * @under: Pointer to a variable to be incremented of the approximation
+ * accuracy range below the precise sum.
+ * @over: Pointer to a variable to be incremented of the approximation
+ * accuracy range above the precise sum.
+ *
+ * Query the accuracy range limits for the counter.
+ * Because of two's complement binary representation, the "under" range is typically
+ * slightly larger than the "over" range.
+ * Those values are derived from the hardware topology and the counter tree batch size.
+ * They are invariant for a given counter tree.
+ * Using this function should not be typically required, see the following functions instead:
+ * * percpu_counter_tree_approximate_compare(),
+ * * percpu_counter_tree_approximate_compare_value(),
+ * * percpu_counter_tree_precise_compare(),
+ * * percpu_counter_tree_precise_compare_value().
+ */
+static inline
+void percpu_counter_tree_approximate_accuracy_range(struct percpu_counter_tree *counter,
+ unsigned long *under, unsigned long *over)
+{
+ *under += counter->approx_accuracy_range.under;
+ *over += counter->approx_accuracy_range.over;
+}
+
+#else /* !CONFIG_SMP */
+
+#define PERCPU_COUNTER_TREE_ITEMS_STATIC_SIZE 0
+
+struct percpu_counter_tree_level_item;
+
+struct percpu_counter_tree {
+ atomic_long_t count;
+};
+
+static inline
+size_t percpu_counter_tree_items_size(void)
+{
+ return 0;
+}
+
+static inline
+int percpu_counter_tree_init_many(struct percpu_counter_tree *counters, struct percpu_counter_tree_level_item *items,
+ unsigned int nr_counters, unsigned long batch_size, gfp_t gfp_flags)
+{
+ for (unsigned int i = 0; i < nr_counters; i++)
+ atomic_long_set(&counters[i].count, 0);
+ return 0;
+}
+
+static inline
+int percpu_counter_tree_init(struct percpu_counter_tree *counter, struct percpu_counter_tree_level_item *items,
+ unsigned long batch_size, gfp_t gfp_flags)
+{
+ return percpu_counter_tree_init_many(counter, items, 1, batch_size, gfp_flags);
+}
+
+static inline
+void percpu_counter_tree_destroy_many(struct percpu_counter_tree *counter, unsigned int nr_counters)
+{
+}
+
+static inline
+void percpu_counter_tree_destroy(struct percpu_counter_tree *counter)
+{
+}
+
+static inline
+long percpu_counter_tree_precise_sum(struct percpu_counter_tree *counter)
+{
+ return atomic_long_read(&counter->count);
+}
+
+static inline
+int percpu_counter_tree_precise_compare(struct percpu_counter_tree *a, struct percpu_counter_tree *b)
+{
+ long count_a = percpu_counter_tree_precise_sum(a),
+ count_b = percpu_counter_tree_precise_sum(b);
+
+ if (count_a == count_b)
+ return 0;
+ if (count_a < count_b)
+ return -1;
+ return 1;
+}
+
+static inline
+int percpu_counter_tree_precise_compare_value(struct percpu_counter_tree *counter, long v)
+{
+ long count = percpu_counter_tree_precise_sum(counter);
+
+ if (count == v)
+ return 0;
+ if (count < v)
+ return -1;
+ return 1;
+}
+
+static inline
+int percpu_counter_tree_approximate_compare(struct percpu_counter_tree *a, struct percpu_counter_tree *b)
+{
+ return percpu_counter_tree_precise_compare(a, b);
+}
+
+static inline
+int percpu_counter_tree_approximate_compare_value(struct percpu_counter_tree *counter, long v)
+{
+ return percpu_counter_tree_precise_compare_value(counter, v);
+}
+
+static inline
+void percpu_counter_tree_set(struct percpu_counter_tree *counter, long v)
+{
+ atomic_long_set(&counter->count, v);
+}
+
+static inline
+void percpu_counter_tree_approximate_accuracy_range(struct percpu_counter_tree *counter,
+ unsigned long *under, unsigned long *over)
+{
+}
+
+static inline
+void percpu_counter_tree_add(struct percpu_counter_tree *counter, long inc)
+{
+ atomic_long_add(inc, &counter->count);
+}
+
+static inline
+long percpu_counter_tree_approximate_sum(struct percpu_counter_tree *counter)
+{
+ return percpu_counter_tree_precise_sum(counter);
+}
+
+static inline
+int percpu_counter_tree_subsystem_init(void)
+{
+ return 0;
+}
+
+#endif /* CONFIG_SMP */
+
+/**
+ * percpu_counter_tree_approximate_sum_positive() - Return a positive approximate counter sum.
+ * @counter: The counter to sum.
+ *
+ * Return an approximate counter sum which is guaranteed to be greater
+ * or equal to 0.
+ *
+ * Return: The current positive approximate counter sum.
+ */
+static inline
+long percpu_counter_tree_approximate_sum_positive(struct percpu_counter_tree *counter)
+{
+ long v = percpu_counter_tree_approximate_sum(counter);
+ return v > 0 ? v : 0;
+}
+
+/**
+ * percpu_counter_tree_precise_sum_positive() - Return a positive precise counter sum.
+ * @counter: The counter to sum.
+ *
+ * Return a precise counter sum which is guaranteed to be greater
+ * or equal to 0.
+ *
+ * Return: The current positive precise counter sum.
+ */
+static inline
+long percpu_counter_tree_precise_sum_positive(struct percpu_counter_tree *counter)
+{
+ long v = percpu_counter_tree_precise_sum(counter);
+ return v > 0 ? v : 0;
+}
+
+/**
+ * percpu_counter_tree_approximate_min_max_range() - Return the approximation min and max precise values.
+ * @approx_sum: Approximated sum.
+ * @under: Tree accuracy range (under).
+ * @over: Tree accuracy range (over).
+ * @precise_min: Minimum possible value for precise sum (output).
+ * @precise_max: Maximum possible value for precise sum (output).
+ *
+ * Calculate the minimum and maximum precise values for a given
+ * approximation and (under, over) accuracy range.
+ *
+ * The range of the approximation as a function of the precise sum is expressed as:
+ *
+ * approx_sum >= precise_sum - approx_accuracy_range.under
+ * approx_sum <= precise_sum + approx_accuracy_range.over
+ *
+ * Therefore, the range of the precise sum as a function of the approximation is expressed as:
+ *
+ * precise_sum <= approx_sum + approx_accuracy_range.under
+ * precise_sum >= approx_sum - approx_accuracy_range.over
+ */
+static inline
+void percpu_counter_tree_approximate_min_max_range(long approx_sum, unsigned long under, unsigned long over,
+ long *precise_min, long *precise_max)
+{
+ *precise_min = approx_sum - over;
+ *precise_max = approx_sum + under;
+}
+
+/**
+ * percpu_counter_tree_approximate_min_max() - Return the tree approximation, min and max possible precise values.
+ * @counter: The counter to sum.
+ * @approx_sum: Approximate sum (output).
+ * @precise_min: Minimum possible value for precise sum (output).
+ * @precise_max: Maximum possible value for precise sum (output).
+ *
+ * Return the approximate sum, minimum and maximum precise values for
+ * a counter.
+ */
+static inline
+void percpu_counter_tree_approximate_min_max(struct percpu_counter_tree *counter,
+ long *approx_sum, long *precise_min, long *precise_max)
+{
+ unsigned long under = 0, over = 0;
+ long v = percpu_counter_tree_approximate_sum(counter);
+
+ percpu_counter_tree_approximate_accuracy_range(counter, &under, &over);
+ percpu_counter_tree_approximate_min_max_range(v, under, over, precise_min, precise_max);
+ *approx_sum = v;
+}
+
+#endif /* _PERCPU_COUNTER_TREE_H */
diff --git a/init/main.c b/init/main.c
index 1cb395dd94e4..13aeb7834111 100644
--- a/init/main.c
+++ b/init/main.c
@@ -105,6 +105,7 @@
#include <linux/ptdump.h>
#include <linux/time_namespace.h>
#include <linux/unaligned.h>
+#include <linux/percpu_counter_tree.h>
#include <net/net_namespace.h>
#include <asm/io.h>
@@ -1067,6 +1068,7 @@ void start_kernel(void)
vfs_caches_init_early();
sort_main_extable();
trap_init();
+ percpu_counter_tree_subsystem_init();
mm_core_init();
maple_tree_init();
poking_init();
diff --git a/lib/Makefile b/lib/Makefile
index 1b9ee167517f..abc32420b581 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -181,6 +181,7 @@ obj-$(CONFIG_TEXTSEARCH_KMP) += ts_kmp.o
obj-$(CONFIG_TEXTSEARCH_BM) += ts_bm.o
obj-$(CONFIG_TEXTSEARCH_FSM) += ts_fsm.o
obj-$(CONFIG_SMP) += percpu_counter.o
+obj-$(CONFIG_SMP) += percpu_counter_tree.o
obj-$(CONFIG_AUDIT_GENERIC) += audit.o
obj-$(CONFIG_AUDIT_COMPAT_GENERIC) += compat_audit.o
diff --git a/lib/percpu_counter_tree.c b/lib/percpu_counter_tree.c
new file mode 100644
index 000000000000..d948eba04c4d
--- /dev/null
+++ b/lib/percpu_counter_tree.c
@@ -0,0 +1,690 @@
+// SPDX-License-Identifier: GPL-2.0+ OR MIT
+// SPDX-FileCopyrightText: 2025 Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
+
+/*
+ * Split Counters With Tree Approximation Propagation
+ *
+ * * Propagation diagram when reaching batch size thresholds (± batch size):
+ *
+ * Example diagram for 8 CPUs:
+ *
+ * log2(8) = 3 levels
+ *
+ * At each level, each pair propagates its values to the next level when
+ * reaching the batch size thresholds.
+ *
+ * Counters at levels 0, 1, 2 can be kept on a single byte ([-128 .. +127] range),
+ * although it may be relevant to keep them on "long" counters for
+ * simplicity. (complexity vs memory footprint tradeoff)
+ *
+ * Counter at level 3 can be kept on a "long" counter.
+ *
+ * Level 0: 0 1 2 3 4 5 6 7
+ * | / | / | / | /
+ * | / | / | / | /
+ * | / | / | / | /
+ * Level 1: 0 1 2 3
+ * | / | /
+ * | / | /
+ * | / | /
+ * Level 2: 0 1
+ * | /
+ * | /
+ * | /
+ * Level 3: 0
+ *
+ * * Approximation accuracy:
+ *
+ * BATCH(level N): Level N batch size.
+ *
+ * Example for BATCH(level 0) = 32.
+ *
+ * BATCH(level 0) = 32
+ * BATCH(level 1) = 64
+ * BATCH(level 2) = 128
+ * BATCH(level N) = BATCH(level 0) * 2^N
+ *
+ * per-counter global
+ * accuracy accuracy
+ * Level 0: [ -32 .. +31] ±256 (8 * 32)
+ * Level 1: [ -64 .. +63] ±256 (4 * 64)
+ * Level 2: [-128 .. +127] ±256 (2 * 128)
+ * Total: ------ ±768 (log2(nr_cpu_ids) * BATCH(level 0) * nr_cpu_ids)
+ *
+ * Note that the global accuracy can be calculated more precisely
+ * by taking into account that the positive accuracy range is
+ * 31 rather than 32.
+ *
+ * -----
+ *
+ * Approximate Sum Carry Propagation
+ *
+ * Let's define a number of counter bits for each level, e.g.:
+ *
+ * log2(BATCH(level 0)) = log2(32) = 5
+ * Let's assume, for this example, a 32-bit architecture (sizeof(long) == 4).
+ *
+ * nr_bit value_mask range
+ * Level 0: 5 bits v 0 .. +31
+ * Level 1: 1 bit (v & ~((1UL << 5) - 1)) 0 .. +63
+ * Level 2: 1 bit (v & ~((1UL << 6) - 1)) 0 .. +127
+ * Level 3: 25 bits (v & ~((1UL << 7) - 1)) 0 .. 2^32-1
+ *
+ * Note: Use a "long" per-cpu counter at level 0 to allow precise sum.
+ *
+ * Note: Use cacheline aligned counters at levels above 0 to prevent false sharing.
+ * If memory footprint is an issue, a specialized allocator could be used
+ * to eliminate padding.
+ *
+ * Example with expanded values:
+ *
+ * counter_add(counter, inc):
+ *
+ * if (!inc)
+ * return;
+ *
+ * res = percpu_add_return(counter @ Level 0, inc);
+ * orig = res - inc;
+ * if (inc < 0) {
+ * inc = -(-inc & ~0b00011111); // Clear used bits
+ * // xor bit 5: underflow
+ * if ((inc ^ orig ^ res) & 0b00100000)
+ * inc -= 0b00100000;
+ * } else {
+ * inc &= ~0b00011111; // Clear used bits
+ * // xor bit 5: overflow
+ * if ((inc ^ orig ^ res) & 0b00100000)
+ * inc += 0b00100000;
+ * }
+ * if (!inc)
+ * return;
+ *
+ * res = atomic_long_add_return(counter @ Level 1, inc);
+ * orig = res - inc;
+ * if (inc < 0) {
+ * inc = -(-inc & ~0b00111111); // Clear used bits
+ * // xor bit 6: underflow
+ * if ((inc ^ orig ^ res) & 0b01000000)
+ * inc -= 0b01000000;
+ * } else {
+ * inc &= ~0b00111111; // Clear used bits
+ * // xor bit 6: overflow
+ * if ((inc ^ orig ^ res) & 0b01000000)
+ * inc += 0b01000000;
+ * }
+ * if (!inc)
+ * return;
+ *
+ * res = atomic_long_add_return(counter @ Level 2, inc);
+ * orig = res - inc;
+ * if (inc < 0) {
+ * inc = -(-inc & ~0b01111111); // Clear used bits
+ * // xor bit 7: underflow
+ * if ((inc ^ orig ^ res) & 0b10000000)
+ * inc -= 0b10000000;
+ * } else {
+ * inc &= ~0b01111111; // Clear used bits
+ * // xor bit 7: overflow
+ * if ((inc ^ orig ^ res) & 0b10000000)
+ * inc += 0b10000000;
+ * }
+ * if (!inc)
+ * return;
+ *
+ * atomic_long_add(counter @ Level 3, inc);
+ */
+
+#include <linux/percpu_counter_tree.h>
+#include <linux/cpumask.h>
+#include <linux/percpu.h>
+#include <linux/atomic.h>
+#include <linux/errno.h>
+#include <linux/slab.h>
+#include <linux/math.h>
+
+#define MAX_NR_LEVELS 5
+
+/*
+ * The counter configuration is selected at boot time based on the
+ * hardware topology.
+ */
+struct counter_config {
+ unsigned int nr_items; /*
+ * nr_items is the number of items in the tree for levels 1
+ * up to and including the final level (approximate sum).
+ * It excludes the level 0 per-CPU counters.
+ */
+ unsigned char nr_levels; /*
+ * nr_levels is the number of hierarchical counter tree levels.
+ * It excludes the final level (approximate sum).
+ */
+ unsigned char n_arity_order[MAX_NR_LEVELS]; /*
+ * n-arity of tree nodes for each level from
+ * 0 to (nr_levels - 1).
+ */
+};
+
+static const struct counter_config per_nr_cpu_order_config[] = {
+ [0] = { .nr_items = 0, .nr_levels = 0, .n_arity_order = { 0 } },
+ [1] = { .nr_items = 1, .nr_levels = 1, .n_arity_order = { 1 } },
+ [2] = { .nr_items = 3, .nr_levels = 2, .n_arity_order = { 1, 1 } },
+ [3] = { .nr_items = 7, .nr_levels = 3, .n_arity_order = { 1, 1, 1 } },
+ [4] = { .nr_items = 7, .nr_levels = 3, .n_arity_order = { 2, 1, 1 } },
+ [5] = { .nr_items = 11, .nr_levels = 3, .n_arity_order = { 2, 2, 1 } },
+ [6] = { .nr_items = 21, .nr_levels = 3, .n_arity_order = { 2, 2, 2 } },
+ [7] = { .nr_items = 21, .nr_levels = 3, .n_arity_order = { 3, 2, 2 } },
+ [8] = { .nr_items = 37, .nr_levels = 3, .n_arity_order = { 3, 3, 2 } },
+ [9] = { .nr_items = 73, .nr_levels = 3, .n_arity_order = { 3, 3, 3 } },
+ [10] = { .nr_items = 149, .nr_levels = 4, .n_arity_order = { 3, 3, 2, 2 } },
+ [11] = { .nr_items = 293, .nr_levels = 4, .n_arity_order = { 3, 3, 3, 2 } },
+ [12] = { .nr_items = 585, .nr_levels = 4, .n_arity_order = { 3, 3, 3, 3 } },
+ [13] = { .nr_items = 1173, .nr_levels = 5, .n_arity_order = { 3, 3, 3, 2, 2 } },
+ [14] = { .nr_items = 2341, .nr_levels = 5, .n_arity_order = { 3, 3, 3, 3, 2 } },
+ [15] = { .nr_items = 4681, .nr_levels = 5, .n_arity_order = { 3, 3, 3, 3, 3 } },
+ [16] = { .nr_items = 4681, .nr_levels = 5, .n_arity_order = { 4, 3, 3, 3, 3 } },
+ [17] = { .nr_items = 8777, .nr_levels = 5, .n_arity_order = { 4, 4, 3, 3, 3 } },
+ [18] = { .nr_items = 17481, .nr_levels = 5, .n_arity_order = { 4, 4, 4, 3, 3 } },
+ [19] = { .nr_items = 34953, .nr_levels = 5, .n_arity_order = { 4, 4, 4, 4, 3 } },
+ [20] = { .nr_items = 69905, .nr_levels = 5, .n_arity_order = { 4, 4, 4, 4, 4 } },
+};
+
+static const struct counter_config *counter_config; /* Hierarchical counter configuration for the hardware topology. */
+static unsigned int nr_cpus_order; /* Order of nr_cpu_ids. */
+static unsigned long accuracy_multiplier; /* Calculate accuracy for a given batch size (multiplication factor). */
+
+static
+int __percpu_counter_tree_init(struct percpu_counter_tree *counter,
+ unsigned long batch_size, gfp_t gfp_flags,
+ unsigned long __percpu *level0,
+ struct percpu_counter_tree_level_item *items)
+{
+ /* Batch size must be greater than 1, and a power of 2. */
+ if (WARN_ON(batch_size <= 1 || (batch_size & (batch_size - 1))))
+ return -EINVAL;
+ counter->batch_size = batch_size;
+ counter->bias = 0;
+ counter->level0 = level0;
+ counter->items = items;
+ if (!nr_cpus_order) {
+ counter->approx_sum.i = per_cpu_ptr(counter->level0, 0);
+ counter->level0_bit_mask = 0;
+ } else {
+ counter->approx_sum.a = &counter->items[counter_config->nr_items - 1].count;
+ counter->level0_bit_mask = 1UL << get_count_order(batch_size);
+ }
+ /*
+ * Each tree item signed integer has a negative range which is
+ * one unit greater than the positive range.
+ */
+ counter->approx_accuracy_range.under = batch_size * accuracy_multiplier;
+ counter->approx_accuracy_range.over = (batch_size - 1) * accuracy_multiplier;
+ return 0;
+}
+
+/**
+ * percpu_counter_tree_init_many() - Initialize many per-CPU counter trees.
+ * @counters: An array of @nr_counters counters to initialize.
+ * Their memory is provided by the caller.
+ * @items: Pointer to memory area where to store tree items.
+ * This memory is provided by the caller.
+ * Its size needs to be at least @nr_counters * percpu_counter_tree_items_size().
+ * @nr_counters: The number of counter trees to initialize
+ * @batch_size: The batch size is the increment step at level 0 which triggers a
+ * carry propagation.
+ * The batch size is required to be greater than 1, and a power of 2.
+ * @gfp_flags: gfp flags to pass to the per-CPU allocator.
+ *
+ * Initialize many per-CPU counter trees using a single per-CPU
+ * allocator invocation for @nr_counters counters.
+ *
+ * Return:
+ * * %0: Success
+ * * %-EINVAL: - Invalid @batch_size argument
+ * * %-ENOMEM: - Out of memory
+ */
+int percpu_counter_tree_init_many(struct percpu_counter_tree *counters, struct percpu_counter_tree_level_item *items,
+ unsigned int nr_counters, unsigned long batch_size, gfp_t gfp_flags)
+{
+ void __percpu *level0, *level0_iter;
+ size_t counter_size = sizeof(*counters->level0),
+ items_size = percpu_counter_tree_items_size();
+ void *items_iter;
+ unsigned int i;
+ int ret;
+
+ memset(items, 0, items_size * nr_counters);
+ level0 = __alloc_percpu_gfp(nr_counters * counter_size,
+ __alignof__(*counters->level0), gfp_flags);
+ if (!level0)
+ return -ENOMEM;
+ level0_iter = level0;
+ items_iter = items;
+ for (i = 0; i < nr_counters; i++) {
+ ret = __percpu_counter_tree_init(&counters[i], batch_size, gfp_flags, level0_iter, items_iter);
+ if (ret)
+ goto free_level0;
+ level0_iter += counter_size;
+ items_iter += items_size;
+ }
+ return 0;
+
+free_level0:
+ free_percpu(level0);
+ return ret;
+}
+
+/**
+ * percpu_counter_tree_init() - Initialize one per-CPU counter tree.
+ * @counter: Counter to initialize.
+ * Its memory is provided by the caller.
+ * @items: Pointer to memory area where to store tree items.
+ * This memory is provided by the caller.
+ * Its size needs to be at least percpu_counter_tree_items_size().
+ * @batch_size: The batch size is the increment step at level 0 which triggers a
+ * carry propagation.
+ * The batch size is required to be greater than 1, and a power of 2.
+ * @gfp_flags: gfp flags to pass to the per-CPU allocator.
+ *
+ * Initialize one per-CPU counter tree.
+ *
+ * Return:
+ * * %0: Success
+ * * %-EINVAL: - Invalid @batch_size argument
+ * * %-ENOMEM: - Out of memory
+ */
+int percpu_counter_tree_init(struct percpu_counter_tree *counter, struct percpu_counter_tree_level_item *items,
+ unsigned long batch_size, gfp_t gfp_flags)
+{
+ return percpu_counter_tree_init_many(counter, items, 1, batch_size, gfp_flags);
+}
+
+/**
+ * percpu_counter_tree_destroy_many() - Destroy many per-CPU counter trees.
+ * @counters: Array of counters trees to destroy.
+ * @nr_counters: The number of counter trees to destroy.
+ *
+ * Release internal resources allocated for @nr_counters per-CPU counter trees.
+ */
+
+void percpu_counter_tree_destroy_many(struct percpu_counter_tree *counters, unsigned int nr_counters)
+{
+ free_percpu(counters->level0);
+}
+
+/**
+ * percpu_counter_tree_destroy() - Destroy one per-CPU counter tree.
+ * @counter: Counter to destroy.
+ *
+ * Release internal resources allocated for one per-CPU counter tree.
+ */
+void percpu_counter_tree_destroy(struct percpu_counter_tree *counter)
+{
+ return percpu_counter_tree_destroy_many(counter, 1);
+}
+
+static
+long percpu_counter_tree_carry(long orig, long res, long inc, unsigned long bit_mask)
+{
+ if (inc < 0) {
+ inc = -(-inc & ~(bit_mask - 1));
+ /*
+ * xor bit_mask: underflow.
+ *
+ * If inc has bit set, decrement an additional bit if
+ * there is _no_ bit transition between orig and res.
+ * Else, inc has bit cleared, decrement an additional
+ * bit if there is a bit transition between orig and
+ * res.
+ */
+ if ((inc ^ orig ^ res) & bit_mask)
+ inc -= bit_mask;
+ } else {
+ inc &= ~(bit_mask - 1);
+ /*
+ * xor bit_mask: overflow.
+ *
+ * If inc has bit set, increment an additional bit if
+ * there is _no_ bit transition between orig and res.
+ * Else, inc has bit cleared, increment an additional
+ * bit if there is a bit transition between orig and
+ * res.
+ */
+ if ((inc ^ orig ^ res) & bit_mask)
+ inc += bit_mask;
+ }
+ return inc;
+}
+
+/*
+ * It does not matter through which path the carry propagates up the
+ * tree, therefore there is no need to disable preemption because the
+ * cpu number is only used to favor cache locality.
+ */
+static
+void percpu_counter_tree_add_slowpath(struct percpu_counter_tree *counter, long inc)
+{
+ unsigned int level_items, nr_levels = counter_config->nr_levels,
+ level, n_arity_order;
+ unsigned long bit_mask;
+ struct percpu_counter_tree_level_item *item = counter->items;
+ unsigned int cpu = raw_smp_processor_id();
+
+ WARN_ON_ONCE(!nr_cpus_order); /* Should never be called for 1 cpu. */
+
+ n_arity_order = counter_config->n_arity_order[0];
+ bit_mask = counter->level0_bit_mask << n_arity_order;
+ level_items = 1U << (nr_cpus_order - n_arity_order);
+
+ for (level = 1; level < nr_levels; level++) {
+ /*
+ * For the purpose of carry propagation, the
+ * intermediate level counters only need to keep track
+ * of the bits relevant for carry propagation. We
+ * therefore don't care about higher order bits.
+ * Note that this optimization is unwanted if the
+ * intended use is to track counters within intermediate
+ * levels of the topology.
+ */
+ if (abs(inc) & (bit_mask - 1)) {
+ atomic_long_t *count = &item[cpu & (level_items - 1)].count;
+ unsigned long orig, res;
+
+ res = atomic_long_add_return_relaxed(inc, count);
+ orig = res - inc;
+ inc = percpu_counter_tree_carry(orig, res, inc, bit_mask);
+ if (likely(!inc))
+ return;
+ }
+ item += level_items;
+ n_arity_order = counter_config->n_arity_order[level];
+ level_items >>= n_arity_order;
+ bit_mask <<= n_arity_order;
+ }
+ atomic_long_add(inc, counter->approx_sum.a);
+}
+
+/**
+ * percpu_counter_tree_add() - Add to a per-CPU counter tree.
+ * @counter: Counter added to.
+ * @inc: Increment value (either positive or negative).
+ *
+ * Add @inc to a per-CPU counter tree. This is a fast-path which will
+ * typically increment per-CPU counters as long as there is no carry
+ * greater or equal to the counter tree batch size.
+ */
+void percpu_counter_tree_add(struct percpu_counter_tree *counter, long inc)
+{
+ unsigned long bit_mask = counter->level0_bit_mask, orig, res;
+
+ res = this_cpu_add_return(*counter->level0, inc);
+ orig = res - inc;
+ inc = percpu_counter_tree_carry(orig, res, inc, bit_mask);
+ if (likely(!inc))
+ return;
+ percpu_counter_tree_add_slowpath(counter, inc);
+}
+
+
+static
+long percpu_counter_tree_precise_sum_unbiased(struct percpu_counter_tree *counter)
+{
+ unsigned long sum = 0;
+ int cpu;
+
+ for_each_possible_cpu(cpu)
+ sum += *per_cpu_ptr(counter->level0, cpu);
+ return (long) sum;
+}
+
+/**
+ * percpu_counter_tree_precise_sum() - Return precise counter sum.
+ * @counter: The counter to sum.
+ *
+ * Querying the precise sum is relatively expensive because it needs to
+ * iterate over all CPUs.
+ * This is meant to be used when accuracy is preferred over speed.
+ *
+ * Return: The current precise counter sum.
+ */
+long percpu_counter_tree_precise_sum(struct percpu_counter_tree *counter)
+{
+ return percpu_counter_tree_precise_sum_unbiased(counter) + READ_ONCE(counter->bias);
+}
+
+static
+int compare_delta(long delta, unsigned long accuracy_neg, unsigned long accuracy_pos)
+{
+ if (delta >= 0) {
+ if (delta <= accuracy_pos)
+ return 0;
+ else
+ return 1;
+ } else {
+ if (-delta <= accuracy_neg)
+ return 0;
+ else
+ return -1;
+ }
+}
+
+/**
+ * percpu_counter_tree_approximate_compare - Approximated comparison of two counter trees.
+ * @a: First counter to compare.
+ * @b: Second counter to compare.
+ *
+ * Evaluate an approximate comparison of two counter trees.
+ * This approximation comparison is fast, and provides an accurate
+ * answer if the counters are found to be either less than or greater
+ * than the other. However, if the approximated comparison returns
+ * 0, the counters respective sums are found to be within the two
+ * counters accuracy range.
+ *
+ * Return:
+ * * %0 - Counters @a and @b do not differ by more than the sum of their respective
+ * accuracy ranges.
+ * * %-1 - Counter @a less than counter @b.
+ * * %1 - Counter @a is greater than counter @b.
+ */
+int percpu_counter_tree_approximate_compare(struct percpu_counter_tree *a, struct percpu_counter_tree *b)
+{
+ return compare_delta(percpu_counter_tree_approximate_sum(a) - percpu_counter_tree_approximate_sum(b),
+ a->approx_accuracy_range.over + b->approx_accuracy_range.under,
+ a->approx_accuracy_range.under + b->approx_accuracy_range.over);
+}
+
+/**
+ * percpu_counter_tree_approximate_compare_value - Approximated comparison of a counter tree against a given value.
+ * @counter: Counter to compare.
+ * @v: Value to compare.
+ *
+ * Evaluate an approximate comparison of a counter tree against a given value.
+ * This approximation comparison is fast, and provides an accurate
+ * answer if the counter is found to be either less than or greater
+ * than the value. However, if the approximated comparison returns
+ * 0, the value is within the counter accuracy range.
+ *
+ * Return:
+ * * %0 - The value @v is within the accuracy range of the counter.
+ * * %-1 - The value @v is less than the counter.
+ * * %1 - The value @v is greater than the counter.
+ */
+int percpu_counter_tree_approximate_compare_value(struct percpu_counter_tree *counter, long v)
+{
+ return compare_delta(v - percpu_counter_tree_approximate_sum(counter),
+ counter->approx_accuracy_range.under,
+ counter->approx_accuracy_range.over);
+}
+
+/**
+ * percpu_counter_tree_precise_compare - Precise comparison of two counter trees.
+ * @a: First counter to compare.
+ * @b: Second counter to compare.
+ *
+ * Evaluate a precise comparison of two counter trees.
+ * As an optimization, it uses the approximate counter comparison
+ * to quickly compare counters which are far apart. Only cases where
+ * counter sums are within the accuracy range require precise counter
+ * sums.
+ *
+ * Return:
+ * * %0 - Counters are equal.
+ * * %-1 - Counter @a less than counter @b.
+ * * %1 - Counter @a is greater than counter @b.
+ */
+int percpu_counter_tree_precise_compare(struct percpu_counter_tree *a, struct percpu_counter_tree *b)
+{
+ long count_a = percpu_counter_tree_approximate_sum(a),
+ count_b = percpu_counter_tree_approximate_sum(b);
+ unsigned long accuracy_a, accuracy_b;
+ long delta = count_a - count_b;
+ int res;
+
+ res = compare_delta(delta,
+ a->approx_accuracy_range.over + b->approx_accuracy_range.under,
+ a->approx_accuracy_range.under + b->approx_accuracy_range.over);
+ /* The values are distanced enough for an accurate approximated comparison. */
+ if (res)
+ return res;
+
+ /*
+ * The approximated comparison is within the accuracy range, therefore at least one
+ * precise sum is needed. Sum the counter which has the largest accuracy first.
+ */
+ if (delta >= 0) {
+ accuracy_a = a->approx_accuracy_range.under;
+ accuracy_b = b->approx_accuracy_range.over;
+ } else {
+ accuracy_a = a->approx_accuracy_range.over;
+ accuracy_b = b->approx_accuracy_range.under;
+ }
+ if (accuracy_b < accuracy_a) {
+ count_a = percpu_counter_tree_precise_sum(a);
+ res = compare_delta(count_a - count_b,
+ b->approx_accuracy_range.under,
+ b->approx_accuracy_range.over);
+ if (res)
+ return res;
+ /* Precise sum of second counter is required. */
+ count_b = percpu_counter_tree_precise_sum(b);
+ } else {
+ count_b = percpu_counter_tree_precise_sum(b);
+ res = compare_delta(count_a - count_b,
+ a->approx_accuracy_range.over,
+ a->approx_accuracy_range.under);
+ if (res)
+ return res;
+ /* Precise sum of second counter is required. */
+ count_a = percpu_counter_tree_precise_sum(a);
+ }
+ if (count_a - count_b < 0)
+ return -1;
+ if (count_a - count_b > 0)
+ return 1;
+ return 0;
+}
+
+/**
+ * percpu_counter_tree_precise_compare_value - Precise comparison of a counter tree against a given value.
+ * @counter: Counter to compare.
+ * @v: Value to compare.
+ *
+ * Evaluate a precise comparison of a counter tree against a given value.
+ * As an optimization, it uses the approximate counter comparison
+ * to quickly identify whether the counter and value are far apart.
+ * Only cases where the value is within the counter accuracy range
+ * require a precise counter sum.
+ *
+ * Return:
+ * * %0 - The value @v is equal to the counter.
+ * * %-1 - The value @v is less than the counter.
+ * * %1 - The value @v is greater than the counter.
+ */
+int percpu_counter_tree_precise_compare_value(struct percpu_counter_tree *counter, long v)
+{
+ long count = percpu_counter_tree_approximate_sum(counter);
+ int res;
+
+ res = compare_delta(v - count,
+ counter->approx_accuracy_range.under,
+ counter->approx_accuracy_range.over);
+ /* The values are distanced enough for an accurate approximated comparison. */
+ if (res)
+ return res;
+
+ /* Precise sum is required. */
+ count = percpu_counter_tree_precise_sum(counter);
+ if (v - count < 0)
+ return -1;
+ if (v - count > 0)
+ return 1;
+ return 0;
+}
+
+static
+void percpu_counter_tree_set_bias(struct percpu_counter_tree *counter, long bias)
+{
+ WRITE_ONCE(counter->bias, bias);
+}
+
+/**
+ * percpu_counter_tree_set - Set the counter tree sum to a given value.
+ * @counter: Counter to set.
+ * @v: Value to set.
+ *
+ * Set the counter sum to a given value. It can be useful for instance
+ * to reset the counter sum to 0. Note that even after setting the
+ * counter sum to a given value, the counter sum approximation can
+ * return any value within the accuracy range around that value.
+ */
+void percpu_counter_tree_set(struct percpu_counter_tree *counter, long v)
+{
+ percpu_counter_tree_set_bias(counter,
+ v - percpu_counter_tree_precise_sum_unbiased(counter));
+}
+
+/*
+ * percpu_counter_tree_items_size - Query the size required for counter tree items.
+ *
+ * Query the size of the memory area required to hold the counter tree
+ * items. This depends on the hardware topology and is invariant after
+ * boot.
+ *
+ * Return: Size required to hold tree items.
+ */
+size_t percpu_counter_tree_items_size(void)
+{
+ if (!nr_cpus_order)
+ return 0;
+ return counter_config->nr_items * sizeof(struct percpu_counter_tree_level_item);
+}
+
+static void __init calculate_accuracy_topology(void)
+{
+ unsigned int nr_levels = counter_config->nr_levels, level;
+ unsigned int level_items = 1U << nr_cpus_order;
+ unsigned long batch_size = 1;
+
+ for (level = 0; level < nr_levels; level++) {
+ unsigned int n_arity_order = counter_config->n_arity_order[level];
+
+ /*
+ * The accuracy multiplier is derived from a batch size of 1
+ * to speed up calculating the accuracy at tree initialization.
+ */
+ accuracy_multiplier += batch_size * level_items;
+ batch_size <<= n_arity_order;
+ level_items >>= n_arity_order;
+ }
+}
+
+int __init percpu_counter_tree_subsystem_init(void)
+{
+ nr_cpus_order = get_count_order(nr_cpu_ids);
+ if (WARN_ON_ONCE(nr_cpus_order >= ARRAY_SIZE(per_nr_cpu_order_config))) {
+ printk(KERN_ERR "Unsupported number of CPUs (%u)\n", nr_cpu_ids);
+ return -1;
+ }
+ counter_config = &per_nr_cpu_order_config[nr_cpus_order];
+ calculate_accuracy_topology();
+ return 0;
+}
--
2.39.5
^ permalink raw reply related
* [PATCH v17 2/3] lib: Test hierarchical per-cpu counters
From: Mathieu Desnoyers @ 2026-02-17 16:10 UTC (permalink / raw)
To: Andrew Morton
Cc: linux-kernel, Mathieu Desnoyers, Paul E. McKenney, Steven Rostedt,
Masami Hiramatsu, Dennis Zhou, Tejun Heo, Christoph Lameter,
Martin Liu, David Rientjes, christian.koenig, Shakeel Butt,
SeongJae Park, Michal Hocko, Johannes Weiner, Sweet Tea Dorminy,
Lorenzo Stoakes, Liam R . Howlett, Mike Rapoport,
Suren Baghdasaryan, Vlastimil Babka, Christian Brauner, Wei Yang,
David Hildenbrand, Miaohe Lin, Al Viro, linux-mm,
linux-trace-kernel, Yu Zhao, Roman Gushchin, Mateusz Guzik,
Matthew Wilcox, Baolin Wang, Aboorva Devarajan
In-Reply-To: <20260217161006.1105611-1-mathieu.desnoyers@efficios.com>
Introduce Kunit tests for hierarchical per-cpu counters.
Keep track of two sets of hierarchical counters, each meant to
have the same precise sum at any time, but distributed differently
across the topology.
Keep track of an atomic counter along with each hierarchical
counter, for sum validation.
Those tests cover:
- Single-threaded (no concurrency) updates.
- Concurrent updates of counters from various CPUs.
Perform the following validations:
- Compare the precise sum of counters with the sum tracked by
an atomic counter.
- Compare the precise sum of two sets of hierarchical counters.
- Approximated comparison of hierarchical counter with atomic counter.
- Approximated comparison of two sets of hierarchical counters.
- Validate the bounds of approximation ranges.
Run with the following .kunit/.kunitconfig:
CONFIG_KUNIT=y
CONFIG_SMP=y
CONFIG_PREEMPT=y
CONFIG_NR_CPUS=32
CONFIG_HOTPLUG_CPU=y
CONFIG_PERCPU_COUNTER_TREE_TEST=y
With the following execution (to use SMP):
./tools/testing/kunit/kunit.py run --arch=x86_64 --qemu_args="-smp 12"
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: "Paul E. McKenney" <paulmck@kernel.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Dennis Zhou <dennis@kernel.org>
Cc: Tejun Heo <tj@kernel.org>
Cc: Christoph Lameter <cl@linux.com>
Cc: Martin Liu <liumartin@google.com>
Cc: David Rientjes <rientjes@google.com>
Cc: christian.koenig@amd.com
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: SeongJae Park <sj@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Sweet Tea Dorminy <sweettea-kernel@dorminy.me>
Cc: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
Cc: "Liam R . Howlett" <liam.howlett@oracle.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Cc: Christian Brauner <brauner@kernel.org>
Cc: Wei Yang <richard.weiyang@gmail.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: Miaohe Lin <linmiaohe@huawei.com>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Cc: linux-mm@kvack.org
Cc: linux-trace-kernel@vger.kernel.org
Cc: Yu Zhao <yuzhao@google.com>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Mateusz Guzik <mjguzik@gmail.com>
Cc: Matthew Wilcox <willy@infradead.org>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Aboorva Devarajan <aboorvad@linux.ibm.com>
---
lib/Kconfig | 12 +
lib/tests/Makefile | 2 +
lib/tests/percpu_counter_tree_kunit.c | 351 ++++++++++++++++++++++++++
3 files changed, 365 insertions(+)
create mode 100644 lib/tests/percpu_counter_tree_kunit.c
diff --git a/lib/Kconfig b/lib/Kconfig
index 0f2fb9610647..0b8241e5b548 100644
--- a/lib/Kconfig
+++ b/lib/Kconfig
@@ -52,6 +52,18 @@ config PACKING_KUNIT_TEST
When in doubt, say N.
+config PERCPU_COUNTER_TREE_TEST
+ tristate "Hierarchical Per-CPU counter test" if !KUNIT_ALL_TESTS
+ depends on KUNIT
+ default KUNIT_ALL_TESTS
+ help
+ This builds Kunit tests for the hierarchical per-cpu counters.
+
+ For more information on KUnit and unit tests in general,
+ please refer to the KUnit documentation in Documentation/dev-tools/kunit/.
+
+ When in doubt, say N.
+
config BITREVERSE
tristate
diff --git a/lib/tests/Makefile b/lib/tests/Makefile
index 05f74edbc62b..d282aa23d273 100644
--- a/lib/tests/Makefile
+++ b/lib/tests/Makefile
@@ -56,4 +56,6 @@ obj-$(CONFIG_UTIL_MACROS_KUNIT) += util_macros_kunit.o
obj-$(CONFIG_RATELIMIT_KUNIT_TEST) += test_ratelimit.o
obj-$(CONFIG_UUID_KUNIT_TEST) += uuid_kunit.o
+obj-$(CONFIG_PERCPU_COUNTER_TREE_TEST) += percpu_counter_tree_kunit.o
+
obj-$(CONFIG_TEST_RUNTIME_MODULE) += module/
diff --git a/lib/tests/percpu_counter_tree_kunit.c b/lib/tests/percpu_counter_tree_kunit.c
new file mode 100644
index 000000000000..6d2cee1c5801
--- /dev/null
+++ b/lib/tests/percpu_counter_tree_kunit.c
@@ -0,0 +1,351 @@
+// SPDX-License-Identifier: GPL-2.0+ OR MIT
+// SPDX-FileCopyrightText: 2026 Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
+
+#include <kunit/test.h>
+#include <linux/percpu_counter_tree.h>
+#include <linux/kthread.h>
+#include <linux/wait.h>
+#include <linux/random.h>
+
+struct multi_thread_test_data {
+ long increment;
+ int nr_inc;
+ int counter_index;
+};
+
+/* Hierarchical per-CPU counter instances. */
+static struct percpu_counter_tree counter[2];
+static struct percpu_counter_tree_level_item *items[2];
+
+/* Global atomic counters for validation. */
+static atomic_long_t global_counter[2];
+
+static struct wait_queue_head kernel_threads_wq;
+static atomic_t kernel_threads_to_run;
+
+static void complete_work(void)
+{
+ if (atomic_dec_and_test(&kernel_threads_to_run))
+ wake_up(&kernel_threads_wq);
+}
+
+static void hpcc_print_info(struct kunit *test)
+{
+ kunit_info(test, "Running test with %d CPUs\n", num_online_cpus());
+}
+
+static void add_to_counter(int counter_index, unsigned int nr_inc, long increment)
+{
+ unsigned int i;
+
+ for (i = 0; i < nr_inc; i++) {
+ percpu_counter_tree_add(&counter[counter_index], increment);
+ atomic_long_add(increment, &global_counter[counter_index]);
+ }
+}
+
+static void check_counters(struct kunit *test)
+{
+ int counter_index;
+
+ /* Compare each counter with its global counter. */
+ for (counter_index = 0; counter_index < 2; counter_index++) {
+ long v = atomic_long_read(&global_counter[counter_index]);
+ long approx_sum = percpu_counter_tree_approximate_sum(&counter[counter_index]);
+ unsigned long under_accuracy = 0, over_accuracy = 0;
+ long precise_min, precise_max;
+
+ /* Precise comparison. */
+ KUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(&counter[counter_index]), v);
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_precise_compare_value(&counter[counter_index], v));
+
+ /* Approximate comparison. */
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_approximate_compare_value(&counter[counter_index], v));
+
+ /* Accuracy limits checks. */
+ percpu_counter_tree_approximate_accuracy_range(&counter[counter_index], &under_accuracy, &over_accuracy);
+
+ KUNIT_EXPECT_GE(test, (long)(approx_sum - (v - under_accuracy)), 0);
+ KUNIT_EXPECT_LE(test, (long)(approx_sum - (v + over_accuracy)), 0);
+ KUNIT_EXPECT_GT(test, (long)(approx_sum - (v - under_accuracy - 1)), 0);
+ KUNIT_EXPECT_LT(test, (long)(approx_sum - (v + over_accuracy + 1)), 0);
+
+ /* Precise min/max range check. */
+ percpu_counter_tree_approximate_min_max_range(approx_sum, under_accuracy, over_accuracy, &precise_min, &precise_max);
+
+ KUNIT_EXPECT_GE(test, v - precise_min, 0);
+ KUNIT_EXPECT_LE(test, v - precise_max, 0);
+ KUNIT_EXPECT_GT(test, v - (precise_min - 1), 0);
+ KUNIT_EXPECT_LT(test, v - (precise_max + 1), 0);
+ }
+ /* Compare each counter with the second counter. */
+ KUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(&counter[0]), percpu_counter_tree_precise_sum(&counter[1]));
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_precise_compare(&counter[0], &counter[1]));
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_approximate_compare(&counter[0], &counter[1]));
+}
+
+static int multi_thread_worker_fn(void *data)
+{
+ struct multi_thread_test_data *td = data;
+
+ add_to_counter(td->counter_index, td->nr_inc, td->increment);
+ complete_work();
+ kfree(td);
+ return 0;
+}
+
+static void test_run_on_specific_cpu(struct kunit *test, int target_cpu, int counter_index, unsigned int nr_inc, long increment)
+{
+ struct task_struct *task;
+ struct multi_thread_test_data *td = kzalloc(sizeof(struct multi_thread_test_data), GFP_KERNEL);
+
+ KUNIT_EXPECT_PTR_NE(test, td, NULL);
+ td->increment = increment;
+ td->nr_inc = nr_inc;
+ td->counter_index = counter_index;
+ atomic_inc(&kernel_threads_to_run);
+ task = kthread_run_on_cpu(multi_thread_worker_fn, td, target_cpu, "kunit_multi_thread_worker");
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, task);
+}
+
+static void init_kthreads(void)
+{
+ atomic_set(&kernel_threads_to_run, 1);
+ init_waitqueue_head(&kernel_threads_wq);
+}
+
+static void fini_kthreads(void)
+{
+ /* Release our own reference. */
+ complete_work();
+ /* Wait for all others threads to run. */
+ __wait_event(kernel_threads_wq, (atomic_read(&kernel_threads_to_run) == 0));
+}
+
+static void test_sync_kthreads(void)
+{
+ fini_kthreads();
+ init_kthreads();
+}
+
+static void init_counters(struct kunit *test, unsigned long batch_size)
+{
+ int i, ret;
+
+ for (i = 0; i < 2; i++) {
+ items[i] = kzalloc(percpu_counter_tree_items_size(), GFP_KERNEL);
+ KUNIT_EXPECT_PTR_NE(test, items[i], NULL);
+ ret = percpu_counter_tree_init(&counter[i], items[i], batch_size, GFP_KERNEL);
+ KUNIT_EXPECT_EQ(test, ret, 0);
+
+ atomic_long_set(&global_counter[i], 0);
+ }
+}
+
+static void fini_counters(void)
+{
+ int i;
+
+ for (i = 0; i < 2; i++) {
+ percpu_counter_tree_destroy(&counter[i]);
+ kfree(items[i]);
+ }
+}
+
+enum up_test_inc_type {
+ INC_ONE,
+ INC_MINUS_ONE,
+ INC_RANDOM,
+};
+
+/*
+ * Single-threaded tests. Those use many threads to run on various CPUs,
+ * but synchronize for completion of each thread before running the
+ * next, effectively making sure there are no concurrent updates.
+ */
+static void do_hpcc_test_single_thread(struct kunit *test, int _cpu0, int _cpu1, enum up_test_inc_type type)
+{
+ unsigned long batch_size_order = 5;
+ int cpu0 = _cpu0;
+ int cpu1 = _cpu1;
+ int i;
+
+ init_counters(test, 1UL << batch_size_order);
+ init_kthreads();
+ for (i = 0; i < 10000; i++) {
+ long increment;
+
+ switch (type) {
+ case INC_ONE:
+ increment = 1;
+ break;
+ case INC_MINUS_ONE:
+ increment = -1;
+ break;
+ case INC_RANDOM:
+ increment = (long) get_random_long() % 50000;
+ break;
+ }
+ if (_cpu0 < 0)
+ cpu0 = cpumask_any_distribute(cpu_online_mask);
+ if (_cpu1 < 0)
+ cpu1 = cpumask_any_distribute(cpu_online_mask);
+ test_run_on_specific_cpu(test, cpu0, 0, 1, increment);
+ test_sync_kthreads();
+ test_run_on_specific_cpu(test, cpu1, 1, 1, increment);
+ test_sync_kthreads();
+ check_counters(test);
+ }
+ fini_kthreads();
+ fini_counters();
+}
+
+static void hpcc_test_single_thread_first(struct kunit *test)
+{
+ int cpu = cpumask_first(cpu_online_mask);
+
+ do_hpcc_test_single_thread(test, cpu, cpu, INC_ONE);
+ do_hpcc_test_single_thread(test, cpu, cpu, INC_MINUS_ONE);
+ do_hpcc_test_single_thread(test, cpu, cpu, INC_RANDOM);
+}
+
+static void hpcc_test_single_thread_first_random(struct kunit *test)
+{
+ int cpu = cpumask_first(cpu_online_mask);
+
+ do_hpcc_test_single_thread(test, cpu, -1, INC_ONE);
+ do_hpcc_test_single_thread(test, cpu, -1, INC_MINUS_ONE);
+ do_hpcc_test_single_thread(test, cpu, -1, INC_RANDOM);
+}
+
+static void hpcc_test_single_thread_random(struct kunit *test)
+{
+ do_hpcc_test_single_thread(test, -1, -1, INC_ONE);
+ do_hpcc_test_single_thread(test, -1, -1, INC_MINUS_ONE);
+ do_hpcc_test_single_thread(test, -1, -1, INC_RANDOM);
+}
+
+/* Multi-threaded SMP tests. */
+
+static void do_hpcc_multi_thread_increment_each_cpu(struct kunit *test, unsigned long batch_size, unsigned int nr_inc, long increment)
+{
+ int cpu;
+
+ init_counters(test, batch_size);
+ init_kthreads();
+ for_each_online_cpu(cpu) {
+ test_run_on_specific_cpu(test, cpu, 0, nr_inc, increment);
+ test_run_on_specific_cpu(test, cpu, 1, nr_inc, increment);
+ }
+ fini_kthreads();
+ check_counters(test);
+ fini_counters();
+}
+
+static void do_hpcc_multi_thread_increment_even_cpus(struct kunit *test, unsigned long batch_size, unsigned int nr_inc, long increment)
+{
+ int cpu;
+
+ init_counters(test, batch_size);
+ init_kthreads();
+ for_each_online_cpu(cpu) {
+ test_run_on_specific_cpu(test, cpu, 0, nr_inc, increment);
+ test_run_on_specific_cpu(test, cpu & ~1, 1, nr_inc, increment); /* even cpus. */
+ }
+ fini_kthreads();
+ check_counters(test);
+ fini_counters();
+}
+
+static void do_hpcc_multi_thread_increment_single_cpu(struct kunit *test, unsigned long batch_size, unsigned int nr_inc, long increment)
+{
+ int cpu;
+
+ init_counters(test, batch_size);
+ init_kthreads();
+ for_each_online_cpu(cpu) {
+ test_run_on_specific_cpu(test, cpu, 0, nr_inc, increment);
+ test_run_on_specific_cpu(test, cpumask_first(cpu_online_mask), 1, nr_inc, increment);
+ }
+ fini_kthreads();
+ check_counters(test);
+ fini_counters();
+}
+
+static void do_hpcc_multi_thread_increment_random_cpu(struct kunit *test, unsigned long batch_size, unsigned int nr_inc, long increment)
+{
+ int cpu;
+
+ init_counters(test, batch_size);
+ init_kthreads();
+ for_each_online_cpu(cpu) {
+ test_run_on_specific_cpu(test, cpu, 0, nr_inc, increment);
+ test_run_on_specific_cpu(test, cpumask_any_distribute(cpu_online_mask), 1, nr_inc, increment);
+ }
+ fini_kthreads();
+ check_counters(test);
+ fini_counters();
+}
+
+static void hpcc_test_multi_thread_batch_increment(struct kunit *test)
+{
+ unsigned long batch_size_order;
+
+ for (batch_size_order = 2; batch_size_order < 10; batch_size_order++) {
+ unsigned int nr_inc;
+
+ for (nr_inc = 1; nr_inc < 1024; nr_inc *= 2) {
+ long increment;
+
+ for (increment = 1; increment < 100000; increment *= 10) {
+ do_hpcc_multi_thread_increment_each_cpu(test, 1UL << batch_size_order, nr_inc, increment);
+ do_hpcc_multi_thread_increment_even_cpus(test, 1UL << batch_size_order, nr_inc, increment);
+ do_hpcc_multi_thread_increment_single_cpu(test, 1UL << batch_size_order, nr_inc, increment);
+ do_hpcc_multi_thread_increment_random_cpu(test, 1UL << batch_size_order, nr_inc, increment);
+ }
+ }
+ }
+}
+
+static void hpcc_test_multi_thread_random_walk(struct kunit *test)
+{
+ unsigned long batch_size_order = 5;
+ int loop;
+
+ for (loop = 0; loop < 100; loop++) {
+ int i;
+
+ init_counters(test, 1UL << batch_size_order);
+ init_kthreads();
+ for (i = 0; i < 1000; i++) {
+ long increment = (long) get_random_long() % 512;
+ unsigned int nr_inc = ((unsigned long) get_random_long()) % 1024;
+
+ test_run_on_specific_cpu(test, cpumask_any_distribute(cpu_online_mask), 0, nr_inc, increment);
+ test_run_on_specific_cpu(test, cpumask_any_distribute(cpu_online_mask), 1, nr_inc, increment);
+ }
+ fini_kthreads();
+ check_counters(test);
+ fini_counters();
+ }
+}
+
+static struct kunit_case hpcc_test_cases[] = {
+ KUNIT_CASE(hpcc_print_info),
+ KUNIT_CASE(hpcc_test_single_thread_first),
+ KUNIT_CASE(hpcc_test_single_thread_first_random),
+ KUNIT_CASE(hpcc_test_single_thread_random),
+ KUNIT_CASE(hpcc_test_multi_thread_batch_increment),
+ KUNIT_CASE(hpcc_test_multi_thread_random_walk),
+ {}
+};
+
+static struct kunit_suite hpcc_test_suite = {
+ .name = "percpu_counter_tree",
+ .test_cases = hpcc_test_cases,
+};
+
+kunit_test_suite(hpcc_test_suite);
+
+MODULE_DESCRIPTION("Test cases for hierarchical per-CPU counters");
+MODULE_LICENSE("Dual MIT/GPL");
--
2.39.5
^ permalink raw reply related
* [PATCH v17 3/3] mm: Improve RSS counter approximation accuracy for proc interfaces
From: Mathieu Desnoyers @ 2026-02-17 16:10 UTC (permalink / raw)
To: Andrew Morton
Cc: linux-kernel, Mathieu Desnoyers, Paul E. McKenney, Steven Rostedt,
Masami Hiramatsu, Dennis Zhou, Tejun Heo, Christoph Lameter,
Martin Liu, David Rientjes, christian.koenig, Shakeel Butt,
SeongJae Park, Michal Hocko, Johannes Weiner, Sweet Tea Dorminy,
Lorenzo Stoakes, Liam R . Howlett, Mike Rapoport,
Suren Baghdasaryan, Vlastimil Babka, Christian Brauner, Wei Yang,
David Hildenbrand, Miaohe Lin, Al Viro, linux-mm,
linux-trace-kernel, Yu Zhao, Roman Gushchin, Mateusz Guzik,
Matthew Wilcox, Baolin Wang, Aboorva Devarajan
In-Reply-To: <20260217161006.1105611-1-mathieu.desnoyers@efficios.com>
Use hierarchical per-cpu counters for RSS tracking to improve the
accuracy of per-mm RSS sum approximation on large many-core systems [1].
This improves the accuracy of the RSS values returned by proc
interfaces.
Here is a (possibly incomplete) list of the prior approaches that were
used or proposed, along with their downside:
1) Per-thread rss tracking: large error on many-thread processes.
2) Per-CPU counters: up to 12% slower for short-lived processes and 9%
increased system time in make test workloads [1]. Moreover, the
inaccuracy increases with O(n^2) with the number of CPUs.
3) Per-NUMA-node counters: requires atomics on fast-path (overhead),
error is high with systems that have lots of NUMA nodes (32 times
the number of NUMA nodes).
4) Use a percise per-cpu counter sum for each counter value query:
Requires iteration on each possible CPUs for each sum, which
adds overhead on large many-core systems running many processes.
The approach proposed here is to replace the per-cpu counters by the
hierarchical per-cpu counters, which bounds the inaccuracy based on the
system topology with O(N*logN).
* Testing results:
Test hardware: 2 sockets AMD EPYC 9654 96-Core Processor (384 logical CPUs total)
Methodology:
Comparing the current upstream implementation with the hierarchical
counters is done by keeping both implementations wired up in parallel,
and running a single-process, single-threaded program which hops
randomly across CPUs in the system, calling mmap(2) and munmap(2) on
random CPUs, keeping track of an array of allocated mappings, randomly
choosing entries to either map or unmap.
get_mm_counter() is instrumented to compare the upstream counter
approximation to the precise value, and print the delta when going over
a given threshold. The delta of the hierarchical counter approximation
to the precise value is also printed for comparison.
After a few minutes running this test, the upstream implementation
counter approximation reaches a 1GB delta from the
precise value, compared to 80MB delta with the hierarchical counter.
The hierarchical counter provides a guaranteed maximum approximation
inaccuracy of 192MB on that hardware topology.
* Fast path implementation comparison
The new inline percpu_counter_tree_add() uses a this_cpu_add_return()
for the fast path (under a certain allocation size threshold). Above
that, it calls a slow path which "trickles up" the carry to upper level
counters with atomic_add_return.
In comparison, the upstream counters implementation calls
percpu_counter_add_batch which uses this_cpu_try_cmpxchg() on the fast
path, and does a raw_spin_lock_irqsave above a certain threshold.
The hierarchical implementation is therefore expected to have less
contention on mid-sized allocations than the upstream counters because
the atomic counters tracking those bits are only shared across nearby
CPUs. In comparison, the upstream counters immediately use a global
spinlock when reaching the threshold.
* Benchmarks
Using will-it-scale page_fault1 benchmarks to compare the upstream
counters to the hierarchical counters. This is done with hyperthreading
disabled. The speedup is within the standard deviation of the upstream
runs, so the overhead is not significant.
upstream hierarchical speedup
page_fault1_processes -s 100 -t 1 614783 615558 +0.1%
page_fault1_threads -s 100 -t 1 612788 612447 -0.1%
page_fault1_processes -s 100 -t 96 37994977 37932035 -0.2%
page_fault1_threads -s 100 -t 96 2484130 2504860 +0.8%
page_fault1_processes -s 100 -t 192 71262917 71118830 -0.2%
page_fault1_threads -s 100 -t 192 2446437 2469296 +0.1%
* Memory Use
The most important parts in terms of memory use are the per-cpu counters
and the tree items which propagate the carry.
In the proposed implementation, the per-cpu counters are allocated
within per-cpu data structures, so they end up using:
nr_possible_cpus * sizeof(unsigned long)
This is in addition to the tree items. The size of those items is
defined by the per_nr_cpu_order_config table "nr_items" field.
Each item is aligned on cacheline size (typically 64 bytes) to minimize
false sharing.
Here is the footprint for a few nr_cpu_ids on a 64-bit arch:
nr_cpu_ids percpu counters (bytes) nr_items items size (bytes) total (bytes)
2 16 1 64 80
4 32 3 192 224
8 64 7 448 512
64 512 21 1344 1856
128 1024 21 1344 2368
256 2048 37 2368 4416
512 4096 73 4672 8768
Compared to this, the upstream percpu counters use a 32-bit integer per-cpu
(4 bytes), and accumulate within a 64-bit global value.
So there is an extra memory footprint added by the current hpcc
implementation, but if it's an issue we have various options to consider
to reduce its footprint.
Link: https://lore.kernel.org/lkml/20250331223516.7810-2-sweettea-kernel@dorminy.me/ # [1]
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: "Paul E. McKenney" <paulmck@kernel.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Dennis Zhou <dennis@kernel.org>
Cc: Tejun Heo <tj@kernel.org>
Cc: Christoph Lameter <cl@linux.com>
Cc: Martin Liu <liumartin@google.com>
Cc: David Rientjes <rientjes@google.com>
Cc: christian.koenig@amd.com
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: SeongJae Park <sj@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Sweet Tea Dorminy <sweettea-kernel@dorminy.me>
Cc: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
Cc: "Liam R . Howlett" <liam.howlett@oracle.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Cc: Christian Brauner <brauner@kernel.org>
Cc: Wei Yang <richard.weiyang@gmail.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: Miaohe Lin <linmiaohe@huawei.com>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Cc: linux-mm@kvack.org
Cc: linux-trace-kernel@vger.kernel.org
Cc: Yu Zhao <yuzhao@google.com>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Mateusz Guzik <mjguzik@gmail.com>
Cc: Matthew Wilcox <willy@infradead.org>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Aboorva Devarajan <aboorvad@linux.ibm.com>
---
Changes since v16:
- Remove references to 2-pass OOM killer algorithm.
Changes since v15:
- Update the commit message to explain that this change is an
improvement to the accuracy of proc interfaces approximated RSS
values, and a preparation step for reducing OOM killer latency.
- Rebase on v2 of "mm: Fix OOM killer and proc stats inaccuracy on
large many-core systems".
Changes since v14:
- This change becomes the preparation for reducing the OOM killer latency.
Changes since v13:
- Change check_mm print format from %d to %ld.
Changes since v10:
- Rebase on top of mm_struct static init fixes.
- Change the alignment of mm_struct flexible array to the alignment of
the rss counters items (which are cacheline aligned on SMP).
- Move the rss counters items to first position within the flexible
array at the end of the mm_struct to place content in decreasing
alignment requirement order.
Changes since v8:
- Use percpu_counter_tree_init_many and
percpu_counter_tree_destroy_many APIs.
- Remove percpu tree items allocation. Extend mm_struct size to include
rss items. Those are handled through the new helpers
get_rss_stat_items() and get_rss_stat_items_size() and passed
as parameter to percpu_counter_tree_init_many().
Changes since v7:
- Use precise sum positive API to handle a scenario where an unlucky
precise sum iteration would observe negative counter values due to
concurrent updates.
Changes since v6:
- Rebased on v6.18-rc3.
- Implement get_mm_counter_sum as percpu_counter_tree_precise_sum for
/proc virtual files memory state queries.
Changes since v5:
- Use percpu_counter_tree_approximate_sum_positive.
Change since v4:
- get_mm_counter needs to return 0 or a positive value.
---
include/linux/mm.h | 19 ++++++++++----
include/linux/mm_types.h | 50 +++++++++++++++++++++++++++----------
include/trace/events/kmem.h | 2 +-
kernel/fork.c | 22 +++++++++-------
4 files changed, 65 insertions(+), 28 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index dc1ad71a2a70..833a60cf6076 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -2847,38 +2847,47 @@ static inline bool get_user_page_fast_only(unsigned long addr,
{
return get_user_pages_fast_only(addr, 1, gup_flags, pagep) == 1;
}
+
+static inline struct percpu_counter_tree_level_item *get_rss_stat_items(struct mm_struct *mm)
+{
+ unsigned long ptr = (unsigned long)mm;
+
+ ptr += offsetof(struct mm_struct, flexible_array);
+ return (struct percpu_counter_tree_level_item *)ptr;
+}
+
/*
* per-process(per-mm_struct) statistics.
*/
static inline unsigned long get_mm_counter(struct mm_struct *mm, int member)
{
- return percpu_counter_read_positive(&mm->rss_stat[member]);
+ return percpu_counter_tree_approximate_sum_positive(&mm->rss_stat[member]);
}
static inline unsigned long get_mm_counter_sum(struct mm_struct *mm, int member)
{
- return percpu_counter_sum_positive(&mm->rss_stat[member]);
+ return percpu_counter_tree_precise_sum_positive(&mm->rss_stat[member]);
}
void mm_trace_rss_stat(struct mm_struct *mm, int member);
static inline void add_mm_counter(struct mm_struct *mm, int member, long value)
{
- percpu_counter_add(&mm->rss_stat[member], value);
+ percpu_counter_tree_add(&mm->rss_stat[member], value);
mm_trace_rss_stat(mm, member);
}
static inline void inc_mm_counter(struct mm_struct *mm, int member)
{
- percpu_counter_inc(&mm->rss_stat[member]);
+ percpu_counter_tree_add(&mm->rss_stat[member], 1);
mm_trace_rss_stat(mm, member);
}
static inline void dec_mm_counter(struct mm_struct *mm, int member)
{
- percpu_counter_dec(&mm->rss_stat[member]);
+ percpu_counter_tree_add(&mm->rss_stat[member], -1);
mm_trace_rss_stat(mm, member);
}
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index 21e6b7814fef..b215e70792fd 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -18,7 +18,7 @@
#include <linux/page-flags-layout.h>
#include <linux/workqueue.h>
#include <linux/seqlock.h>
-#include <linux/percpu_counter.h>
+#include <linux/percpu_counter_tree.h>
#include <linux/types.h>
#include <linux/rseq_types.h>
#include <linux/bitmap.h>
@@ -1117,6 +1117,19 @@ typedef struct {
DECLARE_BITMAP(__mm_flags, NUM_MM_FLAG_BITS);
} __private mm_flags_t;
+/*
+ * The alignment of the mm_struct flexible array is based on the largest
+ * alignment of its content:
+ * __alignof__(struct percpu_counter_tree_level_item) provides a
+ * cacheline aligned alignment on SMP systems, else alignment on
+ * unsigned long on UP systems.
+ */
+#ifdef CONFIG_SMP
+# define __mm_struct_flexible_array_aligned __aligned(__alignof__(struct percpu_counter_tree_level_item))
+#else
+# define __mm_struct_flexible_array_aligned __aligned(__alignof__(unsigned long))
+#endif
+
struct kioctx_table;
struct iommu_mm_data;
struct mm_struct {
@@ -1262,7 +1275,7 @@ struct mm_struct {
unsigned long saved_e_flags;
#endif
- struct percpu_counter rss_stat[NR_MM_COUNTERS];
+ struct percpu_counter_tree rss_stat[NR_MM_COUNTERS];
struct linux_binfmt *binfmt;
@@ -1373,10 +1386,13 @@ struct mm_struct {
} __randomize_layout;
/*
- * The mm_cpumask needs to be at the end of mm_struct, because it
- * is dynamically sized based on nr_cpu_ids.
+ * The rss hierarchical counter items, mm_cpumask, and mm_cid
+ * masks need to be at the end of mm_struct, because they are
+ * dynamically sized based on nr_cpu_ids.
+ * The content of the flexible array needs to be placed in
+ * decreasing alignment requirement order.
*/
- char flexible_array[] __aligned(__alignof__(unsigned long));
+ char flexible_array[] __mm_struct_flexible_array_aligned;
};
/* Copy value to the first system word of mm flags, non-atomically. */
@@ -1415,22 +1431,28 @@ extern struct mm_struct init_mm;
#define MM_STRUCT_FLEXIBLE_ARRAY_INIT \
{ \
- [0 ... sizeof(cpumask_t) + MM_CID_STATIC_SIZE + PERCPU_COUNTER_TREE_ITEMS_STATIC_SIZE - 1] = 0 \
+ [0 ... PERCPU_COUNTER_TREE_ITEMS_STATIC_SIZE + sizeof(cpumask_t) + MM_CID_STATIC_SIZE - 1] = 0 \
}
-/* Pointer magic because the dynamic array size confuses some compilers. */
-static inline void mm_init_cpumask(struct mm_struct *mm)
+static inline size_t get_rss_stat_items_size(void)
{
- unsigned long cpu_bitmap = (unsigned long)mm;
-
- cpu_bitmap += offsetof(struct mm_struct, flexible_array);
- cpumask_clear((struct cpumask *)cpu_bitmap);
+ return percpu_counter_tree_items_size() * NR_MM_COUNTERS;
}
/* Future-safe accessor for struct mm_struct's cpu_vm_mask. */
static inline cpumask_t *mm_cpumask(struct mm_struct *mm)
{
- return (struct cpumask *)&mm->flexible_array;
+ unsigned long ptr = (unsigned long)mm;
+
+ ptr += offsetof(struct mm_struct, flexible_array);
+ /* Skip RSS stats counters. */
+ ptr += get_rss_stat_items_size();
+ return (struct cpumask *)ptr;
+}
+
+static inline void mm_init_cpumask(struct mm_struct *mm)
+{
+ cpumask_clear((struct cpumask *)mm_cpumask(mm));
}
#ifdef CONFIG_LRU_GEN
@@ -1522,6 +1544,8 @@ static inline cpumask_t *mm_cpus_allowed(struct mm_struct *mm)
unsigned long bitmap = (unsigned long)mm;
bitmap += offsetof(struct mm_struct, flexible_array);
+ /* Skip RSS stats counters. */
+ bitmap += get_rss_stat_items_size();
/* Skip cpu_bitmap */
bitmap += cpumask_size();
return (struct cpumask *)bitmap;
diff --git a/include/trace/events/kmem.h b/include/trace/events/kmem.h
index 7f93e754da5c..91c81c44f884 100644
--- a/include/trace/events/kmem.h
+++ b/include/trace/events/kmem.h
@@ -442,7 +442,7 @@ TRACE_EVENT(rss_stat,
__entry->mm_id = mm_ptr_to_hash(mm);
__entry->curr = !!(current->mm == mm);
__entry->member = member;
- __entry->size = (percpu_counter_sum_positive(&mm->rss_stat[member])
+ __entry->size = (percpu_counter_tree_approximate_sum_positive(&mm->rss_stat[member])
<< PAGE_SHIFT);
),
diff --git a/kernel/fork.c b/kernel/fork.c
index e832da9d15a4..bb0c2613a560 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -134,6 +134,11 @@
*/
#define MAX_THREADS FUTEX_TID_MASK
+/*
+ * Batch size of rss stat approximation
+ */
+#define RSS_STAT_BATCH_SIZE 32
+
/*
* Protected counters by write_lock_irq(&tasklist_lock)
*/
@@ -627,14 +632,12 @@ static void check_mm(struct mm_struct *mm)
"Please make sure 'struct resident_page_types[]' is updated as well");
for (i = 0; i < NR_MM_COUNTERS; i++) {
- long x = percpu_counter_sum(&mm->rss_stat[i]);
-
- if (unlikely(x)) {
+ if (unlikely(percpu_counter_tree_precise_compare_value(&mm->rss_stat[i], 0) != 0))
pr_alert("BUG: Bad rss-counter state mm:%p type:%s val:%ld Comm:%s Pid:%d\n",
- mm, resident_page_types[i], x,
+ mm, resident_page_types[i],
+ percpu_counter_tree_precise_sum(&mm->rss_stat[i]),
current->comm,
task_pid_nr(current));
- }
}
if (mm_pgtables_bytes(mm))
@@ -732,7 +735,7 @@ void __mmdrop(struct mm_struct *mm)
put_user_ns(mm->user_ns);
mm_pasid_drop(mm);
mm_destroy_cid(mm);
- percpu_counter_destroy_many(mm->rss_stat, NR_MM_COUNTERS);
+ percpu_counter_tree_destroy_many(mm->rss_stat, NR_MM_COUNTERS);
free_mm(mm);
}
@@ -1124,8 +1127,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p,
if (mm_alloc_cid(mm, p))
goto fail_cid;
- if (percpu_counter_init_many(mm->rss_stat, 0, GFP_KERNEL_ACCOUNT,
- NR_MM_COUNTERS))
+ if (percpu_counter_tree_init_many(mm->rss_stat, get_rss_stat_items(mm),
+ NR_MM_COUNTERS, RSS_STAT_BATCH_SIZE,
+ GFP_KERNEL_ACCOUNT))
goto fail_pcpu;
mm->user_ns = get_user_ns(user_ns);
@@ -3009,7 +3013,7 @@ void __init mm_cache_init(void)
* dynamically sized based on the maximum CPU number this system
* can have, taking hotplug into account (nr_cpu_ids).
*/
- mm_size = sizeof(struct mm_struct) + cpumask_size() + mm_cid_size();
+ mm_size = sizeof(struct mm_struct) + cpumask_size() + mm_cid_size() + get_rss_stat_items_size();
mm_cachep = kmem_cache_create_usercopy("mm_struct",
mm_size, ARCH_MIN_MMSTRUCT_ALIGN,
--
2.39.5
^ permalink raw reply related
* [PATCH v17 0/3] Improve proc RSS accuracy
From: Mathieu Desnoyers @ 2026-02-17 16:10 UTC (permalink / raw)
To: Andrew Morton
Cc: linux-kernel, Mathieu Desnoyers, Paul E. McKenney, Steven Rostedt,
Masami Hiramatsu, Dennis Zhou, Tejun Heo, Christoph Lameter,
Martin Liu, David Rientjes, christian.koenig, Shakeel Butt,
SeongJae Park, Michal Hocko, Johannes Weiner, Sweet Tea Dorminy,
Lorenzo Stoakes, Liam R . Howlett, Mike Rapoport,
Suren Baghdasaryan, Vlastimil Babka, Christian Brauner, Wei Yang,
David Hildenbrand, Miaohe Lin, Al Viro, linux-mm,
linux-trace-kernel, Yu Zhao, Roman Gushchin, Mateusz Guzik,
Matthew Wilcox, Baolin Wang, Aboorva Devarajan
This series introduces the hierarchical tree counter (hpcc) to increase
accuracy of approximated RSS counters exposed through proc interfaces.
With a test program hopping across CPUs doing frequent mmap/munmap
operations, the upstream implementation approximation reaches a 1GB
delta from the precise value after a few minutes, compared to a 80MB
delta with the hierarchical counter. The hierarchical counter provides a
guaranteed maximum approximation inaccuracy of 192MB on that hardware
topology.
This series is based on
commit 0f2acd3148e0 Merge tag 'm68knommu-for-v7.0' of git://git.kernel.org/pub/scm/linux/kernel/git/gerg/m68knommu
The main changes since v16:
- Dropped OOM killer 2-pass task selection algorithm.
- Introduce Kunit tests.
- Only perform atomic increments of intermediate tree nodes when
bits which are significant for carry propagation are being changed.
Andrew, this is meant to target 7.1 after the 7.0 merge window closes.
Thanks,
Mathieu
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: "Paul E. McKenney" <paulmck@kernel.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Dennis Zhou <dennis@kernel.org>
Cc: Tejun Heo <tj@kernel.org>
Cc: Christoph Lameter <cl@linux.com>
Cc: Martin Liu <liumartin@google.com>
Cc: David Rientjes <rientjes@google.com>
Cc: christian.koenig@amd.com
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: SeongJae Park <sj@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Sweet Tea Dorminy <sweettea-kernel@dorminy.me>
Cc: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
Cc: "Liam R . Howlett" <liam.howlett@oracle.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Cc: Christian Brauner <brauner@kernel.org>
Cc: Wei Yang <richard.weiyang@gmail.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: Miaohe Lin <linmiaohe@huawei.com>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Cc: linux-mm@kvack.org
Cc: linux-trace-kernel@vger.kernel.org
Cc: Yu Zhao <yuzhao@google.com>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Mateusz Guzik <mjguzik@gmail.com>
Cc: Matthew Wilcox <willy@infradead.org>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Aboorva Devarajan <aboorvad@linux.ibm.com>
Mathieu Desnoyers (3):
lib: Introduce hierarchical per-cpu counters
lib: Test hierarchical per-cpu counters
mm: Improve RSS counter approximation accuracy for proc interfaces
.../core-api/percpu-counter-tree.rst | 75 ++
include/linux/mm.h | 19 +-
include/linux/mm_types.h | 54 +-
include/linux/percpu_counter_tree.h | 367 ++++++++++
include/trace/events/kmem.h | 2 +-
init/main.c | 2 +
kernel/fork.c | 22 +-
lib/Kconfig | 12 +
lib/Makefile | 1 +
lib/percpu_counter_tree.c | 690 ++++++++++++++++++
lib/tests/Makefile | 2 +
lib/tests/percpu_counter_tree_kunit.c | 351 +++++++++
12 files changed, 1567 insertions(+), 30 deletions(-)
create mode 100644 Documentation/core-api/percpu-counter-tree.rst
create mode 100644 include/linux/percpu_counter_tree.h
create mode 100644 lib/percpu_counter_tree.c
create mode 100644 lib/tests/percpu_counter_tree_kunit.c
--
2.39.5
^ permalink raw reply
* Re: [PATCH v2] blk-mq: use NOIO context to prevent deadlock during debugfs creation
From: Jens Axboe @ 2026-02-17 15:37 UTC (permalink / raw)
To: nilay, ming.lei, hch, Yu Kuai
Cc: yi.zhang, shinichiro.kawasaki, kbusch, rostedt, mhiramat,
mathieu.desnoyers, linux-block, linux-kernel, linux-trace-kernel
In-Reply-To: <20260214054350.2322436-1-yukuai@fnnas.com>
On Sat, 14 Feb 2026 13:43:50 +0800, Yu Kuai wrote:
> Creating debugfs entries can trigger fs reclaim, which can enter back
> into the block layer request_queue. This can cause deadlock if the
> queue is frozen.
>
> Previously, a WARN_ON_ONCE check was used in debugfs_create_files()
> to detect this condition, but it was racy since the queue can be frozen
> from another context at any time.
>
> [...]
Applied, thanks!
[1/1] blk-mq: use NOIO context to prevent deadlock during debugfs creation
(no commit info)
Best regards,
--
Jens Axboe
^ permalink raw reply
* Re: [PATCH v2 1/4] tracing: Fix checking of freed trace_event_file for hist files
From: Steven Rostedt @ 2026-02-17 15:29 UTC (permalink / raw)
To: Petr Pavlu
Cc: Masami Hiramatsu, Mathieu Desnoyers, Tom Zanussi, linux-kernel,
linux-trace-kernel
In-Reply-To: <20260216134247.1311631-2-petr.pavlu@suse.com>
On Mon, 16 Feb 2026 14:41:57 +0100
Petr Pavlu <petr.pavlu@suse.com> wrote:
> diff --git a/include/linux/trace_events.h b/include/linux/trace_events.h
> index 3690221ba3d8..f925034e402d 100644
> --- a/include/linux/trace_events.h
> +++ b/include/linux/trace_events.h
> @@ -683,6 +683,11 @@ static inline void hist_poll_wakeup(void)
>
> #define hist_poll_wait(file, wait) \
> poll_wait(file, &hist_poll_wq, wait)
> +
> +#else
> +static inline void hist_poll_wakeup(void)
> +{
> +}
> #endif
>
> #define __TRACE_EVENT_FLAGS(name, value) \
> diff --git a/kernel/trace/trace_events.c b/kernel/trace/trace_events.c
> index 137b4d9bb116..e8ed6ba155cf 100644
> --- a/kernel/trace/trace_events.c
> +++ b/kernel/trace/trace_events.c
> @@ -1295,6 +1295,9 @@ static void remove_event_file_dir(struct trace_event_file *file)
> free_event_filter(file->filter);
> file->flags |= EVENT_FILE_FL_FREED;
> event_file_put(file);
> +
> + /* Wake up hist poll waiters to notice the EVENT_FILE_FL_FREED flag. */
> + hist_poll_wakeup();
> }
>
> /*
> diff --git a/kernel/trace/trace_events_hist.c b/kernel/trace/trace_events_hist.c
> index c97bb2fda5c0..744c2aa3d668 100644
> --- a/kernel/trace/trace_events_hist.c
> +++ b/kernel/trace/trace_events_hist.c
> @@ -5778,7 +5778,7 @@ static __poll_t event_hist_poll(struct file *file, struct poll_table_struct *wai
>
> guard(mutex)(&event_mutex);
>
> - event_file = event_file_data(file);
> + event_file = event_file_file(file);
> if (!event_file)
> return EPOLLERR;
>
> @@ -5816,7 +5816,7 @@ static int event_hist_open(struct inode *inode, struct file *file)
>
> guard(mutex)(&event_mutex);
>
> - event_file = event_file_data(file);
> + event_file = event_file_file(file);
> if (!event_file) {
> ret = -ENODEV;
> goto err;
> --
This should be broken into two different patches. One for the
hist_poll_wakeup() fix, the other to use event_file_file().
-- Steve
^ permalink raw reply
* Re: [PATCH v2] blk-mq: use NOIO context to prevent deadlock during debugfs creation
From: Shinichiro Kawasaki @ 2026-02-17 7:01 UTC (permalink / raw)
To: Yu Kuai
Cc: axboe@kernel.dk, nilay@linux.ibm.com, ming.lei@redhat.com, hch,
yi.zhang@redhat.com, kbusch@kernel.org, rostedt@goodmis.org,
mhiramat@kernel.org, mathieu.desnoyers@efficios.com,
linux-block@vger.kernel.org, linux-kernel@vger.kernel.org,
linux-trace-kernel@vger.kernel.org
In-Reply-To: <20260214054350.2322436-1-yukuai@fnnas.com>
On Feb 14, 2026 / 13:43, Yu Kuai wrote:
> Creating debugfs entries can trigger fs reclaim, which can enter back
> into the block layer request_queue. This can cause deadlock if the
> queue is frozen.
>
> Previously, a WARN_ON_ONCE check was used in debugfs_create_files()
> to detect this condition, but it was racy since the queue can be frozen
> from another context at any time.
>
> Introduce blk_debugfs_lock()/blk_debugfs_unlock() helpers that combine
> the debugfs_mutex with memalloc_noio_save()/restore() to prevent fs
> reclaim from triggering block I/O. Also add blk_debugfs_lock_nomemsave()
> and blk_debugfs_unlock_nomemrestore() variants for callers that don't
> need NOIO protection (e.g., debugfs removal or read-only operations).
>
> Replace all raw debugfs_mutex lock/unlock pairs with these helpers,
> using the _nomemsave/_nomemrestore variants where appropriate.
>
> Reported-by: Yi Zhang <yi.zhang@redhat.com>
> Closes: https://lore.kernel.org/all/CAHj4cs9gNKEYAPagD9JADfO5UH+OiCr4P7OO2wjpfOYeM-RV=A@mail.gmail.com/
> Reported-by: Shinichiro Kawasaki <shinichiro.kawasaki@wdc.com>
> Closes: https://lore.kernel.org/all/aYWQR7CtYdk3K39g@shinmob/
> Suggested-by: Christoph Hellwig <hch@lst.de>
> Signed-off-by: Yu Kuai <yukuai@fnnas.com>
Yu, thank you for fixing this. I applied this patch on top of the kernel
next-20260211 and confirmed that the WARN is no longer reported for the
test case zbd/012. I also ran whole blktests and observed no regression.
Tested-by: Shinichiro Kawasaki <shinichiro.kawasaki@wdc.com>
^ permalink raw reply
* Re: Enhancing Conditional Filtering for Function Graph Tracer
From: Donglin Peng @ 2026-02-17 3:09 UTC (permalink / raw)
To: Steven Rostedt, Masami Hiramatsu
Cc: linux-trace-kernel, Linux Kernel Mailing List
In-Reply-To: <20260216100935.645ea614@gandalf.local.home>
On Mon, Feb 16, 2026 at 11:09 PM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Sat, 14 Feb 2026 18:36:24 +0800
> Donglin Peng <dolinux.peng@gmail.com> wrote:
>
> > To simplify implementation, I propose extending a new trigger type
> > (e.g., "funcgraph").
> > In ftrace_graph_ignore_func, we could look up the corresponding trace_fprobe and
> > trace_event_file based on trace->func, then decide whether to trace
> > the function using
> > a helper like the following:
> >
> > static bool ftrace_graph_filter(struct trace_fprobe *tf, struct
> > ftrace_regs *fregs,
> > struct trace_event_file *trace_file)
> > {
> > struct fentry_trace_entry_head *entry;
> > struct trace_event_buffer fbuffer;
> > struct event_trigger_data *data;
> > int dsize;
> >
> > dsize = __get_data_size(&tf->tp, fregs, NULL);
> > entry = trace_event_buffer_reserve(&fbuffer, trace_file,
> > sizeof(*entry) + tf->tp.size + dsize);
> > if (!entry)
> > return false;
> >
> > entry = ring_buffer_event_data(fbuffer.event);
> > store_trace_args(&entry[1], &tf->tp, fregs, NULL, sizeof(*entry), dsize);
> >
> > list_for_each_entry_rcu(data, &trace_file->triggers, list) {
> > if (data->cmd_ops->trigger_type == TRIGGER_TYPE_FUNCGRAPH) {
> > struct event_filter *filter = rcu_dereference_sched(data->filter);
> > if (filter && filter_match_preds(filter, entry))
> > return true; // Allow tracing
> > }
> > }
> > return false; // Skip tracing
> > }
> >
> > Does this approach make sense? Any suggestions or concerns?
>
> My biggest concern is with performance. You want to run this against all
> functions being traced?
>
> How is this different than just using fprobes?
Thanks, my initial thought was to extend the functionality of set_graph_function
for more granular tracing control. For instance, consider the function:
vfs_write(struct file *file, const char __user *buf, size_t count, loff_t *pos):
1. Enable tracing when count exceeds a threshold (e.g., 4KB):
echo 'vfs_write if count >= 4096' > set_graph_function
2. Filter by PID and CPU:
echo 'vfs_write if pid == 3456 && cpu == 4' > set_graph_function
To implement this, we would need data structures like trace_probe,
trace_event_call, and trace_event_file. Reusing trace_fprobe might
reduce redundant implementation, however, it's unclear whether this
approach is feasible without introducing unforeseen complexities.
To minimize performance overhead, an intermediate data structure
could be introduced when writing to set_graph_function. For example:
struct funcgraph_data {
unsigned long entry_ip; // Entry address of vfs_write
struct trace_event_file *file; // Associated trace event file
};
This structure would be added to a hash table (funcgraph_hash). At the
entry point of each instrumented function, the execution flow would
proceed as follows:
ftrace_graph_func
→ function_graph_enter_regs
→ trace_graph_entry_args
→ graph_entry
→ ftrace_graph_ignore_fun
→ ftrace_graph_filter(struct ftrace_graph_ent *trace)
Within ftrace_graph_filter, the hash table funcgraph_hash is first queried
using trace->func. If a matching funcgraph_data is found,
ftrace_graph_filter is subsequently invoked with funcgraph_data->file
to apply the filter logic.
>
> -- Steve
^ permalink raw reply
* Re: Enhancing Conditional Filtering for Function Graph Tracer
From: Donglin Peng @ 2026-02-17 3:09 UTC (permalink / raw)
To: Masami Hiramatsu
Cc: Steven Rostedt, linux-trace-kernel, Linux Kernel Mailing List
In-Reply-To: <20260216162716.dc8ec2f3f2fe409b6c11e491@kernel.org>
On Mon, Feb 16, 2026 at 3:27 PM Masami Hiramatsu <mhiramat@kernel.org> wrote:
>
> On Sat, 14 Feb 2026 18:36:24 +0800
> Donglin Peng <dolinux.peng@gmail.com> wrote:
>
> > Hi Steve and Masami,
> >
> > I recently had an idea while using the function graph tracer.
> > Currently, it supports basic
> > filtering via set_ftrace_pid(PID-based) and
> > set_graph_function(function name-based).
> > However, this filtering mechanism feels somewhat limited. Given that
> > we already have
> > features like funcgraph-args, I wonder if we could enhance filtering by allowing
> > conditional tracking based on function parameters — similar to how
> > trace events support
> > filters/triggers.
>
> Agreed. I think it is a good idea.
>
> >
> > To simplify implementation, I propose extending a new trigger type
> > (e.g., "funcgraph").
>
> OK, and I think the question is what do we want to filter. As you know,
> the kernel function call-graph is a very context depending. I think we
> may be better to start with specifying the pid filter via the tringger
> instead of enabling/disabling it on all cpus.
>
> echo "funcgraph:setpid:common_pid" >> events/fprobes/func_foo/trigger
>
> Then, we can also specify a specific task's pid too via sched_switch
> events etc. Also we can add "clearpid". Maybe we also need a special
> PID (e.g. -1) for trace no process instead of using tracing_on.
Thanks, I'm uncertain if modifying triggers is the right approach. My initial
thought was to extend the functionality of set_graph_function for more
granular control, similar to how trace events handle filters. For example,
consider the function
vfs_write(struct file *file, const char __user *buf, size_t count, loff_t *pos):
1. Enable tracing when countmeets a threshold (e.g., ≥4KB):
echo 'vfs_write if count >= 4096' > set_graph_function
2. Filter by PID and CPU:
echo 'vfs_write if pid == 3456 && cpu == 4' > set_graph_function
This usage is somewhat similar to the filter/trigger nodes in trace events,
and it targets function entry tracepoints. I'm considering whether we could
base this on trace_fprobe, as it might align more closely with the existing
filter node functionality. I'm unsure which approach is more appropriate.
Do you have any recommendations?
>
> Thank you,
>
> > In ftrace_graph_ignore_func, we could look up the corresponding trace_fprobe and
> > trace_event_file based on trace->func, then decide whether to trace
> > the function using
> > a helper like the following:
> >
> > static bool ftrace_graph_filter(struct trace_fprobe *tf, struct
> > ftrace_regs *fregs,
> > struct trace_event_file *trace_file)
> > {
> > struct fentry_trace_entry_head *entry;
> > struct trace_event_buffer fbuffer;
> > struct event_trigger_data *data;
> > int dsize;
> >
> > dsize = __get_data_size(&tf->tp, fregs, NULL);
> > entry = trace_event_buffer_reserve(&fbuffer, trace_file,
> > sizeof(*entry) + tf->tp.size + dsize);
> > if (!entry)
> > return false;
> >
> > entry = ring_buffer_event_data(fbuffer.event);
> > store_trace_args(&entry[1], &tf->tp, fregs, NULL, sizeof(*entry), dsize);
> >
> > list_for_each_entry_rcu(data, &trace_file->triggers, list) {
> > if (data->cmd_ops->trigger_type == TRIGGER_TYPE_FUNCGRAPH) {
> > struct event_filter *filter = rcu_dereference_sched(data->filter);
> > if (filter && filter_match_preds(filter, entry))
> > return true; // Allow tracing
> > }
> > }
> > return false; // Skip tracing
> > }
> >
> > Does this approach make sense? Any suggestions or concerns?
> >
> > Thanks,
> > Donglin
>
>
> --
> Masami Hiramatsu (Google) <mhiramat@kernel.org>
^ permalink raw reply
* Re: [PATCH v6 1/3] tracing: Add __event_in_*irq() helpers
From: Steven Rostedt @ 2026-02-17 0:54 UTC (permalink / raw)
To: Usama Arif
Cc: Thomas Ballasi, akpm, axelrasmussen, david, hannes, linux-mm,
linux-trace-kernel, lorenzo.stoakes, mhiramat, mhocko,
shakeel.butt, weixugc, yuanchu, zhengqi.arch
In-Reply-To: <20260216161323.4098568-1-usama.arif@linux.dev>
On Mon, 16 Feb 2026 08:13:21 -0800
Usama Arif <usama.arif@linux.dev> wrote:
> > TP_printk("val=%d %s", __entry->val, __entry_in_irq() ? "(in-irq)" : "")
>
> Should this read __event_in_irq() and not __entry_in_irq()?
Yeah. That's when you should have done "cut-and-paste" but instead just
typed it in by memory :-p
-- Steve
^ permalink raw reply
* Re: [PATCH v3] rv: Fix multiple definition of __pcpu_unique_da_mon_this
From: Gabriele Monaco @ 2026-02-16 19:26 UTC (permalink / raw)
To: Mikhail Gavrilov
Cc: Daniel Bristot de Oliveira, Steven Rostedt, Nam Cao,
linux-trace-kernel, linux-kernel
In-Reply-To: <20260216172707.1441516-1-mikhail.v.gavrilov@gmail.com>
2026-02-16T17:27:23Z Mikhail Gavrilov <mikhail.v.gavrilov@gmail.com>:
> The refactoring in commit 30984ccf31b7 ("rv: Refactor da_monitor to
> minimise macros") replaced per-monitor unique variable names
> (da_mon_##name) with a fixed name (da_mon_this).
>
> While this works for 'static' variables (each translation unit gets its
> own copy), DEFINE_PER_CPU internally generates a non-static dummy
> variable __pcpu_unique_<n> for each per-cpu definition. The requirement
> for this variable to be unique although static exists for modules on
> specific architectures (alpha) and if the kernel is built with
> CONFIG_DEBUG_FORCE_WEAK_PER_CPU (e.g. Fedora's debug kernel).
>
> When multiple per-cpu monitors (e.g. sco and sts) are built-in
> simultaneously, they all produce the same __pcpu_unique_da_mon_this
> symbol, causing a link error:
>
> ld: kernel/trace/rv/monitors/sts/sts.o: multiple definition of
> `__pcpu_unique_da_mon_this';
> kernel/trace/rv/monitors/sco/sco.o: first defined here
>
> Fix this by introducing a DA_MON_NAME macro that expands to a
> per-monitor unique name (da_mon_<MONITOR_NAME>) via the existing
> CONCATENATE helper. This restores the uniqueness that was present
> before the refactoring.
>
Thanks Mikhail,
there was actually no need to send a v3, I believe Steve's was simply a suggestion for future patches.
(Now actually the reader following the link would wonder if there's any difference since V2, I'm assuming there isn't).
I'm going to take this patch and send it once 7.0-rc1 is out. You don't need to do anything else now.
Cheers,
Gabriele
> Fixes: 30984ccf31b7 ("rv: Refactor da_monitor to minimise macros")
> Reviewed-by: Gabriele Monaco <gmonaco@redhat.com>
> Signed-off-by: Mikhail Gavrilov <mikhail.v.gavrilov@gmail.com>
> ---
> Changes since v1: https://lore.kernel.org/all/20260216074727.47292-1-mikhail.v.gavrilov@gmail.com/
> - Updated commit message with details on when __pcpu_unique_ uniqueness
> is required (alpha modules, CONFIG_DEBUG_FORCE_WEAK_PER_CPU)
> - Added code comment above DA_MON_NAME explaining the rationale
>
> include/rv/da_monitor.h | 16 +++++++++++-----
> 1 file changed, 11 insertions(+), 5 deletions(-)
>
> diff --git a/include/rv/da_monitor.h b/include/rv/da_monitor.h
> index db11d41bb438..7511f5464c48 100644
> --- a/include/rv/da_monitor.h
> +++ b/include/rv/da_monitor.h
> @@ -20,6 +20,12 @@
> #include <linux/bug.h>
> #include <linux/sched.h>
>
> +/*
> + * Per-cpu variables require a unique name although static in some
> + * configurations (e.g. CONFIG_DEBUG_FORCE_WEAK_PER_CPU or alpha modules).
> + */
> +#define DA_MON_NAME CONCATENATE(da_mon_, MONITOR_NAME)
> +
> static struct rv_monitor rv_this;
>
> static void react(enum states curr_state, enum events event)
> @@ -183,14 +189,14 @@ static inline bool da_event(struct da_monitor *da_mon, struct task_struct *tsk,
> /*
> * global monitor (a single variable)
> */
> -static struct da_monitor da_mon_this;
> +static struct da_monitor DA_MON_NAME;
>
> /*
> * da_get_monitor - return the global monitor address
> */
> static struct da_monitor *da_get_monitor(void)
> {
> - return &da_mon_this;
> + return &DA_MON_NAME;
> }
>
> /*
> @@ -223,14 +229,14 @@ static inline void da_monitor_destroy(void) { }
> /*
> * per-cpu monitor variables
> */
> -static DEFINE_PER_CPU(struct da_monitor, da_mon_this);
> +static DEFINE_PER_CPU(struct da_monitor, DA_MON_NAME);
>
> /*
> * da_get_monitor - return current CPU monitor address
> */
> static struct da_monitor *da_get_monitor(void)
> {
> - return this_cpu_ptr(&da_mon_this);
> + return this_cpu_ptr(&DA_MON_NAME);
> }
>
> /*
> @@ -242,7 +248,7 @@ static void da_monitor_reset_all(void)
> int cpu;
>
> for_each_cpu(cpu, cpu_online_mask) {
> - da_mon = per_cpu_ptr(&da_mon_this, cpu);
> + da_mon = per_cpu_ptr(&DA_MON_NAME, cpu);
> da_monitor_reset(da_mon);
> }
> }
> --
> 2.53.0
^ permalink raw reply
* Re: [PATCH v2] blk-mq: use NOIO context to prevent deadlock during debugfs creation
From: Mohamed Khalfella @ 2026-02-16 18:28 UTC (permalink / raw)
To: Yu Kuai
Cc: axboe, nilay, ming.lei, hch, yi.zhang, shinichiro.kawasaki,
kbusch, rostedt, mhiramat, mathieu.desnoyers, linux-block,
linux-kernel, linux-trace-kernel
In-Reply-To: <20260214054350.2322436-1-yukuai@fnnas.com>
On Sat 2026-02-14 13:43:50 +0800, Yu Kuai wrote:
> Creating debugfs entries can trigger fs reclaim, which can enter back
> into the block layer request_queue. This can cause deadlock if the
> queue is frozen.
>
> Previously, a WARN_ON_ONCE check was used in debugfs_create_files()
> to detect this condition, but it was racy since the queue can be frozen
> from another context at any time.
>
> Introduce blk_debugfs_lock()/blk_debugfs_unlock() helpers that combine
> the debugfs_mutex with memalloc_noio_save()/restore() to prevent fs
> reclaim from triggering block I/O. Also add blk_debugfs_lock_nomemsave()
> and blk_debugfs_unlock_nomemrestore() variants for callers that don't
> need NOIO protection (e.g., debugfs removal or read-only operations).
>
> Replace all raw debugfs_mutex lock/unlock pairs with these helpers,
> using the _nomemsave/_nomemrestore variants where appropriate.
>
> Reported-by: Yi Zhang <yi.zhang@redhat.com>
> Closes: https://lore.kernel.org/all/CAHj4cs9gNKEYAPagD9JADfO5UH+OiCr4P7OO2wjpfOYeM-RV=A@mail.gmail.com/
> Reported-by: Shinichiro Kawasaki <shinichiro.kawasaki@wdc.com>
> Closes: https://lore.kernel.org/all/aYWQR7CtYdk3K39g@shinmob/
> Suggested-by: Christoph Hellwig <hch@lst.de>
> Signed-off-by: Yu Kuai <yukuai@fnnas.com>
Maybe added Fixes: 65d466b62984 ("blk-mq-debugfs: warn about possible deadlock")?
Reviewed-by: Mohamed Khalfella <mkhalfella@purestorage.com>
^ permalink raw reply
* [PATCH v3] rv: Fix multiple definition of __pcpu_unique_da_mon_this
From: Mikhail Gavrilov @ 2026-02-16 17:27 UTC (permalink / raw)
To: Gabriele Monaco, Daniel Bristot de Oliveira, Steven Rostedt
Cc: Nam Cao, linux-trace-kernel, linux-kernel, Mikhail Gavrilov
The refactoring in commit 30984ccf31b7 ("rv: Refactor da_monitor to
minimise macros") replaced per-monitor unique variable names
(da_mon_##name) with a fixed name (da_mon_this).
While this works for 'static' variables (each translation unit gets its
own copy), DEFINE_PER_CPU internally generates a non-static dummy
variable __pcpu_unique_<n> for each per-cpu definition. The requirement
for this variable to be unique although static exists for modules on
specific architectures (alpha) and if the kernel is built with
CONFIG_DEBUG_FORCE_WEAK_PER_CPU (e.g. Fedora's debug kernel).
When multiple per-cpu monitors (e.g. sco and sts) are built-in
simultaneously, they all produce the same __pcpu_unique_da_mon_this
symbol, causing a link error:
ld: kernel/trace/rv/monitors/sts/sts.o: multiple definition of
`__pcpu_unique_da_mon_this';
kernel/trace/rv/monitors/sco/sco.o: first defined here
Fix this by introducing a DA_MON_NAME macro that expands to a
per-monitor unique name (da_mon_<MONITOR_NAME>) via the existing
CONCATENATE helper. This restores the uniqueness that was present
before the refactoring.
Fixes: 30984ccf31b7 ("rv: Refactor da_monitor to minimise macros")
Reviewed-by: Gabriele Monaco <gmonaco@redhat.com>
Signed-off-by: Mikhail Gavrilov <mikhail.v.gavrilov@gmail.com>
---
Changes since v1: https://lore.kernel.org/all/20260216074727.47292-1-mikhail.v.gavrilov@gmail.com/
- Updated commit message with details on when __pcpu_unique_ uniqueness
is required (alpha modules, CONFIG_DEBUG_FORCE_WEAK_PER_CPU)
- Added code comment above DA_MON_NAME explaining the rationale
include/rv/da_monitor.h | 16 +++++++++++-----
1 file changed, 11 insertions(+), 5 deletions(-)
diff --git a/include/rv/da_monitor.h b/include/rv/da_monitor.h
index db11d41bb438..7511f5464c48 100644
--- a/include/rv/da_monitor.h
+++ b/include/rv/da_monitor.h
@@ -20,6 +20,12 @@
#include <linux/bug.h>
#include <linux/sched.h>
+/*
+ * Per-cpu variables require a unique name although static in some
+ * configurations (e.g. CONFIG_DEBUG_FORCE_WEAK_PER_CPU or alpha modules).
+ */
+#define DA_MON_NAME CONCATENATE(da_mon_, MONITOR_NAME)
+
static struct rv_monitor rv_this;
static void react(enum states curr_state, enum events event)
@@ -183,14 +189,14 @@ static inline bool da_event(struct da_monitor *da_mon, struct task_struct *tsk,
/*
* global monitor (a single variable)
*/
-static struct da_monitor da_mon_this;
+static struct da_monitor DA_MON_NAME;
/*
* da_get_monitor - return the global monitor address
*/
static struct da_monitor *da_get_monitor(void)
{
- return &da_mon_this;
+ return &DA_MON_NAME;
}
/*
@@ -223,14 +229,14 @@ static inline void da_monitor_destroy(void) { }
/*
* per-cpu monitor variables
*/
-static DEFINE_PER_CPU(struct da_monitor, da_mon_this);
+static DEFINE_PER_CPU(struct da_monitor, DA_MON_NAME);
/*
* da_get_monitor - return current CPU monitor address
*/
static struct da_monitor *da_get_monitor(void)
{
- return this_cpu_ptr(&da_mon_this);
+ return this_cpu_ptr(&DA_MON_NAME);
}
/*
@@ -242,7 +248,7 @@ static void da_monitor_reset_all(void)
int cpu;
for_each_cpu(cpu, cpu_online_mask) {
- da_mon = per_cpu_ptr(&da_mon_this, cpu);
+ da_mon = per_cpu_ptr(&DA_MON_NAME, cpu);
da_monitor_reset(da_mon);
}
}
--
2.53.0
^ permalink raw reply related
* Re: [PATCH v6 1/3] tracing: Add __event_in_*irq() helpers
From: Usama Arif @ 2026-02-16 16:13 UTC (permalink / raw)
To: Thomas Ballasi
Cc: Usama Arif, akpm, axelrasmussen, david, hannes, linux-mm,
linux-trace-kernel, lorenzo.stoakes, mhiramat, mhocko, rostedt,
shakeel.butt, weixugc, yuanchu, zhengqi.arch
In-Reply-To: <20260213181537.54350-2-tballasi@linux.microsoft.com>
On Fri, 13 Feb 2026 10:15:35 -0800 Thomas Ballasi <tballasi@linux.microsoft.com> wrote:
> From: Steven Rostedt <rostedt@goodmis.org>
>
> Some trace events want to expose in their output if they were triggered in
> an interrupt or softirq context. Instead of recording this in the event
> structure itself, as this information is stored in the flags portion of
> the event header, add helper macros that can be used in the print format:
>
> TP_printk("val=%d %s", __entry->val, __entry_in_irq() ? "(in-irq)" : "")
Should this read __event_in_irq() and not __entry_in_irq()?
>
> This will output "(in-irq)" for the event in the trace data if the event
> was triggered in hard or soft interrupt context.
>
> Link: https://lore.kernel.org/all/20251229132942.31a2b583@gandalf.local.home/
>
^ permalink raw reply
* Re: [PATCH v6 3/3] mm: vmscan: add PIDs to vmscan tracepoints
From: Usama Arif @ 2026-02-16 16:02 UTC (permalink / raw)
To: Thomas Ballasi
Cc: Usama Arif, akpm, axelrasmussen, david, hannes, linux-mm,
linux-trace-kernel, lorenzo.stoakes, mhiramat, mhocko, rostedt,
shakeel.butt, weixugc, yuanchu, zhengqi.arch
In-Reply-To: <20260213181537.54350-4-tballasi@linux.microsoft.com>
On Fri, 13 Feb 2026 10:15:37 -0800 Thomas Ballasi <tballasi@linux.microsoft.com> wrote:
> The changes aims at adding additionnal tracepoints variables to help
> debuggers attribute them to specific processes.
>
> The PID field uses in_task() to reliably detect when we're in process
> context and can safely access current->pid. When not in process
> context (such as in interrupt or in an asynchronous RCU context), the
> field is set to -1 as a sentinel value.
>
> Signed-off-by: Thomas Ballasi <tballasi@linux.microsoft.com>
> ---
> include/trace/events/vmscan.h | 35 +++++++++++++++++++++++++----------
> 1 file changed, 25 insertions(+), 10 deletions(-)
>
> diff --git a/include/trace/events/vmscan.h b/include/trace/events/vmscan.h
> index 1212f6a7c223e..a68b712ef757a 100644
> --- a/include/trace/events/vmscan.h
> +++ b/include/trace/events/vmscan.h
> @@ -122,18 +122,22 @@ DECLARE_EVENT_CLASS(mm_vmscan_direct_reclaim_begin_template,
> __field( unsigned long, gfp_flags )
> __field( u64, memcg_id )
> __field( int, order )
> + __field( int, pid )
> ),
>
> TP_fast_assign(
> __entry->gfp_flags = (__force unsigned long)gfp_flags;
> __entry->order = order;
> + __entry->pid = current->pid;
> __entry->memcg_id = mem_cgroup_id(memcg);
> ),
>
> - TP_printk("order=%d gfp_flags=%s memcg_id=%llu",
> + TP_printk("order=%d gfp_flags=%s pid=%d memcg_id=%llu %s",
> __entry->order,
> show_gfp_flags(__entry->gfp_flags),
> - __entry->memcg_id)
> + __entry->pid,
> + __entry->memcg_id,
> + __event_in_irq() ? "(in-irq)" : "")
> );
>
> DEFINE_EVENT(mm_vmscan_direct_reclaim_begin_template, mm_vmscan_direct_reclaim_begin,
> @@ -168,16 +172,20 @@ DECLARE_EVENT_CLASS(mm_vmscan_direct_reclaim_end_template,
> TP_STRUCT__entry(
> __field( unsigned long, nr_reclaimed )
> __field( u64, memcg_id )
> + __field( int, pid )
> ),
>
> TP_fast_assign(
> __entry->nr_reclaimed = nr_reclaimed;
> __entry->memcg_id = mem_cgroup_id(memcg);
> + __entry->pid = current->pid;
> ),
>
> - TP_printk("nr_reclaimed=%lu memcg_id=%llu",
> + TP_printk("nr_reclaimed=%lu pid=%d memcg_id=%llu %s",
> __entry->nr_reclaimed,
> - __entry->memcg_id)
> + __entry->pid,
> + __entry->memcg_id,
> + __event_in_irq() ? "(in-irq)" : "")
> );
>
> DEFINE_EVENT(mm_vmscan_direct_reclaim_end_template, mm_vmscan_direct_reclaim_end,
> @@ -220,9 +228,10 @@ TRACE_EVENT(mm_shrink_slab_start,
> __field(unsigned long, cache_items)
> __field(unsigned long long, delta)
> __field(unsigned long, total_scan)
> + __field(u64, memcg_id)
> __field(int, priority)
> __field(int, nid)
> - __field(u64, memcg_id)
> + __field(int, pid)
> ),
>
> TP_fast_assign(
> @@ -236,19 +245,22 @@ TRACE_EVENT(mm_shrink_slab_start,
> __entry->priority = priority;
> __entry->nid = sc->nid;
> __entry->memcg_id = mem_cgroup_id(memcg);
> + __entry->pid = current->pid;
> ),
>
> - TP_printk("%pS %p: nid: %d memcg_id: %llu objects to shrink %ld gfp_flags %s cache items %ld delta %lld total_scan %ld priority %d",
> + TP_printk("%pS %p: nid: %d pid: %d memcg_id: %llu objects to shrink %ld gfp_flags %s cache items %ld delta %lld total_scan %ld priority %d %s",
> __entry->shrink,
> __entry->shr,
> __entry->nid,
> + __entry->pid,
> __entry->memcg_id,
> __entry->nr_objects_to_shrink,
> show_gfp_flags(__entry->gfp_flags),
> __entry->cache_items,
> __entry->delta,
> __entry->total_scan,
> - __entry->priority)
> + __entry->priority,
> + __event_in_irq() ? "(in-irq)" : "")
> );
>
> TRACE_EVENT(mm_shrink_slab_end,
> @@ -266,29 +278,32 @@ TRACE_EVENT(mm_shrink_slab_end,
> __field(long, total_scan)
> __field(int, nid)
> __field(int, retval)
> + __field(int, pid)
> __field(u64, memcg_id)
> ),
>
> TP_fast_assign(
> __entry->shr = shr;
> - __entry->shrink = shr->scan_objects;
__entry->shrink is removed here, but still printed below. Was this an intended
change of this commit?
> __entry->unused_scan = unused_scan_cnt;
> __entry->new_scan = new_scan_cnt;
> __entry->total_scan = total_scan;
> __entry->nid = nid;
> __entry->retval = shrinker_retval;
> + __entry->pid = current->pid;
> __entry->memcg_id = mem_cgroup_id(memcg);
> ),
>
> - TP_printk("%pS %p: nid: %d memcg_id: %llu unused scan count %ld new scan count %ld total_scan %ld last shrinker return val %d",
> + TP_printk("%pS %p: nid: %d pid: %d memcg_id: %llu unused scan count %ld new scan count %ld total_scan %ld last shrinker return val %d %s",
> __entry->shrink,
> __entry->shr,
> __entry->nid,
> + __entry->pid,
> __entry->memcg_id,
> __entry->unused_scan,
> __entry->new_scan,
> __entry->total_scan,
> - __entry->retval)
> + __entry->retval,
> + __event_in_irq() ? "(in-irq)" : "")
> );
>
> TRACE_EVENT(mm_vmscan_lru_isolate,
> --
> 2.33.8
>
>
^ permalink raw reply
* Re: [PATCH mm-unstable v14 08/16] khugepaged: generalize collapse_huge_page for mTHP collapse
From: Lorenzo Stoakes @ 2026-02-16 15:20 UTC (permalink / raw)
To: Nico Pache
Cc: linux-mm, linux-doc, linux-kernel, linux-trace-kernel, akpm,
david, ziy, baolin.wang, Liam.Howlett, ryan.roberts, dev.jain,
baohua, lance.yang, vbabka, rppt, surenb, mhocko, corbet, rostedt,
mhiramat, mathieu.desnoyers, matthew.brost, joshua.hahnjy,
rakie.kim, byungchul, gourry, ying.huang, apopple, jannh,
pfalcato, jackmanb, hannes, willy, peterx, wangkefeng.wang,
usamaarif642, sunnanyong, vishal.moola, thomas.hellstrom, yang,
kas, aarcange, raquini, anshuman.khandual, catalin.marinas, tiwai,
will, dave.hansen, jack, cl, jglisse, zokeefe, rientjes, rdunlap,
hughd, richard.weiyang
In-Reply-To: <CAA1CXcCCbiV9j+_SVNJrkfVRqKPXjGg+Lt3YnyNUhDHWkRjHGQ@mail.gmail.com>
On Wed, Feb 04, 2026 at 03:00:57PM -0700, Nico Pache wrote:
> On Tue, Feb 3, 2026 at 6:13 AM Lorenzo Stoakes
> <lorenzo.stoakes@oracle.com> wrote:
> >
> > On Thu, Jan 22, 2026 at 12:28:33PM -0700, Nico Pache wrote:
> > > Pass an order and offset to collapse_huge_page to support collapsing anon
> > > memory to arbitrary orders within a PMD. order indicates what mTHP size we
> > > are attempting to collapse to, and offset indicates were in the PMD to
> > > start the collapse attempt.
> > >
> > > For non-PMD collapse we must leave the anon VMA write locked until after
> > > we collapse the mTHP-- in the PMD case all the pages are isolated, but in
> > > the mTHP case this is not true, and we must keep the lock to prevent
> > > changes to the VMA from occurring.
> > >
> > > Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
> > > Tested-by: Baolin Wang <baolin.wang@linux.alibaba.com>
> > > Signed-off-by: Nico Pache <npache@redhat.com>
> > > ---
> > > mm/khugepaged.c | 111 +++++++++++++++++++++++++++++++-----------------
> > > 1 file changed, 71 insertions(+), 40 deletions(-)
> > >
> > > diff --git a/mm/khugepaged.c b/mm/khugepaged.c
> > > index 9b7e05827749..76cb17243793 100644
> > > --- a/mm/khugepaged.c
> > > +++ b/mm/khugepaged.c
> > > @@ -1151,44 +1151,54 @@ static enum scan_result alloc_charge_folio(struct folio **foliop, struct mm_stru
> > > return SCAN_SUCCEED;
> > > }
> > >
> > > -static enum scan_result collapse_huge_page(struct mm_struct *mm, unsigned long address,
> > > - int referenced, int unmapped, struct collapse_control *cc)
> > > +static enum scan_result collapse_huge_page(struct mm_struct *mm, unsigned long start_addr,
> > > + int referenced, int unmapped, struct collapse_control *cc,
> > > + bool *mmap_locked, unsigned int order)
> > > {
> > > LIST_HEAD(compound_pagelist);
> > > pmd_t *pmd, _pmd;
> > > - pte_t *pte;
> > > + pte_t *pte = NULL;
> > > pgtable_t pgtable;
> > > struct folio *folio;
> > > spinlock_t *pmd_ptl, *pte_ptl;
> > > enum scan_result result = SCAN_FAIL;
> > > struct vm_area_struct *vma;
> > > struct mmu_notifier_range range;
> > > + bool anon_vma_locked = false;
> > > + const unsigned long nr_pages = 1UL << order;
> > > + const unsigned long pmd_address = start_addr & HPAGE_PMD_MASK;
> > >
> > > - VM_BUG_ON(address & ~HPAGE_PMD_MASK);
> > > + VM_WARN_ON_ONCE(pmd_address & ~HPAGE_PMD_MASK);
> > >
> > > /*
> > > * Before allocating the hugepage, release the mmap_lock read lock.
> > > * The allocation can take potentially a long time if it involves
> > > * sync compaction, and we do not need to hold the mmap_lock during
> > > * that. We will recheck the vma after taking it again in write mode.
> > > + * If collapsing mTHPs we may have already released the read_lock.
> > > */
> > > - mmap_read_unlock(mm);
> > > + if (*mmap_locked) {
> > > + mmap_read_unlock(mm);
> > > + *mmap_locked = false;
> > > + }
> > >
> > > - result = alloc_charge_folio(&folio, mm, cc, HPAGE_PMD_ORDER);
> > > + result = alloc_charge_folio(&folio, mm, cc, order);
> > > if (result != SCAN_SUCCEED)
> > > goto out_nolock;
> > >
> > > mmap_read_lock(mm);
> > > - result = hugepage_vma_revalidate(mm, address, true, &vma, cc,
> > > - HPAGE_PMD_ORDER);
> > > + *mmap_locked = true;
> > > + result = hugepage_vma_revalidate(mm, pmd_address, true, &vma, cc, order);
> >
> > Why would we use the PMD address here rather than the actual start address?
>
> The revalidation relies on the pmd_addr not the start_addr. It (only)
> uses this to make sure the VMA is still at least PMD sized, and it
> uses the order to validate that the target order is allowed. I left a
> small comment about this in the revalidate function.
Yeah having these different addresses is a bit icky, easy to make mistakes here.
Oh how we need to refactor all of these...
>
> >
> > Also please add /*expect_anon=*/ before the 'true' because it's hard to
> > understand what that references.
>
> ack
>
> >
> > > if (result != SCAN_SUCCEED) {
> > > mmap_read_unlock(mm);
> > > + *mmap_locked = false;
> > > goto out_nolock;
> > > }
> > >
> > > - result = find_pmd_or_thp_or_none(mm, address, &pmd);
> > > + result = find_pmd_or_thp_or_none(mm, pmd_address, &pmd);
> > > if (result != SCAN_SUCCEED) {
> > > mmap_read_unlock(mm);
> > > + *mmap_locked = false;
> > > goto out_nolock;
> > > }
> > >
> > > @@ -1198,13 +1208,16 @@ static enum scan_result collapse_huge_page(struct mm_struct *mm, unsigned long a
> > > * released when it fails. So we jump out_nolock directly in
> > > * that case. Continuing to collapse causes inconsistency.
> > > */
> > > - result = __collapse_huge_page_swapin(mm, vma, address, pmd,
> > > - referenced, HPAGE_PMD_ORDER);
> > > - if (result != SCAN_SUCCEED)
> > > + result = __collapse_huge_page_swapin(mm, vma, start_addr, pmd,
> > > + referenced, order);
> > > + if (result != SCAN_SUCCEED) {
> > > + *mmap_locked = false;
> > > goto out_nolock;
> > > + }
> > > }
> > >
> > > mmap_read_unlock(mm);
> > > + *mmap_locked = false;
> > > /*
> > > * Prevent all access to pagetables with the exception of
> > > * gup_fast later handled by the ptep_clear_flush and the VM
> > > @@ -1214,20 +1227,20 @@ static enum scan_result collapse_huge_page(struct mm_struct *mm, unsigned long a
> > > * mmap_lock.
> > > */
> > > mmap_write_lock(mm);
> > > - result = hugepage_vma_revalidate(mm, address, true, &vma, cc,
> > > - HPAGE_PMD_ORDER);
> > > + result = hugepage_vma_revalidate(mm, pmd_address, true, &vma, cc, order);
> > > if (result != SCAN_SUCCEED)
> > > goto out_up_write;
> > > /* check if the pmd is still valid */
> > > vma_start_write(vma);
> > > - result = check_pmd_still_valid(mm, address, pmd);
> > > + result = check_pmd_still_valid(mm, pmd_address, pmd);
> > > if (result != SCAN_SUCCEED)
> > > goto out_up_write;
> > >
> > > anon_vma_lock_write(vma->anon_vma);
> > > + anon_vma_locked = true;
> > >
> > > - mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm, address,
> > > - address + HPAGE_PMD_SIZE);
> > > + mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm, start_addr,
> > > + start_addr + (PAGE_SIZE << order));
> > > mmu_notifier_invalidate_range_start(&range);
> > >
> > > pmd_ptl = pmd_lock(mm, pmd); /* probably unnecessary */
> > > @@ -1239,24 +1252,21 @@ static enum scan_result collapse_huge_page(struct mm_struct *mm, unsigned long a
> > > * Parallel GUP-fast is fine since GUP-fast will back off when
> > > * it detects PMD is changed.
> > > */
> > > - _pmd = pmdp_collapse_flush(vma, address, pmd);
> > > + _pmd = pmdp_collapse_flush(vma, pmd_address, pmd);
> > > spin_unlock(pmd_ptl);
> > > mmu_notifier_invalidate_range_end(&range);
> > > tlb_remove_table_sync_one();
> > >
> > > - pte = pte_offset_map_lock(mm, &_pmd, address, &pte_ptl);
> > > + pte = pte_offset_map_lock(mm, &_pmd, start_addr, &pte_ptl);
> > > if (pte) {
> > > - result = __collapse_huge_page_isolate(vma, address, pte, cc,
> > > - HPAGE_PMD_ORDER,
> > > - &compound_pagelist);
> > > + result = __collapse_huge_page_isolate(vma, start_addr, pte, cc,
> > > + order, &compound_pagelist);
> > > spin_unlock(pte_ptl);
> > > } else {
> > > result = SCAN_NO_PTE_TABLE;
> > > }
> > >
> > > if (unlikely(result != SCAN_SUCCEED)) {
> > > - if (pte)
> > > - pte_unmap(pte);
> > > spin_lock(pmd_ptl);
> > > BUG_ON(!pmd_none(*pmd));
> > > /*
> > > @@ -1266,21 +1276,21 @@ static enum scan_result collapse_huge_page(struct mm_struct *mm, unsigned long a
> > > */
> > > pmd_populate(mm, pmd, pmd_pgtable(_pmd));
> > > spin_unlock(pmd_ptl);
> > > - anon_vma_unlock_write(vma->anon_vma);
> > > goto out_up_write;
> > > }
> > >
> > > /*
> > > - * All pages are isolated and locked so anon_vma rmap
> > > - * can't run anymore.
> > > + * For PMD collapse all pages are isolated and locked so anon_vma
> > > + * rmap can't run anymore. For mTHP collapse we must hold the lock
> > > */
> > > - anon_vma_unlock_write(vma->anon_vma);
> > > + if (is_pmd_order(order)) {
> > > + anon_vma_unlock_write(vma->anon_vma);
> > > + anon_vma_locked = false;
> > > + }
> > >
> > > result = __collapse_huge_page_copy(pte, folio, pmd, _pmd,
> > > - vma, address, pte_ptl,
> > > - HPAGE_PMD_ORDER,
> > > - &compound_pagelist);
> > > - pte_unmap(pte);
> > > + vma, start_addr, pte_ptl,
> > > + order, &compound_pagelist);
> > > if (unlikely(result != SCAN_SUCCEED))
> > > goto out_up_write;
> > >
> > > @@ -1290,20 +1300,42 @@ static enum scan_result collapse_huge_page(struct mm_struct *mm, unsigned long a
> > > * write.
> > > */
> > > __folio_mark_uptodate(folio);
> > > - pgtable = pmd_pgtable(_pmd);
> > > + if (is_pmd_order(order)) { /* PMD collapse */
> > > + pgtable = pmd_pgtable(_pmd);
> > >
> > > - spin_lock(pmd_ptl);
> > > - BUG_ON(!pmd_none(*pmd));
> > > - pgtable_trans_huge_deposit(mm, pmd, pgtable);
> > > - map_anon_folio_pmd_nopf(folio, pmd, vma, address);
> > > + spin_lock(pmd_ptl);
> > > + WARN_ON_ONCE(!pmd_none(*pmd));
> > > + pgtable_trans_huge_deposit(mm, pmd, pgtable);
> > > + map_anon_folio_pmd_nopf(folio, pmd, vma, pmd_address);
> > > + } else { /* mTHP collapse */
> > > + pte_t mthp_pte = mk_pte(folio_page(folio, 0), vma->vm_page_prot);
> > > +
> > > + mthp_pte = maybe_mkwrite(pte_mkdirty(mthp_pte), vma);
> > > + spin_lock(pmd_ptl);
> > > + WARN_ON_ONCE(!pmd_none(*pmd));
> > > + folio_ref_add(folio, nr_pages - 1);
> > > + folio_add_new_anon_rmap(folio, vma, start_addr, RMAP_EXCLUSIVE);
> > > + folio_add_lru_vma(folio, vma);
> > > + set_ptes(vma->vm_mm, start_addr, pte, mthp_pte, nr_pages);
> > > + update_mmu_cache_range(NULL, vma, start_addr, pte, nr_pages);
> > > +
> > > + smp_wmb(); /* make PTEs visible before PMD. See pmd_install() */
> > > + pmd_populate(mm, pmd, pmd_pgtable(_pmd));
> >
> > I seriously hate this being open-coded, can we separate it out into another
> > function?
>
> Yeah I think we've discussed this before. I started to generalize
> this, and apply it to other parts of the kernel that maintain a
> similar pattern, but each potential user of the helper was slightly
> different in its approach and I was unable to find a quick solution to
> make it apply to all. I think it will require a lot more thought to
> cleanly refactor this. I figured I could leave this to the later
> cleanup work, or I could just create a static function just for
> khugepaged for now?
Yeah let's at least separate it out for this logic anyway.
Really we should have done the refactoring in advance of these changes, but that
ship has sailed :)
>
> >
> > > + }
> > > spin_unlock(pmd_ptl);
> > >
> > > folio = NULL;
> > >
> > > result = SCAN_SUCCEED;
> > > out_up_write:
> > > + if (anon_vma_locked)
> > > + anon_vma_unlock_write(vma->anon_vma);
> >
> > Thanks it's much better tracking this specifically.
> >
> > The whole damn thing needs refactoring (by this I mean - khugepaged and really
> > THP in general to be clear :) but it's not your fault.
>
> Yeah it has not been the prettiest code to try and understand/work on!
:)
>
> >
> > Could I ask though whether you might help out with some cleanups after this
> > lands :)
> >
> > I feel like we all need to do our bit to pay down some technical debt!
>
>
> Yes ofc! I had already planned on doing so. I have some in mind, and I
> believe others have already tackled some. After this land, let's
> discuss further plans (discussion thread or THP meeting).
Yeah, I'll get that TODO list discussed in meeting shared soon...
>
> Cheers,
> -- Nico
>
> >
> > > + if (pte)
> > > + pte_unmap(pte);
> > > mmap_write_unlock(mm);
> > > + *mmap_locked = false;
> > > out_nolock:
> > > + WARN_ON_ONCE(*mmap_locked);
> > > if (folio)
> > > folio_put(folio);
> > > trace_mm_collapse_huge_page(mm, result == SCAN_SUCCEED, result);
> > > @@ -1471,9 +1503,8 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
> > > pte_unmap_unlock(pte, ptl);
> > > if (result == SCAN_SUCCEED) {
> > > result = collapse_huge_page(mm, start_addr, referenced,
> > > - unmapped, cc);
> > > - /* collapse_huge_page will return with the mmap_lock released */
> > > - *mmap_locked = false;
> > > + unmapped, cc, mmap_locked,
> > > + HPAGE_PMD_ORDER);
> > > }
> > > out:
> > > trace_mm_khugepaged_scan_pmd(mm, folio, referenced,
> > > --
> > > 2.52.0
> > >
> >
> > Cheers, Lorenzo
> >
>
Cheers, Lorenzo
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox