* Re: [PATCH v3 4/5] perf script: Display PERF_RECORD_CALLCHAIN_DEFERRED
From: Ian Rogers @ 2025-11-14 18:18 UTC (permalink / raw)
To: Namhyung Kim
Cc: Arnaldo Carvalho de Melo, James Clark, Jiri Olsa, Adrian Hunter,
Peter Zijlstra, Ingo Molnar, LKML, linux-perf-users,
Steven Rostedt, Josh Poimboeuf, Indu Bhagat, Jens Remus,
Mathieu Desnoyers, linux-trace-kernel, bpf
In-Reply-To: <20251114070018.160330-5-namhyung@kernel.org>
On Thu, Nov 13, 2025 at 11:02 PM Namhyung Kim <namhyung@kernel.org> wrote:
>
> Handle the deferred callchains in the script output.
>
> $ perf script
> ...
> pwd 2312 121.163435: 249113 cpu/cycles/P:
> ffffffff845b78d8 __build_id_parse.isra.0+0x218 ([kernel.kallsyms])
> ffffffff83bb5bf6 perf_event_mmap+0x2e6 ([kernel.kallsyms])
> ffffffff83c31959 mprotect_fixup+0x1e9 ([kernel.kallsyms])
> ffffffff83c31dc5 do_mprotect_pkey+0x2b5 ([kernel.kallsyms])
> ffffffff83c3206f __x64_sys_mprotect+0x1f ([kernel.kallsyms])
> ffffffff845e6692 do_syscall_64+0x62 ([kernel.kallsyms])
> ffffffff8360012f entry_SYSCALL_64_after_hwframe+0x76 ([kernel.kallsyms])
> b00000006 [unknown] ([unknown])
Does this unknown value correspond to the cookie? Can the cookie and
deferred data be added to the kernel stack trace output?
> pwd 2312 121.163447: DEFERRED CALLCHAIN
> 7f18fe337fa7 mprotect+0x7 (/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
> 7f18fe330e0f _dl_sysdep_start+0x7f (/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
> 7f18fe331448 _dl_start_user+0x0 (/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
Can we display the cookie here so that the user callchain can be
matched with the kernel part?
Thanks,
Ian
> Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> ---
> tools/perf/builtin-script.c | 89 +++++++++++++++++++++++++++++++++++++
> 1 file changed, 89 insertions(+)
>
> diff --git a/tools/perf/builtin-script.c b/tools/perf/builtin-script.c
> index cf0040bbaba9cbc9..3b2896350bad2924 100644
> --- a/tools/perf/builtin-script.c
> +++ b/tools/perf/builtin-script.c
> @@ -2719,6 +2719,93 @@ static int process_sample_event(const struct perf_tool *tool,
> return ret;
> }
>
> +static int process_deferred_sample_event(const struct perf_tool *tool,
> + union perf_event *event,
> + struct perf_sample *sample,
> + struct evsel *evsel,
> + struct machine *machine)
> +{
> + struct perf_script *scr = container_of(tool, struct perf_script, tool);
> + struct perf_event_attr *attr = &evsel->core.attr;
> + struct evsel_script *es = evsel->priv;
> + unsigned int type = output_type(attr->type);
> + struct addr_location al;
> + FILE *fp = es->fp;
> + int ret = 0;
> +
> + if (output[type].fields == 0)
> + return 0;
> +
> + /* Set thread to NULL to indicate addr_al and al are not initialized */
> + addr_location__init(&al);
> +
> + if (perf_time__ranges_skip_sample(scr->ptime_range, scr->range_num,
> + sample->time)) {
> + goto out_put;
> + }
> +
> + if (debug_mode) {
> + if (sample->time < last_timestamp) {
> + pr_err("Samples misordered, previous: %" PRIu64
> + " this: %" PRIu64 "\n", last_timestamp,
> + sample->time);
> + nr_unordered++;
> + }
> + last_timestamp = sample->time;
> + goto out_put;
> + }
> +
> + if (filter_cpu(sample))
> + goto out_put;
> +
> + if (machine__resolve(machine, &al, sample) < 0) {
> + pr_err("problem processing %d event, skipping it.\n",
> + event->header.type);
> + ret = -1;
> + goto out_put;
> + }
> +
> + if (al.filtered)
> + goto out_put;
> +
> + if (!show_event(sample, evsel, al.thread, &al, NULL))
> + goto out_put;
> +
> + if (evswitch__discard(&scr->evswitch, evsel))
> + goto out_put;
> +
> + perf_sample__fprintf_start(scr, sample, al.thread, evsel,
> + PERF_RECORD_CALLCHAIN_DEFERRED, fp);
> + fprintf(fp, "DEFERRED CALLCHAIN");
> +
> + if (PRINT_FIELD(IP)) {
> + struct callchain_cursor *cursor = NULL;
> +
> + if (symbol_conf.use_callchain && sample->callchain) {
> + cursor = get_tls_callchain_cursor();
> + if (thread__resolve_callchain(al.thread, cursor, evsel,
> + sample, NULL, NULL,
> + scripting_max_stack)) {
> + pr_info("cannot resolve deferred callchains\n");
> + cursor = NULL;
> + }
> + }
> +
> + fputc(cursor ? '\n' : ' ', fp);
> + sample__fprintf_sym(sample, &al, 0, output[type].print_ip_opts,
> + cursor, symbol_conf.bt_stop_list, fp);
> + }
> +
> + fprintf(fp, "\n");
> +
> + if (verbose > 0)
> + fflush(fp);
> +
> +out_put:
> + addr_location__exit(&al);
> + return ret;
> +}
> +
> // Used when scr->per_event_dump is not set
> static struct evsel_script es_stdout;
>
> @@ -4320,6 +4407,7 @@ int cmd_script(int argc, const char **argv)
>
> perf_tool__init(&script.tool, !unsorted_dump);
> script.tool.sample = process_sample_event;
> + script.tool.callchain_deferred = process_deferred_sample_event;
> script.tool.mmap = perf_event__process_mmap;
> script.tool.mmap2 = perf_event__process_mmap2;
> script.tool.comm = perf_event__process_comm;
> @@ -4346,6 +4434,7 @@ int cmd_script(int argc, const char **argv)
> script.tool.throttle = process_throttle_event;
> script.tool.unthrottle = process_throttle_event;
> script.tool.ordering_requires_timestamps = true;
> + script.tool.merge_deferred_callchains = false;
> session = perf_session__new(&data, &script.tool);
> if (IS_ERR(session))
> return PTR_ERR(session);
> --
> 2.52.0.rc1.455.g30608eb744-goog
>
>
^ permalink raw reply
* Re: [PATCH v7 00/31] context_tracking,x86: Defer some IPIs until a user->kernel transition
From: Paul E. McKenney @ 2025-11-14 18:14 UTC (permalink / raw)
To: Andy Lutomirski
Cc: Valentin Schneider, Linux Kernel Mailing List, linux-mm, rcu,
the arch/x86 maintainers, linux-arm-kernel, loongarch,
linux-riscv, linux-arch, linux-trace-kernel, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, Dave Hansen, H. Peter Anvin,
Peter Zijlstra (Intel), Arnaldo Carvalho de Melo, Josh Poimboeuf,
Paolo Bonzini, Arnd Bergmann, Frederic Weisbecker, Jason Baron,
Steven Rostedt, Ard Biesheuvel, Sami Tolvanen, David S. Miller,
Neeraj Upadhyay, Joel Fernandes, Josh Triplett, Boqun Feng,
Uladzislau Rezki, Mathieu Desnoyers, Mel Gorman, Andrew Morton,
Masahiro Yamada, Han Shen, Rik van Riel, Jann Horn, Dan Carpenter,
Oleg Nesterov, Juri Lelli, Clark Williams, Yair Podemsky,
Marcelo Tosatti, Daniel Wagner, Petr Tesarik, Shrikanth Hegde
In-Reply-To: <beb46a02-b902-438b-8b6d-b5f0d7ad0ae6@app.fastmail.com>
On Fri, Nov 14, 2025 at 09:22:35AM -0800, Andy Lutomirski wrote:
>
>
> On Fri, Nov 14, 2025, at 8:20 AM, Andy Lutomirski wrote:
> > On Fri, Nov 14, 2025, at 7:01 AM, Valentin Schneider wrote:
> >> Context
> >> =======
> >>
> >> We've observed within Red Hat that isolated, NOHZ_FULL CPUs running a
> >> pure-userspace application get regularly interrupted by IPIs sent from
> >> housekeeping CPUs. Those IPIs are caused by activity on the housekeeping CPUs
> >> leading to various on_each_cpu() calls, e.g.:
> >>
> >
> >> The heart of this series is the thought that while we cannot remove NOHZ_FULL
> >> CPUs from the list of CPUs targeted by these IPIs, they may not have to execute
> >> the callbacks immediately. Anything that only affects kernelspace can wait
> >> until the next user->kernel transition, providing it can be executed "early
> >> enough" in the entry code.
> >>
> >
> > I want to point out that there's another option here, although anyone
> > trying to implement it would be fighting against quite a lot of history.
> >
> > Logically, each CPU is in one of a handful of states: user mode, idle,
> > normal kernel mode (possibly subdivided into IRQ, etc), and a handful
> > of very narrow windows, hopefully uninstrumented and not accessing any
> > PTEs that might be invalid, in the entry and exit paths where any state
> > in memory could be out of sync with actual CPU state. (The latter
> > includes right after the CPU switches to kernel mode, for example.)
> > And NMI and MCE and whatever weird "security" entry types that Intel
> > and AMD love to add.
> >
> > The way the kernel *currently* deals with this has two big historical oddities:
> >
> > 1. The entry and exit code cares about ti_flags, which is per-*task*,
> > which means that atomically poking it from other CPUs involves the
> > runqueue lock or other shenanigans (see the idle nr_polling code for
> > example), and also that it's not accessible from the user page tables
> > if PTI is on.
> >
> > 2. The actual heavyweight atomic part (context tracking) was built for
> > RCU, and it's sort or bolted on, and, as you've observed in this
> > series, it's really quite awkward to do things that aren't RCU using
> > context tracking.
> >
> > If this were a greenfield project, I think there's a straightforward
> > approach that's much nicer: stick everything into a single percpu flags
> > structure. Imagine we have cpu_flags, which tracks both the current
> > state of the CPU and what work needs to be done on state changes. On
> > exit to user mode, we would atomically set the mode to USER and make
> > sure we don't touch anything like vmalloc space after that. On entry
> > back to kernel mode, we would avoid vmalloc space, etc, then atomically
> > switch to kernel mode and read out whatever deferred work is needed.
> > As an optimization, if nothing in the current configuration needs
> > atomic state tracking, the state could be left at USER_OR_KERNEL and
> > the overhead of an extra atomic op at entry and exit could be avoided.
> >
> > And RCU would hook into *that* instead of having its own separate set of hooks.
Please note that RCU needs to sample a given CPU's idle state from other
CPUs, and to have pretty heavy-duty ordering guarantees. This is needed
to avoid RCU needing to wake up idle CPUs on the one hand or relying on
scheduling-clock interrupts waking up idle CPUs on the other.
Or am I missing the point of your suggestion?
> > I think that actually doing this would be a big improvement and would
> > also be a serious project. There's a lot of code that would get
> > touched, and the existing context tracking code is subtle and
> > confusing. And, as mentioned, ti_flags has the wrong scope.
Serious care would certainly be needed! ;-)
> > It's *possible* that one could avoid making ti_flags percpu either by
> > extensive use of the runqueue locks or by borrowing a kludge from the
> > idle code. For the latter, right now, the reason that the
> > wake-from-idle code works is that the optimized path only happens if
> > the idle thread/cpu is "polling", and it's impossible for the idle
> > ti_flags to be polling while the CPU isn't actually idle. We could
> > similarly observe that, if a ti_flags says it's in USER mode *and* is
> > on, say, cpu 3, then cpu 3 is most definitely in USER mode. So someone
> > could try shoving the CPU number into ti_flags :-p (USER means
> > actually user or in the late exit / early entry path.)
> >
> > Anyway, benefits of this whole approach would include considerably
> > (IMO) increased comprehensibility compared to the current tangled ct
> > code and much more straightforward addition of new things that happen
> > to a target CPU conditionally depending on its mode. And, if the flags
> > word was actually per cpu, it could be mapped such that
> > SWITCH_TO_KERNEL_CR3 would use it -- there could be a single CR3 write
> > (and maybe CR4/invpcid depending on whether a zapped mapping is global)
> > and the flush bit could depend on whether a flush is needed. And there
> > would be basically no chance that a bug that accessed
> > invalidated-but-not-flushed kernel data could be undetected -- in PTI
> > mode, any such access would page fault! Similarly, if kernel text
> > pokes deferred the flush and serialization, the only code that could
> > execute before noticing the deferred flush would be the user-CR3 code.
> >
> > Oh, any another primitive would be possible: one CPU could plausibly
> > execute another CPU's interrupts or soft-irqs or whatever by taking a
> > special lock that would effectively pin the remote CPU in user mode --
> > you'd set a flag in the target cpu_flags saying "pin in USER mode" and
> > the transition on that CPU to kernel mode would then spin on entry to
> > kernel mode and wait for the lock to be released. This could plausibly
> > get a lot of the on_each_cpu callers to switch over in one fell swoop:
> > anything that needs to synchronize to the remote CPU but does not need
> > to poke its actual architectural state could be executed locally while
> > the remote CPU is pinned.
It would be necessary to arrange for the remote CPU to remain pinned
while the local CPU executed on its behalf. Does the above approach
make that happen without re-introducing our current context-tracking
overhead and complexity?
> Following up, I think that x86 can do this all with a single atomic (in the common case) per usermode round trip. Imagine:
>
> struct fancy_cpu_state {
> u32 work; // <-- writable by any CPU
> u32 status; // <-- readable anywhere; writable locally
> };
>
> status includes KERNEL, USER, and maybe INDETERMINATE. (INDETERMINATE means USER but we're not committing to doing work.)
>
> Exit to user mode:
>
> atomic_set(&my_state->status, USER);
We need ordering in the RCU nohz_full case. If the grace-period kthread
sees the status as USER, all the preceding KERNEL code's effects must
be visible to the grace-period kthread.
> (or, in the lazy case, set to INDETERMINATE instead.)
>
> Entry from user mode, with IRQs off, before switching to kernel CR3:
>
> if (my_state->status == INDETERMINATE) {
> // we were lazy and we never promised to do work atomically.
> atomic_set(&my_state->status, KERNEL);
> this_entry_work = 0;
> } else {
> // we were not lazy and we promised we would do work atomically
> atomic exchange the entire state to { .work = 0, .status = KERNEL }
> this_entry_work = (whatever we just read);
> }
If this atomic exchange is fully ordered (as opposed to, say, _relaxed),
then this works in that if the grace-period kthread sees USER, its prior
references are guaranteed not to see later kernel-mode references from
that CPU.
> if (PTI) {
> switch to kernel CR3 *and flush if this_entry_work says to flush*
> } else {
> flush if this_entry_work says to flush;
> }
>
> do the rest of the work;
>
>
>
> I suppose that a lot of the stuff in ti_flags could merge into here, but it could be done one bit at a time when people feel like doing so. And I imagine, but I'm very far from confident, that RCU could use this instead of the current context tracking code.
RCU currently needs pretty heavy-duty ordering to reliably detect the
other CPUs' quiescent states without needing to wake them from idle, or,
in the nohz_full case, interrupt their userspace execution. Not saying
it is impossible, but it will need extreme care.
> The idea behind INDETERMINATE is that there are plenty of workloads that frequently switch between user and kernel mode and that would rather accept a few IPIs to avoid the heavyweight atomic operation on user -> kernel transitions. So the default behavior could be to do KERNEL -> INDETERMINATE instead of KERNEL -> USER, but code that wants to be in user mode for a long time could go all the way to USER. We could make it sort of automatic by noticing that we're returning from an IRQ without a context switch and go to USER (so we would get at most one unneeded IPI per normal user entry), and we could have some nice API for a program that intends to hang out in user mode for a very long time (cpu isolation users, for example) to tell the kernel to go immediately into USER mode. (Don't we already have something that could be used for this purpose?)
RCU *could* do an smp_call_function_single() when the CPU failed
to respond, perhaps in a manner similar to how it already forces a
given CPU out of nohz_full state if that CPU has been executing in the
kernel for too long. The real-time guys might not be amused, though.
Especially those real-time guys hitting sub-microsecond latencies.
> Hmm, now I wonder if it would make sense for the default behavior of Linux to be like that. We could call it ONEHZ. It's like NOHZ_FULL except that user threads that don't do syscalls get one single timer tick instead of many or none.
>
>
> Anyway, I think my proposal is pretty good *if* RCU could be made to use it -- the existing context tracking code is fairly expensive, and I don't think we want to invent a new context-tracking-like mechanism if we still need to do the existing thing.
If you build with CONFIG_NO_HZ_FULL=n, do you still get the heavyweight
operations when transitioning between kernel and user execution?
Thanx, Paul
^ permalink raw reply
* Re: [PATCH v3 3/5] perf record: Enable defer_callchain for user callchains
From: Ian Rogers @ 2025-11-14 18:12 UTC (permalink / raw)
To: Namhyung Kim
Cc: Arnaldo Carvalho de Melo, James Clark, Jiri Olsa, Adrian Hunter,
Peter Zijlstra, Ingo Molnar, LKML, linux-perf-users,
Steven Rostedt, Josh Poimboeuf, Indu Bhagat, Jens Remus,
Mathieu Desnoyers, linux-trace-kernel, bpf
In-Reply-To: <CAP-5=fU33sEARn0tc1hSMahBVCvs0cy+Cu-J6+BG0Cm-nuwKnA@mail.gmail.com>
On Fri, Nov 14, 2025 at 10:09 AM Ian Rogers <irogers@google.com> wrote:
>
> On Fri, Nov 14, 2025 at 9:59 AM Ian Rogers <irogers@google.com> wrote:
> >
> > On Thu, Nov 13, 2025 at 11:01 PM Namhyung Kim <namhyung@kernel.org> wrote:
> > >
> > > And add the missing feature detection logic to clear the flag on old
> > > kernels.
> > >
> > > $ perf record -g -vv true
> > > ...
> > > ------------------------------------------------------------
> > > perf_event_attr:
> > > type 0 (PERF_TYPE_HARDWARE)
> > > size 136
> > > config 0 (PERF_COUNT_HW_CPU_CYCLES)
> > > { sample_period, sample_freq } 4000
> > > sample_type IP|TID|TIME|CALLCHAIN|PERIOD
> > > read_format ID|LOST
> > > disabled 1
> > > inherit 1
> > > mmap 1
> > > comm 1
> > > freq 1
> > > enable_on_exec 1
> > > task 1
> > > sample_id_all 1
> > > mmap2 1
> > > comm_exec 1
> > > ksymbol 1
> > > bpf_event 1
> > > defer_callchain 1
> > > defer_output 1
> > > ------------------------------------------------------------
> > > sys_perf_event_open: pid 162755 cpu 0 group_fd -1 flags 0x8
> > > sys_perf_event_open failed, error -22
> > > switching off deferred callchain support
> > >
> > > Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> > > ---
> > > tools/perf/util/evsel.c | 24 ++++++++++++++++++++++++
> > > tools/perf/util/evsel.h | 1 +
> > > 2 files changed, 25 insertions(+)
> > >
> > > diff --git a/tools/perf/util/evsel.c b/tools/perf/util/evsel.c
> > > index 244b3e44d090d413..f5652d00b457d096 100644
> > > --- a/tools/perf/util/evsel.c
> > > +++ b/tools/perf/util/evsel.c
> > > @@ -1061,6 +1061,14 @@ static void __evsel__config_callchain(struct evsel *evsel, struct record_opts *o
> > > }
> > > }
> > >
> > > + if (param->record_mode == CALLCHAIN_FP && !attr->exclude_callchain_user) {
> > > + /*
> > > + * Enable deferred callchains optimistically. It'll be switched
> > > + * off later if the kernel doesn't support it.
> > > + */
> > > + attr->defer_callchain = 1;
> > > + }
> >
> > If a user has requested frame pointer call chains why would they want
> > deferred call chains? The point of deferral to my understanding is to
> > allow the paging in of debug data, but frame pointers don't need that
> > as the stack should be in the page cache.
> >
> > Is this being done for code coverage reasons so that deferral is known
> > to work for later addition of SFrames? In which case this should be an
> > opt-in not default behavior. When there is a record_mode of
> > CALLCHAIN_SFRAME then making deferral the default for that mode makes
> > sense, but not for frame pointers IMO.
>
> Just to be clear. I don't think the behavior of using frame pointers
> should change. Deferral has downsides, for example:
>
> $ perf record -g -a sleep 1
>
> Without deferral kernel stack traces will contain both kernel and user
> traces. With deferral the user stack trace is only generated when the
> system call returns and so there is a chance for kernel stack traces
> to be missing their user part. An obvious behavioral change. I think
> for what you are doing here we can have an option something like:
>
> $ perf record --call-graph fp-deferred -a sleep 1
>
> Which would need a man page update, etc. What is happening with the
> other call-graph modes and deferral? Could the option be something
> like `--call-graph fp,deferred` so that the option is a common one and
> say stack snapshots for dwarf be somehow improved?
Also, making deferral the norm will generate new perf events that
tools, other than perf, processing perf.data files will fail to
consume. So this change would break quite a lot of stuff, so it should
not just be made the default.
Thanks,
Ian
> Thanks,
> Ian
>
> > Thanks,
> > Ian
> >
> > > +
> > > if (function) {
> > > pr_info("Disabling user space callchains for function trace event.\n");
> > > attr->exclude_callchain_user = 1;
> > > @@ -1511,6 +1519,7 @@ void evsel__config(struct evsel *evsel, struct record_opts *opts,
> > > attr->mmap2 = track && !perf_missing_features.mmap2;
> > > attr->comm = track;
> > > attr->build_id = track && opts->build_id;
> > > + attr->defer_output = track;
> > >
> > > /*
> > > * ksymbol is tracked separately with text poke because it needs to be
> > > @@ -2199,6 +2208,10 @@ static int __evsel__prepare_open(struct evsel *evsel, struct perf_cpu_map *cpus,
> > >
> > > static void evsel__disable_missing_features(struct evsel *evsel)
> > > {
> > > + if (perf_missing_features.defer_callchain && evsel->core.attr.defer_callchain)
> > > + evsel->core.attr.defer_callchain = 0;
> > > + if (perf_missing_features.defer_callchain && evsel->core.attr.defer_output)
> > > + evsel->core.attr.defer_output = 0;
> > > if (perf_missing_features.inherit_sample_read && evsel->core.attr.inherit &&
> > > (evsel->core.attr.sample_type & PERF_SAMPLE_READ))
> > > evsel->core.attr.inherit = 0;
> > > @@ -2473,6 +2486,13 @@ static bool evsel__detect_missing_features(struct evsel *evsel, struct perf_cpu
> > >
> > > /* Please add new feature detection here. */
> > >
> > > + attr.defer_callchain = true;
> > > + if (has_attr_feature(&attr, /*flags=*/0))
> > > + goto found;
> > > + perf_missing_features.defer_callchain = true;
> > > + pr_debug2("switching off deferred callchain support\n");
> > > + attr.defer_callchain = false;
> > > +
> > > attr.inherit = true;
> > > attr.sample_type = PERF_SAMPLE_READ | PERF_SAMPLE_TID;
> > > if (has_attr_feature(&attr, /*flags=*/0))
> > > @@ -2584,6 +2604,10 @@ static bool evsel__detect_missing_features(struct evsel *evsel, struct perf_cpu
> > > errno = old_errno;
> > >
> > > check:
> > > + if ((evsel->core.attr.defer_callchain || evsel->core.attr.defer_output) &&
> > > + perf_missing_features.defer_callchain)
> > > + return true;
> > > +
> > > if (evsel->core.attr.inherit &&
> > > (evsel->core.attr.sample_type & PERF_SAMPLE_READ) &&
> > > perf_missing_features.inherit_sample_read)
> > > diff --git a/tools/perf/util/evsel.h b/tools/perf/util/evsel.h
> > > index 3ae4ac8f9a37e009..a08130ff2e47a887 100644
> > > --- a/tools/perf/util/evsel.h
> > > +++ b/tools/perf/util/evsel.h
> > > @@ -221,6 +221,7 @@ struct perf_missing_features {
> > > bool branch_counters;
> > > bool aux_action;
> > > bool inherit_sample_read;
> > > + bool defer_callchain;
> > > };
> > >
> > > extern struct perf_missing_features perf_missing_features;
> > > --
> > > 2.52.0.rc1.455.g30608eb744-goog
> > >
> > >
^ permalink raw reply
* Re: [PATCH v3 3/5] perf record: Enable defer_callchain for user callchains
From: Ian Rogers @ 2025-11-14 18:09 UTC (permalink / raw)
To: Namhyung Kim
Cc: Arnaldo Carvalho de Melo, James Clark, Jiri Olsa, Adrian Hunter,
Peter Zijlstra, Ingo Molnar, LKML, linux-perf-users,
Steven Rostedt, Josh Poimboeuf, Indu Bhagat, Jens Remus,
Mathieu Desnoyers, linux-trace-kernel, bpf
In-Reply-To: <CAP-5=fVEuYXw+P-+Z7bU7Z-+7dsHPPfABh5pdnPtfvH-23u4Qw@mail.gmail.com>
On Fri, Nov 14, 2025 at 9:59 AM Ian Rogers <irogers@google.com> wrote:
>
> On Thu, Nov 13, 2025 at 11:01 PM Namhyung Kim <namhyung@kernel.org> wrote:
> >
> > And add the missing feature detection logic to clear the flag on old
> > kernels.
> >
> > $ perf record -g -vv true
> > ...
> > ------------------------------------------------------------
> > perf_event_attr:
> > type 0 (PERF_TYPE_HARDWARE)
> > size 136
> > config 0 (PERF_COUNT_HW_CPU_CYCLES)
> > { sample_period, sample_freq } 4000
> > sample_type IP|TID|TIME|CALLCHAIN|PERIOD
> > read_format ID|LOST
> > disabled 1
> > inherit 1
> > mmap 1
> > comm 1
> > freq 1
> > enable_on_exec 1
> > task 1
> > sample_id_all 1
> > mmap2 1
> > comm_exec 1
> > ksymbol 1
> > bpf_event 1
> > defer_callchain 1
> > defer_output 1
> > ------------------------------------------------------------
> > sys_perf_event_open: pid 162755 cpu 0 group_fd -1 flags 0x8
> > sys_perf_event_open failed, error -22
> > switching off deferred callchain support
> >
> > Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> > ---
> > tools/perf/util/evsel.c | 24 ++++++++++++++++++++++++
> > tools/perf/util/evsel.h | 1 +
> > 2 files changed, 25 insertions(+)
> >
> > diff --git a/tools/perf/util/evsel.c b/tools/perf/util/evsel.c
> > index 244b3e44d090d413..f5652d00b457d096 100644
> > --- a/tools/perf/util/evsel.c
> > +++ b/tools/perf/util/evsel.c
> > @@ -1061,6 +1061,14 @@ static void __evsel__config_callchain(struct evsel *evsel, struct record_opts *o
> > }
> > }
> >
> > + if (param->record_mode == CALLCHAIN_FP && !attr->exclude_callchain_user) {
> > + /*
> > + * Enable deferred callchains optimistically. It'll be switched
> > + * off later if the kernel doesn't support it.
> > + */
> > + attr->defer_callchain = 1;
> > + }
>
> If a user has requested frame pointer call chains why would they want
> deferred call chains? The point of deferral to my understanding is to
> allow the paging in of debug data, but frame pointers don't need that
> as the stack should be in the page cache.
>
> Is this being done for code coverage reasons so that deferral is known
> to work for later addition of SFrames? In which case this should be an
> opt-in not default behavior. When there is a record_mode of
> CALLCHAIN_SFRAME then making deferral the default for that mode makes
> sense, but not for frame pointers IMO.
Just to be clear. I don't think the behavior of using frame pointers
should change. Deferral has downsides, for example:
$ perf record -g -a sleep 1
Without deferral kernel stack traces will contain both kernel and user
traces. With deferral the user stack trace is only generated when the
system call returns and so there is a chance for kernel stack traces
to be missing their user part. An obvious behavioral change. I think
for what you are doing here we can have an option something like:
$ perf record --call-graph fp-deferred -a sleep 1
Which would need a man page update, etc. What is happening with the
other call-graph modes and deferral? Could the option be something
like `--call-graph fp,deferred` so that the option is a common one and
say stack snapshots for dwarf be somehow improved?
Thanks,
Ian
> Thanks,
> Ian
>
> > +
> > if (function) {
> > pr_info("Disabling user space callchains for function trace event.\n");
> > attr->exclude_callchain_user = 1;
> > @@ -1511,6 +1519,7 @@ void evsel__config(struct evsel *evsel, struct record_opts *opts,
> > attr->mmap2 = track && !perf_missing_features.mmap2;
> > attr->comm = track;
> > attr->build_id = track && opts->build_id;
> > + attr->defer_output = track;
> >
> > /*
> > * ksymbol is tracked separately with text poke because it needs to be
> > @@ -2199,6 +2208,10 @@ static int __evsel__prepare_open(struct evsel *evsel, struct perf_cpu_map *cpus,
> >
> > static void evsel__disable_missing_features(struct evsel *evsel)
> > {
> > + if (perf_missing_features.defer_callchain && evsel->core.attr.defer_callchain)
> > + evsel->core.attr.defer_callchain = 0;
> > + if (perf_missing_features.defer_callchain && evsel->core.attr.defer_output)
> > + evsel->core.attr.defer_output = 0;
> > if (perf_missing_features.inherit_sample_read && evsel->core.attr.inherit &&
> > (evsel->core.attr.sample_type & PERF_SAMPLE_READ))
> > evsel->core.attr.inherit = 0;
> > @@ -2473,6 +2486,13 @@ static bool evsel__detect_missing_features(struct evsel *evsel, struct perf_cpu
> >
> > /* Please add new feature detection here. */
> >
> > + attr.defer_callchain = true;
> > + if (has_attr_feature(&attr, /*flags=*/0))
> > + goto found;
> > + perf_missing_features.defer_callchain = true;
> > + pr_debug2("switching off deferred callchain support\n");
> > + attr.defer_callchain = false;
> > +
> > attr.inherit = true;
> > attr.sample_type = PERF_SAMPLE_READ | PERF_SAMPLE_TID;
> > if (has_attr_feature(&attr, /*flags=*/0))
> > @@ -2584,6 +2604,10 @@ static bool evsel__detect_missing_features(struct evsel *evsel, struct perf_cpu
> > errno = old_errno;
> >
> > check:
> > + if ((evsel->core.attr.defer_callchain || evsel->core.attr.defer_output) &&
> > + perf_missing_features.defer_callchain)
> > + return true;
> > +
> > if (evsel->core.attr.inherit &&
> > (evsel->core.attr.sample_type & PERF_SAMPLE_READ) &&
> > perf_missing_features.inherit_sample_read)
> > diff --git a/tools/perf/util/evsel.h b/tools/perf/util/evsel.h
> > index 3ae4ac8f9a37e009..a08130ff2e47a887 100644
> > --- a/tools/perf/util/evsel.h
> > +++ b/tools/perf/util/evsel.h
> > @@ -221,6 +221,7 @@ struct perf_missing_features {
> > bool branch_counters;
> > bool aux_action;
> > bool inherit_sample_read;
> > + bool defer_callchain;
> > };
> >
> > extern struct perf_missing_features perf_missing_features;
> > --
> > 2.52.0.rc1.455.g30608eb744-goog
> >
> >
^ permalink raw reply
* Re: [PATCH v3 3/5] perf record: Enable defer_callchain for user callchains
From: Ian Rogers @ 2025-11-14 17:59 UTC (permalink / raw)
To: Namhyung Kim
Cc: Arnaldo Carvalho de Melo, James Clark, Jiri Olsa, Adrian Hunter,
Peter Zijlstra, Ingo Molnar, LKML, linux-perf-users,
Steven Rostedt, Josh Poimboeuf, Indu Bhagat, Jens Remus,
Mathieu Desnoyers, linux-trace-kernel, bpf
In-Reply-To: <20251114070018.160330-4-namhyung@kernel.org>
On Thu, Nov 13, 2025 at 11:01 PM Namhyung Kim <namhyung@kernel.org> wrote:
>
> And add the missing feature detection logic to clear the flag on old
> kernels.
>
> $ perf record -g -vv true
> ...
> ------------------------------------------------------------
> perf_event_attr:
> type 0 (PERF_TYPE_HARDWARE)
> size 136
> config 0 (PERF_COUNT_HW_CPU_CYCLES)
> { sample_period, sample_freq } 4000
> sample_type IP|TID|TIME|CALLCHAIN|PERIOD
> read_format ID|LOST
> disabled 1
> inherit 1
> mmap 1
> comm 1
> freq 1
> enable_on_exec 1
> task 1
> sample_id_all 1
> mmap2 1
> comm_exec 1
> ksymbol 1
> bpf_event 1
> defer_callchain 1
> defer_output 1
> ------------------------------------------------------------
> sys_perf_event_open: pid 162755 cpu 0 group_fd -1 flags 0x8
> sys_perf_event_open failed, error -22
> switching off deferred callchain support
>
> Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> ---
> tools/perf/util/evsel.c | 24 ++++++++++++++++++++++++
> tools/perf/util/evsel.h | 1 +
> 2 files changed, 25 insertions(+)
>
> diff --git a/tools/perf/util/evsel.c b/tools/perf/util/evsel.c
> index 244b3e44d090d413..f5652d00b457d096 100644
> --- a/tools/perf/util/evsel.c
> +++ b/tools/perf/util/evsel.c
> @@ -1061,6 +1061,14 @@ static void __evsel__config_callchain(struct evsel *evsel, struct record_opts *o
> }
> }
>
> + if (param->record_mode == CALLCHAIN_FP && !attr->exclude_callchain_user) {
> + /*
> + * Enable deferred callchains optimistically. It'll be switched
> + * off later if the kernel doesn't support it.
> + */
> + attr->defer_callchain = 1;
> + }
If a user has requested frame pointer call chains why would they want
deferred call chains? The point of deferral to my understanding is to
allow the paging in of debug data, but frame pointers don't need that
as the stack should be in the page cache.
Is this being done for code coverage reasons so that deferral is known
to work for later addition of SFrames? In which case this should be an
opt-in not default behavior. When there is a record_mode of
CALLCHAIN_SFRAME then making deferral the default for that mode makes
sense, but not for frame pointers IMO.
Thanks,
Ian
> +
> if (function) {
> pr_info("Disabling user space callchains for function trace event.\n");
> attr->exclude_callchain_user = 1;
> @@ -1511,6 +1519,7 @@ void evsel__config(struct evsel *evsel, struct record_opts *opts,
> attr->mmap2 = track && !perf_missing_features.mmap2;
> attr->comm = track;
> attr->build_id = track && opts->build_id;
> + attr->defer_output = track;
>
> /*
> * ksymbol is tracked separately with text poke because it needs to be
> @@ -2199,6 +2208,10 @@ static int __evsel__prepare_open(struct evsel *evsel, struct perf_cpu_map *cpus,
>
> static void evsel__disable_missing_features(struct evsel *evsel)
> {
> + if (perf_missing_features.defer_callchain && evsel->core.attr.defer_callchain)
> + evsel->core.attr.defer_callchain = 0;
> + if (perf_missing_features.defer_callchain && evsel->core.attr.defer_output)
> + evsel->core.attr.defer_output = 0;
> if (perf_missing_features.inherit_sample_read && evsel->core.attr.inherit &&
> (evsel->core.attr.sample_type & PERF_SAMPLE_READ))
> evsel->core.attr.inherit = 0;
> @@ -2473,6 +2486,13 @@ static bool evsel__detect_missing_features(struct evsel *evsel, struct perf_cpu
>
> /* Please add new feature detection here. */
>
> + attr.defer_callchain = true;
> + if (has_attr_feature(&attr, /*flags=*/0))
> + goto found;
> + perf_missing_features.defer_callchain = true;
> + pr_debug2("switching off deferred callchain support\n");
> + attr.defer_callchain = false;
> +
> attr.inherit = true;
> attr.sample_type = PERF_SAMPLE_READ | PERF_SAMPLE_TID;
> if (has_attr_feature(&attr, /*flags=*/0))
> @@ -2584,6 +2604,10 @@ static bool evsel__detect_missing_features(struct evsel *evsel, struct perf_cpu
> errno = old_errno;
>
> check:
> + if ((evsel->core.attr.defer_callchain || evsel->core.attr.defer_output) &&
> + perf_missing_features.defer_callchain)
> + return true;
> +
> if (evsel->core.attr.inherit &&
> (evsel->core.attr.sample_type & PERF_SAMPLE_READ) &&
> perf_missing_features.inherit_sample_read)
> diff --git a/tools/perf/util/evsel.h b/tools/perf/util/evsel.h
> index 3ae4ac8f9a37e009..a08130ff2e47a887 100644
> --- a/tools/perf/util/evsel.h
> +++ b/tools/perf/util/evsel.h
> @@ -221,6 +221,7 @@ struct perf_missing_features {
> bool branch_counters;
> bool aux_action;
> bool inherit_sample_read;
> + bool defer_callchain;
> };
>
> extern struct perf_missing_features perf_missing_features;
> --
> 2.52.0.rc1.455.g30608eb744-goog
>
>
^ permalink raw reply
* Re: [PATCH v3 2/5] perf tools: Minimal DEFERRED_CALLCHAIN support
From: Ian Rogers @ 2025-11-14 17:52 UTC (permalink / raw)
To: Namhyung Kim
Cc: Arnaldo Carvalho de Melo, James Clark, Jiri Olsa, Adrian Hunter,
Peter Zijlstra, Ingo Molnar, LKML, linux-perf-users,
Steven Rostedt, Josh Poimboeuf, Indu Bhagat, Jens Remus,
Mathieu Desnoyers, linux-trace-kernel, bpf
In-Reply-To: <20251114070018.160330-3-namhyung@kernel.org>
On Thu, Nov 13, 2025 at 11:00 PM Namhyung Kim <namhyung@kernel.org> wrote:
>
> Add a new event type for deferred callchains and a new callback for the
> struct perf_tool. For now it doesn't actually handle the deferred
> callchains but it just marks the sample if it has the PERF_CONTEXT_
> USER_DEFFERED in the callchain array.
>
> At least, perf report can dump the raw data with this change. Actually
> this requires the next commit to enable attr.defer_callchain, but if you
> already have a data file, it'll show the following result.
>
> $ perf report -D
> ...
> 0x2158@perf.data [0x40]: event: 22
> .
> . ... raw event: size 64 bytes
> . 0000: 16 00 00 00 02 00 40 00 06 00 00 00 0b 00 00 00 ......@.........
> . 0010: 03 00 00 00 00 00 00 00 a7 7f 33 fe 18 7f 00 00 ..........3.....
> . 0020: 0f 0e 33 fe 18 7f 00 00 48 14 33 fe 18 7f 00 00 ..3.....H.3.....
> . 0030: 08 09 00 00 08 09 00 00 e6 7a e7 35 1c 00 00 00 .........z.5....
>
> 121163447014 0x2158 [0x40]: PERF_RECORD_CALLCHAIN_DEFERRED(IP, 0x2): 2312/2312: 0xb00000006
> ... FP chain: nr:3
> ..... 0: 00007f18fe337fa7
> ..... 1: 00007f18fe330e0f
> ..... 2: 00007f18fe331448
> : unhandled!
>
> Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> ---
> tools/lib/perf/include/perf/event.h | 8 ++++++++
> tools/perf/util/event.c | 1 +
> tools/perf/util/evsel.c | 19 +++++++++++++++++++
> tools/perf/util/machine.c | 1 +
> tools/perf/util/perf_event_attr_fprintf.c | 2 ++
> tools/perf/util/sample.h | 2 ++
> tools/perf/util/session.c | 20 ++++++++++++++++++++
> tools/perf/util/tool.c | 1 +
> tools/perf/util/tool.h | 3 ++-
> 9 files changed, 56 insertions(+), 1 deletion(-)
>
> diff --git a/tools/lib/perf/include/perf/event.h b/tools/lib/perf/include/perf/event.h
> index aa1e91c97a226e1a..769bc48ca85c0eb8 100644
> --- a/tools/lib/perf/include/perf/event.h
> +++ b/tools/lib/perf/include/perf/event.h
> @@ -151,6 +151,13 @@ struct perf_record_switch {
> __u32 next_prev_tid;
> };
>
> +struct perf_record_callchain_deferred {
> + struct perf_event_header header;
> + __u64 cookie;
Could we add a comment that this value is used to match user and
kernel stack traces together? I don't believe that intent is
immediately obvious from the word "cookie".
> + __u64 nr;
> + __u64 ips[];
> +};
> +
> struct perf_record_header_attr {
> struct perf_event_header header;
> struct perf_event_attr attr;
> @@ -523,6 +530,7 @@ union perf_event {
> struct perf_record_read read;
> struct perf_record_throttle throttle;
> struct perf_record_sample sample;
> + struct perf_record_callchain_deferred callchain_deferred;
> struct perf_record_bpf_event bpf;
> struct perf_record_ksymbol ksymbol;
> struct perf_record_text_poke_event text_poke;
> diff --git a/tools/perf/util/event.c b/tools/perf/util/event.c
> index fcf44149feb20c35..4c92cc1a952c1d9f 100644
> --- a/tools/perf/util/event.c
> +++ b/tools/perf/util/event.c
> @@ -61,6 +61,7 @@ static const char *perf_event__names[] = {
> [PERF_RECORD_CGROUP] = "CGROUP",
> [PERF_RECORD_TEXT_POKE] = "TEXT_POKE",
> [PERF_RECORD_AUX_OUTPUT_HW_ID] = "AUX_OUTPUT_HW_ID",
> + [PERF_RECORD_CALLCHAIN_DEFERRED] = "CALLCHAIN_DEFERRED",
> [PERF_RECORD_HEADER_ATTR] = "ATTR",
> [PERF_RECORD_HEADER_EVENT_TYPE] = "EVENT_TYPE",
> [PERF_RECORD_HEADER_TRACING_DATA] = "TRACING_DATA",
> diff --git a/tools/perf/util/evsel.c b/tools/perf/util/evsel.c
> index 989c56d4a23f74f4..244b3e44d090d413 100644
> --- a/tools/perf/util/evsel.c
> +++ b/tools/perf/util/evsel.c
> @@ -3089,6 +3089,20 @@ int evsel__parse_sample(struct evsel *evsel, union perf_event *event,
> data->data_src = PERF_MEM_DATA_SRC_NONE;
> data->vcpu = -1;
>
> + if (event->header.type == PERF_RECORD_CALLCHAIN_DEFERRED) {
> + const u64 max_callchain_nr = UINT64_MAX / sizeof(u64);
> +
> + data->callchain = (struct ip_callchain *)&event->callchain_deferred.nr;
> + if (data->callchain->nr > max_callchain_nr)
> + return -EFAULT;
> +
> + data->deferred_cookie = event->callchain_deferred.cookie;
> +
> + if (evsel->core.attr.sample_id_all)
> + perf_evsel__parse_id_sample(evsel, event, data);
> + return 0;
> + }
> +
> if (event->header.type != PERF_RECORD_SAMPLE) {
> if (!evsel->core.attr.sample_id_all)
> return 0;
> @@ -3219,6 +3233,11 @@ int evsel__parse_sample(struct evsel *evsel, union perf_event *event,
> if (data->callchain->nr > max_callchain_nr)
> return -EFAULT;
> sz = data->callchain->nr * sizeof(u64);
> + if (evsel->core.attr.defer_callchain && data->callchain->nr >= 2 &&
> + data->callchain->ips[data->callchain->nr - 2] == PERF_CONTEXT_USER_DEFERRED) {
> + data->deferred_cookie = data->callchain->ips[data->callchain->nr - 1];
> + data->deferred_callchain = true;
> + }
It'd be nice to have a comment saying what is going on here. I can see
that if there are 2 stack slots and the 2nd is a magic value then the
first should be read as the "cookie". At a first look this code is
difficult to parse so a comment would add value.
Thanks,
Ian
> OVERFLOW_CHECK(array, sz, max_size);
> array = (void *)array + sz;
> }
> diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c
> index b5dd42588c916d91..841b711d970e9457 100644
> --- a/tools/perf/util/machine.c
> +++ b/tools/perf/util/machine.c
> @@ -2124,6 +2124,7 @@ static int add_callchain_ip(struct thread *thread,
> *cpumode = PERF_RECORD_MISC_KERNEL;
> break;
> case PERF_CONTEXT_USER:
> + case PERF_CONTEXT_USER_DEFERRED:
> *cpumode = PERF_RECORD_MISC_USER;
> break;
> default:
> diff --git a/tools/perf/util/perf_event_attr_fprintf.c b/tools/perf/util/perf_event_attr_fprintf.c
> index 66b666d9ce649dd7..741c3d657a8b6ae7 100644
> --- a/tools/perf/util/perf_event_attr_fprintf.c
> +++ b/tools/perf/util/perf_event_attr_fprintf.c
> @@ -343,6 +343,8 @@ int perf_event_attr__fprintf(FILE *fp, struct perf_event_attr *attr,
> PRINT_ATTRf(inherit_thread, p_unsigned);
> PRINT_ATTRf(remove_on_exec, p_unsigned);
> PRINT_ATTRf(sigtrap, p_unsigned);
> + PRINT_ATTRf(defer_callchain, p_unsigned);
> + PRINT_ATTRf(defer_output, p_unsigned);
>
> PRINT_ATTRn("{ wakeup_events, wakeup_watermark }", wakeup_events, p_unsigned, false);
> PRINT_ATTRf(bp_type, p_unsigned);
> diff --git a/tools/perf/util/sample.h b/tools/perf/util/sample.h
> index fae834144ef42105..a8307b20a9ea8066 100644
> --- a/tools/perf/util/sample.h
> +++ b/tools/perf/util/sample.h
> @@ -107,6 +107,8 @@ struct perf_sample {
> /** @weight3: On x86 holds retire_lat, on powerpc holds p_stage_cyc. */
> u16 weight3;
> bool no_hw_idx; /* No hw_idx collected in branch_stack */
> + bool deferred_callchain; /* Has deferred user callchains */
> + u64 deferred_cookie;
> char insn[MAX_INSN];
> void *raw_data;
> struct ip_callchain *callchain;
> diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
> index 4b0236b2df2913e1..361e15c1f26a96d0 100644
> --- a/tools/perf/util/session.c
> +++ b/tools/perf/util/session.c
> @@ -720,6 +720,7 @@ static perf_event__swap_op perf_event__swap_ops[] = {
> [PERF_RECORD_CGROUP] = perf_event__cgroup_swap,
> [PERF_RECORD_TEXT_POKE] = perf_event__text_poke_swap,
> [PERF_RECORD_AUX_OUTPUT_HW_ID] = perf_event__all64_swap,
> + [PERF_RECORD_CALLCHAIN_DEFERRED] = perf_event__all64_swap,
> [PERF_RECORD_HEADER_ATTR] = perf_event__hdr_attr_swap,
> [PERF_RECORD_HEADER_EVENT_TYPE] = perf_event__event_type_swap,
> [PERF_RECORD_HEADER_TRACING_DATA] = perf_event__tracing_data_swap,
> @@ -854,6 +855,9 @@ static void callchain__printf(struct evsel *evsel,
> for (i = 0; i < callchain->nr; i++)
> printf("..... %2d: %016" PRIx64 "\n",
> i, callchain->ips[i]);
> +
> + if (sample->deferred_callchain)
> + printf("...... (deferred)\n");
> }
>
> static void branch_stack__printf(struct perf_sample *sample,
> @@ -1123,6 +1127,19 @@ static void dump_sample(struct evsel *evsel, union perf_event *event,
> sample_read__printf(sample, evsel->core.attr.read_format);
> }
>
> +static void dump_deferred_callchain(struct evsel *evsel, union perf_event *event,
> + struct perf_sample *sample)
> +{
> + if (!dump_trace)
> + return;
> +
> + printf("(IP, 0x%x): %d/%d: %#" PRIx64 "\n",
> + event->header.misc, sample->pid, sample->tid, sample->deferred_cookie);
> +
> + if (evsel__has_callchain(evsel))
> + callchain__printf(evsel, sample);
> +}
> +
> static void dump_read(struct evsel *evsel, union perf_event *event)
> {
> struct perf_record_read *read_event = &event->read;
> @@ -1353,6 +1370,9 @@ static int machines__deliver_event(struct machines *machines,
> return tool->text_poke(tool, event, sample, machine);
> case PERF_RECORD_AUX_OUTPUT_HW_ID:
> return tool->aux_output_hw_id(tool, event, sample, machine);
> + case PERF_RECORD_CALLCHAIN_DEFERRED:
> + dump_deferred_callchain(evsel, event, sample);
> + return tool->callchain_deferred(tool, event, sample, evsel, machine);
> default:
> ++evlist->stats.nr_unknown_events;
> return -1;
> diff --git a/tools/perf/util/tool.c b/tools/perf/util/tool.c
> index 22a8a4ffe05f778e..f732d33e7f895ed4 100644
> --- a/tools/perf/util/tool.c
> +++ b/tools/perf/util/tool.c
> @@ -287,6 +287,7 @@ void perf_tool__init(struct perf_tool *tool, bool ordered_events)
> tool->read = process_event_sample_stub;
> tool->throttle = process_event_stub;
> tool->unthrottle = process_event_stub;
> + tool->callchain_deferred = process_event_sample_stub;
> tool->attr = process_event_synth_attr_stub;
> tool->event_update = process_event_synth_event_update_stub;
> tool->tracing_data = process_event_synth_tracing_data_stub;
> diff --git a/tools/perf/util/tool.h b/tools/perf/util/tool.h
> index 88337cee1e3e2be3..9b9f0a8cbf3de4b5 100644
> --- a/tools/perf/util/tool.h
> +++ b/tools/perf/util/tool.h
> @@ -44,7 +44,8 @@ enum show_feature_header {
>
> struct perf_tool {
> event_sample sample,
> - read;
> + read,
> + callchain_deferred;
> event_op mmap,
> mmap2,
> comm,
> --
> 2.52.0.rc1.455.g30608eb744-goog
>
^ permalink raw reply
* Re: [PATCH v7 00/31] context_tracking,x86: Defer some IPIs until a user->kernel transition
From: Andy Lutomirski @ 2025-11-14 17:22 UTC (permalink / raw)
To: Valentin Schneider, Linux Kernel Mailing List, linux-mm, rcu,
the arch/x86 maintainers, linux-arm-kernel, loongarch,
linux-riscv, linux-arch, linux-trace-kernel
Cc: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen,
H. Peter Anvin, Peter Zijlstra (Intel), Arnaldo Carvalho de Melo,
Josh Poimboeuf, Paolo Bonzini, Arnd Bergmann, Frederic Weisbecker,
Paul E. McKenney, Jason Baron, Steven Rostedt, Ard Biesheuvel,
Sami Tolvanen, David S. Miller, Neeraj Upadhyay, Joel Fernandes,
Josh Triplett, Boqun Feng, Uladzislau Rezki, Mathieu Desnoyers,
Mel Gorman, Andrew Morton, Masahiro Yamada, Han Shen,
Rik van Riel, Jann Horn, Dan Carpenter, Oleg Nesterov, Juri Lelli,
Clark Williams, Yair Podemsky, Marcelo Tosatti, Daniel Wagner,
Petr Tesarik, Shrikanth Hegde
In-Reply-To: <c4768cf3-f131-44e6-9b25-ebeb633f32ee@app.fastmail.com>
On Fri, Nov 14, 2025, at 8:20 AM, Andy Lutomirski wrote:
> On Fri, Nov 14, 2025, at 7:01 AM, Valentin Schneider wrote:
>> Context
>> =======
>>
>> We've observed within Red Hat that isolated, NOHZ_FULL CPUs running a
>> pure-userspace application get regularly interrupted by IPIs sent from
>> housekeeping CPUs. Those IPIs are caused by activity on the housekeeping CPUs
>> leading to various on_each_cpu() calls, e.g.:
>>
>
>> The heart of this series is the thought that while we cannot remove NOHZ_FULL
>> CPUs from the list of CPUs targeted by these IPIs, they may not have to execute
>> the callbacks immediately. Anything that only affects kernelspace can wait
>> until the next user->kernel transition, providing it can be executed "early
>> enough" in the entry code.
>>
>
> I want to point out that there's another option here, although anyone
> trying to implement it would be fighting against quite a lot of history.
>
> Logically, each CPU is in one of a handful of states: user mode, idle,
> normal kernel mode (possibly subdivided into IRQ, etc), and a handful
> of very narrow windows, hopefully uninstrumented and not accessing any
> PTEs that might be invalid, in the entry and exit paths where any state
> in memory could be out of sync with actual CPU state. (The latter
> includes right after the CPU switches to kernel mode, for example.)
> And NMI and MCE and whatever weird "security" entry types that Intel
> and AMD love to add.
>
> The way the kernel *currently* deals with this has two big historical oddities:
>
> 1. The entry and exit code cares about ti_flags, which is per-*task*,
> which means that atomically poking it from other CPUs involves the
> runqueue lock or other shenanigans (see the idle nr_polling code for
> example), and also that it's not accessible from the user page tables
> if PTI is on.
>
> 2. The actual heavyweight atomic part (context tracking) was built for
> RCU, and it's sort or bolted on, and, as you've observed in this
> series, it's really quite awkward to do things that aren't RCU using
> context tracking.
>
> If this were a greenfield project, I think there's a straightforward
> approach that's much nicer: stick everything into a single percpu flags
> structure. Imagine we have cpu_flags, which tracks both the current
> state of the CPU and what work needs to be done on state changes. On
> exit to user mode, we would atomically set the mode to USER and make
> sure we don't touch anything like vmalloc space after that. On entry
> back to kernel mode, we would avoid vmalloc space, etc, then atomically
> switch to kernel mode and read out whatever deferred work is needed.
> As an optimization, if nothing in the current configuration needs
> atomic state tracking, the state could be left at USER_OR_KERNEL and
> the overhead of an extra atomic op at entry and exit could be avoided.
>
> And RCU would hook into *that* instead of having its own separate set of hooks.
>
> I think that actually doing this would be a big improvement and would
> also be a serious project. There's a lot of code that would get
> touched, and the existing context tracking code is subtle and
> confusing. And, as mentioned, ti_flags has the wrong scope.
>
> It's *possible* that one could avoid making ti_flags percpu either by
> extensive use of the runqueue locks or by borrowing a kludge from the
> idle code. For the latter, right now, the reason that the
> wake-from-idle code works is that the optimized path only happens if
> the idle thread/cpu is "polling", and it's impossible for the idle
> ti_flags to be polling while the CPU isn't actually idle. We could
> similarly observe that, if a ti_flags says it's in USER mode *and* is
> on, say, cpu 3, then cpu 3 is most definitely in USER mode. So someone
> could try shoving the CPU number into ti_flags :-p (USER means
> actually user or in the late exit / early entry path.)
>
> Anyway, benefits of this whole approach would include considerably
> (IMO) increased comprehensibility compared to the current tangled ct
> code and much more straightforward addition of new things that happen
> to a target CPU conditionally depending on its mode. And, if the flags
> word was actually per cpu, it could be mapped such that
> SWITCH_TO_KERNEL_CR3 would use it -- there could be a single CR3 write
> (and maybe CR4/invpcid depending on whether a zapped mapping is global)
> and the flush bit could depend on whether a flush is needed. And there
> would be basically no chance that a bug that accessed
> invalidated-but-not-flushed kernel data could be undetected -- in PTI
> mode, any such access would page fault! Similarly, if kernel text
> pokes deferred the flush and serialization, the only code that could
> execute before noticing the deferred flush would be the user-CR3 code.
>
> Oh, any another primitive would be possible: one CPU could plausibly
> execute another CPU's interrupts or soft-irqs or whatever by taking a
> special lock that would effectively pin the remote CPU in user mode --
> you'd set a flag in the target cpu_flags saying "pin in USER mode" and
> the transition on that CPU to kernel mode would then spin on entry to
> kernel mode and wait for the lock to be released. This could plausibly
> get a lot of the on_each_cpu callers to switch over in one fell swoop:
> anything that needs to synchronize to the remote CPU but does not need
> to poke its actual architectural state could be executed locally while
> the remote CPU is pinned.
Following up, I think that x86 can do this all with a single atomic (in the common case) per usermode round trip. Imagine:
struct fancy_cpu_state {
u32 work; // <-- writable by any CPU
u32 status; // <-- readable anywhere; writable locally
};
status includes KERNEL, USER, and maybe INDETERMINATE. (INDETERMINATE means USER but we're not committing to doing work.)
Exit to user mode:
atomic_set(&my_state->status, USER);
(or, in the lazy case, set to INDETERMINATE instead.)
Entry from user mode, with IRQs off, before switching to kernel CR3:
if (my_state->status == INDETERMINATE) {
// we were lazy and we never promised to do work atomically.
atomic_set(&my_state->status, KERNEL);
this_entry_work = 0;
} else {
// we were not lazy and we promised we would do work atomically
atomic exchange the entire state to { .work = 0, .status = KERNEL }
this_entry_work = (whatever we just read);
}
if (PTI) {
switch to kernel CR3 *and flush if this_entry_work says to flush*
} else {
flush if this_entry_work says to flush;
}
do the rest of the work;
I suppose that a lot of the stuff in ti_flags could merge into here, but it could be done one bit at a time when people feel like doing so. And I imagine, but I'm very far from confident, that RCU could use this instead of the current context tracking code.
The idea behind INDETERMINATE is that there are plenty of workloads that frequently switch between user and kernel mode and that would rather accept a few IPIs to avoid the heavyweight atomic operation on user -> kernel transitions. So the default behavior could be to do KERNEL -> INDETERMINATE instead of KERNEL -> USER, but code that wants to be in user mode for a long time could go all the way to USER. We could make it sort of automatic by noticing that we're returning from an IRQ without a context switch and go to USER (so we would get at most one unneeded IPI per normal user entry), and we could have some nice API for a program that intends to hang out in user mode for a very long time (cpu isolation users, for example) to tell the kernel to go immediately into USER mode. (Don't we already have something that could be used for this purpose?)
Hmm, now I wonder if it would make sense for the default behavior of Linux to be like that. We could call it ONEHZ. It's like NOHZ_FULL except that user threads that don't do syscalls get one single timer tick instead of many or none.
Anyway, I think my proposal is pretty good *if* RCU could be made to use it -- the existing context tracking code is fairly expensive, and I don't think we want to invent a new context-tracking-like mechanism if we still need to do the existing thing.
--Andy
^ permalink raw reply
* [PATCH] unwind: Show that entries of struct unwind_cache is not bound by nr_entries
From: Steven Rostedt @ 2025-11-14 17:13 UTC (permalink / raw)
To: LKML, Linux Trace Kernel
Cc: Thorsten Blum, Josh Poimboeuf, Peter Zijlstra, Kees Cook,
Gustavo A. R. Silva, David Laight
From: Steven Rostedt <rostedt@goodmis.org>
The structure unwind_cache has:
struct unwind_cache {
unsigned long unwind_completed;
unsigned int nr_entries;
unsigned long entries[];
};
Which triggers lots of scripts to convert this to:
struct unwind_cache {
unsigned long unwind_completed;
unsigned int nr_entries;
unsigned long entries[] __counted_by(nr_entries);
};
But that is incorrect. The structure is created via:
#define UNWIND_MAX_ENTRIES \
((SZ_4K - sizeof(struct unwind_cache)) / sizeof(long))
info->cache = kzalloc(struct_size(cache, entries, UNWIND_MAX_ENTRIES), GFP_KERNEL);
Where the size of entries is determined by the size of the rest of the
structure subtracted from 4K. But because the size of entries has a
dependency on the structure itself, it can't be used to define it.
The entries are filled by another function that returns how many entries it
added and that is what nr_entries gets set to. This would most definitely
trigger a false-positive out-of-bounds bug if the __counted_by() was added.
To stop scripts from thinking this needs a counted_by(), move the
UNWIND_MAX_ENTRIES macro to the header, and add a comment in the entries
size:
unsigned long entries[ /* UNWIND_MAX_ENTRIES */ ];
Link: https://lore.kernel.org/all/20251114122748.222833-1-thorsten.blum@linux.dev/
Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
---
include/linux/unwind_deferred_types.h | 6 +++++-
kernel/unwind/deferred.c | 4 ----
2 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/include/linux/unwind_deferred_types.h b/include/linux/unwind_deferred_types.h
index 33b62ac25c86..61496397b0f9 100644
--- a/include/linux/unwind_deferred_types.h
+++ b/include/linux/unwind_deferred_types.h
@@ -2,10 +2,14 @@
#ifndef _LINUX_UNWIND_USER_DEFERRED_TYPES_H
#define _LINUX_UNWIND_USER_DEFERRED_TYPES_H
+/* Make the cache fit in a 4K page */
+#define UNWIND_MAX_ENTRIES \
+ ((SZ_4K - sizeof(struct unwind_cache)) / sizeof(long))
+
struct unwind_cache {
unsigned long unwind_completed;
unsigned int nr_entries;
- unsigned long entries[];
+ unsigned long entries[ /* UNWIND_MAX_ENTRIES */ ];
};
/*
diff --git a/kernel/unwind/deferred.c b/kernel/unwind/deferred.c
index dc6040aae3ee..3284bec6d04b 100644
--- a/kernel/unwind/deferred.c
+++ b/kernel/unwind/deferred.c
@@ -37,10 +37,6 @@ static inline bool try_assign_cnt(struct unwind_task_info *info, u32 cnt)
}
#endif
-/* Make the cache fit in a 4K page */
-#define UNWIND_MAX_ENTRIES \
- ((SZ_4K - sizeof(struct unwind_cache)) / sizeof(long))
-
/* Guards adding to or removing from the list of callbacks */
static DEFINE_MUTEX(callback_mutex);
static LIST_HEAD(callbacks);
--
2.51.0
^ permalink raw reply related
* Re: [PATCH RFC bpf-next 2/7] x86/ftrace: implement DYNAMIC_FTRACE_WITH_JMP
From: Steven Rostedt @ 2025-11-14 16:39 UTC (permalink / raw)
To: Menglong Dong
Cc: ast, daniel, john.fastabend, andrii, martin.lau, eddyz87, song,
yonghong.song, kpsingh, sdf, haoluo, jolsa, mhiramat,
mark.rutland, mathieu.desnoyers, bpf, linux-kernel,
linux-trace-kernel
In-Reply-To: <20251114092450.172024-3-dongml2@chinatelecom.cn>
On Fri, 14 Nov 2025 17:24:45 +0800
Menglong Dong <menglong8.dong@gmail.com> wrote:
> --- a/arch/x86/kernel/ftrace_64.S
> +++ b/arch/x86/kernel/ftrace_64.S
> @@ -285,8 +285,18 @@ SYM_INNER_LABEL(ftrace_regs_caller_end, SYM_L_GLOBAL)
> ANNOTATE_NOENDBR
> RET
>
> +1:
> + testb $1, %al
> + jz 2f
> + andq $0xfffffffffffffffe, %rax
> + movq %rax, MCOUNT_REG_SIZE+8(%rsp)
> + restore_mcount_regs
> + /* Restore flags */
> + popfq
> + RET
> +
> /* Swap the flags with orig_rax */
> -1: movq MCOUNT_REG_SIZE(%rsp), %rdi
> +2: movq MCOUNT_REG_SIZE(%rsp), %rdi
> movq %rdi, MCOUNT_REG_SIZE-8(%rsp)
> movq %rax, MCOUNT_REG_SIZE(%rsp)
>
So in this case we have:
original_caller:
call foo -> foo:
call fentry -> fentry:
[do ftrace callbacks ]
move tramp_addr to stack
RET -> tramp_addr
tramp_addr:
[..]
call foo_body -> foo_body:
[..]
RET -> back to tramp_addr
[..]
RET -> back to original_caller
I guess that looks balanced.
-- Steve
^ permalink raw reply
* Re: [PATCH RFC bpf-next 0/7] bpf trampoline support "jmp" mode
From: Steven Rostedt @ 2025-11-14 16:28 UTC (permalink / raw)
To: Menglong Dong
Cc: Menglong Dong, ast, daniel, john.fastabend, andrii, martin.lau,
eddyz87, song, yonghong.song, kpsingh, sdf, haoluo, jolsa,
mhiramat, mark.rutland, mathieu.desnoyers, bpf, linux-kernel,
linux-trace-kernel
In-Reply-To: <117548898.nniJfEyVGO@7950hx>
On Fri, 14 Nov 2025 21:58:34 +0800
Menglong Dong <menglong.dong@linux.dev> wrote:
> On 2025/11/14 21:38, Steven Rostedt wrote:
> > On Fri, 14 Nov 2025 17:24:43 +0800
> > Menglong Dong <menglong8.dong@gmail.com> wrote:
> >
> > > Therefore, we introduce the "jmp" mode for bpf trampoline, as advised by
> > > Alexei in [1]. And the logic will become this:
> > > call foo -> jmp trampoline -> call foo-body ->
> > > return foo-body -> return foo
> >
> > This obviously only works when there's a single function used by that
> > trampoline. It also doesn't allow tracing of the return side (it's
> > basically just the function tracer for a single function).
>
> Hi, Steven. I think you misunderstand something? For the fentry/fexit,
> the whole process is:
Yeah, I got a bit confused by the notation above.
>
> call foo -> jmp trampoline -> call all the fentry bpf progs ->
> call foo-body -> return foo-body -> call all the fexit bpf progs
> -> return foo.
>
> The "call foo-body" means "origin call", and it will store the
> return value of the traced function to the stack, therefore the
> fexit progs can get it.
>
> So it can trace the return side with the "fexit". And it's almost the
> same as the origin logic of the bpf trampoline:
OK, so this is just the way it always works.
>
> call foo -> call trampoline -> call all the fentry bpf progs ->
> call foo-body -> return foo-body -> call all the fexit bpf progs
> -> skip the rip -> return foo.
>
> What I did here is just replace the "call trampoline" to
> "jmp trampoline".
>
> >
> > Is there any mechanism to make sure that the trampoline being called is
> > only used by that one function? I haven't looked at the code yet, but
> > should there be a test that makes sure a trampoline isn't registered for
> > two or more different functions?
>
> As for now, the bpf trampoline is per-function. Every trampoline
> has a unique key, and we find the trampoline for the target function
> by that key. So it can't be used by two or more different functions.
>
> If the trampoline need to get the ip of the origin call from the stack,
> such as BPF_TRAMP_F_SHARE_IPMODIFY case, we will fallback to the
> "call" mode, as we can't get the rip from the stack in the "jmp" mode.
> And I think this is what you mean "only work for a single function"?
> Yeah, we fallback on such case.
OK, I got lost in the notation. It doesn't need a "call" because each
trampoline is only for a single function. Hence it doesn't need to know the
return address.
-- Steve
^ permalink raw reply
* Re: [PATCH v7 00/31] context_tracking,x86: Defer some IPIs until a user->kernel transition
From: Andy Lutomirski @ 2025-11-14 16:20 UTC (permalink / raw)
To: Valentin Schneider, Linux Kernel Mailing List, linux-mm, rcu,
the arch/x86 maintainers, linux-arm-kernel, loongarch,
linux-riscv, linux-arch, linux-trace-kernel
Cc: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen,
H. Peter Anvin, Peter Zijlstra (Intel), Arnaldo Carvalho de Melo,
Josh Poimboeuf, Paolo Bonzini, Arnd Bergmann, Frederic Weisbecker,
Paul E. McKenney, Jason Baron, Steven Rostedt, Ard Biesheuvel,
Sami Tolvanen, David S. Miller, Neeraj Upadhyay, Joel Fernandes,
Josh Triplett, Boqun Feng, Uladzislau Rezki, Mathieu Desnoyers,
Mel Gorman, Andrew Morton, Masahiro Yamada, Han Shen,
Rik van Riel, Jann Horn, Dan Carpenter, Oleg Nesterov, Juri Lelli,
Clark Williams, Yair Podemsky, Marcelo Tosatti, Daniel Wagner,
Petr Tesarik, Shrikanth Hegde
In-Reply-To: <20251114150133.1056710-1-vschneid@redhat.com>
On Fri, Nov 14, 2025, at 7:01 AM, Valentin Schneider wrote:
> Context
> =======
>
> We've observed within Red Hat that isolated, NOHZ_FULL CPUs running a
> pure-userspace application get regularly interrupted by IPIs sent from
> housekeeping CPUs. Those IPIs are caused by activity on the housekeeping CPUs
> leading to various on_each_cpu() calls, e.g.:
>
> The heart of this series is the thought that while we cannot remove NOHZ_FULL
> CPUs from the list of CPUs targeted by these IPIs, they may not have to execute
> the callbacks immediately. Anything that only affects kernelspace can wait
> until the next user->kernel transition, providing it can be executed "early
> enough" in the entry code.
>
I want to point out that there's another option here, although anyone trying to implement it would be fighting against quite a lot of history.
Logically, each CPU is in one of a handful of states: user mode, idle, normal kernel mode (possibly subdivided into IRQ, etc), and a handful of very narrow windows, hopefully uninstrumented and not accessing any PTEs that might be invalid, in the entry and exit paths where any state in memory could be out of sync with actual CPU state. (The latter includes right after the CPU switches to kernel mode, for example.) And NMI and MCE and whatever weird "security" entry types that Intel and AMD love to add.
The way the kernel *currently* deals with this has two big historical oddities:
1. The entry and exit code cares about ti_flags, which is per-*task*, which means that atomically poking it from other CPUs involves the runqueue lock or other shenanigans (see the idle nr_polling code for example), and also that it's not accessible from the user page tables if PTI is on.
2. The actual heavyweight atomic part (context tracking) was built for RCU, and it's sort or bolted on, and, as you've observed in this series, it's really quite awkward to do things that aren't RCU using context tracking.
If this were a greenfield project, I think there's a straightforward approach that's much nicer: stick everything into a single percpu flags structure. Imagine we have cpu_flags, which tracks both the current state of the CPU and what work needs to be done on state changes. On exit to user mode, we would atomically set the mode to USER and make sure we don't touch anything like vmalloc space after that. On entry back to kernel mode, we would avoid vmalloc space, etc, then atomically switch to kernel mode and read out whatever deferred work is needed. As an optimization, if nothing in the current configuration needs atomic state tracking, the state could be left at USER_OR_KERNEL and the overhead of an extra atomic op at entry and exit could be avoided.
And RCU would hook into *that* instead of having its own separate set of hooks.
I think that actually doing this would be a big improvement and would also be a serious project. There's a lot of code that would get touched, and the existing context tracking code is subtle and confusing. And, as mentioned, ti_flags has the wrong scope.
It's *possible* that one could avoid making ti_flags percpu either by extensive use of the runqueue locks or by borrowing a kludge from the idle code. For the latter, right now, the reason that the wake-from-idle code works is that the optimized path only happens if the idle thread/cpu is "polling", and it's impossible for the idle ti_flags to be polling while the CPU isn't actually idle. We could similarly observe that, if a ti_flags says it's in USER mode *and* is on, say, cpu 3, then cpu 3 is most definitely in USER mode. So someone could try shoving the CPU number into ti_flags :-p (USER means actually user or in the late exit / early entry path.)
Anyway, benefits of this whole approach would include considerably (IMO) increased comprehensibility compared to the current tangled ct code and much more straightforward addition of new things that happen to a target CPU conditionally depending on its mode. And, if the flags word was actually per cpu, it could be mapped such that SWITCH_TO_KERNEL_CR3 would use it -- there could be a single CR3 write (and maybe CR4/invpcid depending on whether a zapped mapping is global) and the flush bit could depend on whether a flush is needed. And there would be basically no chance that a bug that accessed invalidated-but-not-flushed kernel data could be undetected -- in PTI mode, any such access would page fault! Similarly, if kernel text pokes deferred the flush and serialization, the only code that could execute before noticing the deferred flush would be the user-CR3 code.
Oh, any another primitive would be possible: one CPU could plausibly execute another CPU's interrupts or soft-irqs or whatever by taking a special lock that would effectively pin the remote CPU in user mode -- you'd set a flag in the target cpu_flags saying "pin in USER mode" and the transition on that CPU to kernel mode would then spin on entry to kernel mode and wait for the lock to be released. This could plausibly get a lot of the on_each_cpu callers to switch over in one fell swoop: anything that needs to synchronize to the remote CPU but does not need to poke its actual architectural state could be executed locally while the remote CPU is pinned.
--Andy
^ permalink raw reply
* Re: [PATCH 0/2] tracing/wprobe: Fix to avoid inifinite watchpoint exception on arm64
From: Will Deacon @ 2025-11-14 16:17 UTC (permalink / raw)
To: Masami Hiramatsu
Cc: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Steven Rostedt, Mark Rutland, Alexander Shishkin,
Jiri Olsa, Ian Rogers, Adrian Hunter, Mathieu Desnoyers,
linux-perf-users, linux-kernel, linux-trace-kernel,
Catalin Marinas, linux-arm-kernel
In-Reply-To: <20251104223735.4164f224790ea0aea267dff1@kernel.org>
On Tue, Nov 04, 2025 at 10:37:35PM +0900, Masami Hiramatsu wrote:
> Gently ping.
>
> There is a bugfix (or strange behavior) on arm64 hw breakpoint but
> to fix it cleanly, it should change the perf itself (but I'm not
> sure why arm64 changes the behavior only for the default overflow
> handlers.) Anyone knows it?
It's because GDB expects to handle the stepping itself when using the
ptrace interface (with a custom overflow handler to deliver SIGTRAP).
Will
^ permalink raw reply
* [RFC PATCH v7 31/31] x86/entry: Add an option to coalesce TLB flushes
From: Valentin Schneider @ 2025-11-14 15:14 UTC (permalink / raw)
To: linux-kernel, linux-mm, rcu, x86, linux-arm-kernel, loongarch,
linux-riscv, linux-arch, linux-trace-kernel
Cc: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen,
H. Peter Anvin, Andy Lutomirski, Peter Zijlstra,
Arnaldo Carvalho de Melo, Josh Poimboeuf, Paolo Bonzini,
Arnd Bergmann, Frederic Weisbecker, Paul E. McKenney, Jason Baron,
Steven Rostedt, Ard Biesheuvel, Sami Tolvanen, David S. Miller,
Neeraj Upadhyay, Joel Fernandes, Josh Triplett, Boqun Feng,
Uladzislau Rezki, Mathieu Desnoyers, Mel Gorman, Andrew Morton,
Masahiro Yamada, Han Shen, Rik van Riel, Jann Horn, Dan Carpenter,
Oleg Nesterov, Juri Lelli, Clark Williams, Yair Podemsky,
Marcelo Tosatti, Daniel Wagner, Petr Tesarik, Shrikanth Hegde
In-Reply-To: <20251114150133.1056710-1-vschneid@redhat.com>
Previous patches have introduced a mechanism to prevent kernel text updates
from inducing interference on isolated CPUs. A similar action is required
for kernel-range TLB flushes in order to silence the biggest remaining
cause of isolated CPU IPI interference.
These flushes are mostly caused by vmalloc manipulations - e.g. on x86 with
CONFIG_VMAP_STACK, spawning enough processes will easily trigger
flushes. Unfortunately, the newly added context_tracking IPI deferral
mechanism cannot be leveraged for TLB flushes, as the deferred work would
be executed too late. Consider the following execution flow:
<userspace>
!interrupt!
SWITCH_TO_KERNEL_CR3 // vmalloc range becomes accessible
idtentry_func_foo()
irqentry_enter()
irqentry_enter_from_user_mode()
enter_from_user_mode()
[...]
ct_kernel_enter_state()
ct_work_flush() // deferred flush would be done here
Since there is no sane way to assert no stale entry is accessed during
kernel entry, any code executed between SWITCH_TO_KERNEL_CR3 and
ct_work_flush() is at risk of accessing a stale entry. Dave had suggested
hacking up something within SWITCH_TO_KERNEL_CR3 itself, which is what has
been implemented in the previous patches.
Make kernel-range TLB flush deferral available via CONFIG_COALESCE_TLBI.
Signed-off-by: Valentin Schneider <vschneid@redhat.com>
---
arch/x86/Kconfig | 17 +++++++++++++++++
1 file changed, 17 insertions(+)
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index fa9229c0e0939..04f9d6496bbbc 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -2189,6 +2189,23 @@ config ADDRESS_MASKING
The capability can be used for efficient address sanitizers (ASAN)
implementation and for optimizations in JITs.
+config COALESCE_TLBI
+ def_bool n
+ prompt "Coalesce kernel TLB flushes for NOHZ-full CPUs"
+ depends on X86_64 && MITIGATION_PAGE_TABLE_ISOLATION && NO_HZ_FULL
+ help
+ TLB flushes for kernel addresses can lead to IPIs being sent to
+ NOHZ-full CPUs, thus kicking them out of userspace.
+
+ This option coalesces kernel-range TLB flushes for NOHZ-full CPUs into
+ a single flush executed at kernel entry, right after switching to the
+ kernel page table. Note that this flush is unconditionnal, even if no
+ remote flush was issued during the previous userspace execution window.
+
+ This obviously makes the user->kernel transition overhead even worse.
+
+ If unsure, say N.
+
config HOTPLUG_CPU
def_bool y
depends on SMP
--
2.51.0
^ permalink raw reply related
* [RFC PATCH v7 30/31] x86/mm, mm/vmalloc: Defer kernel TLB flush IPIs under CONFIG_COALESCE_TLBI=y
From: Valentin Schneider @ 2025-11-14 15:14 UTC (permalink / raw)
To: linux-kernel, linux-mm, rcu, x86, linux-arm-kernel, loongarch,
linux-riscv, linux-arch, linux-trace-kernel
Cc: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen,
H. Peter Anvin, Andy Lutomirski, Peter Zijlstra,
Arnaldo Carvalho de Melo, Josh Poimboeuf, Paolo Bonzini,
Arnd Bergmann, Frederic Weisbecker, Paul E. McKenney, Jason Baron,
Steven Rostedt, Ard Biesheuvel, Sami Tolvanen, David S. Miller,
Neeraj Upadhyay, Joel Fernandes, Josh Triplett, Boqun Feng,
Uladzislau Rezki, Mathieu Desnoyers, Mel Gorman, Andrew Morton,
Masahiro Yamada, Han Shen, Rik van Riel, Jann Horn, Dan Carpenter,
Oleg Nesterov, Juri Lelli, Clark Williams, Yair Podemsky,
Marcelo Tosatti, Daniel Wagner, Petr Tesarik, Shrikanth Hegde
In-Reply-To: <20251114150133.1056710-1-vschneid@redhat.com>
Previous commits have added an unconditional TLB flush right after
switching to the kernel CR3 on NOHZ_FULL CPUs, and a software signal to
determine whether a CPU has its kernel CR3 loaded.
Using these two components, we can now safely defer kernel TLB flush IPIs
targeting NOHZ_FULL CPUs executing in userspace (i.e. with the user CR3
loaded).
Note that the COALESCE_TLBI config option is introduced in a later commit,
when the whole feature is implemented.
Signed-off-by: Valentin Schneider <vschneid@redhat.com>
---
arch/x86/include/asm/tlbflush.h | 3 +++
arch/x86/mm/tlb.c | 34 ++++++++++++++++++++++++++-------
mm/vmalloc.c | 34 ++++++++++++++++++++++++++++-----
3 files changed, 59 insertions(+), 12 deletions(-)
diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h
index e39ae95b85072..6d533afd70952 100644
--- a/arch/x86/include/asm/tlbflush.h
+++ b/arch/x86/include/asm/tlbflush.h
@@ -321,6 +321,9 @@ extern void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start,
unsigned long end, unsigned int stride_shift,
bool freed_tables);
extern void flush_tlb_kernel_range(unsigned long start, unsigned long end);
+#ifdef CONFIG_COALESCE_TLBI
+extern void flush_tlb_kernel_range_deferrable(unsigned long start, unsigned long end);
+#endif
static inline void flush_tlb_page(struct vm_area_struct *vma, unsigned long a)
{
diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c
index 5d221709353e0..1ce80f8775e7a 100644
--- a/arch/x86/mm/tlb.c
+++ b/arch/x86/mm/tlb.c
@@ -12,6 +12,7 @@
#include <linux/task_work.h>
#include <linux/mmu_notifier.h>
#include <linux/mmu_context.h>
+#include <linux/sched/isolation.h>
#include <asm/tlbflush.h>
#include <asm/mmu_context.h>
@@ -1529,23 +1530,24 @@ static void do_kernel_range_flush(void *info)
flush_tlb_one_kernel(addr);
}
-static void kernel_tlb_flush_all(struct flush_tlb_info *info)
+static void kernel_tlb_flush_all(smp_cond_func_t cond, struct flush_tlb_info *info)
{
if (cpu_feature_enabled(X86_FEATURE_INVLPGB))
invlpgb_flush_all();
else
- on_each_cpu(do_flush_tlb_all, NULL, 1);
+ on_each_cpu_cond(cond, do_flush_tlb_all, NULL, 1);
}
-static void kernel_tlb_flush_range(struct flush_tlb_info *info)
+static void kernel_tlb_flush_range(smp_cond_func_t cond, struct flush_tlb_info *info)
{
if (cpu_feature_enabled(X86_FEATURE_INVLPGB))
invlpgb_kernel_range_flush(info);
else
- on_each_cpu(do_kernel_range_flush, info, 1);
+ on_each_cpu_cond(cond, do_kernel_range_flush, info, 1);
}
-void flush_tlb_kernel_range(unsigned long start, unsigned long end)
+static inline void
+__flush_tlb_kernel_range(smp_cond_func_t cond, unsigned long start, unsigned long end)
{
struct flush_tlb_info *info;
@@ -1555,13 +1557,31 @@ void flush_tlb_kernel_range(unsigned long start, unsigned long end)
TLB_GENERATION_INVALID);
if (info->end == TLB_FLUSH_ALL)
- kernel_tlb_flush_all(info);
+ kernel_tlb_flush_all(cond, info);
else
- kernel_tlb_flush_range(info);
+ kernel_tlb_flush_range(cond, info);
put_flush_tlb_info();
}
+void flush_tlb_kernel_range(unsigned long start, unsigned long end)
+{
+ __flush_tlb_kernel_range(NULL, start, end);
+}
+
+#ifdef CONFIG_COALESCE_TLBI
+static bool flush_tlb_kernel_cond(int cpu, void *info)
+{
+ return housekeeping_cpu(cpu, HK_TYPE_KERNEL_NOISE) ||
+ per_cpu(kernel_cr3_loaded, cpu);
+}
+
+void flush_tlb_kernel_range_deferrable(unsigned long start, unsigned long end)
+{
+ __flush_tlb_kernel_range(flush_tlb_kernel_cond, start, end);
+}
+#endif
+
/*
* This can be used from process context to figure out what the value of
* CR3 is without needing to do a (slow) __read_cr3().
diff --git a/mm/vmalloc.c b/mm/vmalloc.c
index 798b2ed21e460..76ec10d56623b 100644
--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -494,6 +494,30 @@ void vunmap_range_noflush(unsigned long start, unsigned long end)
__vunmap_range_noflush(start, end);
}
+#ifdef CONFIG_COALESCE_TLBI
+/*
+ * !!! BIG FAT WARNING !!!
+ *
+ * The CPU is free to cache any part of the paging hierarchy it wants at any
+ * time. It's also free to set accessed and dirty bits at any time, even for
+ * instructions that may never execute architecturally.
+ *
+ * This means that deferring a TLB flush affecting freed page-table-pages (IOW,
+ * keeping them in a CPU's paging hierarchy cache) is a recipe for disaster.
+ *
+ * This isn't a problem for deferral of TLB flushes in vmalloc, because
+ * page-table-pages used for vmap() mappings are never freed - see how
+ * __vunmap_range_noflush() walks the whole mapping but only clears the leaf PTEs.
+ * If this ever changes, TLB flush deferral will cause misery.
+ */
+void __weak flush_tlb_kernel_range_deferrable(unsigned long start, unsigned long end)
+{
+ flush_tlb_kernel_range(start, end);
+}
+#else
+#define flush_tlb_kernel_range_deferrable(start, end) flush_tlb_kernel_range(start, end)
+#endif
+
/**
* vunmap_range - unmap kernel virtual addresses
* @addr: start of the VM area to unmap
@@ -507,7 +531,7 @@ void vunmap_range(unsigned long addr, unsigned long end)
{
flush_cache_vunmap(addr, end);
vunmap_range_noflush(addr, end);
- flush_tlb_kernel_range(addr, end);
+ flush_tlb_kernel_range_deferrable(addr, end);
}
static int vmap_pages_pte_range(pmd_t *pmd, unsigned long addr,
@@ -2339,7 +2363,7 @@ static bool __purge_vmap_area_lazy(unsigned long start, unsigned long end,
nr_purge_nodes = cpumask_weight(&purge_nodes);
if (nr_purge_nodes > 0) {
- flush_tlb_kernel_range(start, end);
+ flush_tlb_kernel_range_deferrable(start, end);
/* One extra worker is per a lazy_max_pages() full set minus one. */
nr_purge_helpers = atomic_long_read(&vmap_lazy_nr) / lazy_max_pages();
@@ -2442,7 +2466,7 @@ static void free_unmap_vmap_area(struct vmap_area *va)
flush_cache_vunmap(va->va_start, va->va_end);
vunmap_range_noflush(va->va_start, va->va_end);
if (debug_pagealloc_enabled_static())
- flush_tlb_kernel_range(va->va_start, va->va_end);
+ flush_tlb_kernel_range_deferrable(va->va_start, va->va_end);
free_vmap_area_noflush(va);
}
@@ -2890,7 +2914,7 @@ static void vb_free(unsigned long addr, unsigned long size)
vunmap_range_noflush(addr, addr + size);
if (debug_pagealloc_enabled_static())
- flush_tlb_kernel_range(addr, addr + size);
+ flush_tlb_kernel_range_deferrable(addr, addr + size);
spin_lock(&vb->lock);
@@ -2955,7 +2979,7 @@ static void _vm_unmap_aliases(unsigned long start, unsigned long end, int flush)
free_purged_blocks(&purge_list);
if (!__purge_vmap_area_lazy(start, end, false) && flush)
- flush_tlb_kernel_range(start, end);
+ flush_tlb_kernel_range_deferrable(start, end);
mutex_unlock(&vmap_purge_lock);
}
--
2.51.0
^ permalink raw reply related
* [RFC PATCH v7 29/31] x86/mm/pti: Implement a TLB flush immediately after a switch to kernel CR3
From: Valentin Schneider @ 2025-11-14 15:14 UTC (permalink / raw)
To: linux-kernel, linux-mm, rcu, x86, linux-arm-kernel, loongarch,
linux-riscv, linux-arch, linux-trace-kernel
Cc: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen,
H. Peter Anvin, Andy Lutomirski, Peter Zijlstra,
Arnaldo Carvalho de Melo, Josh Poimboeuf, Paolo Bonzini,
Arnd Bergmann, Frederic Weisbecker, Paul E. McKenney, Jason Baron,
Steven Rostedt, Ard Biesheuvel, Sami Tolvanen, David S. Miller,
Neeraj Upadhyay, Joel Fernandes, Josh Triplett, Boqun Feng,
Uladzislau Rezki, Mathieu Desnoyers, Mel Gorman, Andrew Morton,
Masahiro Yamada, Han Shen, Rik van Riel, Jann Horn, Dan Carpenter,
Oleg Nesterov, Juri Lelli, Clark Williams, Yair Podemsky,
Marcelo Tosatti, Daniel Wagner, Petr Tesarik, Shrikanth Hegde
In-Reply-To: <20251114150133.1056710-1-vschneid@redhat.com>
Deferring kernel range TLB flushes requires the guarantee that upon
entering the kernel, no stale entry may be accessed. The simplest way to
provide such a guarantee is to issue an unconditional flush upon switching
to the kernel CR3, as this is the pivoting point where such stale entries
may be accessed.
As this is only relevant to NOHZ_FULL, restrict the mechanism to NOHZ_FULL
CPUs.
Note that the COALESCE_TLBI config option is introduced in a later commit,
when the whole feature is implemented.
Signed-off-by: Valentin Schneider <vschneid@redhat.com>
---
arch/x86/entry/calling.h | 25 ++++++++++++++++++++++---
arch/x86/kernel/asm-offsets.c | 1 +
2 files changed, 23 insertions(+), 3 deletions(-)
diff --git a/arch/x86/entry/calling.h b/arch/x86/entry/calling.h
index 0187c0ea2fddb..620203ef04e9f 100644
--- a/arch/x86/entry/calling.h
+++ b/arch/x86/entry/calling.h
@@ -10,6 +10,7 @@
#include <asm/msr.h>
#include <asm/nospec-branch.h>
#include <asm/jump_label.h>
+#include <asm/invpcid.h>
/*
@@ -171,9 +172,27 @@ For 32-bit we have the following conventions - kernel is built with
andq $(~PTI_USER_PGTABLE_AND_PCID_MASK), \reg
.endm
-.macro COALESCE_TLBI
+.macro COALESCE_TLBI scratch_reg:req
#ifdef CONFIG_COALESCE_TLBI
STATIC_BRANCH_FALSE_LIKELY housekeeping_overridden, .Lend_\@
+ /* No point in doing this for housekeeping CPUs */
+ movslq PER_CPU_VAR(cpu_number), \scratch_reg
+ bt \scratch_reg, tick_nohz_full_mask(%rip)
+ jnc .Lend_tlbi_\@
+
+ ALTERNATIVE "jmp .Lcr4_\@", "", X86_FEATURE_INVPCID
+ movq $(INVPCID_TYPE_ALL_INCL_GLOBAL), \scratch_reg
+ /* descriptor is all zeroes, point at the zero page */
+ invpcid empty_zero_page(%rip), \scratch_reg
+ jmp .Lend_tlbi_\@
+.Lcr4_\@:
+ /* Note: this gives CR4 pinning the finger */
+ movq PER_CPU_VAR(cpu_tlbstate + TLB_STATE_cr4), \scratch_reg
+ xorq $(X86_CR4_PGE), \scratch_reg
+ movq \scratch_reg, %cr4
+ xorq $(X86_CR4_PGE), \scratch_reg
+ movq \scratch_reg, %cr4
+.Lend_tlbi_\@:
movl $1, PER_CPU_VAR(kernel_cr3_loaded)
.Lend_\@:
#endif // CONFIG_COALESCE_TLBI
@@ -192,7 +211,7 @@ For 32-bit we have the following conventions - kernel is built with
mov %cr3, \scratch_reg
ADJUST_KERNEL_CR3 \scratch_reg
mov \scratch_reg, %cr3
- COALESCE_TLBI
+ COALESCE_TLBI \scratch_reg
.Lend_\@:
.endm
@@ -260,7 +279,7 @@ For 32-bit we have the following conventions - kernel is built with
ADJUST_KERNEL_CR3 \scratch_reg
movq \scratch_reg, %cr3
- COALESCE_TLBI
+ COALESCE_TLBI \scratch_reg
.Ldone_\@:
.endm
diff --git a/arch/x86/kernel/asm-offsets.c b/arch/x86/kernel/asm-offsets.c
index 32ba599a51f88..deb92e9c8923d 100644
--- a/arch/x86/kernel/asm-offsets.c
+++ b/arch/x86/kernel/asm-offsets.c
@@ -106,6 +106,7 @@ static void __used common(void)
/* TLB state for the entry code */
OFFSET(TLB_STATE_user_pcid_flush_mask, tlb_state, user_pcid_flush_mask);
+ OFFSET(TLB_STATE_cr4, tlb_state, cr4);
/* Layout info for cpu_entry_area */
OFFSET(CPU_ENTRY_AREA_entry_stack, cpu_entry_area, entry_stack_page);
--
2.51.0
^ permalink raw reply related
* [RFC PATCH v7 28/31] x86/mm/pti: Introduce a kernel/user CR3 software signal
From: Valentin Schneider @ 2025-11-14 15:14 UTC (permalink / raw)
To: linux-kernel, linux-mm, rcu, x86, linux-arm-kernel, loongarch,
linux-riscv, linux-arch, linux-trace-kernel
Cc: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen,
H. Peter Anvin, Andy Lutomirski, Peter Zijlstra,
Arnaldo Carvalho de Melo, Josh Poimboeuf, Paolo Bonzini,
Arnd Bergmann, Frederic Weisbecker, Paul E. McKenney, Jason Baron,
Steven Rostedt, Ard Biesheuvel, Sami Tolvanen, David S. Miller,
Neeraj Upadhyay, Joel Fernandes, Josh Triplett, Boqun Feng,
Uladzislau Rezki, Mathieu Desnoyers, Mel Gorman, Andrew Morton,
Masahiro Yamada, Han Shen, Rik van Riel, Jann Horn, Dan Carpenter,
Oleg Nesterov, Juri Lelli, Clark Williams, Yair Podemsky,
Marcelo Tosatti, Daniel Wagner, Petr Tesarik, Shrikanth Hegde
In-Reply-To: <20251114150133.1056710-1-vschneid@redhat.com>
Later commits will rely on this information to defer kernel TLB flush
IPIs. Update it when switching to and from the kernel CR3.
This will only be really useful for NOHZ_FULL CPUs, but it should be
cheaper to unconditionally update a never-used per-CPU variable living in
its own cacheline than to check a shared cpumask such as
housekeeping_cpumask(HK_TYPE_KERNEL_NOISE)
at every entry.
Note that the COALESCE_TLBI config option is introduced in a later commit,
when the whole feature is implemented.
Signed-off-by: Valentin Schneider <vschneid@redhat.com>
---
Per the cover letter, I really hate this, but couldn't come up with
anything better.
---
arch/x86/entry/calling.h | 21 +++++++++++++++++++++
arch/x86/entry/syscall_64.c | 4 ++++
arch/x86/include/asm/tlbflush.h | 3 +++
3 files changed, 28 insertions(+)
diff --git a/arch/x86/entry/calling.h b/arch/x86/entry/calling.h
index 77e2d920a6407..0187c0ea2fddb 100644
--- a/arch/x86/entry/calling.h
+++ b/arch/x86/entry/calling.h
@@ -9,6 +9,7 @@
#include <asm/ptrace-abi.h>
#include <asm/msr.h>
#include <asm/nospec-branch.h>
+#include <asm/jump_label.h>
/*
@@ -170,11 +171,28 @@ For 32-bit we have the following conventions - kernel is built with
andq $(~PTI_USER_PGTABLE_AND_PCID_MASK), \reg
.endm
+.macro COALESCE_TLBI
+#ifdef CONFIG_COALESCE_TLBI
+ STATIC_BRANCH_FALSE_LIKELY housekeeping_overridden, .Lend_\@
+ movl $1, PER_CPU_VAR(kernel_cr3_loaded)
+.Lend_\@:
+#endif // CONFIG_COALESCE_TLBI
+.endm
+
+.macro NOTE_SWITCH_TO_USER_CR3
+#ifdef CONFIG_COALESCE_TLBI
+ STATIC_BRANCH_FALSE_LIKELY housekeeping_overridden, .Lend_\@
+ movl $0, PER_CPU_VAR(kernel_cr3_loaded)
+.Lend_\@:
+#endif // CONFIG_COALESCE_TLBI
+.endm
+
.macro SWITCH_TO_KERNEL_CR3 scratch_reg:req
ALTERNATIVE "jmp .Lend_\@", "", X86_FEATURE_PTI
mov %cr3, \scratch_reg
ADJUST_KERNEL_CR3 \scratch_reg
mov \scratch_reg, %cr3
+ COALESCE_TLBI
.Lend_\@:
.endm
@@ -182,6 +200,7 @@ For 32-bit we have the following conventions - kernel is built with
PER_CPU_VAR(cpu_tlbstate + TLB_STATE_user_pcid_flush_mask)
.macro SWITCH_TO_USER_CR3 scratch_reg:req scratch_reg2:req
+ NOTE_SWITCH_TO_USER_CR3
mov %cr3, \scratch_reg
ALTERNATIVE "jmp .Lwrcr3_\@", "", X86_FEATURE_PCID
@@ -241,6 +260,7 @@ For 32-bit we have the following conventions - kernel is built with
ADJUST_KERNEL_CR3 \scratch_reg
movq \scratch_reg, %cr3
+ COALESCE_TLBI
.Ldone_\@:
.endm
@@ -257,6 +277,7 @@ For 32-bit we have the following conventions - kernel is built with
bt $PTI_USER_PGTABLE_BIT, \save_reg
jnc .Lend_\@
+ NOTE_SWITCH_TO_USER_CR3
ALTERNATIVE "jmp .Lwrcr3_\@", "", X86_FEATURE_PCID
/*
diff --git a/arch/x86/entry/syscall_64.c b/arch/x86/entry/syscall_64.c
index b6e68ea98b839..2589d232e0ba1 100644
--- a/arch/x86/entry/syscall_64.c
+++ b/arch/x86/entry/syscall_64.c
@@ -83,6 +83,10 @@ static __always_inline bool do_syscall_x32(struct pt_regs *regs, int nr)
return false;
}
+#ifdef CONFIG_COALESCE_TLBI
+DEFINE_PER_CPU(bool, kernel_cr3_loaded) = true;
+#endif
+
/* Returns true to return using SYSRET, or false to use IRET */
__visible noinstr bool do_syscall_64(struct pt_regs *regs, int nr)
{
diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h
index 00daedfefc1b0..e39ae95b85072 100644
--- a/arch/x86/include/asm/tlbflush.h
+++ b/arch/x86/include/asm/tlbflush.h
@@ -17,6 +17,9 @@
#include <asm/pgtable.h>
DECLARE_PER_CPU(u64, tlbstate_untag_mask);
+#ifdef CONFIG_COALESCE_TLBI
+DECLARE_PER_CPU(bool, kernel_cr3_loaded);
+#endif
void __flush_tlb_all(void);
--
2.51.0
^ permalink raw reply related
* [PATCH v7 27/31] x86/mm: Make INVPCID type macros available to assembly
From: Valentin Schneider @ 2025-11-14 15:14 UTC (permalink / raw)
To: linux-kernel, linux-mm, rcu, x86, linux-arm-kernel, loongarch,
linux-riscv, linux-arch, linux-trace-kernel
Cc: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen,
H. Peter Anvin, Andy Lutomirski, Peter Zijlstra,
Arnaldo Carvalho de Melo, Josh Poimboeuf, Paolo Bonzini,
Arnd Bergmann, Frederic Weisbecker, Paul E. McKenney, Jason Baron,
Steven Rostedt, Ard Biesheuvel, Sami Tolvanen, David S. Miller,
Neeraj Upadhyay, Joel Fernandes, Josh Triplett, Boqun Feng,
Uladzislau Rezki, Mathieu Desnoyers, Mel Gorman, Andrew Morton,
Masahiro Yamada, Han Shen, Rik van Riel, Jann Horn, Dan Carpenter,
Oleg Nesterov, Juri Lelli, Clark Williams, Yair Podemsky,
Marcelo Tosatti, Daniel Wagner, Petr Tesarik, Shrikanth Hegde
In-Reply-To: <20251114150133.1056710-1-vschneid@redhat.com>
A later commit will introduce a pure-assembly INVPCID invocation, allow
assembly files to get the type definitions.
Signed-off-by: Valentin Schneider <vschneid@redhat.com>
---
arch/x86/include/asm/invpcid.h | 14 +++++++++-----
1 file changed, 9 insertions(+), 5 deletions(-)
diff --git a/arch/x86/include/asm/invpcid.h b/arch/x86/include/asm/invpcid.h
index 734482afbf81d..27ae75c2d7fed 100644
--- a/arch/x86/include/asm/invpcid.h
+++ b/arch/x86/include/asm/invpcid.h
@@ -2,6 +2,13 @@
#ifndef _ASM_X86_INVPCID
#define _ASM_X86_INVPCID
+#define INVPCID_TYPE_INDIV_ADDR 0
+#define INVPCID_TYPE_SINGLE_CTXT 1
+#define INVPCID_TYPE_ALL_INCL_GLOBAL 2
+#define INVPCID_TYPE_ALL_NON_GLOBAL 3
+
+#ifndef __ASSEMBLER__
+
static inline void __invpcid(unsigned long pcid, unsigned long addr,
unsigned long type)
{
@@ -17,11 +24,6 @@ static inline void __invpcid(unsigned long pcid, unsigned long addr,
:: [desc] "m" (desc), [type] "r" (type) : "memory");
}
-#define INVPCID_TYPE_INDIV_ADDR 0
-#define INVPCID_TYPE_SINGLE_CTXT 1
-#define INVPCID_TYPE_ALL_INCL_GLOBAL 2
-#define INVPCID_TYPE_ALL_NON_GLOBAL 3
-
/* Flush all mappings for a given pcid and addr, not including globals. */
static inline void invpcid_flush_one(unsigned long pcid,
unsigned long addr)
@@ -47,4 +49,6 @@ static inline void invpcid_flush_all_nonglobals(void)
__invpcid(0, 0, INVPCID_TYPE_ALL_NON_GLOBAL);
}
+#endif /* __ASSEMBLER__ */
+
#endif /* _ASM_X86_INVPCID */
--
2.51.0
^ permalink raw reply related
* [PATCH v7 26/31] x86/jump_label: Add ASM support for static_branch_likely()
From: Valentin Schneider @ 2025-11-14 15:14 UTC (permalink / raw)
To: linux-kernel, linux-mm, rcu, x86, linux-arm-kernel, loongarch,
linux-riscv, linux-arch, linux-trace-kernel
Cc: Frederic Weisbecker, Thomas Gleixner, Ingo Molnar,
Borislav Petkov, Dave Hansen, H. Peter Anvin, Andy Lutomirski,
Peter Zijlstra, Arnaldo Carvalho de Melo, Josh Poimboeuf,
Paolo Bonzini, Arnd Bergmann, Paul E. McKenney, Jason Baron,
Steven Rostedt, Ard Biesheuvel, Sami Tolvanen, David S. Miller,
Neeraj Upadhyay, Joel Fernandes, Josh Triplett, Boqun Feng,
Uladzislau Rezki, Mathieu Desnoyers, Mel Gorman, Andrew Morton,
Masahiro Yamada, Han Shen, Rik van Riel, Jann Horn, Dan Carpenter,
Oleg Nesterov, Juri Lelli, Clark Williams, Yair Podemsky,
Marcelo Tosatti, Daniel Wagner, Petr Tesarik, Shrikanth Hegde
In-Reply-To: <20251114150133.1056710-1-vschneid@redhat.com>
A later commit will add some early entry code that only needs to be
executed if nohz_full is present on the cmdline, not just if
CONFIG_NO_HZ_FULL is compiled in. Add an ASM-callable static branch macro.
Note that I haven't found a way to express unlikely (i.e. out-of-line)
static branches in ASM macros without using extra jumps, which kind of
defeats the purpose. Consider:
.macro FOOBAR
// Key enabled: JMP .Ldostuff_\@
// Key disabled: NOP
STATIC_BRANCH_UNLIKELY key, .Ldostuff_\@ // Patched to JMP if enabled
jmp .Lend_\@
.Ldostuff_\@:
<dostuff>
.Lend_\@:
.endm
Instead, this should be expressed as a likely (i.e. in-line) static key:
.macro FOOBAR
// Key enabled: NOP
// Key disabled: JMP .Lend_\@
STATIC_BRANCH_LIKELY key, .Lend\@ // Patched to NOP if enabled
<dostuff>
.Lend_\@:
.endm
Suggested-by: Frederic Weisbecker <frederic@kernel.org>
Signed-off-by: Valentin Schneider <vschneid@redhat.com>
---
arch/x86/include/asm/jump_label.h | 33 ++++++++++++++++++++++++++++++-
1 file changed, 32 insertions(+), 1 deletion(-)
diff --git a/arch/x86/include/asm/jump_label.h b/arch/x86/include/asm/jump_label.h
index 61dd1dee7812e..3c9ba3948e225 100644
--- a/arch/x86/include/asm/jump_label.h
+++ b/arch/x86/include/asm/jump_label.h
@@ -7,7 +7,38 @@
#include <asm/asm.h>
#include <asm/nops.h>
-#ifndef __ASSEMBLER__
+#ifdef __ASSEMBLER__
+
+/*
+ * There isn't a neat way to craft unlikely static branches in ASM, so they
+ * all have to be expressed as likely (inline) static branches. This macro
+ * thus assumes a "likely" usage.
+ */
+.macro ARCH_STATIC_BRANCH_LIKELY_ASM key, label, jump, hack
+1:
+.if \jump || \hack
+ jmp \label
+.else
+ .byte BYTES_NOP5
+.endif
+ .pushsection __jump_table, "aw"
+ _ASM_ALIGN
+ .long 1b - .
+ .long \label - .
+ /* LIKELY so bit0=1, bit1=hack */
+ _ASM_PTR \key + 1 + (\hack << 1) - .
+ .popsection
+.endm
+
+.macro STATIC_BRANCH_TRUE_LIKELY key, label
+ ARCH_STATIC_BRANCH_LIKELY_ASM \key, \label, 0, IS_ENABLED(CONFIG_HAVE_JUMP_LABEL_HACK)
+.endm
+
+.macro STATIC_BRANCH_FALSE_LIKELY key, label
+ ARCH_STATIC_BRANCH_LIKELY_ASM \key, \label, 1, 0
+.endm
+
+#else /* !__ASSEMBLER__ */
#include <linux/stringify.h>
#include <linux/types.h>
--
2.51.0
^ permalink raw reply related
* [PATCH v7 25/31] context_tracking,x86: Defer kernel text patching IPIs
From: Valentin Schneider @ 2025-11-14 15:14 UTC (permalink / raw)
To: linux-kernel, linux-mm, rcu, x86, linux-arm-kernel, loongarch,
linux-riscv, linux-arch, linux-trace-kernel
Cc: Peter Zijlstra (Intel), Nicolas Saenz Julienne,
Frederic Weisbecker, Thomas Gleixner, Ingo Molnar,
Borislav Petkov, Dave Hansen, H. Peter Anvin, Andy Lutomirski,
Arnaldo Carvalho de Melo, Josh Poimboeuf, Paolo Bonzini,
Arnd Bergmann, Paul E. McKenney, Jason Baron, Steven Rostedt,
Ard Biesheuvel, Sami Tolvanen, David S. Miller, Neeraj Upadhyay,
Joel Fernandes, Josh Triplett, Boqun Feng, Uladzislau Rezki,
Mathieu Desnoyers, Mel Gorman, Andrew Morton, Masahiro Yamada,
Han Shen, Rik van Riel, Jann Horn, Dan Carpenter, Oleg Nesterov,
Juri Lelli, Clark Williams, Yair Podemsky, Marcelo Tosatti,
Daniel Wagner, Petr Tesarik, Shrikanth Hegde
In-Reply-To: <20251114150133.1056710-1-vschneid@redhat.com>
text_poke_bp_batch() sends IPIs to all online CPUs to synchronize
them vs the newly patched instruction. CPUs that are executing in userspace
do not need this synchronization to happen immediately, and this is
actually harmful interference for NOHZ_FULL CPUs.
As the synchronization IPIs are sent using a blocking call, returning from
text_poke_bp_batch() implies all CPUs will observe the patched
instruction(s), and this should be preserved even if the IPI is deferred.
In other words, to safely defer this synchronization, any kernel
instruction leading to the execution of the deferred instruction
sync (ct_work_flush()) must *not* be mutable (patchable) at runtime.
This means we must pay attention to mutable instructions in the early entry
code:
- alternatives
- static keys
- static calls
- all sorts of probes (kprobes/ftrace/bpf/???)
The early entry code leading to ct_work_flush() is noinstr, which gets rid
of the probes.
Alternatives are safe, because it's boot-time patching (before SMP is
even brought up) which is before any IPI deferral can happen.
This leaves us with static keys and static calls.
Any static key used in early entry code should be only forever-enabled at
boot time, IOW __ro_after_init (pretty much like alternatives). Exceptions
are explicitly marked as allowed in .noinstr and will always generate an
IPI when flipped.
The same applies to static calls - they should be only updated at boot
time, or manually marked as an exception.
Objtool is now able to point at static keys/calls that don't respect this,
and all static keys/calls used in early entry code have now been verified
as behaving appropriately.
Leverage the new context_tracking infrastructure to defer sync_core() IPIs
to a target CPU's next kernel entry.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: Nicolas Saenz Julienne <nsaenzju@redhat.com>
Signed-off-by: Valentin Schneider <vschneid@redhat.com>
Acked-by: Frederic Weisbecker <frederic@kernel.org>
---
arch/x86/include/asm/context_tracking_work.h | 6 ++-
arch/x86/include/asm/text-patching.h | 1 +
arch/x86/kernel/alternative.c | 39 +++++++++++++++++---
arch/x86/kernel/kprobes/core.c | 4 +-
arch/x86/kernel/kprobes/opt.c | 4 +-
arch/x86/kernel/module.c | 2 +-
include/asm-generic/sections.h | 15 ++++++++
include/linux/context_tracking_work.h | 4 +-
8 files changed, 60 insertions(+), 15 deletions(-)
diff --git a/arch/x86/include/asm/context_tracking_work.h b/arch/x86/include/asm/context_tracking_work.h
index 5f3b2d0977235..485b32881fde5 100644
--- a/arch/x86/include/asm/context_tracking_work.h
+++ b/arch/x86/include/asm/context_tracking_work.h
@@ -2,11 +2,13 @@
#ifndef _ASM_X86_CONTEXT_TRACKING_WORK_H
#define _ASM_X86_CONTEXT_TRACKING_WORK_H
+#include <asm/sync_core.h>
+
static __always_inline void arch_context_tracking_work(enum ct_work work)
{
switch (work) {
- case CT_WORK_n:
- // Do work...
+ case CT_WORK_SYNC:
+ sync_core();
break;
case CT_WORK_MAX:
WARN_ON_ONCE(true);
diff --git a/arch/x86/include/asm/text-patching.h b/arch/x86/include/asm/text-patching.h
index f2d142a0a862e..ca989b0b6c0ae 100644
--- a/arch/x86/include/asm/text-patching.h
+++ b/arch/x86/include/asm/text-patching.h
@@ -33,6 +33,7 @@ extern void text_poke_apply_relocation(u8 *buf, const u8 * const instr, size_t i
*/
extern void *text_poke(void *addr, const void *opcode, size_t len);
extern void smp_text_poke_sync_each_cpu(void);
+extern void smp_text_poke_sync_each_cpu_deferrable(void);
extern void *text_poke_kgdb(void *addr, const void *opcode, size_t len);
extern void *text_poke_copy(void *addr, const void *opcode, size_t len);
#define text_poke_copy text_poke_copy
diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c
index 8ee5ff547357a..ce8989e02ae77 100644
--- a/arch/x86/kernel/alternative.c
+++ b/arch/x86/kernel/alternative.c
@@ -6,6 +6,7 @@
#include <linux/vmalloc.h>
#include <linux/memory.h>
#include <linux/execmem.h>
+#include <linux/context_tracking.h>
#include <asm/text-patching.h>
#include <asm/insn.h>
@@ -2708,9 +2709,24 @@ static void do_sync_core(void *info)
sync_core();
}
+static bool do_sync_core_defer_cond(int cpu, void *info)
+{
+ return !ct_set_cpu_work(cpu, CT_WORK_SYNC);
+}
+
+static void __smp_text_poke_sync_each_cpu(smp_cond_func_t cond_func)
+{
+ on_each_cpu_cond(cond_func, do_sync_core, NULL, 1);
+}
+
void smp_text_poke_sync_each_cpu(void)
{
- on_each_cpu(do_sync_core, NULL, 1);
+ __smp_text_poke_sync_each_cpu(NULL);
+}
+
+void smp_text_poke_sync_each_cpu_deferrable(void)
+{
+ __smp_text_poke_sync_each_cpu(do_sync_core_defer_cond);
}
/*
@@ -2880,6 +2896,7 @@ noinstr int smp_text_poke_int3_handler(struct pt_regs *regs)
*/
void smp_text_poke_batch_finish(void)
{
+ smp_cond_func_t cond = do_sync_core_defer_cond;
unsigned char int3 = INT3_INSN_OPCODE;
unsigned int i;
int do_sync;
@@ -2916,11 +2933,21 @@ void smp_text_poke_batch_finish(void)
* First step: add a INT3 trap to the address that will be patched.
*/
for (i = 0; i < text_poke_array.nr_entries; i++) {
- text_poke_array.vec[i].old = *(u8 *)text_poke_addr(&text_poke_array.vec[i]);
- text_poke(text_poke_addr(&text_poke_array.vec[i]), &int3, INT3_INSN_SIZE);
+ void *addr = text_poke_addr(&text_poke_array.vec[i]);
+
+ /*
+ * There's no safe way to defer IPIs for patching text in
+ * .noinstr, record whether there is at least one such poke.
+ */
+ if (is_kernel_noinstr_text((unsigned long)addr) ||
+ is_module_noinstr_text_address((unsigned long)addr))
+ cond = NULL;
+
+ text_poke_array.vec[i].old = *((u8 *)addr);
+ text_poke(addr, &int3, INT3_INSN_SIZE);
}
- smp_text_poke_sync_each_cpu();
+ __smp_text_poke_sync_each_cpu(cond);
/*
* Second step: update all but the first byte of the patched range.
@@ -2982,7 +3009,7 @@ void smp_text_poke_batch_finish(void)
* not necessary and we'd be safe even without it. But
* better safe than sorry (plus there's not only Intel).
*/
- smp_text_poke_sync_each_cpu();
+ __smp_text_poke_sync_each_cpu(cond);
}
/*
@@ -3003,7 +3030,7 @@ void smp_text_poke_batch_finish(void)
}
if (do_sync)
- smp_text_poke_sync_each_cpu();
+ __smp_text_poke_sync_each_cpu(cond);
/*
* Remove and wait for refs to be zero.
diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c
index 3863d7709386f..51957cd737f52 100644
--- a/arch/x86/kernel/kprobes/core.c
+++ b/arch/x86/kernel/kprobes/core.c
@@ -790,7 +790,7 @@ void arch_arm_kprobe(struct kprobe *p)
u8 int3 = INT3_INSN_OPCODE;
text_poke(p->addr, &int3, 1);
- smp_text_poke_sync_each_cpu();
+ smp_text_poke_sync_each_cpu_deferrable();
perf_event_text_poke(p->addr, &p->opcode, 1, &int3, 1);
}
@@ -800,7 +800,7 @@ void arch_disarm_kprobe(struct kprobe *p)
perf_event_text_poke(p->addr, &int3, 1, &p->opcode, 1);
text_poke(p->addr, &p->opcode, 1);
- smp_text_poke_sync_each_cpu();
+ smp_text_poke_sync_each_cpu_deferrable();
}
void arch_remove_kprobe(struct kprobe *p)
diff --git a/arch/x86/kernel/kprobes/opt.c b/arch/x86/kernel/kprobes/opt.c
index 0aabd4c4e2c4f..eada8dca1c2e8 100644
--- a/arch/x86/kernel/kprobes/opt.c
+++ b/arch/x86/kernel/kprobes/opt.c
@@ -513,11 +513,11 @@ void arch_unoptimize_kprobe(struct optimized_kprobe *op)
JMP32_INSN_SIZE - INT3_INSN_SIZE);
text_poke(addr, new, INT3_INSN_SIZE);
- smp_text_poke_sync_each_cpu();
+ smp_text_poke_sync_each_cpu_deferrable();
text_poke(addr + INT3_INSN_SIZE,
new + INT3_INSN_SIZE,
JMP32_INSN_SIZE - INT3_INSN_SIZE);
- smp_text_poke_sync_each_cpu();
+ smp_text_poke_sync_each_cpu_deferrable();
perf_event_text_poke(op->kp.addr, old, JMP32_INSN_SIZE, new, JMP32_INSN_SIZE);
}
diff --git a/arch/x86/kernel/module.c b/arch/x86/kernel/module.c
index 0ffbae902e2fe..c6c4f391eb465 100644
--- a/arch/x86/kernel/module.c
+++ b/arch/x86/kernel/module.c
@@ -206,7 +206,7 @@ static int write_relocate_add(Elf64_Shdr *sechdrs,
write, apply);
if (!early) {
- smp_text_poke_sync_each_cpu();
+ smp_text_poke_sync_each_cpu_deferrable();
mutex_unlock(&text_mutex);
}
diff --git a/include/asm-generic/sections.h b/include/asm-generic/sections.h
index 0755bc39b0d80..7d2403014010e 100644
--- a/include/asm-generic/sections.h
+++ b/include/asm-generic/sections.h
@@ -199,6 +199,21 @@ static inline bool is_kernel_inittext(unsigned long addr)
addr < (unsigned long)_einittext;
}
+
+/**
+ * is_kernel_noinstr_text - checks if the pointer address is located in the
+ * .noinstr section
+ *
+ * @addr: address to check
+ *
+ * Returns: true if the address is located in .noinstr, false otherwise.
+ */
+static inline bool is_kernel_noinstr_text(unsigned long addr)
+{
+ return addr >= (unsigned long)__noinstr_text_start &&
+ addr < (unsigned long)__noinstr_text_end;
+}
+
/**
* __is_kernel_text - checks if the pointer address is located in the
* .text section
diff --git a/include/linux/context_tracking_work.h b/include/linux/context_tracking_work.h
index 3742f461183ac..abd3c196855f4 100644
--- a/include/linux/context_tracking_work.h
+++ b/include/linux/context_tracking_work.h
@@ -5,12 +5,12 @@
#include <linux/bitops.h>
enum {
- CT_WORK_n_OFFSET,
+ CT_WORK_SYNC_OFFSET,
CT_WORK_MAX_OFFSET
};
enum ct_work {
- CT_WORK_n = BIT(CT_WORK_n_OFFSET),
+ CT_WORK_SYNC = BIT(CT_WORK_SYNC_OFFSET),
CT_WORK_MAX = BIT(CT_WORK_MAX_OFFSET)
};
--
2.51.0
^ permalink raw reply related
* [PATCH v7 24/31] context-tracking: Introduce work deferral infrastructure
From: Valentin Schneider @ 2025-11-14 15:14 UTC (permalink / raw)
To: linux-kernel, linux-mm, rcu, x86, linux-arm-kernel, loongarch,
linux-riscv, linux-arch, linux-trace-kernel
Cc: Nicolas Saenz Julienne, Thomas Gleixner, Ingo Molnar,
Borislav Petkov, Dave Hansen, H. Peter Anvin, Andy Lutomirski,
Peter Zijlstra, Arnaldo Carvalho de Melo, Josh Poimboeuf,
Paolo Bonzini, Arnd Bergmann, Frederic Weisbecker,
Paul E. McKenney, Jason Baron, Steven Rostedt, Ard Biesheuvel,
Sami Tolvanen, David S. Miller, Neeraj Upadhyay, Joel Fernandes,
Josh Triplett, Boqun Feng, Uladzislau Rezki, Mathieu Desnoyers,
Mel Gorman, Andrew Morton, Masahiro Yamada, Han Shen,
Rik van Riel, Jann Horn, Dan Carpenter, Oleg Nesterov, Juri Lelli,
Clark Williams, Yair Podemsky, Marcelo Tosatti, Daniel Wagner,
Petr Tesarik, Shrikanth Hegde
In-Reply-To: <20251114150133.1056710-1-vschneid@redhat.com>
smp_call_function() & friends have the unfortunate habit of sending IPIs to
isolated, NOHZ_FULL, in-userspace CPUs, as they blindly target all online
CPUs.
Some callsites can be bent into doing the right, such as done by commit:
cc9e303c91f5 ("x86/cpu: Disable frequency requests via aperfmperf IPI for nohz_full CPUs")
Unfortunately, not all SMP callbacks can be omitted in this
fashion. However, some of them only affect execution in kernelspace, which
means they don't have to be executed *immediately* if the target CPU is in
userspace: stashing the callback and executing it upon the next kernel entry
would suffice. x86 kernel instruction patching or kernel TLB invalidation
are prime examples of it.
Reduce the RCU dynticks counter width to free up some bits to be used as a
deferred callback bitmask. Add some build-time checks to validate that
setup.
Presence of CT_RCU_WATCHING in the ct_state prevents queuing deferred work.
Later commits introduce the bit:callback mappings.
Link: https://lore.kernel.org/all/20210929151723.162004989@infradead.org/
Signed-off-by: Nicolas Saenz Julienne <nsaenzju@redhat.com>
Signed-off-by: Valentin Schneider <vschneid@redhat.com>
---
arch/Kconfig | 9 +++
arch/x86/Kconfig | 1 +
arch/x86/include/asm/context_tracking_work.h | 16 +++++
include/linux/context_tracking.h | 21 ++++++
include/linux/context_tracking_state.h | 30 +++++---
include/linux/context_tracking_work.h | 24 +++++++
kernel/context_tracking.c | 72 +++++++++++++++++++-
kernel/time/Kconfig | 5 ++
8 files changed, 166 insertions(+), 12 deletions(-)
create mode 100644 arch/x86/include/asm/context_tracking_work.h
create mode 100644 include/linux/context_tracking_work.h
diff --git a/arch/Kconfig b/arch/Kconfig
index 61130b88964b9..6cc3965b8c9eb 100644
--- a/arch/Kconfig
+++ b/arch/Kconfig
@@ -1024,6 +1024,15 @@ config HAVE_CONTEXT_TRACKING_USER_OFFSTACK
- No use of instrumentation, unless instrumentation_begin() got
called.
+config HAVE_CONTEXT_TRACKING_WORK
+ bool
+ help
+ Architecture supports deferring work while not in kernel context.
+ This is especially useful on setups with isolated CPUs that might
+ want to avoid being interrupted to perform housekeeping tasks (for
+ ex. TLB invalidation or icache invalidation). The housekeeping
+ operations are performed upon re-entering the kernel.
+
config HAVE_TIF_NOHZ
bool
help
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index fa3b616af03a2..fa9229c0e0939 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -219,6 +219,7 @@ config X86
select HAVE_CMPXCHG_LOCAL
select HAVE_CONTEXT_TRACKING_USER if X86_64
select HAVE_CONTEXT_TRACKING_USER_OFFSTACK if HAVE_CONTEXT_TRACKING_USER
+ select HAVE_CONTEXT_TRACKING_WORK if X86_64
select HAVE_C_RECORDMCOUNT
select HAVE_OBJTOOL_MCOUNT if HAVE_OBJTOOL
select HAVE_OBJTOOL_NOP_MCOUNT if HAVE_OBJTOOL_MCOUNT
diff --git a/arch/x86/include/asm/context_tracking_work.h b/arch/x86/include/asm/context_tracking_work.h
new file mode 100644
index 0000000000000..5f3b2d0977235
--- /dev/null
+++ b/arch/x86/include/asm/context_tracking_work.h
@@ -0,0 +1,16 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_X86_CONTEXT_TRACKING_WORK_H
+#define _ASM_X86_CONTEXT_TRACKING_WORK_H
+
+static __always_inline void arch_context_tracking_work(enum ct_work work)
+{
+ switch (work) {
+ case CT_WORK_n:
+ // Do work...
+ break;
+ case CT_WORK_MAX:
+ WARN_ON_ONCE(true);
+ }
+}
+
+#endif
diff --git a/include/linux/context_tracking.h b/include/linux/context_tracking.h
index af9fe87a09225..0b0faa040e9b5 100644
--- a/include/linux/context_tracking.h
+++ b/include/linux/context_tracking.h
@@ -5,6 +5,7 @@
#include <linux/sched.h>
#include <linux/vtime.h>
#include <linux/context_tracking_state.h>
+#include <linux/context_tracking_work.h>
#include <linux/instrumentation.h>
#include <asm/ptrace.h>
@@ -137,6 +138,26 @@ static __always_inline unsigned long ct_state_inc(int incby)
return raw_atomic_add_return(incby, this_cpu_ptr(&context_tracking.state));
}
+#ifdef CONFIG_CONTEXT_TRACKING_WORK
+static __always_inline unsigned long ct_state_inc_clear_work(int incby)
+{
+ struct context_tracking *ct = this_cpu_ptr(&context_tracking);
+ unsigned long new, old, state;
+
+ state = arch_atomic_read(&ct->state);
+ do {
+ old = state;
+ new = old & ~CT_WORK_MASK;
+ new += incby;
+ state = arch_atomic_cmpxchg(&ct->state, old, new);
+ } while (old != state);
+
+ return new;
+}
+#else
+#define ct_state_inc_clear_work(x) ct_state_inc(x)
+#endif
+
static __always_inline bool warn_rcu_enter(void)
{
bool ret = false;
diff --git a/include/linux/context_tracking_state.h b/include/linux/context_tracking_state.h
index 0b81248aa03e2..d2c302133672f 100644
--- a/include/linux/context_tracking_state.h
+++ b/include/linux/context_tracking_state.h
@@ -5,6 +5,7 @@
#include <linux/percpu.h>
#include <linux/static_key.h>
#include <linux/context_tracking_irq.h>
+#include <linux/context_tracking_work.h>
/* Offset to allow distinguishing irq vs. task-based idle entry/exit. */
#define CT_NESTING_IRQ_NONIDLE ((LONG_MAX / 2) + 1)
@@ -39,16 +40,19 @@ struct context_tracking {
};
/*
- * We cram two different things within the same atomic variable:
+ * We cram up to three different things within the same atomic variable:
*
- * CT_RCU_WATCHING_START CT_STATE_START
- * | |
- * v v
- * MSB [ RCU watching counter ][ context_state ] LSB
- * ^ ^
- * | |
- * CT_RCU_WATCHING_END CT_STATE_END
+ * CT_RCU_WATCHING_START CT_STATE_START
+ * | CT_WORK_START |
+ * | | |
+ * v v v
+ * MSB [ RCU watching counter ][ context work ][ context_state ] LSB
+ * ^ ^ ^
+ * | | |
+ * | CT_WORK_END |
+ * CT_RCU_WATCHING_END CT_STATE_END
*
+ * The [ context work ] region spans 0 bits if CONFIG_CONTEXT_WORK=n
* Bits are used from the LSB upwards, so unused bits (if any) will always be in
* upper bits of the variable.
*/
@@ -59,18 +63,24 @@ struct context_tracking {
#define CT_STATE_START 0
#define CT_STATE_END (CT_STATE_START + CT_STATE_WIDTH - 1)
-#define CT_RCU_WATCHING_MAX_WIDTH (CT_SIZE - CT_STATE_WIDTH)
+#define CT_WORK_WIDTH (IS_ENABLED(CONFIG_CONTEXT_TRACKING_WORK) ? CT_WORK_MAX_OFFSET : 0)
+#define CT_WORK_START (CT_STATE_END + 1)
+#define CT_WORK_END (CT_WORK_START + CT_WORK_WIDTH - 1)
+
+#define CT_RCU_WATCHING_MAX_WIDTH (CT_SIZE - CT_WORK_WIDTH - CT_STATE_WIDTH)
#define CT_RCU_WATCHING_WIDTH (IS_ENABLED(CONFIG_RCU_DYNTICKS_TORTURE) ? 2 : CT_RCU_WATCHING_MAX_WIDTH)
-#define CT_RCU_WATCHING_START (CT_STATE_END + 1)
+#define CT_RCU_WATCHING_START (CT_WORK_END + 1)
#define CT_RCU_WATCHING_END (CT_RCU_WATCHING_START + CT_RCU_WATCHING_WIDTH - 1)
#define CT_RCU_WATCHING BIT(CT_RCU_WATCHING_START)
#define CT_STATE_MASK GENMASK(CT_STATE_END, CT_STATE_START)
+#define CT_WORK_MASK GENMASK(CT_WORK_END, CT_WORK_START)
#define CT_RCU_WATCHING_MASK GENMASK(CT_RCU_WATCHING_END, CT_RCU_WATCHING_START)
#define CT_UNUSED_WIDTH (CT_RCU_WATCHING_MAX_WIDTH - CT_RCU_WATCHING_WIDTH)
static_assert(CT_STATE_WIDTH +
+ CT_WORK_WIDTH +
CT_RCU_WATCHING_WIDTH +
CT_UNUSED_WIDTH ==
CT_SIZE);
diff --git a/include/linux/context_tracking_work.h b/include/linux/context_tracking_work.h
new file mode 100644
index 0000000000000..3742f461183ac
--- /dev/null
+++ b/include/linux/context_tracking_work.h
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_CONTEXT_TRACKING_WORK_H
+#define _LINUX_CONTEXT_TRACKING_WORK_H
+
+#include <linux/bitops.h>
+
+enum {
+ CT_WORK_n_OFFSET,
+ CT_WORK_MAX_OFFSET
+};
+
+enum ct_work {
+ CT_WORK_n = BIT(CT_WORK_n_OFFSET),
+ CT_WORK_MAX = BIT(CT_WORK_MAX_OFFSET)
+};
+
+#ifdef CONFIG_CONTEXT_TRACKING_WORK
+extern bool ct_set_cpu_work(unsigned int cpu, enum ct_work work);
+#else
+static inline bool
+ct_set_cpu_work(unsigned int cpu, unsigned int work) { return false; }
+#endif
+
+#endif
diff --git a/kernel/context_tracking.c b/kernel/context_tracking.c
index fb5be6e9b423f..5a3cc4ce4d8a6 100644
--- a/kernel/context_tracking.c
+++ b/kernel/context_tracking.c
@@ -25,6 +25,9 @@
#include <linux/kprobes.h>
#include <trace/events/rcu.h>
+#ifdef CONFIG_CONTEXT_TRACKING_WORK
+#include <asm/context_tracking_work.h>
+#endif
DEFINE_PER_CPU(struct context_tracking, context_tracking) = {
#ifdef CONFIG_CONTEXT_TRACKING_IDLE
@@ -72,6 +75,70 @@ static __always_inline void rcu_task_trace_heavyweight_exit(void)
#endif /* #ifdef CONFIG_TASKS_TRACE_RCU */
}
+#ifdef CONFIG_CONTEXT_TRACKING_WORK
+static noinstr void ct_work_flush(unsigned long seq)
+{
+ int bit;
+
+ seq = (seq & CT_WORK_MASK) >> CT_WORK_START;
+
+ /*
+ * arch_context_tracking_work() must be noinstr, non-blocking,
+ * and NMI safe.
+ */
+ for_each_set_bit(bit, &seq, CT_WORK_MAX)
+ arch_context_tracking_work(BIT(bit));
+}
+
+/**
+ * ct_set_cpu_work - set work to be run at next kernel context entry
+ *
+ * If @cpu is not currently executing in kernelspace, it will execute the
+ * callback mapped to @work (see arch_context_tracking_work()) at its next
+ * entry into ct_kernel_enter_state().
+ *
+ * If it is already executing in kernelspace, this will be a no-op.
+ */
+bool ct_set_cpu_work(unsigned int cpu, enum ct_work work)
+{
+ struct context_tracking *ct = per_cpu_ptr(&context_tracking, cpu);
+ unsigned int old;
+ bool ret = false;
+
+ if (!ct->active)
+ return false;
+
+ preempt_disable();
+
+ old = atomic_read(&ct->state);
+
+ /*
+ * The work bit must only be set if the target CPU is not executing
+ * in kernelspace.
+ * CT_RCU_WATCHING is used as a proxy for that - if the bit is set, we
+ * know for sure the CPU is executing in the kernel whether that be in
+ * NMI, IRQ or process context.
+ * Clear CT_RCU_WATCHING here and let the cmpxchg do the check for us;
+ * the state could change between the atomic_read() and the cmpxchg().
+ */
+ old &= ~CT_RCU_WATCHING;
+ /*
+ * Try setting the work until either
+ * - the target CPU has entered kernelspace
+ * - the work has been set
+ */
+ do {
+ ret = atomic_try_cmpxchg(&ct->state, &old, old | (work << CT_WORK_START));
+ } while (!ret && !(old & CT_RCU_WATCHING));
+
+ preempt_enable();
+ return ret;
+}
+#else
+static __always_inline void ct_work_flush(unsigned long work) { }
+static __always_inline void ct_work_clear(struct context_tracking *ct) { }
+#endif
+
/*
* Record entry into an extended quiescent state. This is only to be
* called when not already in an extended quiescent state, that is,
@@ -88,7 +155,7 @@ static noinstr void ct_kernel_exit_state(int offset)
rcu_task_trace_heavyweight_enter(); // Before CT state update!
// RCU is still watching. Better not be in extended quiescent state!
WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !rcu_is_watching_curr_cpu());
- (void)ct_state_inc(offset);
+ (void)ct_state_inc_clear_work(offset);
// RCU is no longer watching.
}
@@ -99,7 +166,7 @@ static noinstr void ct_kernel_exit_state(int offset)
*/
static noinstr void ct_kernel_enter_state(int offset)
{
- int seq;
+ unsigned long seq;
/*
* CPUs seeing atomic_add_return() must see prior idle sojourns,
@@ -107,6 +174,7 @@ static noinstr void ct_kernel_enter_state(int offset)
* critical section.
*/
seq = ct_state_inc(offset);
+ ct_work_flush(seq);
// RCU is now watching. Better not be in an extended quiescent state!
rcu_task_trace_heavyweight_exit(); // After CT state update!
WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !(seq & CT_RCU_WATCHING));
diff --git a/kernel/time/Kconfig b/kernel/time/Kconfig
index 7c6a52f7836ce..1a0c027aad141 100644
--- a/kernel/time/Kconfig
+++ b/kernel/time/Kconfig
@@ -181,6 +181,11 @@ config CONTEXT_TRACKING_USER_FORCE
Say N otherwise, this option brings an overhead that you
don't want in production.
+config CONTEXT_TRACKING_WORK
+ bool
+ depends on HAVE_CONTEXT_TRACKING_WORK && CONTEXT_TRACKING_USER
+ default y
+
config NO_HZ
bool "Old Idle dynticks config"
help
--
2.51.0
^ permalink raw reply related
* [PATCH v7 23/31] module: Add MOD_NOINSTR_TEXT mem_type
From: Valentin Schneider @ 2025-11-14 15:14 UTC (permalink / raw)
To: linux-kernel, linux-mm, rcu, x86, linux-arm-kernel, loongarch,
linux-riscv, linux-arch, linux-trace-kernel
Cc: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen,
H. Peter Anvin, Andy Lutomirski, Peter Zijlstra,
Arnaldo Carvalho de Melo, Josh Poimboeuf, Paolo Bonzini,
Arnd Bergmann, Frederic Weisbecker, Paul E. McKenney, Jason Baron,
Steven Rostedt, Ard Biesheuvel, Sami Tolvanen, David S. Miller,
Neeraj Upadhyay, Joel Fernandes, Josh Triplett, Boqun Feng,
Uladzislau Rezki, Mathieu Desnoyers, Mel Gorman, Andrew Morton,
Masahiro Yamada, Han Shen, Rik van Riel, Jann Horn, Dan Carpenter,
Oleg Nesterov, Juri Lelli, Clark Williams, Yair Podemsky,
Marcelo Tosatti, Daniel Wagner, Petr Tesarik, Shrikanth Hegde
In-Reply-To: <20251114150133.1056710-1-vschneid@redhat.com>
As pointed out by Sean [1], is_kernel_noinstr_text() will return false for
an address contained within a module's .noinstr.text section. A later patch
will require checking whether a text address is noinstr, and this can
unfortunately be the case of modules - KVM is one such case.
A module's .noinstr.text section is already tracked as of commit
66e9b0717102 ("kprobes: Prevent probes in .noinstr.text section")
for kprobe blacklisting purposes, but via an ad-hoc mechanism.
Add a MOD_NOINSTR_TEXT mem_type, and reorganize __layout_sections() so that
it maps all the sections in a single invocation.
[1]: http://lore.kernel.org/r/Z4qQL89GZ_gk0vpu@google.com
Signed-off-by: Valentin Schneider <vschneid@redhat.com>
---
include/linux/module.h | 6 ++--
kernel/kprobes.c | 8 ++---
kernel/module/main.c | 76 ++++++++++++++++++++++++++++++++----------
3 files changed, 66 insertions(+), 24 deletions(-)
diff --git a/include/linux/module.h b/include/linux/module.h
index e135cc79aceea..c0911973337c6 100644
--- a/include/linux/module.h
+++ b/include/linux/module.h
@@ -322,6 +322,7 @@ struct mod_tree_node {
enum mod_mem_type {
MOD_TEXT = 0,
+ MOD_NOINSTR_TEXT,
MOD_DATA,
MOD_RODATA,
MOD_RO_AFTER_INIT,
@@ -492,8 +493,6 @@ struct module {
void __percpu *percpu;
unsigned int percpu_size;
#endif
- void *noinstr_text_start;
- unsigned int noinstr_text_size;
#ifdef CONFIG_TRACEPOINTS
unsigned int num_tracepoints;
@@ -622,12 +621,13 @@ static inline bool module_is_coming(struct module *mod)
return mod->state == MODULE_STATE_COMING;
}
-struct module *__module_text_address(unsigned long addr);
struct module *__module_address(unsigned long addr);
+struct module *__module_text_address(unsigned long addr);
bool is_module_address(unsigned long addr);
bool __is_module_percpu_address(unsigned long addr, unsigned long *can_addr);
bool is_module_percpu_address(unsigned long addr);
bool is_module_text_address(unsigned long addr);
+bool is_module_noinstr_text_address(unsigned long addr);
static inline bool within_module_mem_type(unsigned long addr,
const struct module *mod,
diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index ab8f9fc1f0d17..d60560dddec56 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -2551,9 +2551,9 @@ static void add_module_kprobe_blacklist(struct module *mod)
kprobe_add_area_blacklist(start, end);
}
- start = (unsigned long)mod->noinstr_text_start;
+ start = (unsigned long)mod->mem[MOD_NOINSTR_TEXT].base;
if (start) {
- end = start + mod->noinstr_text_size;
+ end = start + mod->mem[MOD_NOINSTR_TEXT].size;
kprobe_add_area_blacklist(start, end);
}
}
@@ -2574,9 +2574,9 @@ static void remove_module_kprobe_blacklist(struct module *mod)
kprobe_remove_area_blacklist(start, end);
}
- start = (unsigned long)mod->noinstr_text_start;
+ start = (unsigned long)mod->mem[MOD_NOINSTR_TEXT].base;
if (start) {
- end = start + mod->noinstr_text_size;
+ end = start + mod->mem[MOD_NOINSTR_TEXT].size;
kprobe_remove_area_blacklist(start, end);
}
}
diff --git a/kernel/module/main.c b/kernel/module/main.c
index c66b261849362..1f5bfdbb956a7 100644
--- a/kernel/module/main.c
+++ b/kernel/module/main.c
@@ -1653,7 +1653,17 @@ bool module_init_layout_section(const char *sname)
return module_init_section(sname);
}
-static void __layout_sections(struct module *mod, struct load_info *info, bool is_init)
+static bool module_noinstr_layout_section(const char *sname)
+{
+ return strstarts(sname, ".noinstr");
+}
+
+static bool module_default_layout_section(const char *sname)
+{
+ return !module_init_layout_section(sname) && !module_noinstr_layout_section(sname);
+}
+
+static void __layout_sections(struct module *mod, struct load_info *info)
{
unsigned int m, i;
@@ -1662,20 +1672,44 @@ static void __layout_sections(struct module *mod, struct load_info *info, bool i
* Mask of excluded section header flags }
*/
static const unsigned long masks[][2] = {
+ /* Core */
+ { SHF_EXECINSTR | SHF_ALLOC, ARCH_SHF_SMALL },
+ { SHF_EXECINSTR | SHF_ALLOC, ARCH_SHF_SMALL },
+ { SHF_ALLOC, SHF_WRITE | ARCH_SHF_SMALL },
+ { SHF_RO_AFTER_INIT | SHF_ALLOC, ARCH_SHF_SMALL },
+ { SHF_WRITE | SHF_ALLOC, ARCH_SHF_SMALL },
+ { ARCH_SHF_SMALL | SHF_ALLOC, 0 },
+ /* Init */
{ SHF_EXECINSTR | SHF_ALLOC, ARCH_SHF_SMALL },
{ SHF_ALLOC, SHF_WRITE | ARCH_SHF_SMALL },
{ SHF_RO_AFTER_INIT | SHF_ALLOC, ARCH_SHF_SMALL },
{ SHF_WRITE | SHF_ALLOC, ARCH_SHF_SMALL },
- { ARCH_SHF_SMALL | SHF_ALLOC, 0 }
+ { ARCH_SHF_SMALL | SHF_ALLOC, 0 },
};
- static const int core_m_to_mem_type[] = {
+ static bool (*const section_filter[])(const char *) = {
+ /* Core */
+ module_default_layout_section,
+ module_noinstr_layout_section,
+ module_default_layout_section,
+ module_default_layout_section,
+ module_default_layout_section,
+ module_default_layout_section,
+ /* Init */
+ module_init_layout_section,
+ module_init_layout_section,
+ module_init_layout_section,
+ module_init_layout_section,
+ module_init_layout_section,
+ };
+ static const int mem_type_map[] = {
+ /* Core */
MOD_TEXT,
+ MOD_NOINSTR_TEXT,
MOD_RODATA,
MOD_RO_AFTER_INIT,
MOD_DATA,
MOD_DATA,
- };
- static const int init_m_to_mem_type[] = {
+ /* Init */
MOD_INIT_TEXT,
MOD_INIT_RODATA,
MOD_INVALID,
@@ -1684,16 +1718,16 @@ static void __layout_sections(struct module *mod, struct load_info *info, bool i
};
for (m = 0; m < ARRAY_SIZE(masks); ++m) {
- enum mod_mem_type type = is_init ? init_m_to_mem_type[m] : core_m_to_mem_type[m];
+ enum mod_mem_type type = mem_type_map[m];
for (i = 0; i < info->hdr->e_shnum; ++i) {
Elf_Shdr *s = &info->sechdrs[i];
const char *sname = info->secstrings + s->sh_name;
- if ((s->sh_flags & masks[m][0]) != masks[m][0]
- || (s->sh_flags & masks[m][1])
- || s->sh_entsize != ~0UL
- || is_init != module_init_layout_section(sname))
+ if ((s->sh_flags & masks[m][0]) != masks[m][0] ||
+ (s->sh_flags & masks[m][1]) ||
+ s->sh_entsize != ~0UL ||
+ !section_filter[m](sname))
continue;
if (WARN_ON_ONCE(type == MOD_INVALID))
@@ -1733,10 +1767,7 @@ static void layout_sections(struct module *mod, struct load_info *info)
info->sechdrs[i].sh_entsize = ~0UL;
pr_debug("Core section allocation order for %s:\n", mod->name);
- __layout_sections(mod, info, false);
-
- pr_debug("Init section allocation order for %s:\n", mod->name);
- __layout_sections(mod, info, true);
+ __layout_sections(mod, info);
}
static void module_license_taint_check(struct module *mod, const char *license)
@@ -2625,9 +2656,6 @@ static int find_module_sections(struct module *mod, struct load_info *info)
}
#endif
- mod->noinstr_text_start = section_objs(info, ".noinstr.text", 1,
- &mod->noinstr_text_size);
-
#ifdef CONFIG_TRACEPOINTS
mod->tracepoints_ptrs = section_objs(info, "__tracepoints_ptrs",
sizeof(*mod->tracepoints_ptrs),
@@ -3872,12 +3900,26 @@ struct module *__module_text_address(unsigned long addr)
if (mod) {
/* Make sure it's within the text section. */
if (!within_module_mem_type(addr, mod, MOD_TEXT) &&
+ !within_module_mem_type(addr, mod, MOD_NOINSTR_TEXT) &&
!within_module_mem_type(addr, mod, MOD_INIT_TEXT))
mod = NULL;
}
return mod;
}
+bool is_module_noinstr_text_address(unsigned long addr)
+{
+ scoped_guard(preempt) {
+ struct module *mod = __module_address(addr);
+
+ /* Make sure it's within the .noinstr.text section. */
+ if (mod)
+ return within_module_mem_type(addr, mod, MOD_NOINSTR_TEXT);
+ }
+
+ return false;
+}
+
/* Don't grab lock, we're oopsing. */
void print_modules(void)
{
--
2.51.0
^ permalink raw reply related
* [PATCH v7 22/31] objtool: Add noinstr validation for static branches/calls
From: Valentin Schneider @ 2025-11-14 15:14 UTC (permalink / raw)
To: linux-kernel, linux-mm, rcu, x86, linux-arm-kernel, loongarch,
linux-riscv, linux-arch, linux-trace-kernel
Cc: Josh Poimboeuf, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
Dave Hansen, H. Peter Anvin, Andy Lutomirski, Peter Zijlstra,
Arnaldo Carvalho de Melo, Paolo Bonzini, Arnd Bergmann,
Frederic Weisbecker, Paul E. McKenney, Jason Baron,
Steven Rostedt, Ard Biesheuvel, Sami Tolvanen, David S. Miller,
Neeraj Upadhyay, Joel Fernandes, Josh Triplett, Boqun Feng,
Uladzislau Rezki, Mathieu Desnoyers, Mel Gorman, Andrew Morton,
Masahiro Yamada, Han Shen, Rik van Riel, Jann Horn, Dan Carpenter,
Oleg Nesterov, Juri Lelli, Clark Williams, Yair Podemsky,
Marcelo Tosatti, Daniel Wagner, Petr Tesarik, Shrikanth Hegde
In-Reply-To: <20251114150133.1056710-1-vschneid@redhat.com>
From: Josh Poimboeuf <jpoimboe@kernel.org>
Warn about static branches/calls in noinstr regions, unless the
corresponding key is RO-after-init or has been manually whitelisted with
DEFINE_STATIC_KEY_*_NOINSTR(().
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
Signed-off-by: Valentin Schneider <vschneid@redhat.com>
---
include/linux/jump_label.h | 17 +++--
include/linux/objtool.h | 14 ++++
include/linux/static_call.h | 3 +
tools/objtool/Documentation/objtool.txt | 34 +++++++++
tools/objtool/check.c | 92 ++++++++++++++++++++++---
tools/objtool/include/objtool/check.h | 1 +
tools/objtool/include/objtool/elf.h | 1 +
tools/objtool/include/objtool/special.h | 1 +
tools/objtool/special.c | 15 +++-
9 files changed, 162 insertions(+), 16 deletions(-)
diff --git a/include/linux/jump_label.h b/include/linux/jump_label.h
index c4f6240ff4d95..0ea203ebbc493 100644
--- a/include/linux/jump_label.h
+++ b/include/linux/jump_label.h
@@ -76,6 +76,7 @@
#include <linux/types.h>
#include <linux/compiler.h>
#include <linux/cleanup.h>
+#include <linux/objtool.h>
extern bool static_key_initialized;
@@ -376,8 +377,9 @@ struct static_key_false {
#define DEFINE_STATIC_KEY_TRUE(name) \
struct static_key_true name = STATIC_KEY_TRUE_INIT
-#define DEFINE_STATIC_KEY_TRUE_RO(name) \
- struct static_key_true name __ro_after_init = STATIC_KEY_TRUE_INIT
+#define DEFINE_STATIC_KEY_TRUE_RO(name) \
+ struct static_key_true name __ro_after_init = STATIC_KEY_TRUE_INIT; \
+ ANNOTATE_NOINSTR_ALLOWED(name)
#define DECLARE_STATIC_KEY_TRUE(name) \
extern struct static_key_true name
@@ -385,8 +387,9 @@ struct static_key_false {
#define DEFINE_STATIC_KEY_FALSE(name) \
struct static_key_false name = STATIC_KEY_FALSE_INIT
-#define DEFINE_STATIC_KEY_FALSE_RO(name) \
- struct static_key_false name __ro_after_init = STATIC_KEY_FALSE_INIT
+#define DEFINE_STATIC_KEY_FALSE_RO(name) \
+ struct static_key_false name __ro_after_init = STATIC_KEY_FALSE_INIT; \
+ ANNOTATE_NOINSTR_ALLOWED(name)
/*
* The _NOINSTR variants are used to tell objtool the static key is allowed to
@@ -400,10 +403,12 @@ struct static_key_false {
* definition with the rationale.
*/
#define DEFINE_STATIC_KEY_TRUE_NOINSTR(name) \
- DEFINE_STATIC_KEY_TRUE(name)
+ DEFINE_STATIC_KEY_TRUE(name); \
+ ANNOTATE_NOINSTR_ALLOWED(name)
#define DEFINE_STATIC_KEY_FALSE_NOINSTR(name) \
- DEFINE_STATIC_KEY_FALSE(name)
+ DEFINE_STATIC_KEY_FALSE(name); \
+ ANNOTATE_NOINSTR_ALLOWED(name)
#define DECLARE_STATIC_KEY_FALSE(name) \
extern struct static_key_false name
diff --git a/include/linux/objtool.h b/include/linux/objtool.h
index 46ebaa46e6c58..78b73bf65d9a2 100644
--- a/include/linux/objtool.h
+++ b/include/linux/objtool.h
@@ -34,6 +34,19 @@
static void __used __section(".discard.func_stack_frame_non_standard") \
*__func_stack_frame_non_standard_##func = func
+#define __ANNOTATE_NOINSTR_ALLOWED(key) \
+ static void __used __section(".discard.noinstr_allowed") \
+ *__annotate_noinstr_allowed_##key = &key
+
+/*
+ * This is used to tell objtool that a given static key is safe to be used
+ * within .noinstr code, and it doesn't need to generate a warning about it.
+ *
+ * For more information, see tools/objtool/Documentation/objtool.txt,
+ * "non-RO static key usage in noinstr code"
+ */
+#define ANNOTATE_NOINSTR_ALLOWED(key) __ANNOTATE_NOINSTR_ALLOWED(key)
+
/*
* STACK_FRAME_NON_STANDARD_FP() is a frame-pointer-specific function ignore
* for the case where a function is intentionally missing frame pointer setup,
@@ -130,6 +143,7 @@
#define STACK_FRAME_NON_STANDARD_FP(func)
#define __ASM_ANNOTATE(label, type) ""
#define ASM_ANNOTATE(type)
+#define ANNOTATE_NOINSTR_ALLOWED(key)
#else
.macro UNWIND_HINT type:req sp_reg=0 sp_offset=0 signal=0
.endm
diff --git a/include/linux/static_call.h b/include/linux/static_call.h
index ea6ca57e2a829..0d4b16d348501 100644
--- a/include/linux/static_call.h
+++ b/include/linux/static_call.h
@@ -133,6 +133,7 @@
#include <linux/types.h>
#include <linux/cpu.h>
+#include <linux/objtool.h>
#include <linux/static_call_types.h>
#ifdef CONFIG_HAVE_STATIC_CALL
@@ -198,6 +199,7 @@ extern long __static_call_return0(void);
.func = _func, \
.type = 1, \
}; \
+ ANNOTATE_NOINSTR_ALLOWED(STATIC_CALL_TRAMP(name)); \
ARCH_DEFINE_STATIC_CALL_TRAMP(name, _func)
#define DEFINE_STATIC_CALL_NULL(name, _func) \
@@ -214,6 +216,7 @@ extern long __static_call_return0(void);
.func = NULL, \
.type = 1, \
}; \
+ ANNOTATE_NOINSTR_ALLOWED(STATIC_CALL_TRAMP(name)); \
ARCH_DEFINE_STATIC_CALL_NULL_TRAMP(name)
#define DEFINE_STATIC_CALL_RET0(name, _func) \
diff --git a/tools/objtool/Documentation/objtool.txt b/tools/objtool/Documentation/objtool.txt
index 9e97fc25b2d8a..991e085e10d95 100644
--- a/tools/objtool/Documentation/objtool.txt
+++ b/tools/objtool/Documentation/objtool.txt
@@ -456,6 +456,40 @@ the objtool maintainers.
these special names and does not use module_init() / module_exit()
macros to create them.
+13. file.o: warning: func()+0x2a: key: non-RO static key usage in noinstr code
+ file.o: warning: func()+0x2a: key: non-RO static call usage in noinstr code
+
+ This means that noinstr function func() uses a static key or
+ static call named 'key' which can be modified at runtime. This is
+ discouraged because it prevents code patching IPIs from being
+ deferred.
+
+ You have the following options:
+
+ 1) Check whether the static key/call in question is only modified
+ during init. If so, define it as read-only-after-init with
+ DEFINE_STATIC_KEY_*_RO() or DEFINE_STATIC_CALL_RO().
+
+ 2) Avoid the runtime patching. For static keys this can be done by
+ using static_key_enabled() or by getting rid of the static key
+ altogether if performance is not a concern.
+
+ For static calls, something like the following could be done:
+
+ target = static_call_query(foo);
+ if (target == func1)
+ func1();
+ else if (target == func2)
+ func2();
+ ...
+
+ 3) Silence the warning by defining the static key/call with
+ DEFINE_STATIC_*_NOINSTR(). This decision should not
+ be taken lightly as it may result in code patching IPIs getting
+ sent to isolated NOHZ_FULL CPUs running in pure userspace. A
+ comment should be added above the definition explaining the
+ rationale for the decision.
+
If the error doesn't seem to make sense, it could be a bug in objtool.
Feel free to ask objtool maintainers for help.
diff --git a/tools/objtool/check.c b/tools/objtool/check.c
index 1efa9f1bf16ba..474ba2fc87ac6 100644
--- a/tools/objtool/check.c
+++ b/tools/objtool/check.c
@@ -982,6 +982,45 @@ static int create_direct_call_sections(struct objtool_file *file)
return 0;
}
+static int read_noinstr_allowed(struct objtool_file *file)
+{
+ struct section *rsec;
+ struct symbol *sym;
+ struct reloc *reloc;
+
+ rsec = find_section_by_name(file->elf, ".rela.discard.noinstr_allowed");
+ if (!rsec)
+ return 0;
+
+ for_each_reloc(rsec, reloc) {
+ switch (reloc->sym->type) {
+ case STT_OBJECT:
+ case STT_FUNC:
+ sym = reloc->sym;
+ break;
+
+ case STT_SECTION:
+ sym = find_symbol_by_offset(reloc->sym->sec,
+ reloc_addend(reloc));
+ if (!sym) {
+ WARN_FUNC(reloc->sym->sec, reloc_addend(reloc),
+ "can't find static key/call symbol");
+ return -1;
+ }
+ break;
+
+ default:
+ WARN("unexpected relocation symbol type in %s: %d",
+ rsec->name, reloc->sym->type);
+ return -1;
+ }
+
+ sym->noinstr_allowed = 1;
+ }
+
+ return 0;
+}
+
/*
* Warnings shouldn't be reported for ignored functions.
*/
@@ -1868,6 +1907,8 @@ static int handle_jump_alt(struct objtool_file *file,
return -1;
}
+ orig_insn->key = special_alt->key;
+
if (opts.hack_jump_label && special_alt->key_addend & 2) {
struct reloc *reloc = insn_reloc(file, orig_insn);
@@ -2602,6 +2643,10 @@ static int decode_sections(struct objtool_file *file)
if (ret)
return ret;
+ ret = read_noinstr_allowed(file);
+ if (ret)
+ return ret;
+
return 0;
}
@@ -3371,9 +3416,9 @@ static bool pv_call_dest(struct objtool_file *file, struct instruction *insn)
return file->pv_ops[idx].clean;
}
-static inline bool noinstr_call_dest(struct objtool_file *file,
- struct instruction *insn,
- struct symbol *func)
+static inline bool noinstr_call_allowed(struct objtool_file *file,
+ struct instruction *insn,
+ struct symbol *func)
{
/*
* We can't deal with indirect function calls at present;
@@ -3393,10 +3438,10 @@ static inline bool noinstr_call_dest(struct objtool_file *file,
return true;
/*
- * If the symbol is a static_call trampoline, we can't tell.
+ * Only DEFINE_STATIC_CALL_*_RO allowed.
*/
if (func->static_call_tramp)
- return true;
+ return func->noinstr_allowed;
/*
* The __ubsan_handle_*() calls are like WARN(), they only happen when
@@ -3409,14 +3454,29 @@ static inline bool noinstr_call_dest(struct objtool_file *file,
return false;
}
+static char *static_call_name(struct symbol *func)
+{
+ return func->name + strlen("__SCT__");
+}
+
static int validate_call(struct objtool_file *file,
struct instruction *insn,
struct insn_state *state)
{
- if (state->noinstr && state->instr <= 0 &&
- !noinstr_call_dest(file, insn, insn_call_dest(insn))) {
- WARN_INSN(insn, "call to %s() leaves .noinstr.text section", call_dest_name(file, insn));
- return 1;
+ if (state->noinstr && state->instr <= 0) {
+ struct symbol *dest = insn_call_dest(insn);
+
+ if (dest && dest->static_call_tramp) {
+ if (!dest->noinstr_allowed) {
+ WARN_INSN(insn, "%s: non-RO static call usage in noinstr",
+ static_call_name(dest));
+ }
+
+ } else if (dest && !noinstr_call_allowed(file, insn, dest)) {
+ WARN_INSN(insn, "call to %s() leaves .noinstr.text section",
+ call_dest_name(file, insn));
+ return 1;
+ }
}
if (state->uaccess && !func_uaccess_safe(insn_call_dest(insn))) {
@@ -3481,6 +3541,17 @@ static int validate_return(struct symbol *func, struct instruction *insn, struct
return 0;
}
+static int validate_static_key(struct instruction *insn, struct insn_state *state)
+{
+ if (state->noinstr && state->instr <= 0 && !insn->key->noinstr_allowed) {
+ WARN_INSN(insn, "%s: non-RO static key usage in noinstr",
+ insn->key->name);
+ return 1;
+ }
+
+ return 0;
+}
+
static struct instruction *next_insn_to_validate(struct objtool_file *file,
struct instruction *insn)
{
@@ -3673,6 +3744,9 @@ static int validate_branch(struct objtool_file *file, struct symbol *func,
if (handle_insn_ops(insn, next_insn, &state))
return 1;
+ if (insn->key)
+ validate_static_key(insn, &state);
+
switch (insn->type) {
case INSN_RETURN:
diff --git a/tools/objtool/include/objtool/check.h b/tools/objtool/include/objtool/check.h
index 00fb745e72339..d79b08f55bcbc 100644
--- a/tools/objtool/include/objtool/check.h
+++ b/tools/objtool/include/objtool/check.h
@@ -81,6 +81,7 @@ struct instruction {
struct symbol *sym;
struct stack_op *stack_ops;
struct cfi_state *cfi;
+ struct symbol *key;
};
static inline struct symbol *insn_func(struct instruction *insn)
diff --git a/tools/objtool/include/objtool/elf.h b/tools/objtool/include/objtool/elf.h
index df8434d3b7440..545deba57266a 100644
--- a/tools/objtool/include/objtool/elf.h
+++ b/tools/objtool/include/objtool/elf.h
@@ -71,6 +71,7 @@ struct symbol {
u8 frame_pointer : 1;
u8 ignore : 1;
u8 nocfi : 1;
+ u8 noinstr_allowed : 1;
struct list_head pv_target;
struct reloc *relocs;
struct section *group_sec;
diff --git a/tools/objtool/include/objtool/special.h b/tools/objtool/include/objtool/special.h
index 72d09c0adf1a1..e84d704f3f20e 100644
--- a/tools/objtool/include/objtool/special.h
+++ b/tools/objtool/include/objtool/special.h
@@ -18,6 +18,7 @@ struct special_alt {
bool group;
bool jump_or_nop;
u8 key_addend;
+ struct symbol *key;
struct section *orig_sec;
unsigned long orig_off;
diff --git a/tools/objtool/special.c b/tools/objtool/special.c
index c80fed8a840ee..d77f3fa4bbbc9 100644
--- a/tools/objtool/special.c
+++ b/tools/objtool/special.c
@@ -110,13 +110,26 @@ static int get_alt_entry(struct elf *elf, const struct special_entry *entry,
if (entry->key) {
struct reloc *key_reloc;
+ struct symbol *key;
+ s64 key_addend;
key_reloc = find_reloc_by_dest(elf, sec, offset + entry->key);
if (!key_reloc) {
ERROR_FUNC(sec, offset + entry->key, "can't find key reloc");
return -1;
}
- alt->key_addend = reloc_addend(key_reloc);
+
+ key = key_reloc->sym;
+ key_addend = reloc_addend(key_reloc);
+
+ if (key->type == STT_SECTION)
+ key = find_symbol_by_offset(key->sec, key_addend & ~3);
+
+ /* embedded keys not supported */
+ if (key) {
+ alt->key = key;
+ alt->key_addend = key_addend;
+ }
}
return 0;
--
2.51.0
^ permalink raw reply related
* [PATCH v7 21/31] stackleack: Mark stack_erasing_bypass key as allowed in .noinstr
From: Valentin Schneider @ 2025-11-14 15:14 UTC (permalink / raw)
To: linux-kernel, linux-mm, rcu, x86, linux-arm-kernel, loongarch,
linux-riscv, linux-arch, linux-trace-kernel
Cc: Josh Poimboeuf, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
Dave Hansen, H. Peter Anvin, Andy Lutomirski, Peter Zijlstra,
Arnaldo Carvalho de Melo, Paolo Bonzini, Arnd Bergmann,
Frederic Weisbecker, Paul E. McKenney, Jason Baron,
Steven Rostedt, Ard Biesheuvel, Sami Tolvanen, David S. Miller,
Neeraj Upadhyay, Joel Fernandes, Josh Triplett, Boqun Feng,
Uladzislau Rezki, Mathieu Desnoyers, Mel Gorman, Andrew Morton,
Masahiro Yamada, Han Shen, Rik van Riel, Jann Horn, Dan Carpenter,
Oleg Nesterov, Juri Lelli, Clark Williams, Yair Podemsky,
Marcelo Tosatti, Daniel Wagner, Petr Tesarik, Shrikanth Hegde
In-Reply-To: <20251114150133.1056710-1-vschneid@redhat.com>
Later commits will cause objtool to warn about static keys being used in
.noinstr sections in order to safely defer instruction patching IPIs
targeted at NOHZ_FULL CPUs.
stack_erasing_bypass is used in .noinstr code, and can be modified at runtime
(proc/sys/kernel/stack_erasing write). However it is not expected that it
will be flipped during latency-sensitive operations, and thus shouldn't be
a source of interference wrt the text patching IPI.
Mark it to let objtool know not to warn about it.
Reported-by: Josh Poimboeuf <jpoimboe@kernel.org>
Signed-off-by: Valentin Schneider <vschneid@redhat.com>
---
kernel/kstack_erase.c | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/kernel/kstack_erase.c b/kernel/kstack_erase.c
index e49bb88b4f0a3..99ba44cf939bf 100644
--- a/kernel/kstack_erase.c
+++ b/kernel/kstack_erase.c
@@ -19,7 +19,11 @@
#include <linux/sysctl.h>
#include <linux/init.h>
-static DEFINE_STATIC_KEY_FALSE(stack_erasing_bypass);
+/*
+ * NOINSTR: This static key can only be modified via its sysctl interface. It is
+ * expected it will remain stable during latency-senstive operations.
+ */
+static DEFINE_STATIC_KEY_FALSE_NOINSTR(stack_erasing_bypass);
#ifdef CONFIG_SYSCTL
static int stack_erasing_sysctl(const struct ctl_table *table, int write,
--
2.51.0
^ permalink raw reply related
* [PATCH v7 20/31] KVM: VMX: Mark vmx_l1d_should flush and vmx_l1d_flush_cond keys as allowed in .noinstr
From: Valentin Schneider @ 2025-11-14 15:10 UTC (permalink / raw)
To: linux-kernel, linux-mm, rcu, x86, linux-arm-kernel, loongarch,
linux-riscv, linux-arch, linux-trace-kernel
Cc: Josh Poimboeuf, Sean Christopherson, Thomas Gleixner, Ingo Molnar,
Borislav Petkov, Dave Hansen, H. Peter Anvin, Andy Lutomirski,
Peter Zijlstra, Arnaldo Carvalho de Melo, Paolo Bonzini,
Arnd Bergmann, Frederic Weisbecker, Paul E. McKenney, Jason Baron,
Steven Rostedt, Ard Biesheuvel, Sami Tolvanen, David S. Miller,
Neeraj Upadhyay, Joel Fernandes, Josh Triplett, Boqun Feng,
Uladzislau Rezki, Mathieu Desnoyers, Mel Gorman, Andrew Morton,
Masahiro Yamada, Han Shen, Rik van Riel, Jann Horn, Dan Carpenter,
Oleg Nesterov, Juri Lelli, Clark Williams, Yair Podemsky,
Marcelo Tosatti, Daniel Wagner, Petr Tesarik, Shrikanth Hegde
In-Reply-To: <20251114150133.1056710-1-vschneid@redhat.com>
Later commits will cause objtool to warn about static keys being used in
.noinstr sections in order to safely defer instruction patching IPIs
targeted at NOHZ_FULL CPUs.
The VMX keys are used in .noinstr code, and can be modified at runtime
(/proc/kernel/vmx* write). However it is not expected that they will be
flipped during latency-sensitive operations, and thus shouldn't be a source
of interference for NOHZ_FULL CPUs wrt the text patching IPI.
Note, smp_text_poke_batch_finish() never defers IPIs if noinstr code is
being patched, i.e. this is purely to tell objtool we're okay with updates
to that key causing IPIs and to silence the associated objtool warning.
Reported-by: Josh Poimboeuf <jpoimboe@kernel.org>
Signed-off-by: Valentin Schneider <vschneid@redhat.com>
Acked-by: Sean Christopherson <seanjc@google.com>
---
arch/x86/kvm/vmx/vmx.c | 11 +++++++++--
1 file changed, 9 insertions(+), 2 deletions(-)
diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
index 91b6f2f3edc2a..99936a2af6641 100644
--- a/arch/x86/kvm/vmx/vmx.c
+++ b/arch/x86/kvm/vmx/vmx.c
@@ -203,8 +203,15 @@ module_param(pt_mode, int, S_IRUGO);
struct x86_pmu_lbr __ro_after_init vmx_lbr_caps;
-static DEFINE_STATIC_KEY_FALSE(vmx_l1d_should_flush);
-static DEFINE_STATIC_KEY_FALSE(vmx_l1d_flush_cond);
+/*
+ * NOINSTR: Both of these static keys end up being used in .noinstr sections,
+ * however they are only modified:
+ * - at init
+ * - from a /proc/kernel/vmx* write
+ * thus during latency-sensitive operations they should remain stable.
+ */
+static DEFINE_STATIC_KEY_FALSE_NOINSTR(vmx_l1d_should_flush);
+static DEFINE_STATIC_KEY_FALSE_NOINSTR(vmx_l1d_flush_cond);
static DEFINE_MUTEX(vmx_l1d_flush_mutex);
/* Storage for pre module init parameter parsing */
--
2.51.0
^ permalink raw reply related
* [PATCH v7 19/31] sched/clock, x86: Mark __sched_clock_stable key as allowed in .noinstr
From: Valentin Schneider @ 2025-11-14 15:10 UTC (permalink / raw)
To: linux-kernel, linux-mm, rcu, x86, linux-arm-kernel, loongarch,
linux-riscv, linux-arch, linux-trace-kernel
Cc: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen,
H. Peter Anvin, Andy Lutomirski, Peter Zijlstra,
Arnaldo Carvalho de Melo, Josh Poimboeuf, Paolo Bonzini,
Arnd Bergmann, Frederic Weisbecker, Paul E. McKenney, Jason Baron,
Steven Rostedt, Ard Biesheuvel, Sami Tolvanen, David S. Miller,
Neeraj Upadhyay, Joel Fernandes, Josh Triplett, Boqun Feng,
Uladzislau Rezki, Mathieu Desnoyers, Mel Gorman, Andrew Morton,
Masahiro Yamada, Han Shen, Rik van Riel, Jann Horn, Dan Carpenter,
Oleg Nesterov, Juri Lelli, Clark Williams, Yair Podemsky,
Marcelo Tosatti, Daniel Wagner, Petr Tesarik, Shrikanth Hegde
In-Reply-To: <20251114150133.1056710-1-vschneid@redhat.com>
Later commits will cause objtool to warn about static keys being used in
.noinstr sections in order to safely defer instruction patching IPIs
targeted at NOHZ_FULL CPUs.
__sched_clock_stable is used in .noinstr code, and can be modified at
runtime (e.g. time_cpufreq_notifier()). Suppressing the text_poke_sync()
IPI has little benefits for this key, as NOHZ_FULL is incompatible with an
unstable TSC anyway.
Mark it to let objtool know not to warn about it.
Signed-off-by: Valentin Schneider <vschneid@redhat.com>
---
kernel/sched/clock.c | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/kernel/sched/clock.c b/kernel/sched/clock.c
index c1a028e99d2cd..5ee27349a4811 100644
--- a/kernel/sched/clock.c
+++ b/kernel/sched/clock.c
@@ -78,8 +78,11 @@ static DEFINE_STATIC_KEY_FALSE_RO(sched_clock_running);
*
* Similarly we start with __sched_clock_stable_early, thereby assuming we
* will become stable, such that there's only a single 1 -> 0 transition.
+ *
+ * NOINSTR: an unstable TLC is incompatible with NOHZ_FULL, thus the text
+ * patching IPI would be the least of our concerns.
*/
-static DEFINE_STATIC_KEY_FALSE(__sched_clock_stable);
+static DEFINE_STATIC_KEY_FALSE_NOINSTR(__sched_clock_stable);
static int __sched_clock_stable_early = 1;
/*
--
2.51.0
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox