* Re: [PATCH v2 2/3] Documentation: tracing: Add PCI controller event documentation
From: Bjorn Helgaas @ 2026-01-09 20:18 UTC (permalink / raw)
To: Shawn Lin
Cc: Manivannan Sadhasivam, Bjorn Helgaas, linux-rockchip, linux-pci,
linux-trace-kernel, linux-doc, Steven Rostedt, Masami Hiramatsu
In-Reply-To: <1767929389-143957-3-git-send-email-shawn.lin@rock-chips.com>
On Fri, Jan 09, 2026 at 11:29:48AM +0800, Shawn Lin wrote:
> The available tracepoint, pcie_ltssm_state_transition, monitors the LTSSM state
> transistion for debugging purpose. Add description about it.
s/transistion/transition/
> Documentation/trace/events-pci-conotroller.rst | 41 ++++++++++++++++++++++++++
s/events-pci-conotroller.rst/events-pci-controller.rst/
> +The PCI controller tracing system provides tracepoints to monitor controller level
> +information for debugging purpose. The events normally show up here:
> +up here:
Wrap text to fit in 80 columns (unless it's a command line or similar
that shouldn't be split).
^ permalink raw reply
* Re: [PATCH v5] tracing: Guard __DECLARE_TRACE() use of __DO_TRACE_CALL() with SRCU-fast
From: Mathieu Desnoyers @ 2026-01-09 20:21 UTC (permalink / raw)
To: Steven Rostedt, Alexei Starovoitov
Cc: LKML, Linux trace kernel, bpf, Masami Hiramatsu, Paul E. McKenney,
Sebastian Andrzej Siewior, Thomas Gleixner
In-Reply-To: <20260109141930.6deb2a0a@gandalf.local.home>
On 2026-01-09 14:19, Steven Rostedt wrote:
> On Fri, 9 Jan 2026 11:10:16 -0800
> Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:
>
> \> >
>>> We also have to consider that migrate disable is *not* cheap at all
>>> compared to preempt disable.
>>
>> Looks like your complaint comes from lack of engagement in kernel
>> development.
>
> No need to make comments like that. The Linux kernel is an ocean of code.
> It's very hard to keep up on everything that is happening. I knew of work
> being done on migrate_disable but I didn't know what the impacts of that
> work was. Mathieu is still very much involved and engaged in kernel
> development.
Thanks Steven. I guess Alexei missed my recent involvement in other
areas of the kernel.
As Steven pointed out, the kernel is vast, so I cannot keep up with
the progress on every single topic. That being said, I very recently
(about 1 month ago) tried using migrate disable for the RSS tracking
improvements (hierarchical percpu counters), and found that the overhead
of migrate disable was large compared to preempt disable. The generated
assembler is also orders of magnitude larger (on x86-64).
Creating small placeholder functions which just call preempt/migrate
disable and enable for a preempt RT build:
0000000000002a20 <test_preempt_disable>:
2a20: f3 0f 1e fa endbr64
2a24: 65 ff 05 00 00 00 00 incl %gs:0x0(%rip) # 2a2b <test_preempt_disable+0xb>
2a2b: e9 00 00 00 00 jmp 2a30 <test_preempt_disable+0x10>
0000000000002a40 <test_preempt_enable>:
2a40: f3 0f 1e fa endbr64
2a44: 65 ff 0d 00 00 00 00 decl %gs:0x0(%rip) # 2a4b <test_preempt_enable+0xb>
2a4b: 74 05 je 2a52 <test_preempt_enable+0x12>
2a4d: e9 00 00 00 00 jmp 2a52 <test_preempt_enable+0x12>
2a52: e8 00 00 00 00 call 2a57 <test_preempt_enable+0x17>
2a57: e9 00 00 00 00 jmp 2a5c <test_preempt_enable+0x1c>
0000000000002920 <test_migrate_disable>:
2920: f3 0f 1e fa endbr64
2924: 65 48 8b 15 00 00 00 mov %gs:0x0(%rip),%rdx # 292c <test_migrate_disable+0xc>
292b: 00
292c: 0f b7 82 38 07 00 00 movzwl 0x738(%rdx),%eax
2933: 66 85 c0 test %ax,%ax
2936: 74 0f je 2947 <test_migrate_disable+0x27>
2938: 83 c0 01 add $0x1,%eax
293b: 66 89 82 38 07 00 00 mov %ax,0x738(%rdx)
2942: e9 00 00 00 00 jmp 2947 <test_migrate_disable+0x27>
2947: 65 ff 05 00 00 00 00 incl %gs:0x0(%rip) # 294e <test_migrate_disable+0x2e>
294e: 65 48 8b 05 00 00 00 mov %gs:0x0(%rip),%rax # 2956 <test_migrate_disable+0x36>
2955: 00
2956: 83 80 00 00 00 00 01 addl $0x1,0x0(%rax)
295d: b8 01 00 00 00 mov $0x1,%eax
2962: 66 89 82 38 07 00 00 mov %ax,0x738(%rdx)
2969: 65 ff 0d 00 00 00 00 decl %gs:0x0(%rip) # 2970 <test_migrate_disable+0x50>
2970: 74 05 je 2977 <test_migrate_disable+0x57>
2972: e9 00 00 00 00 jmp 2977 <test_migrate_disable+0x57>
2977: e8 00 00 00 00 call 297c <test_migrate_disable+0x5c>
297c: e9 00 00 00 00 jmp 2981 <test_migrate_disable+0x61>
00000000000029a0 <test_migrate_enable>:
29a0: f3 0f 1e fa endbr64
29a4: 65 48 8b 15 00 00 00 mov %gs:0x0(%rip),%rdx # 29ac <test_migrate_enable+0xc>
29ab: 00
29ac: 0f b7 82 38 07 00 00 movzwl 0x738(%rdx),%eax
29b3: 66 85 c0 test %ax,%ax
29b6: 74 0f je 29c7 <test_migrate_enable+0x27>
29b8: 83 c0 01 add $0x1,%eax
29bb: 66 89 82 38 07 00 00 mov %ax,0x738(%rdx)
29c2: e9 00 00 00 00 jmp 29c7 <test_migrate_enable+0x27>
29c7: 65 ff 05 00 00 00 00 incl %gs:0x0(%rip) # 29ce <test_migrate_enable+0x2e>
29ce: 65 48 8b 05 00 00 00 mov %gs:0x0(%rip),%rax # 29d6 <test_migrate_enable+0x36>
29d5: 00
29d6: 83 80 00 00 00 00 01 addl $0x1,0x0(%rax)
29dd: b8 01 00 00 00 mov $0x1,%eax
29e2: 66 89 82 38 07 00 00 mov %ax,0x738(%rdx)
29e9: 65 ff 0d 00 00 00 00 decl %gs:0x0(%rip) # 29f0 <test_migrate_enable+0x50>
29f0: 74 05 je 29f7 <test_migrate_enable+0x57>
29f2: e9 00 00 00 00 jmp 29f7 <test_migrate_enable+0x57>
29f7: e8 00 00 00 00 call 29fc <test_migrate_enable+0x5c>
29fc: e9 00 00 00 00 jmp 2a01 <test_migrate_enable+0x61>
>
>> migrate_disable _was_ not cheap.
>> Try to benchmark it now.
>> It's inlined. It's a fraction of extra overhead on top of preempt_disable.
>
> It would be good to have a benchmark of the two. What about fast_srcu? Is
> that fast enough to replace the preempt_disable()? If so, then could we
> just make this the same for both RT and !RT?
I've modified kernel/rcu/refscale.c to compare those:
AMD EPYC 9654 96-Core Processor, kernel baseline: v6.18.1
CONFIG_PREEMPT=y
# CONFIG_PREEMPT_LAZY is not set
# CONFIG_PREEMPT_RT is not set
* preempt disable/enable pair: 1.1 ns
* srcu-fast lock/unlock: 1.5 ns
CONFIG_RCU_REF_SCALE_TEST=y
* migrate disable/enable pair: 3.0 ns
* calls to migrate disable/enable pair within noinline functions: 17.0 ns
CONFIG_RCU_REF_SCALE_TEST=m
* migrate disable/enable pair: 22.0 ns
When I attempted using migrate disable, I configured refscale as
a module, which gave me the appalling 22 ns overhead. It looks like
the implementation of migrate disable/enable now differs depending on
whether it's used from the core kernel or from a module. That's rather
unexpected.
It seems to be done on purpose though (INSTANTIATE_EXPORTED_MIGRATE_DISABLE)
to work around the fact that it is not possible to export the runqueues
variable.
That's the kind of compilation context dependent overhead variability I'd
rather avoid in the implementation of the tracepoint instrumentation API.
Thanks,
Mathieu
--
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com
^ permalink raw reply
* Re: [PATCH v5] tracing: Guard __DECLARE_TRACE() use of __DO_TRACE_CALL() with SRCU-fast
From: Steven Rostedt @ 2026-01-09 21:02 UTC (permalink / raw)
To: Mathieu Desnoyers
Cc: Alexei Starovoitov, LKML, Linux trace kernel, bpf,
Masami Hiramatsu, Paul E. McKenney, Sebastian Andrzej Siewior,
Thomas Gleixner, Peter Zijlstra, Linus Torvalds
In-Reply-To: <3c0df437-f6e5-47c6-aed5-f4cc26fe627a@efficios.com>
On Fri, 9 Jan 2026 15:21:19 -0500
Mathieu Desnoyers <mathieu.desnoyers@efficios.com> wrote:
> * preempt disable/enable pair: 1.1 ns
> * srcu-fast lock/unlock: 1.5 ns
>
> CONFIG_RCU_REF_SCALE_TEST=y
> * migrate disable/enable pair: 3.0 ns
> * calls to migrate disable/enable pair within noinline functions: 17.0 ns
>
> CONFIG_RCU_REF_SCALE_TEST=m
> * migrate disable/enable pair: 22.0 ns
OUCH! So migrate disable/enable has a much larger overhead when executed in
a module than in the kernel? This means all spin_locks() in modules
converted to mutexes in PREEMPT_RT are taking this hit!
It looks like it has to allow access to the rq->nr_pinned. There's a hack to
expose this part of the rq struct for in-kernel by the following:
kernel/sched/rq-offsets.c: DEFINE(RQ_nr_pinned, offsetof(struct rq, nr_pinned));
Then for the in-kernel code we have:
#define this_rq_raw() arch_raw_cpu_ptr(&runqueues)
#else
#define this_rq_raw() PERCPU_PTR(&runqueues)
#endif
#define this_rq_pinned() (*(unsigned int *)((void *)this_rq_raw() + RQ_nr_pinned))
Looking at the scheduler code, the rq->nr_pinned is referenced by a static
function with:
static inline bool rq_has_pinned_tasks(struct rq *rq)
{
return rq->nr_pinned;
}
Which is only referenced in hotplug code and a balance_push() path in load
balancing. Does this variable really need to be in the runqueue struct?
Why not just make it a per-cpu variable. Maybe call it cpu_nr_pinned_tasks,
and export that for all to use?
It will not only fix the discrepancy between the overhead of
migrate_disable/enable in modules vs in-kernel. But it also removes the
hack to expose a portion of the runqueue.
-- Steve
^ permalink raw reply
* Re: [PATCH 3/5] tracing: Have tracer option be instance specific
From: Dan Carpenter @ 2026-01-09 21:40 UTC (permalink / raw)
To: Steven Rostedt
Cc: oe-kbuild, linux-kernel, linux-trace-kernel, lkp, oe-kbuild-all,
Masami Hiramatsu, Mark Rutland, Mathieu Desnoyers, Andrew Morton,
Linux Memory Management List
In-Reply-To: <20260109102301.19daeb27@gandalf.local.home>
On Fri, Jan 09, 2026 at 10:23:01AM -0500, Steven Rostedt wrote:
> On Mon, 5 Jan 2026 18:23:31 +0300
> Dan Carpenter <dan.carpenter@linaro.org> wrote:
>
> > Hi Steven,
>
> Hi Dan,
>
> >
> > kernel test robot noticed the following build warnings:
> >
> > https://git-scm.com/docs/git-format-patch#_base_tree_information]
> >
> > url: https://github.com/intel-lab-lkp/linux/commits/Steven-Rostedt/tracing-Remove-dummy-options-and-flags/20251106-010511
> > base: https://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace for-next
> > patch link: https://lore.kernel.org/r/20251105161935.545400234%40kernel.org
>
> This is an old patch (v1) and v3 fixed this issue:
> https://lore.kernel.org/all/20251110234341.767135255@kernel.org/
>
> and v4 was applied and is now in mainline.
>
> The patch you are reporting on was sent Nov 5th, the fix was sent Nov 10th.
>
> Why is this sending reports about an old patch that is obsolete? Is there
> something wrong with your setup?
>
> -- Steve
>
Yeah. I appologize. That was my bad. I accidentally was looking
at old emails. I think this was the only one I sent, though.
regards,
dan carpenter
^ permalink raw reply
* Re: [PATCH v5] tracing: Guard __DECLARE_TRACE() use of __DO_TRACE_CALL() with SRCU-fast
From: Alexei Starovoitov @ 2026-01-09 21:54 UTC (permalink / raw)
To: Mathieu Desnoyers
Cc: Steven Rostedt, LKML, Linux trace kernel, bpf, Masami Hiramatsu,
Paul E. McKenney, Sebastian Andrzej Siewior, Thomas Gleixner
In-Reply-To: <3c0df437-f6e5-47c6-aed5-f4cc26fe627a@efficios.com>
On Fri, Jan 9, 2026 at 12:21 PM Mathieu Desnoyers
<mathieu.desnoyers@efficios.com> wrote:
>
>
> * preempt disable/enable pair: 1.1 ns
> * srcu-fast lock/unlock: 1.5 ns
>
> CONFIG_RCU_REF_SCALE_TEST=y
> * migrate disable/enable pair: 3.0 ns
.. and you're arguing that 3ns vs 1ns difference is so important
for your out-of-tree tracer that in-tree tracers need to do
some workarounds?! wtf
^ permalink raw reply
* Re: [PATCH v5] tracing: Guard __DECLARE_TRACE() use of __DO_TRACE_CALL() with SRCU-fast
From: Steven Rostedt @ 2026-01-09 22:00 UTC (permalink / raw)
To: Alexei Starovoitov
Cc: Mathieu Desnoyers, LKML, Linux trace kernel, bpf,
Masami Hiramatsu, Paul E. McKenney, Sebastian Andrzej Siewior,
Thomas Gleixner
In-Reply-To: <CAADnVQLeCLRhx1Oe5DdJCT0e+WWq4L3Rdee1Ky0JNNh3LdozeQ@mail.gmail.com>
On Fri, 9 Jan 2026 13:54:34 -0800
Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:
> On Fri, Jan 9, 2026 at 12:21 PM Mathieu Desnoyers
> <mathieu.desnoyers@efficios.com> wrote:
> >
> >
> > * preempt disable/enable pair: 1.1 ns
> > * srcu-fast lock/unlock: 1.5 ns
> >
> > CONFIG_RCU_REF_SCALE_TEST=y
> > * migrate disable/enable pair: 3.0 ns
>
> .. and you're arguing that 3ns vs 1ns difference is so important
> for your out-of-tree tracer that in-tree tracers need to do
> some workarounds?! wtf
This has nothing to do with out of tree tracers. The overhead of the
22ns is for any tracepoint in an in-tree module. That's because the
rq->nr_pinned isn't exported for modules to use.
-- Steve
^ permalink raw reply
* Re: [PATCH v5] tracing: Guard __DECLARE_TRACE() use of __DO_TRACE_CALL() with SRCU-fast
From: Alexei Starovoitov @ 2026-01-09 22:18 UTC (permalink / raw)
To: Steven Rostedt
Cc: Mathieu Desnoyers, LKML, Linux trace kernel, bpf,
Masami Hiramatsu, Paul E. McKenney, Sebastian Andrzej Siewior,
Thomas Gleixner
In-Reply-To: <20260109170028.0068a14d@fedora>
On Fri, Jan 9, 2026 at 2:00 PM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Fri, 9 Jan 2026 13:54:34 -0800
> Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:
>
> > On Fri, Jan 9, 2026 at 12:21 PM Mathieu Desnoyers
> > <mathieu.desnoyers@efficios.com> wrote:
> > >
> > >
> > > * preempt disable/enable pair: 1.1 ns
> > > * srcu-fast lock/unlock: 1.5 ns
> > >
> > > CONFIG_RCU_REF_SCALE_TEST=y
> > > * migrate disable/enable pair: 3.0 ns
> >
> > .. and you're arguing that 3ns vs 1ns difference is so important
> > for your out-of-tree tracer that in-tree tracers need to do
> > some workarounds?! wtf
>
> This has nothing to do with out of tree tracers. The overhead of the
> 22ns is for any tracepoint in an in-tree module. That's because the
> rq->nr_pinned isn't exported for modules to use.
None of the driver's tracepoints are in the critical path.
You perfectly know that Mathieu argued about not slowing down lttng.
^ permalink raw reply
* Re: [PATCH v5] tracing: Guard __DECLARE_TRACE() use of __DO_TRACE_CALL() with SRCU-fast
From: Mathieu Desnoyers @ 2026-01-09 22:31 UTC (permalink / raw)
To: Alexei Starovoitov, Steven Rostedt
Cc: LKML, Linux trace kernel, bpf, Masami Hiramatsu, Paul E. McKenney,
Sebastian Andrzej Siewior, Thomas Gleixner
In-Reply-To: <CAADnVQKGm-t2SdN_vFVMn0tNiQ5Fs6FutD2Au-jO69aGdhKS7Q@mail.gmail.com>
On 2026-01-09 17:18, Alexei Starovoitov wrote:
> On Fri, Jan 9, 2026 at 2:00 PM Steven Rostedt <rostedt@goodmis.org> wrote:
>>
>> On Fri, 9 Jan 2026 13:54:34 -0800
>> Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:
>>
>>> On Fri, Jan 9, 2026 at 12:21 PM Mathieu Desnoyers
>>> <mathieu.desnoyers@efficios.com> wrote:
>>>>
>>>>
>>>> * preempt disable/enable pair: 1.1 ns
>>>> * srcu-fast lock/unlock: 1.5 ns
>>>>
>>>> CONFIG_RCU_REF_SCALE_TEST=y
>>>> * migrate disable/enable pair: 3.0 ns
>>>
>>> .. and you're arguing that 3ns vs 1ns difference is so important
>>> for your out-of-tree tracer that in-tree tracers need to do
>>> some workarounds?! wtf
>>
>> This has nothing to do with out of tree tracers. The overhead of the
>> 22ns is for any tracepoint in an in-tree module. That's because the
>> rq->nr_pinned isn't exported for modules to use.
>
> None of the driver's tracepoints are in the critical path.
> You perfectly know that Mathieu argued about not slowing down lttng.
My argument is about not slowing down high-throughput tracers
on preempt-rt kernels. This affects ftrace as well as lttng,
so both in-tree and OOT tracers just alike.
Are you so sure that no tracepoint instrumentation whatsoever can
be compiled into a module which is a critical path for some workload ?
That's a bold statement.
Thanks,
Mathieu
--
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com
^ permalink raw reply
* Re: [PATCH v5] tracing: Guard __DECLARE_TRACE() use of __DO_TRACE_CALL() with SRCU-fast
From: Steven Rostedt @ 2026-01-09 22:33 UTC (permalink / raw)
To: Alexei Starovoitov
Cc: Mathieu Desnoyers, LKML, Linux trace kernel, bpf,
Masami Hiramatsu, Paul E. McKenney, Sebastian Andrzej Siewior,
Thomas Gleixner
In-Reply-To: <CAADnVQKGm-t2SdN_vFVMn0tNiQ5Fs6FutD2Au-jO69aGdhKS7Q@mail.gmail.com>
On Fri, 9 Jan 2026 14:18:36 -0800
Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:
> On Fri, Jan 9, 2026 at 2:00 PM Steven Rostedt <rostedt@goodmis.org> wrote:
> >
> > On Fri, 9 Jan 2026 13:54:34 -0800
> > Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:
> >
> > > On Fri, Jan 9, 2026 at 12:21 PM Mathieu Desnoyers
> > > <mathieu.desnoyers@efficios.com> wrote:
> > > >
> > > >
> > > > * preempt disable/enable pair: 1.1 ns
> > > > * srcu-fast lock/unlock: 1.5 ns
> > > >
> > > > CONFIG_RCU_REF_SCALE_TEST=y
> > > > * migrate disable/enable pair: 3.0 ns
> > >
> > > .. and you're arguing that 3ns vs 1ns difference is so important
> > > for your out-of-tree tracer that in-tree tracers need to do
> > > some workarounds?! wtf
> >
> > This has nothing to do with out of tree tracers. The overhead of the
> > 22ns is for any tracepoint in an in-tree module. That's because the
> > rq->nr_pinned isn't exported for modules to use.
>
> None of the driver's tracepoints are in the critical path.
> You perfectly know that Mathieu argued about not slowing down lttng.
How is this about lttng? Sure he cares about that, but even tracepoints
that lttng uses doesn't get affected any more than ftrace or bpf.
Because lttng is one of the callbacks. The migrate disable happens in
the in-tree portion of the code.
So you are saying that all the tracepoints for xfs are not in a fastpath?
-- Steve
^ permalink raw reply
* Re: [PATCH v5] tracing: Guard __DECLARE_TRACE() use of __DO_TRACE_CALL() with SRCU-fast
From: Steven Rostedt @ 2026-01-09 22:39 UTC (permalink / raw)
To: Alexei Starovoitov
Cc: Mathieu Desnoyers, LKML, Linux trace kernel, bpf,
Masami Hiramatsu, Paul E. McKenney, Sebastian Andrzej Siewior,
Thomas Gleixner
In-Reply-To: <20260109173326.616e873c@fedora>
On Fri, 9 Jan 2026 17:33:26 -0500
Steven Rostedt <rostedt@goodmis.org> wrote:
> How is this about lttng? Sure he cares about that, but even tracepoints
> that lttng uses doesn't get affected any more than ftrace or bpf.
> Because lttng is one of the callbacks. The migrate disable happens in
> the in-tree portion of the code.
>
> So you are saying that all the tracepoints for xfs are not in a fastpath?
Regardless of tracing. I now have my RT hat on. The spin_locks that are
converted to mutex use migrate disable. The fact that migrate_disable
in modules are close to 10x slower than the same code in-kernel is
troubling to say the least. It means that modules in RT take a hit
every time they take a spin_lock().
The migrate disable being slow for modules is no longer just a tracing
issue. It's a PREEMPT_RT issue.
-- Steve
^ permalink raw reply
* Re: [PATCH v3] selftests/tracing: Fix test_multiple_writes stall
From: Shuah Khan @ 2026-01-09 22:51 UTC (permalink / raw)
To: Fushuai Wang, rostedt, mhiramat, mathieu.desnoyers, shuah,
wangfushuai
Cc: linux-kernel, linux-trace-kernel, linux-kselftest, Shuah Khan
In-Reply-To: <20260109033620.25727-1-fushuai.wang@linux.dev>
On 1/8/26 20:36, Fushuai Wang wrote:
> From: Fushuai Wang <wangfushuai@baidu.com>
>
> When /sys/kernel/tracing/buffer_size_kb is less than 12KB,
> the test_multiple_writes test will stall and wait for more
> input due to insufficient buffer space.
>
> Check current buffer_size_kb value before the test. If it is
> less than 12KB, it temporarily increase the buffer to 12KB,
> and restore the original value after the tests are completed.
>
> Fixes: 37f46601383a ("selftests/tracing: Add basic test for trace_marker_raw file")
> Suggested-by: Steven Rostedt <rostedt@goodmis.org>
> Signed-off-by: Fushuai Wang <wangfushuai@baidu.com>
> Acked-by: Steven Rostedt (Google) <rostedt@goodmis.org>
> ---
> V2 -> V3: Make the From and SoB match.
> V1 -> V2: Restore buffer_size_kb outside of awk script.
Thank you. Applied to linux-kselftest fixes branch.
I will send this up for the next rc.
thanks,
-- Shuah
^ permalink raw reply
* Re: [PATCH v5] tracing: Guard __DECLARE_TRACE() use of __DO_TRACE_CALL() with SRCU-fast
From: Alexei Starovoitov @ 2026-01-10 0:35 UTC (permalink / raw)
To: Steven Rostedt
Cc: Mathieu Desnoyers, LKML, Linux trace kernel, bpf,
Masami Hiramatsu, Paul E. McKenney, Sebastian Andrzej Siewior,
Thomas Gleixner
In-Reply-To: <20260109173915.1e8a784e@fedora>
On Fri, Jan 9, 2026 at 2:39 PM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Fri, 9 Jan 2026 17:33:26 -0500
> Steven Rostedt <rostedt@goodmis.org> wrote:
>
> > How is this about lttng? Sure he cares about that, but even tracepoints
> > that lttng uses doesn't get affected any more than ftrace or bpf.
> > Because lttng is one of the callbacks. The migrate disable happens in
> > the in-tree portion of the code.
> >
> > So you are saying that all the tracepoints for xfs are not in a fastpath?
>
> Regardless of tracing. I now have my RT hat on. The spin_locks that are
> converted to mutex use migrate disable. The fact that migrate_disable
> in modules are close to 10x slower than the same code in-kernel is
> troubling to say the least. It means that modules in RT take a hit
> every time they take a spin_lock().
>
> The migrate disable being slow for modules is no longer just a tracing
> issue. It's a PREEMPT_RT issue.
migrate_enable/disable() wasn't inlined for a long time.
It bothered us enough, since sleepable bpf is the main user
of it besides RT, so we made an effort to inline it.
RT, at the same time, doesn't inline rt_spin_lock() itself
so inlining migrate_disable() or not is not 10x at all.
Benchmark spin_lock on RT in-tree and in-module and I bet
there won't be a big difference.
^ permalink raw reply
* Re: [PATCHv6 bpf-next 1/9] ftrace,bpf: Remove FTRACE_OPS_FL_JMP ftrace_ops flag
From: Andrii Nakryiko @ 2026-01-10 0:36 UTC (permalink / raw)
To: Jiri Olsa
Cc: Steven Rostedt, Florent Revest, Mark Rutland, bpf, linux-kernel,
linux-trace-kernel, linux-arm-kernel, Alexei Starovoitov,
Daniel Borkmann, Andrii Nakryiko, Menglong Dong, Song Liu
In-Reply-To: <20251230145010.103439-2-jolsa@kernel.org>
On Tue, Dec 30, 2025 at 6:50 AM Jiri Olsa <jolsa@kernel.org> wrote:
>
> At the moment the we allow the jmp attach only for ftrace_ops that
> has FTRACE_OPS_FL_JMP set. This conflicts with following changes
> where we use single ftrace_ops object for all direct call sites,
> so all could be be attached via just call or jmp.
>
> We already limit the jmp attach support with config option and bit
> (LSB) set on the trampoline address. It turns out that's actually
> enough to limit the jmp attach for architecture and only for chosen
> addresses (with LSB bit set).
>
> Each user of register_ftrace_direct or modify_ftrace_direct can set
> the trampoline bit (LSB) to indicate it has to be attached by jmp.
>
> The bpf trampoline generation code uses trampoline flags to generate
> jmp-attach specific code and ftrace inner code uses the trampoline
> bit (LSB) to handle return from jmp attachment, so there's no harm
> to remove the FTRACE_OPS_FL_JMP bit.
>
> The fexit/fmodret performance stays the same (did not drop),
> current code:
>
> fentry : 77.904 ± 0.546M/s
> fexit : 62.430 ± 0.554M/s
> fmodret : 66.503 ± 0.902M/s
>
> with this change:
>
> fentry : 80.472 ± 0.061M/s
> fexit : 63.995 ± 0.127M/s
> fmodret : 67.362 ± 0.175M/s
>
> Fixes: 25e4e3565d45 ("ftrace: Introduce FTRACE_OPS_FL_JMP")
> Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> ---
> include/linux/ftrace.h | 1 -
> kernel/bpf/trampoline.c | 32 ++++++++++++++------------------
> kernel/trace/ftrace.c | 14 --------------
> 3 files changed, 14 insertions(+), 33 deletions(-)
>
I don't see anything wrong with this from BPF side
Acked-by: Andrii Nakryiko <andrii@kernel.org>
[...]
^ permalink raw reply
* Re: [PATCHv6 bpf-next 7/9] bpf: Add trampoline ip hash table
From: Andrii Nakryiko @ 2026-01-10 0:36 UTC (permalink / raw)
To: Jiri Olsa
Cc: Steven Rostedt, Florent Revest, Mark Rutland, bpf, linux-kernel,
linux-trace-kernel, linux-arm-kernel, Alexei Starovoitov,
Daniel Borkmann, Andrii Nakryiko, Menglong Dong, Song Liu
In-Reply-To: <20251230145010.103439-8-jolsa@kernel.org>
On Tue, Dec 30, 2025 at 6:51 AM Jiri Olsa <jolsa@kernel.org> wrote:
>
> Following changes need to lookup trampoline based on its ip address,
> adding hash table for that.
>
> Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> ---
> include/linux/bpf.h | 7 +++++--
> kernel/bpf/trampoline.c | 30 +++++++++++++++++++-----------
> 2 files changed, 24 insertions(+), 13 deletions(-)
>
> diff --git a/include/linux/bpf.h b/include/linux/bpf.h
> index 4e7d72dfbcd4..c85677aae865 100644
> --- a/include/linux/bpf.h
> +++ b/include/linux/bpf.h
> @@ -1325,14 +1325,17 @@ struct bpf_tramp_image {
> };
>
> struct bpf_trampoline {
> - /* hlist for trampoline_table */
> - struct hlist_node hlist;
> + /* hlist for trampoline_key_table */
> + struct hlist_node hlist_key;
> + /* hlist for trampoline_ip_table */
> + struct hlist_node hlist_ip;
> struct ftrace_ops *fops;
> /* serializes access to fields of this trampoline */
> struct mutex mutex;
> refcount_t refcnt;
> u32 flags;
> u64 key;
> + unsigned long ip;
> struct {
> struct btf_func_model model;
> void *addr;
> diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c
> index 789ff4e1f40b..bdac9d673776 100644
> --- a/kernel/bpf/trampoline.c
> +++ b/kernel/bpf/trampoline.c
> @@ -24,9 +24,10 @@ const struct bpf_prog_ops bpf_extension_prog_ops = {
> #define TRAMPOLINE_HASH_BITS 10
> #define TRAMPOLINE_TABLE_SIZE (1 << TRAMPOLINE_HASH_BITS)
>
> -static struct hlist_head trampoline_table[TRAMPOLINE_TABLE_SIZE];
> +static struct hlist_head trampoline_key_table[TRAMPOLINE_TABLE_SIZE];
> +static struct hlist_head trampoline_ip_table[TRAMPOLINE_TABLE_SIZE];
>
> -/* serializes access to trampoline_table */
> +/* serializes access to trampoline tables */
> static DEFINE_MUTEX(trampoline_mutex);
>
> #ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS
> @@ -135,15 +136,15 @@ void bpf_image_ksym_del(struct bpf_ksym *ksym)
> PAGE_SIZE, true, ksym->name);
> }
>
> -static struct bpf_trampoline *bpf_trampoline_lookup(u64 key)
> +static struct bpf_trampoline *bpf_trampoline_lookup(u64 key, unsigned long ip)
> {
> struct bpf_trampoline *tr;
> struct hlist_head *head;
> int i;
>
> mutex_lock(&trampoline_mutex);
> - head = &trampoline_table[hash_64(key, TRAMPOLINE_HASH_BITS)];
> - hlist_for_each_entry(tr, head, hlist) {
> + head = &trampoline_key_table[hash_64(key, TRAMPOLINE_HASH_BITS)];
> + hlist_for_each_entry(tr, head, hlist_key) {
> if (tr->key == key) {
> refcount_inc(&tr->refcnt);
> goto out;
> @@ -164,8 +165,12 @@ static struct bpf_trampoline *bpf_trampoline_lookup(u64 key)
> #endif
>
> tr->key = key;
> - INIT_HLIST_NODE(&tr->hlist);
> - hlist_add_head(&tr->hlist, head);
> + tr->ip = ftrace_location(ip);
> + INIT_HLIST_NODE(&tr->hlist_key);
> + INIT_HLIST_NODE(&tr->hlist_ip);
> + hlist_add_head(&tr->hlist_key, head);
> + head = &trampoline_ip_table[hash_64(tr->ip, TRAMPOLINE_HASH_BITS)];
For key lookups we check that there is no existing trampoline for the
given key. Can it happen that we have two trampolines at the same IP
but using two different keys?
> + hlist_add_head(&tr->hlist_ip, head);
> refcount_set(&tr->refcnt, 1);
> mutex_init(&tr->mutex);
> for (i = 0; i < BPF_TRAMP_MAX; i++)
> @@ -846,7 +851,7 @@ void bpf_trampoline_unlink_cgroup_shim(struct bpf_prog *prog)
> prog->aux->attach_btf_id);
>
> bpf_lsm_find_cgroup_shim(prog, &bpf_func);
> - tr = bpf_trampoline_lookup(key);
> + tr = bpf_trampoline_lookup(key, 0);
> if (WARN_ON_ONCE(!tr))
> return;
>
> @@ -866,7 +871,7 @@ struct bpf_trampoline *bpf_trampoline_get(u64 key,
> {
> struct bpf_trampoline *tr;
>
> - tr = bpf_trampoline_lookup(key);
> + tr = bpf_trampoline_lookup(key, tgt_info->tgt_addr);
> if (!tr)
> return NULL;
>
> @@ -902,7 +907,8 @@ void bpf_trampoline_put(struct bpf_trampoline *tr)
> * fexit progs. The fentry-only trampoline will be freed via
> * multiple rcu callbacks.
> */
> - hlist_del(&tr->hlist);
> + hlist_del(&tr->hlist_key);
> + hlist_del(&tr->hlist_ip);
> if (tr->fops) {
> ftrace_free_filter(tr->fops);
> kfree(tr->fops);
> @@ -1175,7 +1181,9 @@ static int __init init_trampolines(void)
> int i;
>
> for (i = 0; i < TRAMPOLINE_TABLE_SIZE; i++)
> - INIT_HLIST_HEAD(&trampoline_table[i]);
> + INIT_HLIST_HEAD(&trampoline_key_table[i]);
> + for (i = 0; i < TRAMPOLINE_TABLE_SIZE; i++)
> + INIT_HLIST_HEAD(&trampoline_ip_table[i]);
> return 0;
> }
> late_initcall(init_trampolines);
> --
> 2.52.0
>
^ permalink raw reply
* Re: [PATCHv6 bpf-next 9/9] bpf,x86: Use single ftrace_ops for direct calls
From: Andrii Nakryiko @ 2026-01-10 0:36 UTC (permalink / raw)
To: Jiri Olsa
Cc: Steven Rostedt, Florent Revest, Mark Rutland, bpf, linux-kernel,
linux-trace-kernel, linux-arm-kernel, Alexei Starovoitov,
Daniel Borkmann, Andrii Nakryiko, Menglong Dong, Song Liu
In-Reply-To: <20251230145010.103439-10-jolsa@kernel.org>
On Tue, Dec 30, 2025 at 6:51 AM Jiri Olsa <jolsa@kernel.org> wrote:
>
> Using single ftrace_ops for direct calls update instead of allocating
> ftrace_ops object for each trampoline.
>
> With single ftrace_ops object we can use update_ftrace_direct_* api
> that allows multiple ip sites updates on single ftrace_ops object.
>
> Adding HAVE_SINGLE_FTRACE_DIRECT_OPS config option to be enabled on
> each arch that supports this.
>
> At the moment we can enable this only on x86 arch, because arm relies
> on ftrace_ops object representing just single trampoline image (stored
> in ftrace_ops::direct_call). Archs that do not support this will continue
> to use *_ftrace_direct api.
>
> Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> ---
> arch/x86/Kconfig | 1 +
> kernel/bpf/trampoline.c | 220 ++++++++++++++++++++++++++++++++++------
> kernel/trace/Kconfig | 3 +
> kernel/trace/ftrace.c | 7 +-
> 4 files changed, 200 insertions(+), 31 deletions(-)
>
As far as I can follow, everything looks reasonable
Acked-by: Andrii Nakryiko <andrii@kernel.org>
[...]
^ permalink raw reply
* [PATCH] kprobes: Use dedicated kthread for kprobe optimizer
From: Masami Hiramatsu (Google) @ 2026-01-10 14:33 UTC (permalink / raw)
To: Steven Rostedt, Naveen N Rao, David S . Miller, Masami Hiramatsu
Cc: linux-kernel, linux-trace-kernel
From: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Instead of using generic workqueue, use a dedicated kthread for optimizing
kprobes, because it can wait (sleep) for a long time inside the process
by synchronize_rcu_task(). This means other works can be stopped until it
finishes.
Suggested-by: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
---
kernel/kprobes.c | 105 ++++++++++++++++++++++++++++++++++++++++++++----------
1 file changed, 85 insertions(+), 20 deletions(-)
diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index ab8f9fc1f0d1..40e3d6af4370 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -32,6 +32,7 @@
#include <linux/debugfs.h>
#include <linux/sysctl.h>
#include <linux/kdebug.h>
+#include <linux/kthread.h>
#include <linux/memory.h>
#include <linux/ftrace.h>
#include <linux/cpu.h>
@@ -40,6 +41,7 @@
#include <linux/perf_event.h>
#include <linux/execmem.h>
#include <linux/cleanup.h>
+#include <linux/wait.h>
#include <asm/sections.h>
#include <asm/cacheflush.h>
@@ -514,8 +516,17 @@ static LIST_HEAD(optimizing_list);
static LIST_HEAD(unoptimizing_list);
static LIST_HEAD(freeing_list);
-static void kprobe_optimizer(struct work_struct *work);
-static DECLARE_DELAYED_WORK(optimizing_work, kprobe_optimizer);
+static struct task_struct *kprobe_optimizer_task;
+static wait_queue_head_t kprobe_optimizer_wait;
+static atomic_t optimizer_state;
+enum {
+ OPTIMIZER_ST_IDLE = 0,
+ OPTIMIZER_ST_KICKED = 1,
+ OPTIMIZER_ST_FLUSHING = 2,
+};
+
+static DECLARE_COMPLETION(optimizer_completion);
+
#define OPTIMIZE_DELAY 5
/*
@@ -597,14 +608,10 @@ static void do_free_cleaned_kprobes(void)
}
}
-/* Start optimizer after OPTIMIZE_DELAY passed */
-static void kick_kprobe_optimizer(void)
-{
- schedule_delayed_work(&optimizing_work, OPTIMIZE_DELAY);
-}
+static void kick_kprobe_optimizer(void);
/* Kprobe jump optimizer */
-static void kprobe_optimizer(struct work_struct *work)
+static void kprobe_optimizer(void)
{
guard(mutex)(&kprobe_mutex);
@@ -635,9 +642,52 @@ static void kprobe_optimizer(struct work_struct *work)
do_free_cleaned_kprobes();
}
- /* Step 5: Kick optimizer again if needed */
+ /* Step 5: Kick optimizer again if needed. But if there is a flush requested, */
+ if (completion_done(&optimizer_completion))
+ complete(&optimizer_completion);
+
if (!list_empty(&optimizing_list) || !list_empty(&unoptimizing_list))
- kick_kprobe_optimizer();
+ kick_kprobe_optimizer(); /*normal kick*/
+}
+
+static int kprobe_optimizer_thread(void *data)
+{
+ set_freezable();
+ while (!kthread_should_stop()) {
+ wait_event_freezable(kprobe_optimizer_wait,
+ atomic_read(&optimizer_state) != OPTIMIZER_ST_IDLE ||
+ kthread_should_stop());
+
+ if (kthread_should_stop())
+ break;
+
+ if (try_to_freeze())
+ continue;
+
+ /*
+ * If it was a normal kick, wait for OPTIMIZE_DELAY.
+ * This wait can be interrupted by a flush request.
+ */
+ if (atomic_read(&optimizer_state) == 1)
+ wait_event_freezable_timeout(kprobe_optimizer_wait,
+ atomic_read(&optimizer_state) == OPTIMIZER_ST_FLUSHING ||
+ kthread_should_stop(),
+ OPTIMIZE_DELAY);
+
+ atomic_set(&optimizer_state, OPTIMIZER_ST_IDLE);
+
+ kprobe_optimizer();
+ }
+ return 0;
+}
+
+/* Start optimizer after OPTIMIZE_DELAY passed */
+static void kick_kprobe_optimizer(void)
+{
+ lockdep_assert_held(&kprobe_mutex);
+ if (atomic_cmpxchg(&optimizer_state,
+ OPTIMIZER_ST_IDLE, OPTIMIZER_ST_KICKED) == OPTIMIZER_ST_IDLE)
+ wake_up(&kprobe_optimizer_wait);
}
static void wait_for_kprobe_optimizer_locked(void)
@@ -645,13 +695,17 @@ static void wait_for_kprobe_optimizer_locked(void)
lockdep_assert_held(&kprobe_mutex);
while (!list_empty(&optimizing_list) || !list_empty(&unoptimizing_list)) {
- mutex_unlock(&kprobe_mutex);
-
- /* This will also make 'optimizing_work' execute immmediately */
- flush_delayed_work(&optimizing_work);
- /* 'optimizing_work' might not have been queued yet, relax */
- cpu_relax();
+ init_completion(&optimizer_completion);
+ /*
+ * Set state to OPTIMIZER_ST_FLUSHING and wake up the thread if it's
+ * idle. If it's already kicked, it will see the state change.
+ */
+ if (atomic_xchg_acquire(&optimizer_state,
+ OPTIMIZER_ST_FLUSHING) != OPTIMIZER_ST_FLUSHING)
+ wake_up(&kprobe_optimizer_wait);
+ mutex_unlock(&kprobe_mutex);
+ wait_for_completion(&optimizer_completion);
mutex_lock(&kprobe_mutex);
}
}
@@ -1010,8 +1064,21 @@ static void __disarm_kprobe(struct kprobe *p, bool reopt)
*/
}
+static void __init init_optprobe(void)
+{
+#ifdef __ARCH_WANT_KPROBES_INSN_SLOT
+ /* Init 'kprobe_optinsn_slots' for allocation */
+ kprobe_optinsn_slots.insn_size = MAX_OPTINSN_SIZE;
+#endif
+
+ init_waitqueue_head(&kprobe_optimizer_wait);
+ atomic_set(&optimizer_state, 0);
+ kprobe_optimizer_task = kthread_run(kprobe_optimizer_thread, NULL,
+ "kprobe-optimizer");
+}
#else /* !CONFIG_OPTPROBES */
+#define init_optprobe() do {} while (0)
#define optimize_kprobe(p) do {} while (0)
#define unoptimize_kprobe(p, f) do {} while (0)
#define kill_optimized_kprobe(p) do {} while (0)
@@ -2694,10 +2761,8 @@ static int __init init_kprobes(void)
/* By default, kprobes are armed */
kprobes_all_disarmed = false;
-#if defined(CONFIG_OPTPROBES) && defined(__ARCH_WANT_KPROBES_INSN_SLOT)
- /* Init 'kprobe_optinsn_slots' for allocation */
- kprobe_optinsn_slots.insn_size = MAX_OPTINSN_SIZE;
-#endif
+ /* Initialize the optimization infrastructure */
+ init_optprobe();
err = arch_init_kprobes();
if (!err)
^ permalink raw reply related
* Re: [PATCH v5] tracing: Guard __DECLARE_TRACE() use of __DO_TRACE_CALL() with SRCU-fast
From: Steven Rostedt @ 2026-01-10 16:14 UTC (permalink / raw)
To: Alexei Starovoitov
Cc: Mathieu Desnoyers, LKML, Linux trace kernel, bpf,
Masami Hiramatsu, Paul E. McKenney, Sebastian Andrzej Siewior,
Thomas Gleixner
In-Reply-To: <CAADnVQKB4dAWtX7T15yh31NYNcBUugoqcnTZ3U9APo8SZkTuwg@mail.gmail.com>
On Fri, 9 Jan 2026 16:35:10 -0800
Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:
> migrate_enable/disable() wasn't inlined for a long time.
> It bothered us enough, since sleepable bpf is the main user
> of it besides RT, so we made an effort to inline it.
It did bother us too. it went through lots of iterations to become more
efficient over the years (it was really bad in the beginning while
still in the rt-patch), and hopefully that will continue.
>
> RT, at the same time, doesn't inline rt_spin_lock() itself
> so inlining migrate_disable() or not is not 10x at all.
> Benchmark spin_lock on RT in-tree and in-module and I bet
> there won't be a big difference.
I'll put that on my todo list. But still, having migrate_disable a
function for modules and 100% inlined for in-kernel code just because
it needs access to a field in the run queue that doesn't need to be in
the run queue seems like it should be fixed.
As for tracepoints, BPF is the only one that needs migrate disable.
It's not needed for ftrace or perf (although perf uses preempt
disable). It should be moved into the BPF callback code as perf has its
preempt disable in its callback code.
If BPF doesn't care about the extra overhead of migrate_disable() for
modules, then why should XFS suffer from that too?
-- Steve
^ permalink raw reply
* Re: [PATCHv2 bpf-next 1/2] arm64/ftrace,bpf: Fix partial regs after bpf_prog_run
From: Jiri Olsa @ 2026-01-10 17:06 UTC (permalink / raw)
To: Will Deacon
Cc: Masami Hiramatsu, Steven Rostedt, Mahe Tardy, Peter Zijlstra, bpf,
linux-trace-kernel, linux-arm-kernel, x86, Yonghong Song,
Song Liu, Andrii Nakryiko, Mark Rutland
In-Reply-To: <aWEG685zlaV0o7M7@willie-the-truck>
On Fri, Jan 09, 2026 at 01:47:23PM +0000, Will Deacon wrote:
> On Fri, Jan 09, 2026 at 10:34:53AM +0100, Jiri Olsa wrote:
> > Mahe reported issue with bpf_override_return helper not working when
> > executed from kprobe.multi bpf program on arm.
> >
> > The problem is that on arm we use alternate storage for pt_regs object
> > that is passed to bpf_prog_run and if any register is changed (which
> > is the case of bpf_override_return) it's not propagated back to actual
> > pt_regs object.
> >
> > Fixing this by introducing and calling ftrace_partial_regs_update function
> > to propagate the values of changed registers (ip and stack).
> >
> > Reported-by: Mahe Tardy <mahe.tardy@gmail.com>
> > Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> > ---
> > v2 changes:
> > - moved ftrace_partial_regs_update to generic code [Will]
> >
> > include/linux/ftrace_regs.h | 25 +++++++++++++++++++++++++
> > kernel/trace/bpf_trace.c | 1 +
> > 2 files changed, 26 insertions(+)
> >
> > diff --git a/include/linux/ftrace_regs.h b/include/linux/ftrace_regs.h
> > index 15627ceea9bc..f9a7c009cdae 100644
> > --- a/include/linux/ftrace_regs.h
> > +++ b/include/linux/ftrace_regs.h
> > @@ -33,6 +33,31 @@ struct ftrace_regs;
> > #define ftrace_regs_get_frame_pointer(fregs) \
> > frame_pointer(&arch_ftrace_regs(fregs)->regs)
> >
> > +static __always_inline void
> > +ftrace_partial_regs_update(const struct ftrace_regs *fregs, struct pt_regs *regs) { }
> > +
> > +#else
> > +
> > +/*
> > + * ftrace_partial_regs_update - update the original ftrace_regs from regs
> > + * @fregs: The ftrace_regs to update from @regs
> > + * @regs: The partial regs from ftrace_partial_regs() that was updated
> > + *
> > + * Some architectures have the partial regs living in the ftrace_regs
> > + * structure, whereas other architectures need to make a different copy
> > + * of the @regs. If a partial @regs is retrieved by ftrace_partial_regs() and
> > + * if the code using @regs updates a field (like the instruction pointer or
> > + * stack pointer) it may need to propagate that change to the original @fregs
> > + * it retrieved the partial @regs from. Use this function to guarantee that
> > + * update happens.
> > + */
> > +static __always_inline void
> > +ftrace_partial_regs_update(const struct ftrace_regs *fregs, struct pt_regs *regs)
> > +{
> > + ftrace_regs_set_instruction_pointer(fregs, instruction_pointer(regs));
> > + ftrace_regs_set_return_value(fregs, regs_return_value(regs));
> > +}
>
> I think the AI thingy is right about dropping the const qualifier here
yes, will resend, thanks
jirka
> but overall I prefer this to the previous revisions. Thanks for sticking
> with it!
>
> Acked-by: Will Deacon <will@kernel.org>
>
> Will
^ permalink raw reply
* Re: [PATCH V4 2/2] mm/khugepaged: retry with sync writeback for MADV_COLLAPSE
From: Garg, Shivank @ 2026-01-10 18:20 UTC (permalink / raw)
To: David Hildenbrand (Red Hat), Andrew Morton, Lorenzo Stoakes
Cc: Zi Yan, Baolin Wang, Liam R . Howlett, Nico Pache, Ryan Roberts,
Dev Jain, Barry Song, Lance Yang, Steven Rostedt,
Masami Hiramatsu, Mathieu Desnoyers, Zach O'Keefe, linux-mm,
linux-kernel, linux-trace-kernel, Branden Moore
In-Reply-To: <3cc27720-8f53-4b6f-9202-42b3b73928b8@kernel.org>
On 1/9/2026 8:16 PM, David Hildenbrand (Red Hat) wrote:
> On 12/15/25 09:46, Shivank Garg wrote:
>> When MADV_COLLAPSE is called on file-backed mappings (e.g., executable
>> text sections), the pages may still be dirty from recent writes.
>> collapse_file() will trigger async writeback and fail with
>> SCAN_PAGE_DIRTY_OR_WRITEBACK (-EAGAIN).
>>
>> MADV_COLLAPSE is a synchronous operation where userspace expects
>> immediate results. If the collapse fails due to dirty pages, perform
>> synchronous writeback on the specific range and retry once.
>>
>> This avoids spurious failures for freshly written executables while
>> avoiding unnecessary synchronous I/O for mappings that are already clean.
>>
>> Reported-by: Branden Moore <Branden.Moore@amd.com>
>> Closes: https://lore.kernel.org/all/4e26fe5e-7374-467c-a333-9dd48f85d7cc@amd.com
>> Fixes: 34488399fa08 ("mm/madvise: add file and shmem support to MADV_COLLAPSE")
>> Suggested-by: David Hildenbrand <david@kernel.org>
>> Tested-by: Lance Yang <lance.yang@linux.dev>
>> Signed-off-by: Shivank Garg <shivankg@amd.com>
>> ---
>> mm/khugepaged.c | 40 ++++++++++++++++++++++++++++++++++++++++
>> 1 file changed, 40 insertions(+)
>>
>> diff --git a/mm/khugepaged.c b/mm/khugepaged.c
>> index 219dfa2e523c..6c8c35d3e0c9 100644
>> --- a/mm/khugepaged.c
>> +++ b/mm/khugepaged.c
>> @@ -22,6 +22,7 @@
>> #include <linux/dax.h>
>> #include <linux/ksm.h>
>> #include <linux/pgalloc.h>
>> +#include <linux/backing-dev.h>
>> #include <asm/tlb.h>
>> #include "internal.h"
>> @@ -2787,9 +2788,11 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
>> hend = end & HPAGE_PMD_MASK;
>> for (addr = hstart; addr < hend; addr += HPAGE_PMD_SIZE) {
>> + bool retried = false;
>> int result = SCAN_FAIL;
>> if (!mmap_locked) {
>> +retry:
>
> Jumping into an if block is nasty :)
>
>> cond_resched();
>> mmap_read_lock(mm);
>> mmap_locked = true;
>> @@ -2819,6 +2822,43 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
>> if (!mmap_locked)
>> *lock_dropped = true;
>> + /*
>> + * If the file-backed VMA has dirty pages, the scan triggers
>> + * async writeback and returns SCAN_PAGE_DIRTY_OR_WRITEBACK.
>> + * Since MADV_COLLAPSE is sync, we force sync writeback and
>> + * retry once.
>> + */
>> + if (result == SCAN_PAGE_DIRTY_OR_WRITEBACK && !retried) {
>> + /*
>> + * File scan drops the lock. We must re-acquire it to
>> + * safely inspect the VMA and hold the file reference.
>> + */
>> + if (!mmap_locked) {
>> + cond_resched();
>> + mmap_read_lock(mm);
>> + mmap_locked = true;
>> + result = hugepage_vma_revalidate(mm, addr, false, &vma, cc);
>> + if (result != SCAN_SUCCEED)
>> + goto handle_result;
>> + }
>> +
>> + if (!vma_is_anonymous(vma) && vma->vm_file &&
>> + mapping_can_writeback(vma->vm_file->f_mapping)) {
>> + struct file *file = get_file(vma->vm_file);
>> + pgoff_t pgoff = linear_page_index(vma, addr);
>> + loff_t lstart = (loff_t)pgoff << PAGE_SHIFT;
>> + loff_t lend = lstart + HPAGE_PMD_SIZE - 1;
>> +
>> + mmap_read_unlock(mm);
>> + mmap_locked = false;
>> + *lock_dropped = true;
>> + filemap_write_and_wait_range(file->f_mapping, lstart, lend);
>> + fput(file);
>> + retried = true;
>> + goto retry;
>> + }
>> + }
>> +
>
> This looks a bit complicated. Can't we move that handing up, where we have most of that
> information already? Or am I missing something important?
>
> diff --git a/mm/khugepaged.c b/mm/khugepaged.c
> index 97d1b2824386f..c7271877c5220 100644
> --- a/mm/khugepaged.c
> +++ b/mm/khugepaged.c
> @@ -22,6 +22,7 @@
> #include <linux/dax.h>
> #include <linux/ksm.h>
> #include <linux/pgalloc.h>
> +#include <linux/backing-dev.h>
>
> #include <asm/tlb.h>
> #include "internal.h"
> @@ -2786,7 +2787,9 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
>
> for (addr = hstart; addr < hend; addr += HPAGE_PMD_SIZE) {
> int result = SCAN_FAIL;
> + bool triggered_wb = false;
>
> +retry:
> if (!mmap_locked) {
> cond_resched();
> mmap_read_lock(mm);
> @@ -2809,6 +2812,16 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
> mmap_locked = false;
*lock_dropped = true;
> result = hpage_collapse_scan_file(mm, addr, file, pgoff,
> cc);
> +
> + if (result == SCAN_PAGE_DIRTY_OR_WRITEBACK && !triggered_wb &&
> + mapping_can_writeback(file->f_mapping)) {
> + loff_t lstart = (loff_t)pgoff << PAGE_SHIFT;
> + loff_t lend = lstart + HPAGE_PMD_SIZE - 1;
> +
> + filemap_write_and_wait_range(file->f_mapping, lstart, lend);
> + triggered_wb = true;
fput(file);
> + goto retry;
> + }
> fput(file);
> } else {
> result = hpage_collapse_scan_pmd(mm, vma, addr,
>
>
Thank you for the suggestion, this approach looks much simpler.
There are two small nits I observed:
1. In the retry loop, it is possible that we reacquire the mmap_lock and set
mmap_locked to true. This can cause issues later when we do:
if (!mmap_locked)
*lock_dropped = true;
because the caller would no longer see that the lock was dropped earlier.
2. We need an fput() to balance the file reference taken at line 2795.
fixed patch: (tested it no longer reproduce the issue):
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 87ca577c2668..b88bbb54f109 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -21,6 +21,7 @@
#include <linux/shmem_fs.h>
#include <linux/dax.h>
#include <linux/ksm.h>
+#include <linux/backing-dev.h>
#include <asm/tlb.h>
#include <asm/pgalloc.h>
@@ -2771,7 +2772,9 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
for (addr = hstart; addr < hend; addr += HPAGE_PMD_SIZE) {
int result = SCAN_FAIL;
+ bool triggered_wb = false;
+retry:
if (!mmap_locked) {
cond_resched();
mmap_read_lock(mm);
@@ -2794,8 +2797,20 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
mmap_read_unlock(mm);
mmap_locked = false;
+ *lock_dropped = true;
result = hpage_collapse_scan_file(mm, addr, file, pgoff,
cc);
+
+ if (result == SCAN_PAGE_DIRTY_OR_WRITEBACK && !triggered_wb &&
+ mapping_can_writeback(file->f_mapping)) {
+ loff_t lstart = (loff_t)pgoff << PAGE_SHIFT;
+ loff_t lend = lstart + HPAGE_PMD_SIZE - 1;
+
+ filemap_write_and_wait_range(file->f_mapping, lstart, lend);
+ triggered_wb = true;
+ fput(file);
+ goto retry;
+ }
fput(file);
} else {
result = hpage_collapse_scan_pmd(mm, vma, addr,
^ permalink raw reply related
* Re: [v3 PATCH 0/1] tracing: Add bitmask-list option for human-readable bitmask display
From: Aaron Tomlin @ 2026-01-10 21:06 UTC (permalink / raw)
To: rostedt, mhiramat, mark.rutland, mathieu.desnoyers, corbet
Cc: sean, linux-kernel, linux-trace-kernel, linux-doc
In-Reply-To: <20251226160724.2246493-1-atomlin@atomlin.com>
[-- Attachment #1: Type: text/plain, Size: 1038 bytes --]
On Fri, Dec 26, 2025 at 11:07:23AM -0500, Aaron Tomlin wrote:
> Hi Steve,
>
> This patch adds support for displaying bitmasks in human-readable list
> format (e.g., 0,2-5,7) in addition to the default hexadecimal bitmap
> representation. This is particularly useful when tracing CPU masks and
> other large bitmasks where individual bit positions are more meaningful
> than their hexadecimal encoding.
>
> When the "bitmask-list" option is enabled, the printk "%*pbl" format
> specifier is used to render bitmasks as comma-separated ranges, making
> trace output easier to interpret for complex CPU configurations and
> large bitmask values.
>
> This iteration incorporates the use of iter->tmp_seq to ensure the
> implementation is robust, instance-aware, and free from buffer contention
> or duplication issues.
>
> Please let me know your thoughts.
Hi Steve,
I would like to ask if this iteration is suitable for inclusion, or should
any further refinements be made?
Kind regards,
--
Aaron Tomlin
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply
* Re: [v3 PATCH 0/1] tracing: Add bitmask-list option for human-readable bitmask display
From: Steven Rostedt @ 2026-01-10 21:10 UTC (permalink / raw)
To: Aaron Tomlin
Cc: mhiramat, mark.rutland, mathieu.desnoyers, corbet, sean,
linux-kernel, linux-trace-kernel, linux-doc
In-Reply-To: <sm4xoemzv7d5pkyudybpxydriukvnlkwiaxrbh3qgevpnxphha@x7gberqhyjdo>
On Sat, 10 Jan 2026 16:06:02 -0500
Aaron Tomlin <atomlin@atomlin.com> wrote:
> On Fri, Dec 26, 2025 at 11:07:23AM -0500, Aaron Tomlin wrote:
> > Hi Steve,
> >
> > This patch adds support for displaying bitmasks in human-readable list
> > format (e.g., 0,2-5,7) in addition to the default hexadecimal bitmap
> > representation. This is particularly useful when tracing CPU masks and
> > other large bitmasks where individual bit positions are more meaningful
> > than their hexadecimal encoding.
> >
> > When the "bitmask-list" option is enabled, the printk "%*pbl" format
> > specifier is used to render bitmasks as comma-separated ranges, making
> > trace output easier to interpret for complex CPU configurations and
> > large bitmask values.
> >
> > This iteration incorporates the use of iter->tmp_seq to ensure the
> > implementation is robust, instance-aware, and free from buffer contention
> > or duplication issues.
> >
> > Please let me know your thoughts.
>
> Hi Steve,
>
> I would like to ask if this iteration is suitable for inclusion, or should
> any further refinements be made?
>
You can always check the status of a patch from patchwork. This one is here:
https://patchwork.kernel.org/project/linux-trace-kernel/patch/20251226160724.2246493-2-atomlin@atomlin.com/
The states are:
New - I haven't looked at it.
Under Review - I have it in a local queue (can still be rejected)
Queued - It's moved to linux-next
Accepted - It's in Linus's tree
Other states I will most likely have a response to.
This one is currently in the "Under Review" state. It also means I haven't
found anything wrong with it. It just hasn't gone through all my tests.
-- Steve
>
> Kind regards,
^ permalink raw reply
* Re: [PATCH V4 2/2] mm/khugepaged: retry with sync writeback for MADV_COLLAPSE
From: David Hildenbrand (Red Hat) @ 2026-01-11 11:29 UTC (permalink / raw)
To: Garg, Shivank, Andrew Morton, Lorenzo Stoakes
Cc: Zi Yan, Baolin Wang, Liam R . Howlett, Nico Pache, Ryan Roberts,
Dev Jain, Barry Song, Lance Yang, Steven Rostedt,
Masami Hiramatsu, Mathieu Desnoyers, Zach O'Keefe, linux-mm,
linux-kernel, linux-trace-kernel, Branden Moore
In-Reply-To: <dbbfc5f1-d9e5-4d39-a2df-e35d6ba73063@amd.com>
On 1/10/26 19:20, Garg, Shivank wrote:
>
>
> On 1/9/2026 8:16 PM, David Hildenbrand (Red Hat) wrote:
>> On 12/15/25 09:46, Shivank Garg wrote:
>>> When MADV_COLLAPSE is called on file-backed mappings (e.g., executable
>>> text sections), the pages may still be dirty from recent writes.
>>> collapse_file() will trigger async writeback and fail with
>>> SCAN_PAGE_DIRTY_OR_WRITEBACK (-EAGAIN).
>>>
>>> MADV_COLLAPSE is a synchronous operation where userspace expects
>>> immediate results. If the collapse fails due to dirty pages, perform
>>> synchronous writeback on the specific range and retry once.
>>>
>>> This avoids spurious failures for freshly written executables while
>>> avoiding unnecessary synchronous I/O for mappings that are already clean.
>>>
>>> Reported-by: Branden Moore <Branden.Moore@amd.com>
>>> Closes: https://lore.kernel.org/all/4e26fe5e-7374-467c-a333-9dd48f85d7cc@amd.com
>>> Fixes: 34488399fa08 ("mm/madvise: add file and shmem support to MADV_COLLAPSE")
>>> Suggested-by: David Hildenbrand <david@kernel.org>
>>> Tested-by: Lance Yang <lance.yang@linux.dev>
>>> Signed-off-by: Shivank Garg <shivankg@amd.com>
>>> ---
>>> mm/khugepaged.c | 40 ++++++++++++++++++++++++++++++++++++++++
>>> 1 file changed, 40 insertions(+)
>>>
>>> diff --git a/mm/khugepaged.c b/mm/khugepaged.c
>>> index 219dfa2e523c..6c8c35d3e0c9 100644
>>> --- a/mm/khugepaged.c
>>> +++ b/mm/khugepaged.c
>>> @@ -22,6 +22,7 @@
>>> #include <linux/dax.h>
>>> #include <linux/ksm.h>
>>> #include <linux/pgalloc.h>
>>> +#include <linux/backing-dev.h>
>>> #include <asm/tlb.h>
>>> #include "internal.h"
>>> @@ -2787,9 +2788,11 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
>>> hend = end & HPAGE_PMD_MASK;
>>> for (addr = hstart; addr < hend; addr += HPAGE_PMD_SIZE) {
>>> + bool retried = false;
>>> int result = SCAN_FAIL;
>>> if (!mmap_locked) {
>>> +retry:
>>
>> Jumping into an if block is nasty :)
>>
>>> cond_resched();
>>> mmap_read_lock(mm);
>>> mmap_locked = true;
>>> @@ -2819,6 +2822,43 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
>>> if (!mmap_locked)
>>> *lock_dropped = true;
>>> + /*
>>> + * If the file-backed VMA has dirty pages, the scan triggers
>>> + * async writeback and returns SCAN_PAGE_DIRTY_OR_WRITEBACK.
>>> + * Since MADV_COLLAPSE is sync, we force sync writeback and
>>> + * retry once.
>>> + */
>>> + if (result == SCAN_PAGE_DIRTY_OR_WRITEBACK && !retried) {
>>> + /*
>>> + * File scan drops the lock. We must re-acquire it to
>>> + * safely inspect the VMA and hold the file reference.
>>> + */
>>> + if (!mmap_locked) {
>>> + cond_resched();
>>> + mmap_read_lock(mm);
>>> + mmap_locked = true;
>>> + result = hugepage_vma_revalidate(mm, addr, false, &vma, cc);
>>> + if (result != SCAN_SUCCEED)
>>> + goto handle_result;
>>> + }
>>> +
>>> + if (!vma_is_anonymous(vma) && vma->vm_file &&
>>> + mapping_can_writeback(vma->vm_file->f_mapping)) {
>>> + struct file *file = get_file(vma->vm_file);
>>> + pgoff_t pgoff = linear_page_index(vma, addr);
>>> + loff_t lstart = (loff_t)pgoff << PAGE_SHIFT;
>>> + loff_t lend = lstart + HPAGE_PMD_SIZE - 1;
>>> +
>>> + mmap_read_unlock(mm);
>>> + mmap_locked = false;
>>> + *lock_dropped = true;
>>> + filemap_write_and_wait_range(file->f_mapping, lstart, lend);
>>> + fput(file);
>>> + retried = true;
>>> + goto retry;
>>> + }
>>> + }
>>> +
>>
>> This looks a bit complicated. Can't we move that handing up, where we have most of that
>> information already? Or am I missing something important?
>>
>> diff --git a/mm/khugepaged.c b/mm/khugepaged.c
>> index 97d1b2824386f..c7271877c5220 100644
>> --- a/mm/khugepaged.c
>> +++ b/mm/khugepaged.c
>> @@ -22,6 +22,7 @@
>> #include <linux/dax.h>
>> #include <linux/ksm.h>
>> #include <linux/pgalloc.h>
>> +#include <linux/backing-dev.h>
>>
>> #include <asm/tlb.h>
>> #include "internal.h"
>> @@ -2786,7 +2787,9 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
>>
>> for (addr = hstart; addr < hend; addr += HPAGE_PMD_SIZE) {
>> int result = SCAN_FAIL;
>> + bool triggered_wb = false;
>>
>> +retry:
>> if (!mmap_locked) {
>> cond_resched();
>> mmap_read_lock(mm);
>> @@ -2809,6 +2812,16 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
>> mmap_locked = false;
>
> *lock_dropped = true;
>> result = hpage_collapse_scan_file(mm, addr, file, pgoff,
>> cc);
>> +
>> + if (result == SCAN_PAGE_DIRTY_OR_WRITEBACK && !triggered_wb &&
>> + mapping_can_writeback(file->f_mapping)) {
>> + loff_t lstart = (loff_t)pgoff << PAGE_SHIFT;
>> + loff_t lend = lstart + HPAGE_PMD_SIZE - 1;
>> +
>> + filemap_write_and_wait_range(file->f_mapping, lstart, lend);
>> + triggered_wb = true;
>
> fput(file);
>
>> + goto retry;
>> + }
>> fput(file);
>> } else {
>> result = hpage_collapse_scan_pmd(mm, vma, addr,
>>
>>
>
> Thank you for the suggestion, this approach looks much simpler.
>
> There are two small nits I observed:
Yeah, was a quick untested hack to see if this can be simplified :)
>
> 1. In the retry loop, it is possible that we reacquire the mmap_lock and set
> mmap_locked to true. This can cause issues later when we do:
>
> if (!mmap_locked)
> *lock_dropped = true;
That whole logic of having two variables that express whether locks have
been taken/dropped is just absolutely confusing. Any way we can clean
that up?
>
> because the caller would no longer see that the lock was dropped earlier.
>
> 2. We need an fput() to balance the file reference taken at line 2795.
Ah, yes, makes sense. Having a single fput() would be nicer, but that
would require yet another temporary variable.
--
Cheers
David
^ permalink raw reply
* [PATCH] function_graph: Fix missing FGRAPH_ENTRY_ARGS() call in print_graph_retval()
From: Donglin Peng @ 2026-01-11 14:33 UTC (permalink / raw)
To: rostedt; +Cc: mhiramat, linux-trace-kernel, linux-kernel, Donglin Peng
From: Donglin Peng <pengdonglin@xiaomi.com>
Call FGRAPH_ENTRY_ARGS(entry) to retrieve the correct arguments pointer
when displaying function parameters in print_graph_retval().
Fixes: f83ac7544fbf ("function_graph: Enable funcgraph-args and funcgraph-retaddr to work simultaneously")
Signed-off-by: Donglin Peng <pengdonglin@xiaomi.com>
---
kernel/trace/trace_functions_graph.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/kernel/trace/trace_functions_graph.c b/kernel/trace/trace_functions_graph.c
index b1e9c9913309..1de6f1573621 100644
--- a/kernel/trace/trace_functions_graph.c
+++ b/kernel/trace/trace_functions_graph.c
@@ -901,7 +901,7 @@ static void print_graph_retval(struct trace_seq *s, struct ftrace_graph_ent_entr
trace_seq_printf(s, "%ps", func);
if (args_size >= FTRACE_REGS_MAX_ARGS * sizeof(long)) {
- print_function_args(s, entry->args, (unsigned long)func);
+ print_function_args(s, FGRAPH_ENTRY_ARGS(entry), (unsigned long)func);
trace_seq_putc(s, ';');
} else
trace_seq_puts(s, "();");
--
2.34.1
^ permalink raw reply related
* [PATCH v12 0/3] mm: Fix OOM killer inaccuracy on large many-core systems
From: Mathieu Desnoyers @ 2026-01-11 15:02 UTC (permalink / raw)
To: Andrew Morton
Cc: linux-kernel, Mathieu Desnoyers, Paul E. McKenney, Steven Rostedt,
Masami Hiramatsu, Dennis Zhou, Tejun Heo, Christoph Lameter,
Martin Liu, David Rientjes, christian.koenig, Shakeel Butt,
SeongJae Park, Michal Hocko, Johannes Weiner, Sweet Tea Dorminy,
Lorenzo Stoakes, Liam R . Howlett, Mike Rapoport,
Suren Baghdasaryan, Vlastimil Babka, Christian Brauner, Wei Yang,
David Hildenbrand, Miaohe Lin, Al Viro, linux-mm,
linux-trace-kernel, Yu Zhao, Roman Gushchin, Mateusz Guzik,
Matthew Wilcox, Baolin Wang, Aboorva Devarajan
Introduce hierarchical per-cpu counters and use them for RSS tracking to
fix the per-mm RSS tracking which has become too inaccurate for OOM
killer purposes on large many-core systems.
The following rss tracking issues were noted by Sweet Tea Dorminy [1],
which lead to picking wrong tasks as OOM kill target:
Recently, several internal services had an RSS usage regression as part of a
kernel upgrade. Previously, they were on a pre-6.2 kernel and were able to
read RSS statistics in a backup watchdog process to monitor and decide if
they'd overrun their memory budget. Now, however, a representative service
with five threads, expected to use about a hundred MB of memory, on a 250-cpu
machine had memory usage tens of megabytes different from the expected amount
-- this constituted a significant percentage of inaccuracy, causing the
watchdog to act.
This was a result of commit f1a7941243c1 ("mm: convert mm's rss stats
into percpu_counter") [1]. Previously, the memory error was bounded by
64*nr_threads pages, a very livable megabyte. Now, however, as a result of
scheduler decisions moving the threads around the CPUs, the memory error could
be as large as a gigabyte.
This is a really tremendous inaccuracy for any few-threaded program on a
large machine and impedes monitoring significantly. These stat counters are
also used to make OOM killing decisions, so this additional inaccuracy could
make a big difference in OOM situations -- either resulting in the wrong
process being killed, or in less memory being returned from an OOM-kill than
expected.
The approach proposed here is to replace this by the hierarchical
per-cpu counters, which bounds the inaccuracy based on the system
topology with O(N*logN).
Notable changes for v12:
- Reduce per-CPU counters memory allocation size to sizeof long
(fixing mixup with sizeof intermediate cache line aligned items).
- Use "long" counters types rather than "int".
- get_mm_counter_sum() returns a precise sum.
- Introduce and use functions to calculate the min/max possible precise
sum values associated with an approximate sum.
I've done moderate testing of this series on a 256-core VM with 128GB
RAM. Figuring out whether this indeed helps solve issues with real-life
workloads will require broader feedback from the community.
This series is based on v6.19-rc4, on top of the following two
preparation series:
https://lore.kernel.org/linux-mm/20251224173358.647691-1-mathieu.desnoyers@efficios.com/T/#t
https://lore.kernel.org/linux-mm/20251224173810.648699-1-mathieu.desnoyers@efficios.com/T/#t
Andrew, this series replaces v11, for testing in mm-new.
Thanks!
Mathieu
Link: https://lore.kernel.org/lkml/20250331223516.7810-2-sweettea-kernel@dorminy.me/ # [1]
To: Andrew Morton <akpm@linux-foundation.org>
Cc: "Paul E. McKenney" <paulmck@kernel.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Dennis Zhou <dennis@kernel.org>
Cc: Tejun Heo <tj@kernel.org>
Cc: Christoph Lameter <cl@linux.com>
Cc: Martin Liu <liumartin@google.com>
Cc: David Rientjes <rientjes@google.com>
Cc: christian.koenig@amd.com
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: SeongJae Park <sj@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Sweet Tea Dorminy <sweettea-kernel@dorminy.me>
Cc: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
Cc: "Liam R . Howlett" <liam.howlett@oracle.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Cc: Christian Brauner <brauner@kernel.org>
Cc: Wei Yang <richard.weiyang@gmail.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: Miaohe Lin <linmiaohe@huawei.com>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Cc: linux-mm@kvack.org
Cc: linux-trace-kernel@vger.kernel.org
Cc: Yu Zhao <yuzhao@google.com>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Mateusz Guzik <mjguzik@gmail.com>
Cc: Matthew Wilcox <willy@infradead.org>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Aboorva Devarajan <aboorvad@linux.ibm.com>
Mathieu Desnoyers (3):
lib: Introduce hierarchical per-cpu counters
mm: Fix OOM killer inaccuracy on large many-core systems
mm: Implement precise OOM killer task selection
fs/proc/base.c | 2 +-
include/linux/mm.h | 49 +-
include/linux/mm_types.h | 54 ++-
include/linux/oom.h | 11 +-
include/linux/percpu_counter_tree.h | 344 ++++++++++++++
include/trace/events/kmem.h | 2 +-
init/main.c | 2 +
kernel/fork.c | 22 +-
lib/Makefile | 1 +
lib/percpu_counter_tree.c | 702 ++++++++++++++++++++++++++++
mm/oom_kill.c | 82 +++-
11 files changed, 1222 insertions(+), 49 deletions(-)
create mode 100644 include/linux/percpu_counter_tree.h
create mode 100644 lib/percpu_counter_tree.c
--
2.39.5
^ permalink raw reply
* [PATCH v12 3/3] mm: Implement precise OOM killer task selection
From: Mathieu Desnoyers @ 2026-01-11 15:02 UTC (permalink / raw)
To: Andrew Morton
Cc: linux-kernel, Mathieu Desnoyers, Paul E. McKenney, Steven Rostedt,
Masami Hiramatsu, Dennis Zhou, Tejun Heo, Christoph Lameter,
Martin Liu, David Rientjes, christian.koenig, Shakeel Butt,
SeongJae Park, Michal Hocko, Johannes Weiner, Sweet Tea Dorminy,
Lorenzo Stoakes, Liam R . Howlett, Mike Rapoport,
Suren Baghdasaryan, Vlastimil Babka, Christian Brauner, Wei Yang,
David Hildenbrand, Miaohe Lin, Al Viro, linux-mm,
linux-trace-kernel, Yu Zhao, Roman Gushchin, Mateusz Guzik,
Matthew Wilcox, Baolin Wang, Aboorva Devarajan
In-Reply-To: <20260111150249.1222944-1-mathieu.desnoyers@efficios.com>
Use the hierarchical tree counter approximation to implement the OOM
killer task selection with a 2-pass algorithm. The first pass selects
the process that has the highest badness points approximation, and the
second pass compares each process using the current max badness points
approximation.
The second pass uses an approximate comparison to eliminate all processes
which are below the current max badness points approximation accuracy
range.
Summing the per-CPU counters to calculate the precise badness of tasks
is only required for tasks with an approximate badness within the
accuracy range of the current max points value.
Limit to 16 the maximum number of badness sums allowed for an OOM killer
task selection before falling back to the approximated comparison. This
ensures bounded execution time for scenarios where many tasks have
badness within the accuracy of the maximum badness approximation.
Tested with the following script:
#!/bin/sh
for a in $(seq 1 10); do (tail /dev/zero &); done
sleep 5
for a in $(seq 1 10); do (tail /dev/zero &); done
sleep 2
for a in $(seq 1 10); do (tail /dev/zero &); done
echo "Waiting for tasks to finish"
wait
Results: OOM kill order on a 128GB memory system
================================================
* systemd and sd-pam are chosen first due to their oom_score_ajd:100:
Out of memory: Killed process 3502 (systemd) total-vm:20096kB, anon-rss:0kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:72kB oom_score_adj:100
Out of memory: Killed process 3503 ((sd-pam)) total-vm:21432kB, anon-rss:0kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:76kB oom_score_adj:100
* The first batch of 10 processes are gradually killed, consecutively
picking the one that uses the most memory. The fact that we are
freeing memory from the previous processes increases the threshold
at which the remaining processes of that group are killed. Processes
from the second and third batches of 10 processes have time to start
before we complete killing the first 10 processes:
Out of memory: Killed process 3703 (tail) total-vm:6591280kB, anon-rss:6578176kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:12936kB oom_score_adj:0
Out of memory: Killed process 3705 (tail) total-vm:6731716kB, anon-rss:6709248kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:13212kB oom_score_adj:0
Out of memory: Killed process 3707 (tail) total-vm:6977216kB, anon-rss:6946816kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:13692kB oom_score_adj:0
Out of memory: Killed process 3699 (tail) total-vm:7205640kB, anon-rss:7184384kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:14136kB oom_score_adj:0
Out of memory: Killed process 3713 (tail) total-vm:7463204kB, anon-rss:7438336kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:14644kB oom_score_adj:0
Out of memory: Killed process 3701 (tail) total-vm:7739204kB, anon-rss:7716864kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:15180kB oom_score_adj:0
Out of memory: Killed process 3709 (tail) total-vm:8050176kB, anon-rss:8028160kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:15792kB oom_score_adj:0
Out of memory: Killed process 3711 (tail) total-vm:8362236kB, anon-rss:8339456kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:16404kB oom_score_adj:0
Out of memory: Killed process 3715 (tail) total-vm:8649360kB, anon-rss:8634368kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:16972kB oom_score_adj:0
Out of memory: Killed process 3697 (tail) total-vm:8951788kB, anon-rss:8929280kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:17560kB oom_score_adj:0
* Even though there is a 2 seconds delay between the 2nd and 3rd batches
those appear to execute in mixed order. Therefore, let's consider them
as a single batch of 20 processes. We are hitting oom at a lower
memory threshold because at this point the 20 remaining proceses are
running rather than the previous 10. The process with highest memory
usage is selected for oom, thus making room for the remaining
processes so they can use more memory before they fill the available
memory, thus explaining why the memory use for selected processes
gradually increases, until all system memory is used by the last one:
Out of memory: Killed process 3731 (tail) total-vm:7089868kB, anon-rss:7077888kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:13912kB oom_score_adj:0
Out of memory: Killed process 3721 (tail) total-vm:7417248kB, anon-rss:7405568kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:14556kB oom_score_adj:0
Out of memory: Killed process 3729 (tail) total-vm:7795864kB, anon-rss:7766016kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:15300kB oom_score_adj:0
Out of memory: Killed process 3723 (tail) total-vm:8259620kB, anon-rss:8224768kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:16208kB oom_score_adj:0
Out of memory: Killed process 3737 (tail) total-vm:8695984kB, anon-rss:8667136kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:17060kB oom_score_adj:0
Out of memory: Killed process 3735 (tail) total-vm:9295980kB, anon-rss:9265152kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:18240kB oom_score_adj:0
Out of memory: Killed process 3727 (tail) total-vm:9907900kB, anon-rss:9895936kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:19428kB oom_score_adj:0
Out of memory: Killed process 3719 (tail) total-vm:10631248kB, anon-rss:10600448kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:20844kB oom_score_adj:0
Out of memory: Killed process 3733 (tail) total-vm:11341720kB, anon-rss:11321344kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:22232kB oom_score_adj:0
Out of memory: Killed process 3725 (tail) total-vm:12348124kB, anon-rss:12320768kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:24204kB oom_score_adj:0
Out of memory: Killed process 3759 (tail) total-vm:12978888kB, anon-rss:12967936kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:25440kB oom_score_adj:0
Out of memory: Killed process 3751 (tail) total-vm:14386412kB, anon-rss:14352384kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:28196kB oom_score_adj:0
Out of memory: Killed process 3741 (tail) total-vm:16153168kB, anon-rss:16130048kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:31652kB oom_score_adj:0
Out of memory: Killed process 3753 (tail) total-vm:18414856kB, anon-rss:18391040kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:36076kB oom_score_adj:0
Out of memory: Killed process 3745 (tail) total-vm:21389456kB, anon-rss:21356544kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:41904kB oom_score_adj:0
Out of memory: Killed process 3747 (tail) total-vm:25659348kB, anon-rss:25632768kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:50260kB oom_score_adj:0
Out of memory: Killed process 3755 (tail) total-vm:32030820kB, anon-rss:32006144kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:62720kB oom_score_adj:0
Out of memory: Killed process 3743 (tail) total-vm:42648456kB, anon-rss:42614784kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:83504kB oom_score_adj:0
Out of memory: Killed process 3757 (tail) total-vm:63971028kB, anon-rss:63938560kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:125228kB oom_score_adj:0
Out of memory: Killed process 3749 (tail) total-vm:127799660kB, anon-rss:127778816kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:250140kB oom_score_adj:0
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: "Paul E. McKenney" <paulmck@kernel.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Dennis Zhou <dennis@kernel.org>
Cc: Tejun Heo <tj@kernel.org>
Cc: Christoph Lameter <cl@linux.com>
Cc: Martin Liu <liumartin@google.com>
Cc: David Rientjes <rientjes@google.com>
Cc: christian.koenig@amd.com
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: SeongJae Park <sj@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Sweet Tea Dorminy <sweettea-kernel@dorminy.me>
Cc: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
Cc: "Liam R . Howlett" <liam.howlett@oracle.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Cc: Christian Brauner <brauner@kernel.org>
Cc: Wei Yang <richard.weiyang@gmail.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: Miaohe Lin <linmiaohe@huawei.com>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Cc: linux-mm@kvack.org
Cc: linux-trace-kernel@vger.kernel.org
Cc: Yu Zhao <yuzhao@google.com>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Mateusz Guzik <mjguzik@gmail.com>
Cc: Matthew Wilcox <willy@infradead.org>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Aboorva Devarajan <aboorvad@linux.ibm.com>
---
Changes since v11:
- get_mm_counter_sum() returns a precise sum.
- Use unsigned long type rather than unsigned int for accuracy.
- Use precise sum min/max calculation to compare the chosen vs
current points.
- The first pass finds the maximum task's min points. The
second pass eliminates all tasks for which the max points
are below the currently chosen min points, and uses a precise
sum to validate the candidates which are possibly in range.
---
fs/proc/base.c | 2 +-
include/linux/mm.h | 34 ++++++++++++++++---
include/linux/oom.h | 11 +++++-
kernel/fork.c | 2 +-
mm/oom_kill.c | 82 +++++++++++++++++++++++++++++++++++++--------
5 files changed, 109 insertions(+), 22 deletions(-)
diff --git a/fs/proc/base.c b/fs/proc/base.c
index 4eec684baca9..d75d0ce97032 100644
--- a/fs/proc/base.c
+++ b/fs/proc/base.c
@@ -589,7 +589,7 @@ static int proc_oom_score(struct seq_file *m, struct pid_namespace *ns,
unsigned long points = 0;
long badness;
- badness = oom_badness(task, totalpages);
+ badness = oom_badness(task, totalpages, false, NULL, NULL);
/*
* Special case OOM_SCORE_ADJ_MIN for all others scale the
* badness value into [0, 2000] range which we have been
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 6d938b3e3709..680f2811702e 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -2855,14 +2855,32 @@ static inline struct percpu_counter_tree_level_item *get_rss_stat_items(struct m
/*
* per-process(per-mm_struct) statistics.
*/
+static inline unsigned long __get_mm_counter(struct mm_struct *mm, int member, bool approximate,
+ unsigned long *accuracy_under, unsigned long *accuracy_over)
+{
+ if (approximate) {
+ if (accuracy_under && accuracy_over) {
+ unsigned long under, over;
+
+ percpu_counter_tree_approximate_accuracy_range(&mm->rss_stat[member], &under, &over);
+ *accuracy_under += under;
+ *accuracy_over += over;
+ }
+ return percpu_counter_tree_approximate_sum_positive(&mm->rss_stat[member]);
+ } else {
+ return percpu_counter_tree_precise_sum_positive(&mm->rss_stat[member]);
+ }
+}
+
static inline unsigned long get_mm_counter(struct mm_struct *mm, int member)
{
- return percpu_counter_tree_approximate_sum_positive(&mm->rss_stat[member]);
+ return __get_mm_counter(mm, member, true, NULL, NULL);
}
+
static inline unsigned long get_mm_counter_sum(struct mm_struct *mm, int member)
{
- return percpu_counter_tree_precise_sum_positive(&mm->rss_stat[member]);
+ return __get_mm_counter(mm, member, false, NULL, NULL);
}
void mm_trace_rss_stat(struct mm_struct *mm, int member);
@@ -2903,11 +2921,17 @@ static inline int mm_counter(struct folio *folio)
return mm_counter_file(folio);
}
+static inline unsigned long __get_mm_rss(struct mm_struct *mm, bool approximate,
+ unsigned long *accuracy_under, unsigned long *accuracy_over)
+{
+ return __get_mm_counter(mm, MM_FILEPAGES, approximate, accuracy_under, accuracy_over) +
+ __get_mm_counter(mm, MM_ANONPAGES, approximate, accuracy_under, accuracy_over) +
+ __get_mm_counter(mm, MM_SHMEMPAGES, approximate, accuracy_under, accuracy_over);
+}
+
static inline unsigned long get_mm_rss(struct mm_struct *mm)
{
- return get_mm_counter(mm, MM_FILEPAGES) +
- get_mm_counter(mm, MM_ANONPAGES) +
- get_mm_counter(mm, MM_SHMEMPAGES);
+ return __get_mm_rss(mm, true, NULL, NULL);
}
static inline unsigned long get_mm_hiwater_rss(struct mm_struct *mm)
diff --git a/include/linux/oom.h b/include/linux/oom.h
index 7b02bc1d0a7e..f8e5bfaf7b39 100644
--- a/include/linux/oom.h
+++ b/include/linux/oom.h
@@ -48,6 +48,12 @@ struct oom_control {
unsigned long totalpages;
struct task_struct *chosen;
long chosen_points;
+ bool approximate;
+ /*
+ * Number of precise badness points sums performed by this task
+ * selection.
+ */
+ int nr_precise;
/* Used to print the constraint info. */
enum oom_constraint constraint;
@@ -97,7 +103,10 @@ static inline vm_fault_t check_stable_address_space(struct mm_struct *mm)
}
long oom_badness(struct task_struct *p,
- unsigned long totalpages);
+ unsigned long totalpages,
+ bool approximate,
+ unsigned long *accuracy_under,
+ unsigned long *accuracy_over);
extern bool out_of_memory(struct oom_control *oc);
diff --git a/kernel/fork.c b/kernel/fork.c
index 949ac019a7b1..8b56d81af734 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -632,7 +632,7 @@ static void check_mm(struct mm_struct *mm)
for (i = 0; i < NR_MM_COUNTERS; i++) {
if (unlikely(percpu_counter_tree_precise_compare_value(&mm->rss_stat[i], 0) != 0))
- pr_alert("BUG: Bad rss-counter state mm:%p type:%s val:%d Comm:%s Pid:%d\n",
+ pr_alert("BUG: Bad rss-counter state mm:%p type:%s val:%ld Comm:%s Pid:%d\n",
mm, resident_page_types[i],
percpu_counter_tree_precise_sum(&mm->rss_stat[i]),
current->comm,
diff --git a/mm/oom_kill.c b/mm/oom_kill.c
index 5eb11fbba704..740891be3267 100644
--- a/mm/oom_kill.c
+++ b/mm/oom_kill.c
@@ -53,6 +53,14 @@
#define CREATE_TRACE_POINTS
#include <trace/events/oom.h>
+/*
+ * Maximum number of badness sums allowed before using an approximated
+ * comparison. This ensures bounded execution time for scenarios where
+ * many tasks have badness within the accuracy of the maximum badness
+ * approximation.
+ */
+static int max_precise_badness_sums = 16;
+
static int sysctl_panic_on_oom;
static int sysctl_oom_kill_allocating_task;
static int sysctl_oom_dump_tasks = 1;
@@ -194,12 +202,16 @@ static bool should_dump_unreclaim_slab(void)
* oom_badness - heuristic function to determine which candidate task to kill
* @p: task struct of which task we should calculate
* @totalpages: total present RAM allowed for page allocation
+ * @approximate: whether the value can be approximated
+ * @accuracy_under: accuracy of the badness value approximation (under value)
+ * @accuracy_over: accuracy of the badness value approximation (over value)
*
* The heuristic for determining which task to kill is made to be as simple and
* predictable as possible. The goal is to return the highest value for the
* task consuming the most memory to avoid subsequent oom failures.
*/
-long oom_badness(struct task_struct *p, unsigned long totalpages)
+long oom_badness(struct task_struct *p, unsigned long totalpages, bool approximate,
+ unsigned long *accuracy_under, unsigned long *accuracy_over)
{
long points;
long adj;
@@ -228,7 +240,8 @@ long oom_badness(struct task_struct *p, unsigned long totalpages)
* The baseline for the badness score is the proportion of RAM that each
* task's rss, pagetable and swap space use.
*/
- points = get_mm_rss(p->mm) + get_mm_counter(p->mm, MM_SWAPENTS) +
+ points = __get_mm_rss(p->mm, approximate, accuracy_under, accuracy_over) +
+ __get_mm_counter(p->mm, MM_SWAPENTS, approximate, accuracy_under, accuracy_over) +
mm_pgtables_bytes(p->mm) / PAGE_SIZE;
task_unlock(p);
@@ -309,7 +322,8 @@ static enum oom_constraint constrained_alloc(struct oom_control *oc)
static int oom_evaluate_task(struct task_struct *task, void *arg)
{
struct oom_control *oc = arg;
- long points;
+ unsigned long accuracy_under = 0, accuracy_over = 0;
+ long points, points_min, points_max;
if (oom_unkillable_task(task))
goto next;
@@ -339,16 +353,43 @@ static int oom_evaluate_task(struct task_struct *task, void *arg)
goto select;
}
- points = oom_badness(task, oc->totalpages);
- if (points == LONG_MIN || points < oc->chosen_points)
- goto next;
+ points = oom_badness(task, oc->totalpages, true, &accuracy_under, &accuracy_over);
+ if (points != LONG_MIN) {
+ percpu_counter_tree_approximate_min_max_range(points,
+ accuracy_under, accuracy_over,
+ &points_min, &points_max);
+ }
+ if (oc->approximate) {
+ /*
+ * Keep the process which has the highest minimum
+ * possible points value based on approximation.
+ */
+ if (points == LONG_MIN || points_min < oc->chosen_points)
+ goto next;
+ } else {
+ /*
+ * Eliminate processes which are certainly below the
+ * chosen points minimum possible value with an
+ * approximation.
+ */
+ if (points == LONG_MIN || (long)(points_max - oc->chosen_points) < 0)
+ goto next;
+
+ if (oc->nr_precise < max_precise_badness_sums) {
+ oc->nr_precise++;
+ /* Precise evaluation. */
+ points_min = points_max = points = oom_badness(task, oc->totalpages, false, NULL, NULL);
+ if (points == LONG_MIN || (long)(points - oc->chosen_points) < 0)
+ goto next;
+ }
+ }
select:
if (oc->chosen)
put_task_struct(oc->chosen);
get_task_struct(task);
oc->chosen = task;
- oc->chosen_points = points;
+ oc->chosen_points = points_min;
next:
return 0;
abort:
@@ -358,14 +399,8 @@ static int oom_evaluate_task(struct task_struct *task, void *arg)
return 1;
}
-/*
- * Simple selection loop. We choose the process with the highest number of
- * 'points'. In case scan was aborted, oc->chosen is set to -1.
- */
-static void select_bad_process(struct oom_control *oc)
+static void select_bad_process_iter(struct oom_control *oc)
{
- oc->chosen_points = LONG_MIN;
-
if (is_memcg_oom(oc))
mem_cgroup_scan_tasks(oc->memcg, oom_evaluate_task, oc);
else {
@@ -379,6 +414,25 @@ static void select_bad_process(struct oom_control *oc)
}
}
+/*
+ * Simple selection loop. We choose the process with the highest number of
+ * 'points'. In case scan was aborted, oc->chosen is set to -1.
+ */
+static void select_bad_process(struct oom_control *oc)
+{
+ oc->chosen_points = LONG_MIN;
+ oc->nr_precise = 0;
+
+ /* Approximate scan. */
+ oc->approximate = true;
+ select_bad_process_iter(oc);
+ if (oc->chosen == (void *)-1UL)
+ return;
+ /* Precise scan. */
+ oc->approximate = false;
+ select_bad_process_iter(oc);
+}
+
static int dump_task(struct task_struct *p, void *arg)
{
struct oom_control *oc = arg;
--
2.39.5
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox