* Re: [RFC PATCH 1/2] locking: add mutex_lock_nospin()
From: Yafang Shao @ 2026-03-04 14:25 UTC (permalink / raw)
To: Peter Zijlstra
Cc: mingo, will, boqun, longman, rostedt, mhiramat, mark.rutland,
mathieu.desnoyers, linux-kernel, linux-trace-kernel, bpf
In-Reply-To: <20260304124148.GA2277644@noisy.programming.kicks-ass.net>
On Wed, Mar 4, 2026 at 8:41 PM Peter Zijlstra <peterz@infradead.org> wrote:
>
> On Wed, Mar 04, 2026 at 07:52:06PM +0800, Yafang Shao wrote:
> > On Wed, Mar 4, 2026 at 6:11 PM Peter Zijlstra <peterz@infradead.org> wrote:
> > >
> > > On Wed, Mar 04, 2026 at 05:37:31PM +0800, Yafang Shao wrote:
> > > > On Wed, Mar 4, 2026 at 5:03 PM Peter Zijlstra <peterz@infradead.org> wrote:
> > > > >
> > > > > On Wed, Mar 04, 2026 at 03:46:49PM +0800, Yafang Shao wrote:
> > > > > > Introduce mutex_lock_nospin(), a helper that disables optimistic spinning
> > > > > > on the owner for specific heavy locks. This prevents long spinning times
> > > > > > that can lead to latency spikes for other tasks on the same runqueue.
> > > > >
> > > > > This makes no sense; spinning stops on need_resched().
> > > >
> > > > Hello Peter,
> > > >
> > > > The condition to stop spinning on need_resched() relies on the mutex
> > > > owner remaining unchanged. However, when multiple tasks contend for
> > > > the same lock, the owner can change frequently. This creates a
> > > > potential TOCTOU (Time of Check to Time of Use) issue.
> > > >
> > > > mutex_optimistic_spin
> > > > owner = __mutex_trylock_or_owner(lock);
> > > > mutex_spin_on_owner
> > > > // the __mutex_owner(lock) might get a new owner.
> > > > while (__mutex_owner(lock) == owner)
> > > >
> > >
> > > How do these new owners become the owner? Are they succeeding the
> > > __mutex_trylock() that sits before mutex_optimistic_spin() and
> > > effectively starving the spinner?
> > >
> > > Something like the below would make a difference if that were so.
> >
> > The following change made no difference; concurrent runs still result
> > in prolonged system time.
> >
> > real 0m5.265s user 0m0.000s sys 0m4.921s
> > real 0m5.295s user 0m0.002s sys 0m4.697s
> > real 0m5.293s user 0m0.003s sys 0m4.844s
> > real 0m5.303s user 0m0.001s sys 0m4.511s
> > real 0m5.303s user 0m0.000s sys 0m4.694s
> > real 0m5.302s user 0m0.002s sys 0m4.677s
> > real 0m5.313s user 0m0.000s sys 0m4.837s
> > real 0m5.327s user 0m0.000s sys 0m4.808s
> > real 0m5.330s user 0m0.001s sys 0m4.893s
> > real 0m5.358s user 0m0.005s sys 0m4.919s
> >
> > Our kernel is not built with CONFIG_PREEMPT enabled, so prolonged
> > system time can lead to CPU pressure and potential latency spikes.
> > Since we can reliably reproduce this unnecessary spinning, why not
> > improve it to reduce the overhead?
>
> If you cannot explain what the problem is (you haven't), there is
> nothing to fix.
>
> Also, current kernels cannot be build without PREEMPT; and if you care
> about latency running a PREEMPT=n kernel is daft. That said,
> TIF_NEED_RESCHED should work irrespective of PREEMPT settings, the
> PREEMPT settings just affect when and how you end up in schedule().
>
> Even without PREEMPT, if there is a task waiting either the wakeup or
> the tick will set TIF_NEED_RESCHED and it should stop spinning. If there
> is no task waiting, there is no actual latency, just burning CPU time,
> and that isn't a problem per-se.
>
> What should happen is that the first spinner gets the lock next, the
> next spinner is then promoted to first spinner and so on.
>
> This chain continues, which means the lock owner is always
> on-cpu and good progress is being made and there is no CPU contention,
> or the spinner gets marked for preemption (as said, this does not
> require PREEMPT=y) and will stop spinning and go sleep, or the owner
> goes to sleep and all the spinners stop and also go sleep.
>
> Again, you have not said anything specific enough to figure out what
> happens on your end. You said the owner changes, this means there is
> progress made. What isn't clear is if any one particular spinner is
> starved (that would be a problem)
As far as I can tell, no spinner is starved. The spinner and the
impacted task are interleaved, which is expected behavior.
> or if this latency spike you observe
> is worse than would be from running a while(1) loop, in which case,
> that's just how it is.
The latency spike occurs because the impacted task must wait for the
spinner to voluntarily yield the CPU. In effect, the spinner behaves
similarly to a while (1) {} loop.
So the real problem here is that we should avoid unnecessary spinning.
Is there any reason we have to spin to speed up the ftrace_lock? I
believe not.
>
> What is not sane, is marking random locks with random properties just
> because random workload.
--
Regards
Yafang
^ permalink raw reply
* Re: [PATCH v2 001/110] vfs: introduce kino_t typedef and PRIino format macro
From: Christian Brauner @ 2026-03-04 14:10 UTC (permalink / raw)
To: Jeff Layton
Cc: Christoph Hellwig, Darrick J. Wong, Theodore Tso, Alexander Viro,
Jan Kara, Steven Rostedt, Masami Hiramatsu, Mathieu Desnoyers,
Dan Williams, Matthew Wilcox, Eric Biggers, Muchun Song,
Oscar Salvador, David Hildenbrand, David Howells, Paulo Alcantara,
Andreas Dilger, Jan Kara, Jaegeuk Kim, Chao Yu, Trond Myklebust,
Anna Schumaker, Chuck Lever, NeilBrown, Olga Kornievskaia,
Dai Ngo, Tom Talpey, Steve French, Ronnie Sahlberg,
Shyam Prasad N, Bharath SM, Alexander Aring, Ryusuke Konishi,
Viacheslav Dubeyko, Eric Van Hensbergen, Latchesar Ionkov,
Dominique Martinet, Christian Schoenebeck, David Sterba,
Marc Dionne, Ian Kent, Luis de Bethencourt, Salah Triki,
Tigran A. Aivazian, Ilya Dryomov, Alex Markuze, Jan Harkes, coda,
Nicolas Pitre, Tyler Hicks, Amir Goldstein,
John Paul Adrian Glaubitz, Yangtao Li, Mikulas Patocka,
David Woodhouse, Richard Weinberger, Dave Kleikamp,
Konstantin Komarov, Mark Fasheh, Joel Becker, Joseph Qi,
Mike Marshall, Martin Brandenburg, Miklos Szeredi, Anders Larsen,
Zhihao Cheng, Damien Le Moal, Naohiro Aota, Johannes Thumshirn,
John Johansen, Paul Moore, James Morris, Serge E. Hallyn,
Mimi Zohar, Roberto Sassu, Dmitry Kasatkin, Eric Snowberg, Fan Wu,
Stephen Smalley, Ondrej Mosnacek, Casey Schaufler, Alex Deucher,
Christian König, David Airlie, Simona Vetter, Sumit Semwal,
Eric Dumazet, Kuniyuki Iwashima, Paolo Abeni, Willem de Bruijn,
David S. Miller, Jakub Kicinski, Simon Horman, Oleg Nesterov,
Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Mark Rutland, Alexander Shishkin, Jiri Olsa,
Ian Rogers, Adrian Hunter, James Clark, Martin Schiller,
Eric Paris, Joerg Reuter, Marcel Holtmann, Johan Hedberg,
Luiz Augusto von Dentz, Oliver Hartkopp, Marc Kleine-Budde,
David Ahern, Neal Cardwell, Steffen Klassert, Herbert Xu,
Remi Denis-Courmont, Marcelo Ricardo Leitner, Xin Long,
Magnus Karlsson, Maciej Fijalkowski, Stanislav Fomichev,
Alexei Starovoitov, Daniel Borkmann, Jesper Dangaard Brouer,
John Fastabend, linux-fsdevel, linux-kernel, linux-trace-kernel,
nvdimm, fsverity, linux-mm, netfs, linux-ext4, linux-f2fs-devel,
linux-nfs, linux-cifs, samba-technical, linux-nilfs, v9fs,
linux-afs, autofs, ceph-devel, codalist, ecryptfs, linux-mtd,
jfs-discussion, ntfs3, ocfs2-devel, devel, linux-unionfs,
apparmor, linux-security-module, linux-integrity, selinux,
amd-gfx, dri-devel, linux-media, linaro-mm-sig, netdev,
linux-perf-users, linux-fscrypt, linux-xfs, linux-hams, linux-x25,
audit, linux-bluetooth, linux-can, linux-sctp, bpf
In-Reply-To: <4d3b9b92da613ad329b822f3f6043fa08f534451.camel@kernel.org>
On Tue, Mar 03, 2026 at 10:14:27AM -0500, Jeff Layton wrote:
> On Tue, 2026-03-03 at 06:30 -0800, Christoph Hellwig wrote:
> > On Tue, Mar 03, 2026 at 09:19:42AM -0500, Jeff Layton wrote:
> > > On Tue, 2026-03-03 at 05:59 -0800, Christoph Hellwig wrote:
> > > > On Tue, Mar 03, 2026 at 08:43:15AM -0500, Jeff Layton wrote:
> > > > > On Tue, 2026-03-03 at 05:37 -0800, Christoph Hellwig wrote:
> > > > > > On Tue, Mar 03, 2026 at 05:53:39AM -0500, Jeff Layton wrote:
> > > > > > > Like I said to Ted, this is just temporary scaffolding for the change.
> > > > > > > The PRIino macro is removed in the end. Given that, perhaps you can
> > > > > > > overlook the bikeshed's color in this instance?
> > > > > >
> > > > > > So why add it in the first place?
> > > > >
> > > > > Bisectability. The first version I did of this would have broken the
> > > > > ability to bisect properly across these changes. I don't love the
> > > > > "churn" here either, but this should be cleanly bisectable.
> > > >
> > > > What do you need to bisect in format string changes? Splitting
> > > > every variable type change outside of the main i_ino out - sure.
> > > > But bisecting that "change to u64 in ext4" really broke ext4 and
> > > > not "change to u64" is not very useful. Commits should do one
> > > > well defined thing. Adding a weird transition layer for a format
> > > > thing that just gets dropped is not one well defined thing.
> > >
> > > In the middle stages of the series, you will get warnings or errors on
> > > 32-bit hosts when i_ino's type doesn't match what the format string
> > > expects.
> > >
> > > There are really only three options here:
> > >
> > > 1/ Do (almost) all of the changes in one giant patch
> > >
> > > 2/ Accept that the build may break during the interim stages
> > >
> > > 3/ This series: using a typedef and macro to work around the breakage
> > > until the type can be changed, at the expense of some extra churn in
> > > the codebase
> > >
> > > 3 seems like the lesser evil.
> >
> > No, 1 is by far the least evil. Note that it's not really almost all,
> > as all the local variables can easily and sanely be split out. It's
> > all of the format strings, and that makes sense. The only "regressions"
> > there are incorrect format strings which have good warnings and can
> > be fixed easily.
>
> Well, I've done 2 and 3 already. Why not 1? :)
>
> It's not so much the regressions that are a problem here, but the merge
> conflicts for anyone wanting to backport later patches that are near
> these format changes. Having that change broken up by subsystem makes
> it easier to handle that piecemeal later.
>
> I think we'll be looking at close to a 1000 line patch that touches
> nearly 200 files if go that route. Roughly:
>
> 182 files changed, 910 insertions(+), 912 deletions(-)
>
> There are some tracepoint changes in some of the per-subsystem patches
> that will need to be split out, so the count isn't exact, but it'll be
> fairly close.
>
> Since Christian will probably end up taking this series, I'd like to
> get his opinion before I respin anything.
I'm kinda surprised that we suddenly started caring about the amount of
individual patches. I personally don't care either way. Do it in one
giant patch if this moves us forward. I've done 1 and 3 and what you
did. And I'd be really annoyed if during a bisect I start to get
pointless build failures because someone did 2.
^ permalink raw reply
* Re: [PATCH v3 12/18] rtla: Enforce exact match for time unit suffixes
From: Tomas Glozar @ 2026-03-04 13:57 UTC (permalink / raw)
To: Wander Lairson Costa
Cc: Steven Rostedt, Crystal Wood, Ivan Pravdin, Costa Shulyupin,
John Kacur, Tiezhu Yang, Daniel Wagner,
Daniel Bristot de Oliveira,
open list:Real-time Linux Analysis (RTLA) tools,
open list:Real-time Linux Analysis (RTLA) tools,
open list:BPF [MISC]:Keyword:(?:b|_)bpf(?:b|_)
In-Reply-To: <20260115163650.118910-13-wander@redhat.com>
čt 15. 1. 2026 v 18:28 odesílatel Wander Lairson Costa
<wander@redhat.com> napsal:
>
> The parse_ns_duration() function currently uses prefix matching for
> detecting time units. This approach is problematic as it silently
> accepts malformed strings such as "100nsx" or "100us_invalid" by
> ignoring the trailing characters, leading to potential configuration
> errors.
>
> Switch to using strcmp() for suffix comparison to enforce exact matches.
> This ensures that the parser strictly validates the time unit and
> rejects any input containing invalid trailing characters, thereby
> improving the robustness of the configuration parsing.
This solution is incorrect. We need to be able to parse deadline
priority correctly, whose format includes two suffixes:
d:runtime[us|ms|s]:period[us|ms|s]
(see manpages)
and is parsed like this:
int parse_prio(char *arg, struct sched_attr *sched_param)
{
...
switch (arg[0]) {
case 'd':
case 'D':
/* d:runtime:period */
if (strlen(arg) < 4)
return -1;
runtime = get_long_ns_after_colon(arg);
if (runtime == INVALID_VAL)
return -1;
period = get_long_ns_after_colon(&arg[2]);
if (period == INVALID_VAL)
return -1;
...
Your commit breaks that:
$ rtla timerlat -P d:10ms:100ms
Invalid -P priority
Tomas
^ permalink raw reply
* Re: [PATCH v4 4/5] mm: rename zone->lock to zone->_lock
From: Dmitry Ilvokhin @ 2026-03-04 13:01 UTC (permalink / raw)
To: SeongJae Park
Cc: Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Liam R. Howlett, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Axel Rasmussen, Yuanchu Xie,
Wei Xu, Steven Rostedt, Masami Hiramatsu, Mathieu Desnoyers,
Rafael J. Wysocki, Pavel Machek, Len Brown, Brendan Jackman,
Johannes Weiner, Zi Yan, Oscar Salvador, Qi Zheng, Shakeel Butt,
linux-kernel, linux-mm, linux-trace-kernel, linux-pm
In-Reply-To: <20260304015035.84839-1-sj@kernel.org>
On Tue, Mar 03, 2026 at 05:50:34PM -0800, SeongJae Park wrote:
> On Tue, 3 Mar 2026 14:25:55 +0000 Dmitry Ilvokhin <d@ilvokhin.com> wrote:
>
> > On Mon, Mar 02, 2026 at 02:37:43PM -0800, Andrew Morton wrote:
> > > On Mon, 2 Mar 2026 15:10:03 +0100 "Vlastimil Babka (SUSE)" <vbabka@kernel.org> wrote:
> > >
> > > > On 2/27/26 17:00, Dmitry Ilvokhin wrote:
> > > > > This intentionally breaks direct users of zone->lock at compile time so
> > > > > all call sites are converted to the zone lock wrappers. Without the
> > > > > rename, present and future out-of-tree code could continue using
> > > > > spin_lock(&zone->lock) and bypass the wrappers and tracing
> > > > > infrastructure.
> > > > >
> > > > > No functional change intended.
> > > > >
> > > > > Suggested-by: Andrew Morton <akpm@linux-foundation.org>
> > > > > Signed-off-by: Dmitry Ilvokhin <d@ilvokhin.com>
> > > > > Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
> > > > > Acked-by: SeongJae Park <sj@kernel.org>
> > > >
> > > > I see some more instances of 'zone->lock' in comments in
> > > > include/linux/mmzone.h and under Documentation/ but otherwise LGTM.
> > > >
> > >
> > > I fixed (most of) that in the previous version but my fix was lost.
> >
> > Thanks for the fixups, Andrew.
> >
> > I still see a few 'zone->lock' references in Documentation remain on
> > mm-new. This patch cleans them up, as noted by Vlastimil.
> >
> > I'm happy to adjust this patch if anything else needs attention.
> >
> > From 9142d5a8b60038fa424a6033253960682e5a51f4 Mon Sep 17 00:00:00 2001
> > From: Dmitry Ilvokhin <d@ilvokhin.com>
> > Date: Tue, 3 Mar 2026 06:13:13 -0800
> > Subject: [PATCH] mm: fix remaining zone->lock references
> >
> > Signed-off-by: Dmitry Ilvokhin <d@ilvokhin.com>
> > ---
> > Documentation/mm/physical_memory.rst | 4 ++--
> > Documentation/trace/events-kmem.rst | 8 ++++----
> > 2 files changed, 6 insertions(+), 6 deletions(-)
> >
> > diff --git a/Documentation/mm/physical_memory.rst b/Documentation/mm/physical_memory.rst
> > index b76183545e5b..e344f93515b6 100644
> > --- a/Documentation/mm/physical_memory.rst
> > +++ b/Documentation/mm/physical_memory.rst
> > @@ -500,11 +500,11 @@ General
> > ``nr_isolate_pageblock``
> > Number of isolated pageblocks. It is used to solve incorrect freepage counting
> > problem due to racy retrieving migratetype of pageblock. Protected by
> > - ``zone->lock``. Defined only when ``CONFIG_MEMORY_ISOLATION`` is enabled.
> > + ``zone_lock``. Defined only when ``CONFIG_MEMORY_ISOLATION`` is enabled.
>
> Dmitry's original patch [1] was doing 's/zone->lock/zone->_lock/', which aligns
> to my expectation. But this patch is doing 's/zone->lock/zone_lock/'. Same
> for the rest of this patch.
>
> I was initially thinking this is just a mistake, but I also found Andrew is
> doing same change [2], so I'm bit confused. Is this an intentional change?
>
> [1] https://lore.kernel.org/d61500c5784c64e971f4d328c57639303c475f81.1772206930.git.d@ilvokhin.com
> [2] https://lore.kernel.org/20260302143743.220eed4feb36d7572fe726cc@linux-foundation.org
>
Good catch, thanks for pointing this out, SJ.
Originally the mechanical rename was indeed zone->lock -> zone->_lock.
However, in Documentation I intentionally switched references to
zone_lock instead of zone->_lock. The reasoning is that _lock is now an
internal implementation detail, and direct access is discouraged. The
intended interface is via the zone_lock_*() / zone_unlock_*() wrappers,
so referencing zone_lock in documentation felt more appropriate than
mentioning the private struct field (zone->_lock).
That said, I agree this creates inconsistency with the mechanical
rename, and I'm happy to adjust either way: either consistently refer
to the wrapper API, or keep documentation aligned with zone->_lock.
I slightly prefer referring to the wrapper API, but don't have a strong
preference as long as we're consistent.
>
> Thanks,
> SJ
>
> [...]
^ permalink raw reply
* Re: [RFC PATCH 1/2] locking: add mutex_lock_nospin()
From: Peter Zijlstra @ 2026-03-04 12:41 UTC (permalink / raw)
To: Yafang Shao
Cc: mingo, will, boqun, longman, rostedt, mhiramat, mark.rutland,
mathieu.desnoyers, linux-kernel, linux-trace-kernel, bpf
In-Reply-To: <CALOAHbDmmbcK-0mG=44rw3wbaTwS8drzYD-s2gg8g+rPQZNLCg@mail.gmail.com>
On Wed, Mar 04, 2026 at 07:52:06PM +0800, Yafang Shao wrote:
> On Wed, Mar 4, 2026 at 6:11 PM Peter Zijlstra <peterz@infradead.org> wrote:
> >
> > On Wed, Mar 04, 2026 at 05:37:31PM +0800, Yafang Shao wrote:
> > > On Wed, Mar 4, 2026 at 5:03 PM Peter Zijlstra <peterz@infradead.org> wrote:
> > > >
> > > > On Wed, Mar 04, 2026 at 03:46:49PM +0800, Yafang Shao wrote:
> > > > > Introduce mutex_lock_nospin(), a helper that disables optimistic spinning
> > > > > on the owner for specific heavy locks. This prevents long spinning times
> > > > > that can lead to latency spikes for other tasks on the same runqueue.
> > > >
> > > > This makes no sense; spinning stops on need_resched().
> > >
> > > Hello Peter,
> > >
> > > The condition to stop spinning on need_resched() relies on the mutex
> > > owner remaining unchanged. However, when multiple tasks contend for
> > > the same lock, the owner can change frequently. This creates a
> > > potential TOCTOU (Time of Check to Time of Use) issue.
> > >
> > > mutex_optimistic_spin
> > > owner = __mutex_trylock_or_owner(lock);
> > > mutex_spin_on_owner
> > > // the __mutex_owner(lock) might get a new owner.
> > > while (__mutex_owner(lock) == owner)
> > >
> >
> > How do these new owners become the owner? Are they succeeding the
> > __mutex_trylock() that sits before mutex_optimistic_spin() and
> > effectively starving the spinner?
> >
> > Something like the below would make a difference if that were so.
>
> The following change made no difference; concurrent runs still result
> in prolonged system time.
>
> real 0m5.265s user 0m0.000s sys 0m4.921s
> real 0m5.295s user 0m0.002s sys 0m4.697s
> real 0m5.293s user 0m0.003s sys 0m4.844s
> real 0m5.303s user 0m0.001s sys 0m4.511s
> real 0m5.303s user 0m0.000s sys 0m4.694s
> real 0m5.302s user 0m0.002s sys 0m4.677s
> real 0m5.313s user 0m0.000s sys 0m4.837s
> real 0m5.327s user 0m0.000s sys 0m4.808s
> real 0m5.330s user 0m0.001s sys 0m4.893s
> real 0m5.358s user 0m0.005s sys 0m4.919s
>
> Our kernel is not built with CONFIG_PREEMPT enabled, so prolonged
> system time can lead to CPU pressure and potential latency spikes.
> Since we can reliably reproduce this unnecessary spinning, why not
> improve it to reduce the overhead?
If you cannot explain what the problem is (you haven't), there is
nothing to fix.
Also, current kernels cannot be build without PREEMPT; and if you care
about latency running a PREEMPT=n kernel is daft. That said,
TIF_NEED_RESCHED should work irrespective of PREEMPT settings, the
PREEMPT settings just affect when and how you end up in schedule().
Even without PREEMPT, if there is a task waiting either the wakeup or
the tick will set TIF_NEED_RESCHED and it should stop spinning. If there
is no task waiting, there is no actual latency, just burning CPU time,
and that isn't a problem per-se.
What should happen is that the first spinner gets the lock next, the
next spinner is then promoted to first spinner and so on.
This chain continues, which means the lock owner is always
on-cpu and good progress is being made and there is no CPU contention,
or the spinner gets marked for preemption (as said, this does not
require PREEMPT=y) and will stop spinning and go sleep, or the owner
goes to sleep and all the spinners stop and also go sleep.
Again, you have not said anything specific enough to figure out what
happens on your end. You said the owner changes, this means there is
progress made. What isn't clear is if any one particular spinner is
starved (that would be a problem) or if this latency spike you observe
is worse than would be from running a while(1) loop, in which case,
that's just how it is.
What is not sane, is marking random locks with random properties just
because random workload.
^ permalink raw reply
* Re: [RFC PATCH 1/2] locking: add mutex_lock_nospin()
From: Yafang Shao @ 2026-03-04 11:52 UTC (permalink / raw)
To: Peter Zijlstra
Cc: mingo, will, boqun, longman, rostedt, mhiramat, mark.rutland,
mathieu.desnoyers, linux-kernel, linux-trace-kernel, bpf
In-Reply-To: <20260304101111.GQ606826@noisy.programming.kicks-ass.net>
On Wed, Mar 4, 2026 at 6:11 PM Peter Zijlstra <peterz@infradead.org> wrote:
>
> On Wed, Mar 04, 2026 at 05:37:31PM +0800, Yafang Shao wrote:
> > On Wed, Mar 4, 2026 at 5:03 PM Peter Zijlstra <peterz@infradead.org> wrote:
> > >
> > > On Wed, Mar 04, 2026 at 03:46:49PM +0800, Yafang Shao wrote:
> > > > Introduce mutex_lock_nospin(), a helper that disables optimistic spinning
> > > > on the owner for specific heavy locks. This prevents long spinning times
> > > > that can lead to latency spikes for other tasks on the same runqueue.
> > >
> > > This makes no sense; spinning stops on need_resched().
> >
> > Hello Peter,
> >
> > The condition to stop spinning on need_resched() relies on the mutex
> > owner remaining unchanged. However, when multiple tasks contend for
> > the same lock, the owner can change frequently. This creates a
> > potential TOCTOU (Time of Check to Time of Use) issue.
> >
> > mutex_optimistic_spin
> > owner = __mutex_trylock_or_owner(lock);
> > mutex_spin_on_owner
> > // the __mutex_owner(lock) might get a new owner.
> > while (__mutex_owner(lock) == owner)
> >
>
> How do these new owners become the owner? Are they succeeding the
> __mutex_trylock() that sits before mutex_optimistic_spin() and
> effectively starving the spinner?
>
> Something like the below would make a difference if that were so.
The following change made no difference; concurrent runs still result
in prolonged system time.
real 0m5.265s user 0m0.000s sys 0m4.921s
real 0m5.295s user 0m0.002s sys 0m4.697s
real 0m5.293s user 0m0.003s sys 0m4.844s
real 0m5.303s user 0m0.001s sys 0m4.511s
real 0m5.303s user 0m0.000s sys 0m4.694s
real 0m5.302s user 0m0.002s sys 0m4.677s
real 0m5.313s user 0m0.000s sys 0m4.837s
real 0m5.327s user 0m0.000s sys 0m4.808s
real 0m5.330s user 0m0.001s sys 0m4.893s
real 0m5.358s user 0m0.005s sys 0m4.919s
Our kernel is not built with CONFIG_PREEMPT enabled, so prolonged
system time can lead to CPU pressure and potential latency spikes.
Since we can reliably reproduce this unnecessary spinning, why not
improve it to reduce the overhead?
--
Regards
Yafang
^ permalink raw reply
* Re: [PATCH v3 16/18] rtla/trace: Fix I/O handling in save_trace_to_file()
From: Tomas Glozar @ 2026-03-04 10:30 UTC (permalink / raw)
To: Wander Lairson Costa
Cc: Steven Rostedt, Ivan Pravdin, Crystal Wood, Costa Shulyupin,
John Kacur, Haiyong Sun, Tiezhu Yang, Daniel Wagner,
Daniel Bristot de Oliveira,
open list:Real-time Linux Analysis (RTLA) tools,
open list:Real-time Linux Analysis (RTLA) tools,
open list:BPF [MISC]:Keyword:(?:b|_)bpf(?:b|_)
In-Reply-To: <20260115163650.118910-17-wander@redhat.com>
čt 15. 1. 2026 v 18:29 odesílatel Wander Lairson Costa
<wander@redhat.com> napsal:
> diff --git a/tools/tracing/rtla/src/trace.c b/tools/tracing/rtla/src/trace.c
> index fed3362527b08..8e93b48d33ef8 100644
> --- a/tools/tracing/rtla/src/trace.c
> +++ b/tools/tracing/rtla/src/trace.c
> @@ -73,6 +73,7 @@ int save_trace_to_file(struct tracefs_instance *inst, const char *filename)
> char buffer[4096];
> int out_fd, in_fd;
> int retval = -1;
> + ssize_t n_read;
>
> if (!inst || !filename)
> return 0;
> @@ -90,15 +91,30 @@ int save_trace_to_file(struct tracefs_instance *inst, const char *filename)
> goto out_close_in;
> }
>
> - do {
> - retval = read(in_fd, buffer, sizeof(buffer));
> - if (retval <= 0)
> + for (;;) {
> + n_read = read(in_fd, buffer, sizeof(buffer));
> + if (n_read < 0) {
> + if (errno == EINTR)
> + continue;
> + err_msg("Error reading trace file: %s\n", strerror(errno));
> goto out_close;
> + }
> + if (n_read == 0)
> + break;
>
> - retval = write(out_fd, buffer, retval);
> - if (retval < 0)
> - goto out_close;
> - } while (retval > 0);
> + ssize_t n_written = 0;
Why break the style of declaring all variables at the beginning of the
function? n_read, added in the same commit, keeps the style.
This also applies to the previous patch.
Tomas
^ permalink raw reply
* Re: [RFC PATCH 1/2] locking: add mutex_lock_nospin()
From: Peter Zijlstra @ 2026-03-04 10:11 UTC (permalink / raw)
To: Yafang Shao
Cc: mingo, will, boqun, longman, rostedt, mhiramat, mark.rutland,
mathieu.desnoyers, linux-kernel, linux-trace-kernel, bpf
In-Reply-To: <CALOAHbC1AHc2eZrzafz7ynrW7NFPrVmervCY_jjWsRTo0gBwQQ@mail.gmail.com>
On Wed, Mar 04, 2026 at 05:37:31PM +0800, Yafang Shao wrote:
> On Wed, Mar 4, 2026 at 5:03 PM Peter Zijlstra <peterz@infradead.org> wrote:
> >
> > On Wed, Mar 04, 2026 at 03:46:49PM +0800, Yafang Shao wrote:
> > > Introduce mutex_lock_nospin(), a helper that disables optimistic spinning
> > > on the owner for specific heavy locks. This prevents long spinning times
> > > that can lead to latency spikes for other tasks on the same runqueue.
> >
> > This makes no sense; spinning stops on need_resched().
>
> Hello Peter,
>
> The condition to stop spinning on need_resched() relies on the mutex
> owner remaining unchanged. However, when multiple tasks contend for
> the same lock, the owner can change frequently. This creates a
> potential TOCTOU (Time of Check to Time of Use) issue.
>
> mutex_optimistic_spin
> owner = __mutex_trylock_or_owner(lock);
> mutex_spin_on_owner
> // the __mutex_owner(lock) might get a new owner.
> while (__mutex_owner(lock) == owner)
>
How do these new owners become the owner? Are they succeeding the
__mutex_trylock() that sits before mutex_optimistic_spin() and
effectively starving the spinner?
Something like the below would make a difference if that were so.
---
diff --git a/kernel/locking/mutex.c b/kernel/locking/mutex.c
index c867f6c15530..0796e77a8c3b 100644
--- a/kernel/locking/mutex.c
+++ b/kernel/locking/mutex.c
@@ -521,7 +521,7 @@ static __always_inline bool
mutex_optimistic_spin(struct mutex *lock, struct ww_acquire_ctx *ww_ctx,
struct mutex_waiter *waiter)
{
- return false;
+ return __mutex_trylock(lock);
}
#endif
@@ -614,8 +614,7 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclas
mutex_acquire_nest(&lock->dep_map, subclass, 0, nest_lock, ip);
trace_contention_begin(lock, LCB_F_MUTEX | LCB_F_SPIN);
- if (__mutex_trylock(lock) ||
- mutex_optimistic_spin(lock, ww_ctx, NULL)) {
+ if (mutex_optimistic_spin(lock, ww_ctx, NULL)) {
/* got the lock, yay! */
lock_acquired(&lock->dep_map, ip);
if (ww_ctx)
^ permalink raw reply related
* Re: [RFC PATCH 1/2] locking: add mutex_lock_nospin()
From: David Laight @ 2026-03-04 9:54 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Yafang Shao, mingo, will, boqun, longman, rostedt, mhiramat,
mark.rutland, mathieu.desnoyers, linux-kernel, linux-trace-kernel,
bpf
In-Reply-To: <20260304090249.GN606826@noisy.programming.kicks-ass.net>
On Wed, 4 Mar 2026 10:02:49 +0100
Peter Zijlstra <peterz@infradead.org> wrote:
> On Wed, Mar 04, 2026 at 03:46:49PM +0800, Yafang Shao wrote:
> > Introduce mutex_lock_nospin(), a helper that disables optimistic spinning
> > on the owner for specific heavy locks. This prevents long spinning times
> > that can lead to latency spikes for other tasks on the same runqueue.
>
> This makes no sense; spinning stops on need_resched().
>
That might still be an issue if a high priority process is spinning.
But a %sys spike doesn't imply a latency spike.
Is this using the osq_lock.c code?
That will have problems on overprovisioned VMs, it tries to find out
whether the hypervisor has switched out - but ISTR that is flawed.
In reality a spin lock shouldn't be held for long enough to cause
any kind latency issue.
So something in the code that reads the list of filter functions
needs to be done differently so that the lock isn't held for as long.
David
^ permalink raw reply
* Re: [RFC PATCH 1/2] locking: add mutex_lock_nospin()
From: Yafang Shao @ 2026-03-04 9:37 UTC (permalink / raw)
To: Peter Zijlstra
Cc: mingo, will, boqun, longman, rostedt, mhiramat, mark.rutland,
mathieu.desnoyers, linux-kernel, linux-trace-kernel, bpf
In-Reply-To: <20260304090249.GN606826@noisy.programming.kicks-ass.net>
On Wed, Mar 4, 2026 at 5:03 PM Peter Zijlstra <peterz@infradead.org> wrote:
>
> On Wed, Mar 04, 2026 at 03:46:49PM +0800, Yafang Shao wrote:
> > Introduce mutex_lock_nospin(), a helper that disables optimistic spinning
> > on the owner for specific heavy locks. This prevents long spinning times
> > that can lead to latency spikes for other tasks on the same runqueue.
>
> This makes no sense; spinning stops on need_resched().
Hello Peter,
The condition to stop spinning on need_resched() relies on the mutex
owner remaining unchanged. However, when multiple tasks contend for
the same lock, the owner can change frequently. This creates a
potential TOCTOU (Time of Check to Time of Use) issue.
mutex_optimistic_spin
owner = __mutex_trylock_or_owner(lock);
mutex_spin_on_owner
// the __mutex_owner(lock) might get a new owner.
while (__mutex_owner(lock) == owner)
We observed high CPU pressure in production when this scenario occurred.
Below are the benchmark results when running 10 concurrent tasks:
for i in `seq 0 9`; do
time cat /sys/kernel/debug/tracing/available_filter_functions
> /dev/null &
done
- before this patch
real 0m4.636s user 0m0.001s sys 0m3.773s
real 0m5.157s user 0m0.001s sys 0m4.362s
real 0m5.205s user 0m0.000s sys 0m4.538s
real 0m5.212s user 0m0.001s sys 0m4.700s
real 0m5.246s user 0m0.001s sys 0m4.501s
real 0m5.254s user 0m0.003s sys 0m4.335s
real 0m5.260s user 0m0.003s sys 0m4.525s
real 0m5.267s user 0m0.004s sys 0m4.482s
real 0m5.273s user 0m0.002s sys 0m4.215s
real 0m5.285s user 0m0.003s sys 0m4.373s
- after this patch
real 0m4.733s user 0m0.002s sys 0m0.511s
real 0m4.740s user 0m0.001s sys 0m0.509s
real 0m4.862s user 0m0.001s sys 0m0.513s
real 0m4.884s user 0m0.000s sys 0m0.507s
real 0m4.888s user 0m0.003s sys 0m0.513s
real 0m4.888s user 0m0.000s sys 0m0.511s
real 0m4.886s user 0m0.003s sys 0m0.508s
real 0m4.952s user 0m0.000s sys 0m0.513s
real 0m4.973s user 0m0.001s sys 0m0.510s
real 0m5.042s user 0m0.002s sys 0m0.515s
The results show that system time dropped dramatically from ~4.5
seconds to ~0.5 seconds, confirming that the patch can help reduce the
issue.
Please correct me if I've misunderstood anything.
--
Regards
Yafang
^ permalink raw reply
* Re: [PATCH v2 000/110] vfs: change inode->i_ino from unsigned long to u64
From: David Laight @ 2026-03-04 9:30 UTC (permalink / raw)
To: NeilBrown
Cc: Jeff Layton, linux-fsdevel, linux-kernel, linux-trace-kernel,
nvdimm, fsverity, linux-mm, netfs, linux-ext4, linux-f2fs-devel,
linux-nfs, linux-cifs, samba-technical, linux-nilfs, v9fs,
linux-afs, autofs, ceph-devel, codalist, ecryptfs, linux-mtd,
jfs-discussion, ntfs3, ocfs2-devel, devel, linux-unionfs,
apparmor, linux-security-module, linux-integrity, selinux,
amd-gfx, dri-devel, linux-media, linaro-mm-sig, netdev,
linux-perf-users, linux-fscrypt, linux-xfs, linux-hams, linux-x25,
audit, linux-bluetooth, linux-can, linux-sctp, bpf
In-Reply-To: <177260561903.7472.14075475865748618717@noble.neil.brown.name>
On Wed, 04 Mar 2026 17:26:59 +1100
NeilBrown <neilb@ownmail.net> wrote:
> On Tue, 03 Mar 2026, Jeff Layton wrote:
> > On Tue, 2026-03-03 at 10:55 +0000, David Howells wrote:
> > > Jeff Layton <jlayton@kernel.org> wrote:
> > >
> > > > This version splits the change up to be more bisectable. It first adds a
> > > > new kino_t typedef and a new "PRIino" macro to hold the width specifier
> > > > for format strings. The conversion is done, and then everything is
> > > > changed to remove the new macro and typedef.
> > >
> > > Why remove the typedef? It might be better to keep it.
> > >
> >
> > Why? After this change, internel kernel inodes will be u64's -- full
> > stop. I don't see what the macro or typedef will buy us at that point.
>
> Implicit documentation?
> ktime_t is (now) always s64, but we still keep the typedef;
>
> It would be cool if we could teach vsprintf to understand some new
> specifier to mean "kinode_t" or "ktime_t" etc. But that would trigger
> gcc warnings.
A more interesting one would be something that made gcc re-write the
format with the correct 'length modifier' for the parameter.
That would save a lot of effort!
David
>
> NeilBrown
>
^ permalink raw reply
* [PATCH] kprobes: Remove unneeded warnings from __arm_kprobe_ftrace()
From: Masami Hiramatsu (Google) @ 2026-03-04 9:08 UTC (permalink / raw)
To: Naveen N Rao, Masami Hiramatsu, Steven Rostedt
Cc: Zw Tang, linux-kernel, linux-trace-kernel, linux-perf-users,
Arnaldo Carvalho de Melo, Sasha Levin, David S . Miller
From: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Remove unneeded warnings for handled errors from __arm_kprobe_ftrace()
because all caller handled the error correctly.
Reported-by: Zw Tang <shicenci@gmail.com>
Closes: https://lore.kernel.org/all/CAPHJ_V+J6YDb_wX2nhXU6kh466Dt_nyDSas-1i_Y8s7tqY-Mzw@mail.gmail.com/
Fixes: 9c89bb8e3272 ("kprobes: treewide: Cleanup the error messages for kprobes")
Cc: stable@vger.kernel.org
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
---
kernel/kprobes.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index ab25b4aa9095..2c2b7bd24dd4 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -1144,12 +1144,12 @@ static int __arm_kprobe_ftrace(struct kprobe *p, struct ftrace_ops *ops,
lockdep_assert_held(&kprobe_mutex);
ret = ftrace_set_filter_ip(ops, (unsigned long)p->addr, 0, 0);
- if (WARN_ONCE(ret < 0, "Failed to arm kprobe-ftrace at %pS (error %d)\n", p->addr, ret))
+ if (ret < 0)
return ret;
if (*cnt == 0) {
ret = register_ftrace_function(ops);
- if (WARN(ret < 0, "Failed to register kprobe-ftrace (error %d)\n", ret)) {
+ if (ret < 0) {
/*
* At this point, sinec ops is not registered, we should be sefe from
* registering empty filter.
^ permalink raw reply related
* Re: [PATCH 1/2] tracing: Have futex syscall trace event show specific user data
From: Peter Zijlstra @ 2026-03-04 9:07 UTC (permalink / raw)
To: Steven Rostedt
Cc: linux-kernel, linux-trace-kernel, Masami Hiramatsu, Mark Rutland,
Mathieu Desnoyers, Andrew Morton, Thomas Gleixner, Brian Geffon,
John Stultz, Ian Rogers, Suleiman Souhlal
In-Reply-To: <20260303214942.428502100@kernel.org>
On Tue, Mar 03, 2026 at 04:47:36PM -0500, Steven Rostedt wrote:
> From: Steven Rostedt <rostedt@goodmis.org>
>
> Add specific reporting of the futex system call. This allows for debugging
> the futex code a bit easier. Instead of just showing the values passed
> into the futex system call, read the value of the user space memory
> pointed to by the addr parameter.
>
> Also make the op parameter more readable by parsing the values to show
> what the command is:
>
> futex_requeue_p-3251 [002] ..... 2101.068479: sys_futex(uaddr: 0x55e79a4da834 (0x80000cb1), FUTEX_LOCK_PI|FUTEX_PRIVATE_FLAG, val: 0)
> futex_requeue_p-3248 [001] ..... 2101.068970: sys_futex(uaddr: 0x7f859072f990 (0xcb2), FUTEX_WAIT_BITSET|FUTEX_CLOCK_REALTIME, val: 3250)
> futex_requeue_p-3252 [005] ..... 2101.069108: sys_futex(uaddr: 0x55e79a4da838 (0), FUTEX_WAIT_REQUEUE_PI|FUTEX_PRIVATE_FLAG, val: 0, timespec: 0x7ffe61076aa0, uaddr2: 0x55e79a4da834, uaddr2: 94453214586932, val3: 0)
> futex_requeue_p-3252 [005] ..... 2101.069410: sys_futex(uaddr: 0x55e79a4da834 (0x80000cb1), FUTEX_LOCK_PI|FUTEX_PRIVATE_FLAG, val: 0)
>
> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
> ---
> kernel/trace/trace_syscalls.c | 266 +++++++++++++++++++++++++++++++++-
> 1 file changed, 263 insertions(+), 3 deletions(-)
Egads, I really dislike how all sorts of syscall crud is 'duplicated' in
this file, rather than near or in the actual syscall definition.
^ permalink raw reply
* Re: [RFC PATCH 1/2] locking: add mutex_lock_nospin()
From: Peter Zijlstra @ 2026-03-04 9:02 UTC (permalink / raw)
To: Yafang Shao
Cc: mingo, will, boqun, longman, rostedt, mhiramat, mark.rutland,
mathieu.desnoyers, linux-kernel, linux-trace-kernel, bpf
In-Reply-To: <20260304074650.58165-2-laoar.shao@gmail.com>
On Wed, Mar 04, 2026 at 03:46:49PM +0800, Yafang Shao wrote:
> Introduce mutex_lock_nospin(), a helper that disables optimistic spinning
> on the owner for specific heavy locks. This prevents long spinning times
> that can lead to latency spikes for other tasks on the same runqueue.
This makes no sense; spinning stops on need_resched().
^ permalink raw reply
* Re: [syzbot] [bpf?] [trace?] KASAN: slab-use-after-free Read in bpf_trace_run3 (2)
From: Qing Wang @ 2026-03-04 8:07 UTC (permalink / raw)
To: syzbot+9ea7c90be2b24e189592
Cc: andrii, ast, bpf, daniel, eddyz87, haoluo, john.fastabend, jolsa,
kpsingh, linux-kernel, linux-trace-kernel, martin.lau,
mathieu.desnoyers, mattbobrowski, mhiramat, rostedt, sdf, song,
syzkaller-bugs, wangqing7171, yonghong.song
In-Reply-To: <69a7e4df.050a0220.21ae90.0014.GAE@google.com>
On Wed, 04 Mar 2026 at 15:53, syzbot <syzbot+9ea7c90be2b24e189592@syzkaller.appspotmail.com> wrote:
> Hello,
>
> syzbot has tested the proposed patch and the reproducer did not trigger any issue:
>
> Reported-by: syzbot+9ea7c90be2b24e189592@syzkaller.appspotmail.com
> Tested-by: syzbot+9ea7c90be2b24e189592@syzkaller.appspotmail.com
>
> Tested on:
>
> commit: 0031c068 Merge tag 'cgroup-for-7.0-rc2-fixes' of git:/..
> git tree: upstream
> console output: https://syzkaller.appspot.com/x/log.txt?x=12afb006580000
> kernel config: https://syzkaller.appspot.com/x/.config?x=c5c49ee0942d1cdb
> dashboard link: https://syzkaller.appspot.com/bug?extid=9ea7c90be2b24e189592
> compiler: Debian clang version 21.1.8 (++20251221033036+2078da43e25a-1~exp1~20251221153213.50), Debian LLD 21.1.8
> patch: https://syzkaller.appspot.com/x/patch.diff?x=1566e552580000
>
> Note: testing is done by a robot and is best-effort only.
It's same as [0].
[0] https://syzkaller.appspot.com/bug?extid=b4c5ad098c821bf8d8bc
I sent the fixed patch [1] to [0].
[1] https://lore.kernel.org/all/20260304070927.178464-1-wangqing7171@gmail.com/T/
--
Qing
^ permalink raw reply
* Re: [syzbot] [bpf?] [trace?] KASAN: slab-use-after-free Read in bpf_trace_run3 (2)
From: syzbot @ 2026-03-04 7:53 UTC (permalink / raw)
To: andrii, ast, bpf, daniel, eddyz87, haoluo, john.fastabend, jolsa,
kpsingh, linux-kernel, linux-trace-kernel, martin.lau,
mathieu.desnoyers, mattbobrowski, mhiramat, rostedt, sdf, song,
syzkaller-bugs, wangqing7171, yonghong.song
In-Reply-To: <20260304072616.185060-1-wangqing7171@gmail.com>
Hello,
syzbot has tested the proposed patch and the reproducer did not trigger any issue:
Reported-by: syzbot+9ea7c90be2b24e189592@syzkaller.appspotmail.com
Tested-by: syzbot+9ea7c90be2b24e189592@syzkaller.appspotmail.com
Tested on:
commit: 0031c068 Merge tag 'cgroup-for-7.0-rc2-fixes' of git:/..
git tree: upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=12afb006580000
kernel config: https://syzkaller.appspot.com/x/.config?x=c5c49ee0942d1cdb
dashboard link: https://syzkaller.appspot.com/bug?extid=9ea7c90be2b24e189592
compiler: Debian clang version 21.1.8 (++20251221033036+2078da43e25a-1~exp1~20251221153213.50), Debian LLD 21.1.8
patch: https://syzkaller.appspot.com/x/patch.diff?x=1566e552580000
Note: testing is done by a robot and is best-effort only.
^ permalink raw reply
* [RFC PATCH 2/2] ftrace: disable optimistic spinning for ftrace_lock
From: Yafang Shao @ 2026-03-04 7:46 UTC (permalink / raw)
To: peterz, mingo, will, boqun, longman, rostedt, mhiramat,
mark.rutland, mathieu.desnoyers
Cc: linux-kernel, linux-trace-kernel, bpf, Yafang Shao
In-Reply-To: <20260304074650.58165-1-laoar.shao@gmail.com>
mutex_lock_nospin() is used for ftrace_lock to selectively disable
optimistic spinning.
Signed-off-by: Yafang Shao <laoar.shao@gmail.com>
---
kernel/trace/ftrace.c | 52 +++++++++++++++++++++----------------------
1 file changed, 26 insertions(+), 26 deletions(-)
diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
index 827fb9a0bf0d..b8cca4f76118 100644
--- a/kernel/trace/ftrace.c
+++ b/kernel/trace/ftrace.c
@@ -1284,7 +1284,7 @@ static void clear_ftrace_mod_list(struct list_head *head)
if (!head)
return;
- mutex_lock(&ftrace_lock);
+ mutex_lock_nospin(&ftrace_lock);
list_for_each_entry_safe(p, n, head, list)
free_ftrace_mod(p);
mutex_unlock(&ftrace_lock);
@@ -4254,7 +4254,7 @@ static void *t_start(struct seq_file *m, loff_t *pos)
void *p = NULL;
loff_t l;
- mutex_lock(&ftrace_lock);
+ mutex_lock_nospin(&ftrace_lock);
if (unlikely(ftrace_disabled))
return NULL;
@@ -4362,7 +4362,7 @@ static __init void ftrace_check_work_func(struct work_struct *work)
struct ftrace_page *pg;
struct dyn_ftrace *rec;
- mutex_lock(&ftrace_lock);
+ mutex_lock_nospin(&ftrace_lock);
do_for_each_ftrace_rec(pg, rec) {
test_for_valid_rec(rec);
} while_for_each_ftrace_rec();
@@ -5123,7 +5123,7 @@ static void process_mod_list(struct list_head *head, struct ftrace_ops *ops,
if (!new_hash)
goto out; /* warn? */
- mutex_lock(&ftrace_lock);
+ mutex_lock_nospin(&ftrace_lock);
list_for_each_entry_safe(ftrace_mod, n, head, list) {
@@ -5159,7 +5159,7 @@ static void process_mod_list(struct list_head *head, struct ftrace_ops *ops,
if (enable && list_empty(head))
new_hash->flags &= ~FTRACE_HASH_FL_MOD;
- mutex_lock(&ftrace_lock);
+ mutex_lock_nospin(&ftrace_lock);
ftrace_hash_move_and_update_ops(ops, orig_hash,
new_hash, enable);
@@ -5465,7 +5465,7 @@ register_ftrace_function_probe(char *glob, struct trace_array *tr,
return -EINVAL;
- mutex_lock(&ftrace_lock);
+ mutex_lock_nospin(&ftrace_lock);
/* Check if the probe_ops is already registered */
list_for_each_entry(iter, &tr->func_probes, list) {
if (iter->probe_ops == probe_ops) {
@@ -5540,7 +5540,7 @@ register_ftrace_function_probe(char *glob, struct trace_array *tr,
}
}
- mutex_lock(&ftrace_lock);
+ mutex_lock_nospin(&ftrace_lock);
if (!count) {
/* Nothing was added? */
@@ -5619,7 +5619,7 @@ unregister_ftrace_function_probe_func(char *glob, struct trace_array *tr,
return -EINVAL;
}
- mutex_lock(&ftrace_lock);
+ mutex_lock_nospin(&ftrace_lock);
/* Check if the probe_ops is already registered */
list_for_each_entry(iter, &tr->func_probes, list) {
if (iter->probe_ops == probe_ops) {
@@ -5679,7 +5679,7 @@ unregister_ftrace_function_probe_func(char *glob, struct trace_array *tr,
goto out_unlock;
}
- mutex_lock(&ftrace_lock);
+ mutex_lock_nospin(&ftrace_lock);
WARN_ON(probe->ref < count);
@@ -5943,7 +5943,7 @@ ftrace_set_hash(struct ftrace_ops *ops, unsigned char *buf, int len,
goto out_regex_unlock;
}
- mutex_lock(&ftrace_lock);
+ mutex_lock_nospin(&ftrace_lock);
ret = ftrace_hash_move_and_update_ops(ops, orig_hash, hash, enable);
mutex_unlock(&ftrace_lock);
@@ -6205,7 +6205,7 @@ __modify_ftrace_direct(struct ftrace_ops *ops, unsigned long addr)
* Now the ftrace_ops_list_func() is called to do the direct callers.
* We can safely change the direct functions attached to each entry.
*/
- mutex_lock(&ftrace_lock);
+ mutex_lock_nospin(&ftrace_lock);
size = 1 << hash->size_bits;
for (i = 0; i < size; i++) {
@@ -6625,7 +6625,7 @@ int update_ftrace_direct_mod(struct ftrace_ops *ops, struct ftrace_hash *hash, b
* Now the ftrace_ops_list_func() is called to do the direct callers.
* We can safely change the direct functions attached to each entry.
*/
- mutex_lock(&ftrace_lock);
+ mutex_lock_nospin(&ftrace_lock);
size = 1 << hash->size_bits;
for (i = 0; i < size; i++) {
@@ -6980,7 +6980,7 @@ int ftrace_regex_release(struct inode *inode, struct file *file)
} else
orig_hash = &iter->ops->func_hash->notrace_hash;
- mutex_lock(&ftrace_lock);
+ mutex_lock_nospin(&ftrace_lock);
ftrace_hash_move_and_update_ops(iter->ops, orig_hash,
iter->hash, filter_hash);
mutex_unlock(&ftrace_lock);
@@ -7464,7 +7464,7 @@ void ftrace_create_filter_files(struct ftrace_ops *ops,
*/
void ftrace_destroy_filter_files(struct ftrace_ops *ops)
{
- mutex_lock(&ftrace_lock);
+ mutex_lock_nospin(&ftrace_lock);
if (ops->flags & FTRACE_OPS_FL_ENABLED)
ftrace_shutdown(ops, 0);
ops->flags |= FTRACE_OPS_FL_DELETED;
@@ -7571,7 +7571,7 @@ static int ftrace_process_locs(struct module *mod,
if (!start_pg)
return -ENOMEM;
- mutex_lock(&ftrace_lock);
+ mutex_lock_nospin(&ftrace_lock);
/*
* Core and each module needs their own pages, as
@@ -7868,7 +7868,7 @@ void ftrace_release_mod(struct module *mod)
struct ftrace_page *tmp_page = NULL;
struct ftrace_page *pg;
- mutex_lock(&ftrace_lock);
+ mutex_lock_nospin(&ftrace_lock);
/*
* To avoid the UAF problem after the module is unloaded, the
@@ -7938,7 +7938,7 @@ void ftrace_module_enable(struct module *mod)
struct dyn_ftrace *rec;
struct ftrace_page *pg;
- mutex_lock(&ftrace_lock);
+ mutex_lock_nospin(&ftrace_lock);
if (ftrace_disabled)
goto out_unlock;
@@ -8267,7 +8267,7 @@ void ftrace_free_mem(struct module *mod, void *start_ptr, void *end_ptr)
key.ip = start;
key.flags = end; /* overload flags, as it is unsigned long */
- mutex_lock(&ftrace_lock);
+ mutex_lock_nospin(&ftrace_lock);
/*
* If we are freeing module init memory, then check if
@@ -8686,7 +8686,7 @@ static void clear_ftrace_pids(struct trace_array *tr, int type)
void ftrace_clear_pids(struct trace_array *tr)
{
- mutex_lock(&ftrace_lock);
+ mutex_lock_nospin(&ftrace_lock);
clear_ftrace_pids(tr, TRACE_PIDS | TRACE_NO_PIDS);
@@ -8695,7 +8695,7 @@ void ftrace_clear_pids(struct trace_array *tr)
static void ftrace_pid_reset(struct trace_array *tr, int type)
{
- mutex_lock(&ftrace_lock);
+ mutex_lock_nospin(&ftrace_lock);
clear_ftrace_pids(tr, type);
ftrace_update_pid_func();
@@ -8713,7 +8713,7 @@ static void *fpid_start(struct seq_file *m, loff_t *pos)
struct trace_pid_list *pid_list;
struct trace_array *tr = m->private;
- mutex_lock(&ftrace_lock);
+ mutex_lock_nospin(&ftrace_lock);
rcu_read_lock_sched();
pid_list = rcu_dereference_sched(tr->function_pids);
@@ -8766,7 +8766,7 @@ static void *fnpid_start(struct seq_file *m, loff_t *pos)
struct trace_pid_list *pid_list;
struct trace_array *tr = m->private;
- mutex_lock(&ftrace_lock);
+ mutex_lock_nospin(&ftrace_lock);
rcu_read_lock_sched();
pid_list = rcu_dereference_sched(tr->function_no_pids);
@@ -9057,7 +9057,7 @@ static int prepare_direct_functions_for_ipmodify(struct ftrace_ops *ops)
unsigned long ip = entry->ip;
bool found_op = false;
- mutex_lock(&ftrace_lock);
+ mutex_lock_nospin(&ftrace_lock);
do_for_each_ftrace_op(op, ftrace_ops_list) {
if (!(op->flags & FTRACE_OPS_FL_DIRECT))
continue;
@@ -9106,7 +9106,7 @@ static void cleanup_direct_functions_after_ipmodify(struct ftrace_ops *ops)
unsigned long ip = entry->ip;
bool found_op = false;
- mutex_lock(&ftrace_lock);
+ mutex_lock_nospin(&ftrace_lock);
do_for_each_ftrace_op(op, ftrace_ops_list) {
if (!(op->flags & FTRACE_OPS_FL_DIRECT))
continue;
@@ -9153,7 +9153,7 @@ static int register_ftrace_function_nolock(struct ftrace_ops *ops)
ftrace_ops_init(ops);
- mutex_lock(&ftrace_lock);
+ mutex_lock_nospin(&ftrace_lock);
ret = ftrace_startup(ops, 0);
@@ -9200,7 +9200,7 @@ int unregister_ftrace_function(struct ftrace_ops *ops)
{
int ret;
- mutex_lock(&ftrace_lock);
+ mutex_lock_nospin(&ftrace_lock);
ret = ftrace_shutdown(ops, 0);
mutex_unlock(&ftrace_lock);
--
2.47.3
^ permalink raw reply related
* [RFC PATCH 1/2] locking: add mutex_lock_nospin()
From: Yafang Shao @ 2026-03-04 7:46 UTC (permalink / raw)
To: peterz, mingo, will, boqun, longman, rostedt, mhiramat,
mark.rutland, mathieu.desnoyers
Cc: linux-kernel, linux-trace-kernel, bpf, Yafang Shao
In-Reply-To: <20260304074650.58165-1-laoar.shao@gmail.com>
Introduce mutex_lock_nospin(), a helper that disables optimistic spinning
on the owner for specific heavy locks. This prevents long spinning times
that can lead to latency spikes for other tasks on the same runqueue.
Signed-off-by: Yafang Shao <laoar.shao@gmail.com>
---
include/linux/mutex.h | 3 +++
kernel/locking/mutex.c | 39 ++++++++++++++++++++++++++++++++-------
2 files changed, 35 insertions(+), 7 deletions(-)
diff --git a/include/linux/mutex.h b/include/linux/mutex.h
index ecaa0440f6ec..1e488bb24b57 100644
--- a/include/linux/mutex.h
+++ b/include/linux/mutex.h
@@ -189,11 +189,13 @@ extern int __must_check mutex_lock_interruptible_nested(struct mutex *lock,
extern int __must_check _mutex_lock_killable(struct mutex *lock,
unsigned int subclass, struct lockdep_map *nest_lock) __cond_acquires(0, lock);
extern void mutex_lock_io_nested(struct mutex *lock, unsigned int subclass) __acquires(lock);
+extern void mutex_lock_nospin_nested(struct mutex *lock, unsigned int subclass);
#define mutex_lock(lock) mutex_lock_nested(lock, 0)
#define mutex_lock_interruptible(lock) mutex_lock_interruptible_nested(lock, 0)
#define mutex_lock_killable(lock) _mutex_lock_killable(lock, 0, NULL)
#define mutex_lock_io(lock) mutex_lock_io_nested(lock, 0)
+#define mutex_lock_nospin(lock) mutex_lock_nospin_nested(lock, 0)
#define mutex_lock_nest_lock(lock, nest_lock) \
do { \
@@ -215,6 +217,7 @@ extern void mutex_lock(struct mutex *lock) __acquires(lock);
extern int __must_check mutex_lock_interruptible(struct mutex *lock) __cond_acquires(0, lock);
extern int __must_check mutex_lock_killable(struct mutex *lock) __cond_acquires(0, lock);
extern void mutex_lock_io(struct mutex *lock) __acquires(lock);
+extern void mutex_lock_nospin(struct mutex *lock) __acquires(lock);
# define mutex_lock_nested(lock, subclass) mutex_lock(lock)
# define mutex_lock_interruptible_nested(lock, subclass) mutex_lock_interruptible(lock)
diff --git a/kernel/locking/mutex.c b/kernel/locking/mutex.c
index 2a1d165b3167..03d3b0749882 100644
--- a/kernel/locking/mutex.c
+++ b/kernel/locking/mutex.c
@@ -290,6 +290,14 @@ void __sched mutex_lock(struct mutex *lock)
__mutex_lock_slowpath(lock);
}
EXPORT_SYMBOL(mutex_lock);
+
+void __sched mutex_lock_nospin(struct mutex *lock)
+{
+ might_sleep();
+
+ if (!__mutex_trylock_fast(lock))
+ __mutex_lock_nospin(lock);
+}
#endif
#include "ww_mutex.h"
@@ -443,8 +451,11 @@ static inline int mutex_can_spin_on_owner(struct mutex *lock)
*/
static __always_inline bool
mutex_optimistic_spin(struct mutex *lock, struct ww_acquire_ctx *ww_ctx,
- struct mutex_waiter *waiter)
+ struct mutex_waiter *waiter, const bool nospin)
{
+ if (nospin)
+ return false;
+
if (!waiter) {
/*
* The purpose of the mutex_can_spin_on_owner() function is
@@ -577,7 +588,8 @@ EXPORT_SYMBOL(ww_mutex_unlock);
static __always_inline int __sched
__mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclass,
struct lockdep_map *nest_lock, unsigned long ip,
- struct ww_acquire_ctx *ww_ctx, const bool use_ww_ctx)
+ struct ww_acquire_ctx *ww_ctx, const bool use_ww_ctx,
+ const bool nospin)
{
DEFINE_WAKE_Q(wake_q);
struct mutex_waiter waiter;
@@ -615,7 +627,7 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclas
trace_contention_begin(lock, LCB_F_MUTEX | LCB_F_SPIN);
if (__mutex_trylock(lock) ||
- mutex_optimistic_spin(lock, ww_ctx, NULL)) {
+ mutex_optimistic_spin(lock, ww_ctx, NULL, nospin)) {
/* got the lock, yay! */
lock_acquired(&lock->dep_map, ip);
if (ww_ctx)
@@ -716,7 +728,7 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclas
* to run.
*/
clear_task_blocked_on(current, lock);
- if (mutex_optimistic_spin(lock, ww_ctx, &waiter))
+ if (mutex_optimistic_spin(lock, ww_ctx, &waiter, nospin))
break;
set_task_blocked_on(current, lock);
trace_contention_begin(lock, LCB_F_MUTEX);
@@ -773,14 +785,21 @@ static int __sched
__mutex_lock(struct mutex *lock, unsigned int state, unsigned int subclass,
struct lockdep_map *nest_lock, unsigned long ip)
{
- return __mutex_lock_common(lock, state, subclass, nest_lock, ip, NULL, false);
+ return __mutex_lock_common(lock, state, subclass, nest_lock, ip, NULL, false, false);
+}
+
+static int __sched
+__mutex_lock_nospin(struct mutex *lock, unsigned int state, unsigned int subclass,
+ struct lockdep_map *nest_lock, unsigned long ip)
+{
+ return __mutex_lock_common(lock, state, subclass, nest_lock, ip, NULL, false, true);
}
static int __sched
__ww_mutex_lock(struct mutex *lock, unsigned int state, unsigned int subclass,
unsigned long ip, struct ww_acquire_ctx *ww_ctx)
{
- return __mutex_lock_common(lock, state, subclass, NULL, ip, ww_ctx, true);
+ return __mutex_lock_common(lock, state, subclass, NULL, ip, ww_ctx, true, false);
}
/**
@@ -861,11 +880,17 @@ mutex_lock_io_nested(struct mutex *lock, unsigned int subclass)
token = io_schedule_prepare();
__mutex_lock_common(lock, TASK_UNINTERRUPTIBLE,
- subclass, NULL, _RET_IP_, NULL, 0);
+ subclass, NULL, _RET_IP_, NULL, 0, false);
io_schedule_finish(token);
}
EXPORT_SYMBOL_GPL(mutex_lock_io_nested);
+void __sched
+mutex_lock_nospin_nested(struct mutex *lock, unsigned int subclass)
+{
+ __mutex_lock_nospin(lock, TASK_UNINTERRUPTIBLE, subclass, NULL, _RET_IP_);
+}
+
static inline int
ww_mutex_deadlock_injection(struct ww_mutex *lock, struct ww_acquire_ctx *ctx)
{
--
2.47.3
^ permalink raw reply related
* [RFC PATCH 0/2] disable optimistic spinning for ftrace_lock
From: Yafang Shao @ 2026-03-04 7:46 UTC (permalink / raw)
To: peterz, mingo, will, boqun, longman, rostedt, mhiramat,
mark.rutland, mathieu.desnoyers
Cc: linux-kernel, linux-trace-kernel, bpf, Yafang Shao
Background
==========
One of our latency-sensitive services reported random CPU %sys spikes.
After a thorough investigation, we finally identified the root cause of the
CPU %sys spikes. The key kernel stacks are as follows:
- Task A
2026-02-14-16:53:40.938243: [CPU198] 2156302(bpftrace) cgrp:4019437 pod:4019253
find_kallsyms_symbol+142
module_address_lookup+104
kallsyms_lookup_buildid+203
kallsyms_lookup+20
print_rec+64
t_show+67
seq_read_iter+709
seq_read+165
vfs_read+165
ksys_read+103
__x64_sys_read+25
do_syscall_64+56
entry_SYSCALL_64_after_hwframe+100
This task (2156302, bpftrace) is reading the
/sys/kernel/debug/tracing/available_filter_functions to check if a function
is traceable:
https://github.com/bpftrace/bpftrace/blob/master/src/tracefs/tracefs.h#L21
Reading the available_filter_functions file is time-consuming, as it
contains tens of thousands of functions:
$ cat /sys/kernel/debug/tracing/available_filter_functions | wc -l
61081
$ time cat /sys/kernel/debug/tracing/available_filter_functions > /dev/null
real 0m0.452s
user 0m0.000s
sys 0m0.452s
Consequently, the ftrace_lock is held by this task for an extended period.
- Other Tasks
2026-02-14-16:53:41.437094: [CPU79] 2156308(bpftrace) cgrp:4019437 pod:4019253
mutex_spin_on_owner+108
__mutex_lock.constprop.0+1132
__mutex_lock_slowpath+19
mutex_lock+56
t_start+51
seq_read_iter+250
seq_read+165
vfs_read+165
ksys_read+103
__x64_sys_read+25
do_syscall_64+56
entry_SYSCALL_64_after_hwframe+100
Since ftrace_lock is held by Task-A and Task-A is actively running on a
CPU, all other tasks waiting for the same lock will spin on their
respective CPUs. This leads to increased CPU pressure.
Reproduction
============
This issue can be reproduced simply by running
`cat available_filter_functions`.
- Single process reading available_filter_functions:
$ time cat /sys/kernel/tracing/available_filter_functions > /dev/null
real 0m0.452s
user 0m0.001s
sys 0m0.451s
- Six processes reading available_filter_functions simultaneously:
for i in `seq 0 5`; do
time cat /sys/kernel/tracing/available_filter_functions > /dev/null &
done
The results are as follows:
real 0m1.801s
user 0m0.000s
sys 0m1.779s
real 0m1.804s
user 0m0.001s
sys 0m1.791s
real 0m1.805s
user 0m0.000s
sys 0m1.792s
real 0m1.806s
user 0m0.001s
sys 0m1.796s
As more processes are added, the system time increases correspondingly.
Solution
========
One approach is to optimize the reading of available_filter_functions to
make it as fast as possible. However, the risk lies in the contention
caused by optimistic spin locking.
Therefore, we need to consider an alternative solution that avoids
optimistic spinning for heavy mutexes that may be held for long durations.
Note that we do not want to disable CONFIG_MUTEX_SPIN_ON_OWNER entirely, as
that could lead to unexpected performance regressions.
In this patch, a new wrapper mutex_lock_nospin() is used for ftrace_lock to
selectively disable optimistic spinning.
Yafang Shao (2):
locking: add mutex_lock_nospin()
ftrace: disable optimistic spinning for ftrace_lock
include/linux/mutex.h | 3 +++
kernel/locking/mutex.c | 39 +++++++++++++++++++++++++------
kernel/trace/ftrace.c | 52 +++++++++++++++++++++---------------------
3 files changed, 61 insertions(+), 33 deletions(-)
--
2.47.3
^ permalink raw reply
* Re: [syzbot] [bpf?] [trace?] KASAN: slab-use-after-free Read in bpf_trace_run3 (2)
From: Qing Wang @ 2026-03-04 7:26 UTC (permalink / raw)
To: syzbot+9ea7c90be2b24e189592
Cc: andrii, ast, bpf, daniel, eddyz87, haoluo, john.fastabend, jolsa,
kpsingh, linux-kernel, linux-trace-kernel, martin.lau,
mathieu.desnoyers, mattbobrowski, mhiramat, rostedt, sdf, song,
syzkaller-bugs, wangqing7171, yonghong.song
In-Reply-To: <69a3e5ad.050a0220.3a55be.0056.GAE@google.com>
#syz test
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index 0378e83b4099..b1e3e9f59930 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -3783,6 +3783,8 @@ static void bpf_raw_tp_link_release(struct bpf_link *link)
bpf_probe_unregister(raw_tp->btp, raw_tp);
bpf_put_raw_tracepoint(raw_tp->btp);
+
+ tracepoint_synchronize_unregister();
}
static void bpf_raw_tp_link_dealloc(struct bpf_link *link)
^ permalink raw reply related
* Re: [PATCH v2 000/110] vfs: change inode->i_ino from unsigned long to u64
From: NeilBrown @ 2026-03-04 6:26 UTC (permalink / raw)
To: Jeff Layton
Cc: David Howells, Alexander Viro, Christian Brauner, Jan Kara,
Steven Rostedt, Masami Hiramatsu, Mathieu Desnoyers, Dan Williams,
Matthew Wilcox, Eric Biggers, Theodore Y. Ts'o, Muchun Song,
Oscar Salvador, David Hildenbrand, Paulo Alcantara,
Andreas Dilger, Jan Kara, Jaegeuk Kim, Chao Yu, Trond Myklebust,
Anna Schumaker, Chuck Lever, Olga Kornievskaia, Dai Ngo,
Tom Talpey, Steve French, Ronnie Sahlberg, Shyam Prasad N,
Bharath SM, Alexander Aring, Ryusuke Konishi, Viacheslav Dubeyko,
Eric Van Hensbergen, Latchesar Ionkov, Dominique Martinet,
Christian Schoenebeck, David Sterba, Marc Dionne, Ian Kent,
Luis de Bethencourt, Salah Triki, Tigran A. Aivazian,
Ilya Dryomov, Alex Markuze, Jan Harkes, coda, Nicolas Pitre,
Tyler Hicks, Amir Goldstein, Christoph Hellwig,
John Paul Adrian Glaubitz, Yangtao Li, Mikulas Patocka,
David Woodhouse, Richard Weinberger, Dave Kleikamp,
Konstantin Komarov, Mark Fasheh, Joel Becker, Joseph Qi,
Mike Marshall, Martin Brandenburg, Miklos Szeredi, Anders Larsen,
Zhihao Cheng, Damien Le Moal, Naohiro Aota, Johannes Thumshirn,
John Johansen, Paul Moore, James Morris, Serge E. Hallyn,
Mimi Zohar, Roberto Sassu, Dmitry Kasatkin, Eric Snowberg, Fan Wu,
Stephen Smalley, Ondrej Mosnacek, Casey Schaufler, Alex Deucher,
Christian König, David Airlie, Simona Vetter, Sumit Semwal,
Eric Dumazet, Kuniyuki Iwashima, Paolo Abeni, Willem de Bruijn,
David S. Miller, Jakub Kicinski, Simon Horman, Oleg Nesterov,
Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Namhyung Kim, Mark Rutland, Alexander Shishkin, Jiri Olsa,
Ian Rogers, Adrian Hunter, James Clark, Darrick J. Wong,
Martin Schiller, Eric Paris, Joerg Reuter, Marcel Holtmann,
Johan Hedberg, Luiz Augusto von Dentz, Oliver Hartkopp,
Marc Kleine-Budde, David Ahern, Neal Cardwell, Steffen Klassert,
Herbert Xu, Remi Denis-Courmont, Marcelo Ricardo Leitner,
Xin Long, Magnus Karlsson, Maciej Fijalkowski, Stanislav Fomichev,
Alexei Starovoitov, Daniel Borkmann, Jesper Dangaard Brouer,
John Fastabend, linux-fsdevel, linux-kernel, linux-trace-kernel,
nvdimm, fsverity, linux-mm, netfs, linux-ext4, linux-f2fs-devel,
linux-nfs, linux-cifs, samba-technical, linux-nilfs, v9fs,
linux-afs, autofs, ceph-devel, codalist, ecryptfs, linux-mtd,
jfs-discussion, ntfs3, ocfs2-devel, devel, linux-unionfs,
apparmor, linux-security-module, linux-integrity, selinux,
amd-gfx, dri-devel, linux-media, linaro-mm-sig, netdev,
linux-perf-users, linux-fscrypt, linux-xfs, linux-hams, linux-x25,
audit, linux-bluetooth, linux-can, linux-sctp, bpf
In-Reply-To: <1c28e34c7167acf4e20c3e201476504135aa44e8.camel@kernel.org>
On Tue, 03 Mar 2026, Jeff Layton wrote:
> On Tue, 2026-03-03 at 10:55 +0000, David Howells wrote:
> > Jeff Layton <jlayton@kernel.org> wrote:
> >
> > > This version splits the change up to be more bisectable. It first adds a
> > > new kino_t typedef and a new "PRIino" macro to hold the width specifier
> > > for format strings. The conversion is done, and then everything is
> > > changed to remove the new macro and typedef.
> >
> > Why remove the typedef? It might be better to keep it.
> >
>
> Why? After this change, internel kernel inodes will be u64's -- full
> stop. I don't see what the macro or typedef will buy us at that point.
Implicit documentation?
ktime_t is (now) always s64, but we still keep the typedef;
It would be cool if we could teach vsprintf to understand some new
specifier to mean "kinode_t" or "ktime_t" etc. But that would trigger
gcc warnings.
NeilBrown
^ permalink raw reply
* Re: [syzbot] [bpf?] [trace?] KASAN: slab-use-after-free Read in bpf_trace_run9
From: syzbot @ 2026-03-04 4:06 UTC (permalink / raw)
To: andrii, ast, bpf, daniel, eddyz87, haoluo, john.fastabend, jolsa,
kpsingh, linux-kernel, linux-trace-kernel, martin.lau,
mathieu.desnoyers, mattbobrowski, mhiramat, rostedt, sdf, song,
syzkaller-bugs, wangqing7171, yonghong.song
In-Reply-To: <20260304033822.107697-1-wangqing7171@gmail.com>
Hello,
syzbot has tested the proposed patch and the reproducer did not trigger any issue:
Reported-by: syzbot+b4c5ad098c821bf8d8bc@syzkaller.appspotmail.com
Tested-by: syzbot+b4c5ad098c821bf8d8bc@syzkaller.appspotmail.com
Tested on:
commit: 0031c068 Merge tag 'cgroup-for-7.0-rc2-fixes' of git:/..
git tree: upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=1434f0ba580000
kernel config: https://syzkaller.appspot.com/x/.config?x=c5c49ee0942d1cdb
dashboard link: https://syzkaller.appspot.com/bug?extid=b4c5ad098c821bf8d8bc
compiler: Debian clang version 21.1.8 (++20251221033036+2078da43e25a-1~exp1~20251221153213.50), Debian LLD 21.1.8
patch: https://syzkaller.appspot.com/x/patch.diff?x=17e5bb5a580000
Note: testing is done by a robot and is best-effort only.
^ permalink raw reply
* Re: [syzbot] [bpf?] [trace?] KASAN: slab-use-after-free Read in bpf_trace_run9
From: Qing Wang @ 2026-03-04 3:38 UTC (permalink / raw)
To: syzbot+b4c5ad098c821bf8d8bc
Cc: andrii, ast, bpf, daniel, eddyz87, haoluo, john.fastabend, jolsa,
kpsingh, linux-kernel, linux-trace-kernel, martin.lau,
mathieu.desnoyers, mattbobrowski, mhiramat, rostedt, sdf, song,
syzkaller-bugs, yonghong.song
In-Reply-To: <69a7276a.a70a0220.b118c.0011.GAE@google.com>
#syz test
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index 0378e83b4099..b1e3e9f59930 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -3783,6 +3783,8 @@ static void bpf_raw_tp_link_release(struct bpf_link *link)
bpf_probe_unregister(raw_tp->btp, raw_tp);
bpf_put_raw_tracepoint(raw_tp->btp);
+
+ tracepoint_synchronize_unregister();
}
static void bpf_raw_tp_link_dealloc(struct bpf_link *link)
^ permalink raw reply related
* Re: [syzbot] [bpf?] [trace?] KASAN: slab-use-after-free Read in bpf_trace_run9
From: Qing Wang @ 2026-03-04 3:33 UTC (permalink / raw)
To: syzbot+b4c5ad098c821bf8d8bc
Cc: andrii, ast, bpf, daniel, eddyz87, haoluo, john.fastabend, jolsa,
kpsingh, linux-kernel, linux-trace-kernel, martin.lau,
mathieu.desnoyers, mattbobrowski, mhiramat, rostedt, sdf, song,
syzkaller-bugs, yonghong.song
In-Reply-To: <69a7276a.a70a0220.b118c.0011.GAE@google.com>
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index 0378e83b4099..b1e3e9f59930 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -3783,6 +3783,8 @@ static void bpf_raw_tp_link_release(struct bpf_link *link)
bpf_probe_unregister(raw_tp->btp, raw_tp);
bpf_put_raw_tracepoint(raw_tp->btp);
+
+ tracepoint_synchronize_unregister();
}
static void bpf_raw_tp_link_dealloc(struct bpf_link *link)
^ permalink raw reply related
* [PATCH] tracing: Disable preemption in the tracepoint callbacks handling filtered pids
From: Steven Rostedt @ 2026-03-04 2:57 UTC (permalink / raw)
To: LKML, Linux trace kernel; +Cc: Masami Hiramatsu, Mathieu Desnoyers
From: "Masami Hiramatsu (Google)" <mhiramat@kernel.org>
Filtering PIDs for events triggered the following during selftests:
[37] event tracing - restricts events based on pid notrace filtering
[ 155.874095]
[ 155.874869] =============================
[ 155.876037] WARNING: suspicious RCU usage
[ 155.877287] 7.0.0-rc1-00004-g8cd473a19bc7 #7 Not tainted
[ 155.879263] -----------------------------
[ 155.882839] kernel/trace/trace_events.c:1057 suspicious rcu_dereference_check() usage!
[ 155.889281]
[ 155.889281] other info that might help us debug this:
[ 155.889281]
[ 155.894519]
[ 155.894519] rcu_scheduler_active = 2, debug_locks = 1
[ 155.898068] no locks held by ftracetest/4364.
[ 155.900524]
[ 155.900524] stack backtrace:
[ 155.902645] CPU: 1 UID: 0 PID: 4364 Comm: ftracetest Not tainted 7.0.0-rc1-00004-g8cd473a19bc7 #7 PREEMPT(lazy)
[ 155.902648] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-debian-1.17.0-1 04/01/2014
[ 155.902651] Call Trace:
[ 155.902655] <TASK>
[ 155.902659] dump_stack_lvl+0x67/0x90
[ 155.902665] lockdep_rcu_suspicious+0x154/0x1a0
[ 155.902672] event_filter_pid_sched_process_fork+0x9a/0xd0
[ 155.902678] kernel_clone+0x367/0x3a0
[ 155.902689] __x64_sys_clone+0x116/0x140
[ 155.902696] do_syscall_64+0x158/0x460
[ 155.902700] ? entry_SYSCALL_64_after_hwframe+0x77/0x7f
[ 155.902702] ? trace_irq_disable+0x1d/0xc0
[ 155.902709] entry_SYSCALL_64_after_hwframe+0x77/0x7f
[ 155.902711] RIP: 0033:0x4697c3
[ 155.902716] Code: 1f 84 00 00 00 00 00 64 48 8b 04 25 10 00 00 00 45 31 c0 31 d2 31 f6 bf 11 00 20 01 4c 8d 90 d0 02 00 00 b8 38 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 35 89 c2 85 c0 75 2c 64 48 8b 04 25 10 00 00
[ 155.902718] RSP: 002b:00007ffc41150428 EFLAGS: 00000246 ORIG_RAX: 0000000000000038
[ 155.902721] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00000000004697c3
[ 155.902722] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000001200011
[ 155.902724] RBP: 0000000000000000 R08: 0000000000000000 R09: 000000003fccf990
[ 155.902725] R10: 000000003fccd690 R11: 0000000000000246 R12: 0000000000000001
[ 155.902726] R13: 000000003fce8103 R14: 0000000000000001 R15: 0000000000000000
[ 155.902733] </TASK>
[ 155.902747]
The tracepoint callbacks recently were changed to allow preemption. The
event PID filtering callbacks that were attached to the fork and exit
tracepoints expected preemption disabled in order to access the RCU
protected PID lists.
Add a guard(preempt)() to protect the references to the PID list.
Cc: stable@vger.kernel.org
Fixes: a46023d5616e ("tracing: Guard __DECLARE_TRACE() use of __DO_TRACE_CALL() with SRCU-fast")
Link: https://patch.msgid.link/20260303131706.96057f61a48a34c43ce1e396@kernel.org
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
---
kernel/trace/trace_events.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/kernel/trace/trace_events.c b/kernel/trace/trace_events.c
index e989362c7709..a4c6469e15d0 100644
--- a/kernel/trace/trace_events.c
+++ b/kernel/trace/trace_events.c
@@ -1039,6 +1039,7 @@ event_filter_pid_sched_process_exit(void *data, struct task_struct *task)
struct trace_pid_list *pid_list;
struct trace_array *tr = data;
+ guard(preempt)();
pid_list = rcu_dereference_raw(tr->filtered_pids);
trace_filter_add_remove_task(pid_list, NULL, task);
@@ -1054,6 +1055,7 @@ event_filter_pid_sched_process_fork(void *data,
struct trace_pid_list *pid_list;
struct trace_array *tr = data;
+ guard(preempt)();
pid_list = rcu_dereference_sched(tr->filtered_pids);
trace_filter_add_remove_task(pid_list, self, task);
--
2.51.0
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox