[GIT PULL] tracing: Updates for 7.2

[GIT PULL] tracing: Updates for 7.2

[Steven Rostedt]

Linus,

tracing: Updates for v7.2:

- Remove a redundant IS_ERR() check

  trace_pipe_open() already checks for IS_ERR() and does it again in the
  return path. Remove the return check.

- Export seq_buf_putmem_hex() to allow kunit tests against them

  To add Kunit tests on seq_buf_putmem_hex(), it needs to be exported.

- Replace strcat() and strcpy() with seq_buf() logic

  The code for synthetic events uses a series of strcat() and strcpy() which
  can be error prone. Replace them with seq_buf() logic that does all the
  necessary bound checking.

- Add a lockdep rcu_is_watching() to trace_##event##_enabled() call

  The trace_##event##_enabled() is a static branch that is true if the
  "event" is enabled. But this can hide bugs if this logic is in a location
  where RCU is disabled and not "watching". It would only trigger if lockdep
  is enabled and the event is enabled.

  Add a "rcu_is_watching()" warning if lockdep is enabled in that helper
  function to trigger regardless if the event is enabled or not.

- Remove the local variable in the trace_printk() macro

  For name space integrity, remove the _______STR variable in the
  trace_printk() macro for using the sizeof() macro directly.

- Use guard()s for the trace_recursion_record.c file

- Fix typo in a comment of eventfs_callback() kerneldoc

- Use trace_call__##event() in events called within trace_##event##_enabled()

  A couple of events are called within an if block guarded by
  trace_##event##_enabled(). That is a static key that is only enabled when
  the event is enabled. The trace_call_##event() calls the tracepoint code
  directly without adding a redundant static key for that check.

- Allow perf to read synthetic events

  Currently, perf does not have the ability to enable a synthetic event. If
  it does, it will either cause a kernel warning or error with "No such
  device". Synthetic events are not much different than kprobes and perf can
  handle fine with a few modifications.

- Replace printk(KERN_WARNING ...) with pr_warn()

- Replace krealloc() on an array with krealloc_array()

- Fix README file path name for synthetic events

- Change tracing_map tracing_map_array to use a flexible array

  Instead of allocating a separate pointer to hold the pages field of
  tracing_map_array, allocate the pages field as a flexible array when
  allocating the structure.

- Fold trace_iterator_increment() into trace_find_next_entry_inc()

  The function trace_iterator_increment() was only used by
  trace_find_next_entry_inc(). It's not big enough to be a helper function
  for one user. Fold it into its caller.

- Make field_var_str field a flexible array of hist_elt_data

  Instead of allocating a separate pointer for the field_var_str array of
  the hist_elt_data structure, allocate it as a flexible array when
  allocating the structure.

- Disable KCOV for trace_irqsoff.c

  Like trace_preemptirq.c, trace_irqsoff.c has code that will crash when
  KCOV is enabled on ARM. The irqsoff tracing can be called on ARM because
  the irqsoff tracing code can be run from early interrupt code and produce
  coverage unrelated to syscall inputs.

- Fix warning in __unregister_ftrace_function() called by perf

  Perf calls unregister_ftrace_function() without checking if its ftrace_ops
  has already been unregistered. There's an error path where on clean up it
  will unregister the ftrace_ops even if it wasn't registered and causes a
  warning.


Please pull the latest trace-v7.2 tree, which can be found at:

NOTE: I did check to make sure this merges cleanly into your tree.

  git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace.git
trace-v7.2

Tag SHA1: d65b33218c1565f4819c4f06a26556e79de7d8ee
Head SHA1: 9581123304b23049437324038698af9fb56ee663


Ao Sun (1):
      tracing: Fix README path for synthetic_events

David Carlier (1):
      tracepoint: Add lockdep rcu_is_watching() check to trace_##name##_enabled()

Karl Mehltretter (1):
      tracing: Disable KCOV instrumentation for trace_irqsoff.o

Martin Kaiser (1):
      tracefs: Fix typo in a comment of eventfs_callback() kerneldoc

Pengpeng Hou (1):
      tracing: Bound synthetic-field strings with seq_buf

Qian-Yu Lin (1):
      tracing: Remove local variable for argument detection from trace_printk()

Rik van Riel (1):
      perf/ftrace: Fix WARNING in __unregister_ftrace_function

Rosen Penev (2):
      tracing: Simplify pages allocation for tracing_map logic
      tracing: Turn hist_elt_data field_var_str into a flexible array

Shuvam Pandey (1):
      seq_buf: Export seq_buf_putmem_hex() and add KUnit tests

Steven Rostedt (2):
      tracing: Allow perf to read synthetic events
      tracing: Move trace_iterator_increment() into trace_find_next_entry_inc()

Vineeth Pillai (2):
      cpufreq: amd-pstate: Use trace_call__##name() at guarded tracepoint call site
      HID: Use trace_call__##name() at guarded tracepoint call sites

Yash Suthar (2):
      tracing: Remove redundant IS_ERR() check in trace_pipe_open()
      tracing: Switch trace_recursion_record.c code over to use guard()

Yu Peng (2):
      tracing/branch: Use pr_warn() instead of printk(KERN_WARNING)
      tracing: Use krealloc_array() for trace option array growth

----
 drivers/cpufreq/amd-pstate.c            |   3 +-
 drivers/hid/intel-ish-hid/ipc/pci-ish.c |   2 +-
 include/linux/trace_printk.h            |   3 +-
 include/linux/tracefs.h                 |   2 +-
 include/linux/tracepoint.h              |   4 ++
 kernel/trace/Makefile                   |   5 +-
 kernel/trace/trace.c                    |  25 +++----
 kernel/trace/trace_branch.c             |   9 +--
 kernel/trace/trace_event_perf.c         |  12 +++-
 kernel/trace/trace_events_hist.c        |  72 ++++++++++---------
 kernel/trace/trace_events_synth.c       | 121 +++++++++++++++++++++++++-------
 kernel/trace/trace_recursion_record.c   |   8 +--
 kernel/trace/trace_remote.c             |   2 +-
 kernel/trace/tracing_map.c              |  30 +++-----
 kernel/trace/tracing_map.h              |   2 +-
 lib/seq_buf.c                           |   1 +
 lib/tests/seq_buf_kunit.c               |  34 +++++++++
 17 files changed, 222 insertions(+), 113 deletions(-)
---------------------------

Re: [GIT PULL] tracing: Updates for 7.2

[Linus Torvalds]

On Tue, 16 Jun 2026 at 15:01, Steven Rostedt <rostedt@goodmis.org> wrote:
>
> tracing: Updates for v7.2:

I wondered why the heck this causes the whole kernel to be rebuilt,
when the diffstat looked fairly innocuous.

It's because it changed <linux/trace_printk.h>, which used to be part
of <linux/kernel.h> but was split out in commit bec261fec6d4
("tracing: move tracing declarations from kernel.h to a dedicated
header").

But the second stage of splitting it out - not including it from
<linux/kernel.h> - was never done, so now while the diffstat *looks*
innocuous, it is still as deadly to the build as an actual
<linux/kernel.h> change would be.

The intent was clearly to make people who actually needed the
trace_printk() functionality to include the header - that's what the
commit log says. But that's not how any of it works ;(

So all that commit did was to make things look like they wouldn't hurt
as much as they do, and that trivial change to that file causes a full
rebuild of the kernel.

I'm not seeing any patches even being discussed that would make only
files that need it to include that file and remove it from
linux/kernel.h that *everybody* uses. Yet that was the *only* point of
making it a separate header in the first place. Yury?

           Linus

Re: [GIT PULL] tracing: Updates for 7.2

[pr-tracker-bot]

The pull request you sent on Tue, 16 Jun 2026 18:01:22 -0400:

> git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace.git trace-v7.2

has been merged into torvalds/linux.git:
https://git.kernel.org/torvalds/c/23b5d045ae5df0a2d509915cedcd82f93261d7bc

Thank you!

-- 
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/prtracker.html

Re: [GIT PULL] tracing: Updates for 7.2

[Steven Rostedt]

On Thu, 18 Jun 2026 21:23:04 -0700
Linus Torvalds <torvalds@linux-foundation.org> wrote:

> The intent was clearly to make people who actually needed the
> trace_printk() functionality to include the header - that's what the
> commit log says. But that's not how any of it works ;(
> 
> So all that commit did was to make things look like they wouldn't hurt
> as much as they do, and that trivial change to that file causes a full
> rebuild of the kernel.
> 
> I'm not seeing any patches even being discussed that would make only
> files that need it to include that file and remove it from
> linux/kernel.h that *everybody* uses. Yet that was the *only* point of
> making it a separate header in the first place. Yury?

Many developers use trace_printk() over printk(). Requiring people to
add that include for every use case would severely limit debugging of
the kernel. Would you be OK if we did the same for printk? That is, if
you want to add a printk to a function, you had to add a include
printk.h to the file before doing so?

-- Steve

Re: [GIT PULL] tracing: Updates for 7.2

[Linus Torvalds]

[-- Attachment #1: Type: text/plain, Size: 1380 bytes --]

On Fri, 19 Jun 2026 at 05:15, Steven Rostedt <rostedt@goodmis.org> wrote:
>
> Many developers use trace_printk() over printk().

You're on some bad drugs, Steven.

By "many developers" you clearly must have meant "almost nobody".

> Requiring people to
> add that include for every use case would severely limit debugging of
> the kernel. Would you be OK if we did the same for printk?

Stop making inane and silly arguments that have nothing to do with reality.

The two are not the same:

  $ git grep -wl printk | wc -l
  3212
  $ git grep -wl trace_printk | wc -l
  50

and making them sound remotely similar is just making you look dishonst.

Look, the attached patch makes my defconfig build cleanly, and makes a

  $ touch include/linux/trace_printk.h

cause my recompile to build just a handful of files. While touching
linux/kernel.h causes thousands of files to be re-build.

That's a whopping

 6 files changed, 6 insertions(+), 1 deletion(-)

to make my tree build without including <linux/trace_printk.h> everywhere.

Now, that's _just_ my defconfig, and the full build clearly has a
handful of other cases. From a quick look, mostly in the 'samples'
directory.

But it looks like a *handful* of other cases.

Now, if the same was true for printk(), I'd happily split that into
some other header too.

But it isn't.

Because we live in reality.

               Linus

[-- Attachment #2: patch.diff --]
[-- Type: text/x-patch, Size: 2236 bytes --]

 drivers/tty/sysrq.c    | 1 +
 include/linux/kernel.h | 1 -
 include/linux/trace.h  | 2 ++
 kernel/panic.c         | 1 +
 kernel/rcu/rcu.h       | 1 +
 lib/sys_info.c         | 1 +
 6 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/drivers/tty/sysrq.c b/drivers/tty/sysrq.c
index c2e4b31b699a..9d080e10baa3 100644
--- a/drivers/tty/sysrq.c
+++ b/drivers/tty/sysrq.c
@@ -51,6 +51,7 @@
 #include <linux/syscalls.h>
 #include <linux/of.h>
 #include <linux/rcupdate.h>
+#include <linux/trace_printk.h>
 
 #include <asm/ptrace.h>
 #include <asm/irq_regs.h>
diff --git a/include/linux/kernel.h b/include/linux/kernel.h
index e5570a16cbb1..e87a40fbd152 100644
--- a/include/linux/kernel.h
+++ b/include/linux/kernel.h
@@ -31,7 +31,6 @@
 #include <linux/build_bug.h>
 #include <linux/sprintf.h>
 #include <linux/static_call_types.h>
-#include <linux/trace_printk.h>
 #include <linux/util_macros.h>
 #include <linux/wordpart.h>
 
diff --git a/include/linux/trace.h b/include/linux/trace.h
index 7eaad857dee0..070a406a7f05 100644
--- a/include/linux/trace.h
+++ b/include/linux/trace.h
@@ -2,6 +2,8 @@
 #ifndef _LINUX_TRACE_H
 #define _LINUX_TRACE_H
 
+#include <linux/trace_printk.h>
+
 #define TRACE_EXPORT_FUNCTION	BIT(0)
 #define TRACE_EXPORT_EVENT	BIT(1)
 #define TRACE_EXPORT_MARKER	BIT(2)
diff --git a/kernel/panic.c b/kernel/panic.c
index 213725b612aa..983ad272d622 100644
--- a/kernel/panic.c
+++ b/kernel/panic.c
@@ -37,6 +37,7 @@
 #include <linux/context_tracking.h>
 #include <linux/seq_buf.h>
 #include <linux/sys_info.h>
+#include <linux/trace_printk.h>
 #include <trace/events/error_report.h>
 #include <asm/sections.h>
 #include <kunit/test-bug.h>
diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h
index fa6d30ce73d1..7db85c9f6c73 100644
--- a/kernel/rcu/rcu.h
+++ b/kernel/rcu/rcu.h
@@ -11,6 +11,7 @@
 #define __LINUX_RCU_H
 
 #include <linux/slab.h>
+#include <linux/trace_printk.h>
 #include <trace/events/rcu.h>
 
 /*
diff --git a/lib/sys_info.c b/lib/sys_info.c
index f32a06ec9ed4..7ded4e7f3671 100644
--- a/lib/sys_info.c
+++ b/lib/sys_info.c
@@ -10,6 +10,7 @@
 #include <linux/sched/debug.h>
 #include <linux/string.h>
 #include <linux/sysctl.h>
+#include <linux/trace_printk.h>
 
 #include <linux/sys_info.h>
 

Re: [GIT PULL] tracing: Updates for 7.2

[Yury Norov]

On Thu, Jun 18, 2026 at 09:23:04PM -0700, Linus Torvalds wrote:
> On Tue, 16 Jun 2026 at 15:01, Steven Rostedt <rostedt@goodmis.org> wrote:
> >
> > tracing: Updates for v7.2:
> 
> I wondered why the heck this causes the whole kernel to be rebuilt,
> when the diffstat looked fairly innocuous.
> 
> It's because it changed <linux/trace_printk.h>, which used to be part
> of <linux/kernel.h> but was split out in commit bec261fec6d4
> ("tracing: move tracing declarations from kernel.h to a dedicated
> header").
> 
> But the second stage of splitting it out - not including it from
> <linux/kernel.h> - was never done, so now while the diffstat *looks*
> innocuous, it is still as deadly to the build as an actual
> <linux/kernel.h> change would be.
> 
> The intent was clearly to make people who actually needed the
> trace_printk() functionality to include the header - that's what the
> commit log says. But that's not how any of it works ;(
> 
> So all that commit did was to make things look like they wouldn't hurt
> as much as they do, and that trivial change to that file causes a full
> rebuild of the kernel.
> 
> I'm not seeing any patches even being discussed that would make only
> files that need it to include that file and remove it from
> linux/kernel.h that *everybody* uses. Yet that was the *only* point of
> making it a separate header in the first place. Yury?

Yes, you are right. The original intention was to decouple trace_printk()
and kernel.h, and I have the patch for it in v4:

https://lore.kernel.org/all/aVvoe5fQN3EUtEAJ@yury/

The Steven's point is the convenience for developers to throw
trace_printk() here and there without including any headers.

I'm not one of those trace_printk() users, but I take that. Even
though, I don't think it's a strong argument, pretty much for the
same reasons that you've described above.

I suggested adding a debugging flag, so that trace_printk() would
be included in kernel.h conditionally, if it's absolutely necessary
to have the API included "automatically":

> >         make D=1 // trace_printk() is available
> >         make D=0 // trace_printk() is not available
> >         make     // trace_printk() is not available

That was rejected too.

I even measured 1.5-2% compile time decrease back then. Surely need to
retest it to make sure.

I'm still for decoupling trace_printk() and kernel.h, so I can resend
if needed.

Thanks,
Yury

Re: [GIT PULL] tracing: Updates for 7.2

[Linus Torvalds]

On Fri, 19 Jun 2026 at 08:19, Yury Norov <yury.norov@gmail.com> wrote:
>
> The Steven's point is the convenience for developers to throw
> trace_printk() here and there without including any headers.

Steven's point isn't a point. It's just fantasy.

trace_printk() isn't special. It's not even common. And people who add
debugging printk's can add a header file to their own hackery, rather
than hurt everybody else.

                    Linus

Re: [GIT PULL] tracing: Updates for 7.2

[Sebastian Andrzej Siewior]

On 2026-06-19 07:35:15 [-0700], Linus Torvalds wrote:
> On Fri, 19 Jun 2026 at 05:15, Steven Rostedt <rostedt@goodmis.org> wrote:
> >
> > Many developers use trace_printk() over printk().
> 
> You're on some bad drugs, Steven.
> 
> By "many developers" you clearly must have meant "almost nobody".

I do use it for debugging all the time. The alternative is printk but it
is "slower" if it throws everything to the UART which I prefer to limit
to BUG/WARN kind of messages. Also it does not record the context where
this comes from which is often helpful.
Afaik it is the only to debug printk.

> > Requiring people to add that include for every use case would
> > severely limit debugging of the kernel. Would you be OK if we did
> > the same for printk?
> 
> Stop making inane and silly arguments that have nothing to do with
> reality.
> 
> The two are not the same:
> 
>   $ git grep -wl printk | wc -l 3212 $ git grep -wl trace_printk | wc
>   -l 50
> 
> and making them sound remotely similar is just making you look dishonst.
> 
> Look, the attached patch makes my defconfig build cleanly, and makes a
> 
>   $ touch include/linux/trace_printk.h
> 
> cause my recompile to build just a handful of files. While touching
> linux/kernel.h causes thousands of files to be re-build.
> 
> That's a whopping
> 
>  6 files changed, 6 insertions(+), 1 deletion(-)
> 
> to make my tree build without including <linux/trace_printk.h> everywhere.
> 
> Now, that's _just_ my defconfig, and the full build clearly has a
> handful of other cases. From a quick look, mostly in the 'samples'
> directory.
> 
> But it looks like a *handful* of other cases.
> 
> Now, if the same was true for printk(), I'd happily split that into
> some other header too.
> 
> But it isn't.
> 
> Because we live in reality.
> 
>                Linus

Sebastian

Re: [GIT PULL] tracing: Updates for 7.2

[Linus Torvalds]

On Fri, 19 Jun 2026 at 08:40, Sebastian Andrzej Siewior
<bigeasy@linutronix.de> wrote:
>
> I do use it for debugging all the time. The alternative is printk but it
> is "slower" if it throws everything to the UART which I prefer to limit
> to BUG/WARN kind of messages. Also it does not record the context where
> this comes from which is often helpful.

Go ahead and use it. But don't make everybody else hurt just because
you do. If you add local hacks, add the local hack header.

Don't say "I'M SO IMPORTANT THAT I CAN'T BE BOTHERED, SO I'LL BOTHER
EVERYBODY ELSE BY MAKING THEIR BUILDS POINTLESSLY SLOWER"

Do you really want to be that person? Do you really want to make that argument?

              Linus

Re: [GIT PULL] tracing: Updates for 7.2

[Steven Rostedt]

On Fri, 19 Jun 2026 07:35:15 -0700
Linus Torvalds <torvalds@linux-foundation.org> wrote:

> 
> Now, if the same was true for printk(), I'd happily split that into
> some other header too.
> 
> But it isn't.
> 
> Because we live in reality.

I'm talking about using it for debugging and not to print info. In
fact, I try hard to *prevent* people keeping trace_printk() *in* the
kernel code. If you add it, it triggers that nasty banner.

The point of having it in kernel.h is for debugging purposes. That's
probably why printk.h is also there too. Many others use the "shotgun"
approach where they add trace_printk()s all over the place to track down
a bug.

Your patch now requires developers from remembering to add a
#include <linux/trace_printk.h> in every file the touch. Or they may
have to touch kernel.h triggering a full build as well. It moves burden
from people who build the kernel and have to wait a little longer to
developers that now need to remember to add includes that they didn't
need to before!

trace_printk() hardly ever changes, so it should not trigger these
builds on changing kernels. The only reason it was changed this time
was because the compilers now have more warnings about printk formats
that required annotation to trace_printk(). As well as a clean up to
remove a variable from a macro.

When I said people use trace_printk() over printk() I meant I know
several (including myself) the seldom ever use printk for debugging the
kernel. If I need to see what's happening, I use trace_printk().
Because it can be used in any context without worrying about slowing
down the system, and less likely to cause heissenbugs.

-- Steve

Re: [GIT PULL] tracing: Updates for 7.2

[Linus Torvalds]

On Fri, 19 Jun 2026 at 08:54, Steven Rostedt <rostedt@goodmis.org> wrote:
>
> The point of having it in kernel.h is for debugging purposes. That's
> probably why printk.h is also there too.

No, Steven.

printk.h is there because EVERYTHING USES IT.

trace_printk.h is getting removed from there because ALMOST NOTHING USES IT.

See the subtle but important difference? I'm trying to highlight it here.

            Linus

Re: [GIT PULL] tracing: Updates for 7.2

[Sebastian Andrzej Siewior]

On 2026-06-19 08:43:14 [-0700], Linus Torvalds wrote:
> Go ahead and use it. But don't make everybody else hurt just because
> you do. If you add local hacks, add the local hack header.
> 
> Don't say "I'M SO IMPORTANT THAT I CAN'T BE BOTHERED, SO I'LL BOTHER
> EVERYBODY ELSE BY MAKING THEIR BUILDS POINTLESSLY SLOWER"
> 
> Do you really want to be that person? Do you really want to make that argument?

Well, no.
This started as "Steven you must be crazy to think someone is using
trace_printk". That is not the case. There should be no in-tree users as
nothing should use it - there are trace events for regular users.

So if this is just about splitting trace_printk away from kernel.h,
okay. Even container_of() moved out and is included explicitly which is
used a lot. If this next so be it.

>               Linus

Sebastian

Re: [GIT PULL] tracing: Updates for 7.2

[Linus Torvalds]

On Fri, 19 Jun 2026 at 11:30, Sebastian Andrzej Siewior
<bigeasy@linutronix.de> wrote:
>
> So if this is just about splitting trace_printk away from kernel.h,
> okay. Even container_of() moved out and is included explicitly which is
> used a lot. If this next so be it.

Absolutely.

We should aim to never have the insanity that is a typical C++ build
system, where you have headers that include the world because of how
the C++ method and object models work.

That way lies insanity. It's not only "random small header changes
rebuild everything", it's also simply "everything is slower because
the compiler has to parse way too much irrelevant garbage".

Now, we're not exactly doing great in this area, I'm afraid. To pick a
random small core kernel file, we have this kind of disgusting
situation:

  $ make kernel/pid.i
  $ wc -c kernel/pid.c kernel/pid.i
    24460 kernel/pid.c
  5271779 kernel/pid.i

ie a relatively tiny sub-1kloc file balloons by a factor of 215x and
becomes 5MB worth of noise to parse for the compiler because of header
inclusion.

And the build cost is pretty much directly related to that kind of
expansion (admittedly linking is bad too, mainly because our linking
phases tend to be more serialized than building most C files)

Now, I picked a small file on purpose and this is with 'allmodconfig'
which makes a bad situation even worse by about 20%. That small file
expands to "only" 4.2MB in a more sedate config.

But even our *biggest* C file in the kernel subdirectory sees a 20x
expansion due to pre-processing.

And yes, <kernel/trace_printk.c> is part of that, but certainly not a
very big part - it's only 6kB and 202 lines in that mess.

A pittance. Hardly noticeable.

But "it's a small part of the problem" doesn't mean that it shouldn't
be fixed when it looks like it's *trivial* to fix.

Now, the biggest single cause of horrible expansion to be some of our
more complex nested macros.  So it's not always about the size of the
header and how many other headers it includes, we have some other
structural issues that make it worse.

We had very real build performance issues due to the deep nesting of
the min/max macros to the point that they even caused build failures
on machines with limited memory.

But we mostly fixed the worst part of that mess, and the type checking
in those macros at least add real value to normal code.

But in general, if the question is "should we split this header file
up if it's straightforward and easy", the answer is *ALWAYS* a
resounding "YES".

Most of the time it's just a pain to unravel all the type
dependencies. This case look strivial compared to some we've done.

One of the things I at one point hoped sparse would do was a "you're
including this header for no good reason" tooling.

But it's non-trivial to do, particularly with lots of different
architectures that aren't very consistent in where they declare or use
things.

                     Linus

Re: [GIT PULL] tracing: Updates for 7.2

[Thomas Gleixner]

On Fri, Jun 19 2026 at 12:07, Linus Torvalds wrote:
> On Fri, 19 Jun 2026 at 11:30, Sebastian Andrzej Siewior
> <bigeasy@linutronix.de> wrote:
> But in general, if the question is "should we split this header file
> up if it's straightforward and easy", the answer is *ALWAYS* a
> resounding "YES".

Full ACK.

> Most of the time it's just a pain to unravel all the type
> dependencies. This case look strivial compared to some we've done.
>
> One of the things I at one point hoped sparse would do was a "you're
> including this header for no good reason" tooling.
>
> But it's non-trivial to do, particularly with lots of different
> architectures that aren't very consistent in where they declare or use
> things.

That's only one part of the overall problem.

People are really careless when they add new things. It took me
literally weeks to handle all the 0-day compile fallout when I tried to
disentangle the __percpu mess so that a header using a __percpu
annotation in a struct declaration is not required to include the world
and running into recursive header hell.

See c06cd66387da ("percpu: Sanitize __percpu_qual include hell")

I'm still fighting to resolve the last remaining issue to get rid of the
asm/xtime.h dependency in some other cleanup project. That header is so
deeply ingrained in some of our header chains for the very wrong
reasons.

I've tried to utilize AI for analyzing it, but that ended up in an even
bigger disaster.

One thing I'm seeing is that the kernel is patently bad in separating
data type declarations from actual APIs, where the APIs usually just
need a forward declaration of the pointer type. Ditto for struct
declarations with pointer types. 

After doing quite some of these disentanglements manually and trying to
utilize all kinds of tooling, I still have no good answer to this
unfortunately.

Thanks,

        tglx

Re: [GIT PULL] tracing: Updates for 7.2

[Linus Torvalds]

On Fri, 19 Jun 2026 at 13:28, Thomas Gleixner <tglx@kernel.org> wrote:
>
> One thing I'm seeing is that the kernel is patently bad in separating
> data type declarations from actual APIs, where the APIs usually just
> need a forward declaration of the pointer type. Ditto for struct
> declarations with pointer types.

We tend to do a lot of inline functions, and that often makes it
impossible to do the simple "just forward-declare the type".

And often there are fairly good reasons for the inline functions, in
that they tend to be much safer than macros, both for type checking
and for the whole "use argument once" reason (they also avoid compiler
warnings when the argument _isn't_ used).

But yes, it's often really nasty to try to split things up, even when
it *looks* straightforward.

             Linus

Re: [GIT PULL] tracing: Updates for 7.2

[Yury Norov]

On Fri, Jun 19, 2026 at 08:40:12AM -0700, Linus Torvalds wrote:
> On Fri, 19 Jun 2026 at 08:19, Yury Norov <yury.norov@gmail.com> wrote:
> >
> > The Steven's point is the convenience for developers to throw
> > trace_printk() here and there without including any headers.
> 
> Steven's point isn't a point. It's just fantasy.
> 
> trace_printk() isn't special. It's not even common. And people who add
> debugging printk's can add a header file to their own hackery, rather
> than hurt everybody else.

OK, then I'll resend my patch. I tested it with build bots against
various architectures and configs, so I believe it's complete.

Your version includes trace_printk.h in linux/trace.h. It looks
reasonable, and it allows to drop explicit trace_printk.h
inclusion from:
  - kernel/trace/trace.c;
  - kernel/trace/trace_events_trigger.c;
  - kernel/trace/trace_functions.c;
  - kernel/trace/trace_printk.c.

With that, the total number of files including the trace_printk.h 
is 30, which looks reasonable.

I need a couple days to get feedback from build bots, so not sure I'll
submit it before the end of the week.

If somebody wants to give it a try, check here:

https://github.com/norov/linux/tree/trace_printk

Thanks,
Yury

Re: [GIT PULL] tracing: Updates for 7.2

[Linus Torvalds]

On Fri, 19 Jun 2026 at 12:07, Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
> Now, we're not exactly doing great in this area, I'm afraid. To pick a
> random small core kernel file, we have this kind of disgusting
> situation:

Actually, we have much worse cases.

Doing a "make allmodconfig" will do all those '*.mod.c" files, which
are tiny generated files that just have some mod-info structure etc
for linking into the module.

They're like ten lines of code. MODULE_INFO() and a 'struct module'
that gets linked in with the actual code. Auto-generated by
scropts/mod/modpost.c.

And then they have

   #include <linux/module.h>
   #include <linux/export-internal.h>
   #include <linux/compiler.h>

and those three lines of include make them balloon into 3MB of code,
so building those *tiny* things takes forever.

(Where "forever" is obviously relative - but it's disgustingly long
for something that should be instant).

In fact, just a single

  #include <linux/module.h>

and absolutely nothing else, will give you 2.8MB of output.

(In contrast, the two other headers we include - export-internal.h and
compiler.h - result in 13kB of output. Yesh, it's 550 lines of pretty
random stuff, but it's not really all that bad. It's mostly just
really core type declarations, so it's all quite understandable)

That module.h header situation is beyond disgusting - and yes, it's
very much noticeable in my build time. Adding some module maintainer
people, because this is bad, and I think it needs to be fixed.

             Linus

Re: [GIT PULL] tracing: Updates for 7.2

[Willy Tarreau]

On Fri, Jun 19, 2026 at 01:55:15PM -0700, Linus Torvalds wrote:
> On Fri, 19 Jun 2026 at 13:28, Thomas Gleixner <tglx@kernel.org> wrote:
> >
> > One thing I'm seeing is that the kernel is patently bad in separating
> > data type declarations from actual APIs, where the APIs usually just
> > need a forward declaration of the pointer type. Ditto for struct
> > declarations with pointer types.
> 
> We tend to do a lot of inline functions, and that often makes it
> impossible to do the simple "just forward-declare the type".
> 
> And often there are fairly good reasons for the inline functions, in
> that they tend to be much safer than macros, both for type checking
> and for the whole "use argument once" reason (they also avoid compiler
> warnings when the argument _isn't_ used).

In haproxy we've had a hard time dealing with circular dependencies and
build time issue for a while and a few years ago we found what looks
like a mostly optimal arrangement. Each include file exists in two
versions:
  - one suffixed with "-t" which contains only type declarations
    (#define as well as enums, structs etc)
  - the main one without "-t" that includes the first one and contains
    declarations (extern variables & functions as well as inline)

And all this with a strict rule: no -t file may ever include any of the
other ones. It now works pretty well, and we could significantly reduce
the depth of includes and no longer need to resolve broken inline-to-inline
references. And we discovered that many C files don't even need the main
file and only need types definitions, further reducing build time. Also
we figured that certain -t files only included another one for a single
struct definition that appeared as a pointer, so these were sometimes
replaced with just a forward-declaration before use and that was done.

We made mistakes too, most of our macros that replaced inline functions
tend to be in the main files instead of the -t one as the rule implies,
encouraging to place further macro definitions at the wrong place. But
moving them is no big deal.

If I would change something, it would probably be around long macro
definitions (e.g. series of flags) that are often not needed by other
-t files and only by functions. It's just difficult for me to express
that distinction in a way that's clear enough for eveyrone to respect,
but this could definitely further reduce the preprocessing time of the
types-only "-t" files.

Willy

Re: [GIT PULL] tracing: Updates for 7.2

[Julia Lawall]

On Fri, 19 Jun 2026, Linus Torvalds wrote:

> On Fri, 19 Jun 2026 at 05:15, Steven Rostedt <rostedt@goodmis.org> wrote:
> >
> > Many developers use trace_printk() over printk().
>
> You're on some bad drugs, Steven.
>
> By "many developers" you clearly must have meant "almost nobody".

I use trace_printk on a daily basis.  Tracepoints are useful to get an
overview of what is going on, and then trace_printk is useful for
interleaving that with information relevant to the specific problem at
hand.

julia


>
> > Requiring people to
> > add that include for every use case would severely limit debugging of
> > the kernel. Would you be OK if we did the same for printk?
>
> Stop making inane and silly arguments that have nothing to do with reality.
>
> The two are not the same:
>
>   $ git grep -wl printk | wc -l
>   3212
>   $ git grep -wl trace_printk | wc -l
>   50
>
> and making them sound remotely similar is just making you look dishonst.
>
> Look, the attached patch makes my defconfig build cleanly, and makes a
>
>   $ touch include/linux/trace_printk.h
>
> cause my recompile to build just a handful of files. While touching
> linux/kernel.h causes thousands of files to be re-build.
>
> That's a whopping
>
>  6 files changed, 6 insertions(+), 1 deletion(-)
>
> to make my tree build without including <linux/trace_printk.h> everywhere.
>
> Now, that's _just_ my defconfig, and the full build clearly has a
> handful of other cases. From a quick look, mostly in the 'samples'
> directory.
>
> But it looks like a *handful* of other cases.
>
> Now, if the same was true for printk(), I'd happily split that into
> some other header too.
>
> But it isn't.
>
> Because we live in reality.
>
>                Linus
>

Re: [GIT PULL] tracing: Updates for 7.2

[Steven Rostedt]

On Fri, 19 Jun 2026 07:35:15 -0700
Linus Torvalds <torvalds@linux-foundation.org> wrote:

> The two are not the same:
> 
>   $ git grep -wl printk | wc -l
>   3212
>   $ git grep -wl trace_printk | wc -l
>   50
> 
> and making them sound remotely similar is just making you look dishonst.
> 
> Look, the attached patch makes my defconfig build cleanly, and makes a
> 
>   $ touch include/linux/trace_printk.h
> 
> cause my recompile to build just a handful of files. While touching
> linux/kernel.h causes thousands of files to be re-build.
> 
> That's a whopping
> 
>  6 files changed, 6 insertions(+), 1 deletion(-)
> 
> to make my tree build without including <linux/trace_printk.h> everywhere.
> 
> Now, that's _just_ my defconfig, and the full build clearly has a
> handful of other cases. From a quick look, mostly in the 'samples'
> directory.
> 
> But it looks like a *handful* of other cases.
> 
> Now, if the same was true for printk(), I'd happily split that into
> some other header too.

I totally agree that trace_printk.h should not be added to kernel.h for
debugging if printk.h is also not included for that purpose. But you
are stating that it is included because everything "uses" it. But
really, if we are worried about getting rid of adding headers that are
not used, I think it would be a worth while exercise to remove printk.h
from kernel.h to have it included only where it is used.

That said, I'm also happy to get rid of trace_printk.h from kernel.h if
there's an easy way to have it included for debugging purposes.

Are you OK with the following approach? Basically, it adds
"CONFIG_TRACE_PRINTK_DEBUGGING" config to be set by those that want to
use it for debugging. When set, it will add to KBUILD_CFLAGS:

  -include $(srctree)/include/linux/trace_printk.h

which is basically the same as having it in kernel.h but without adding
the "burden" to those that don't want it (like yourself).

This would allow the best of both words:

diff --git a/Makefile b/Makefile
index d33a7cadd237..84236890a806 100644
--- a/Makefile
+++ b/Makefile
@@ -839,6 +839,10 @@ ifdef CONFIG_FUNCTION_TRACER
   CC_FLAGS_FTRACE := -pg
 endif
 
+ifdef CONFIG_TRACE_PRINTK_DEBUGGING
+  LINUXINCLUDE += -include $(srctree)/include/linux/trace_printk.h
+endif
+
 ifdef CONFIG_TRACEPOINTS
 # To check for unused tracepoints (tracepoints that are defined but never
 # called), run with:
diff --git a/include/linux/trace_printk.h b/include/linux/trace_printk.h
index 2670ec7f4262..3fe5fc5c6a2a 100644
--- a/include/linux/trace_printk.h
+++ b/include/linux/trace_printk.h
@@ -1,11 +1,12 @@
 /* SPDX-License-Identifier: GPL-2.0 */
 #ifndef _LINUX_TRACE_PRINTK_H
 #define _LINUX_TRACE_PRINTK_H
+#if !defined(__ASSEMBLY__) && !defined(__GENKSYMS__) && !defined(BUILD_VDSO)
 
-#include <linux/compiler_attributes.h>
 #include <linux/instruction_pointer.h>
 #include <linux/stddef.h>
 #include <linux/stringify.h>
+#include <linux/stdarg.h>
 
 /*
  * General tracing related utility functions - trace_printk(),
@@ -199,5 +200,5 @@ ftrace_vprintk(const char *fmt, va_list ap)
 }
 static inline void ftrace_dump(enum ftrace_dump_mode oops_dump_mode) { }
 #endif /* CONFIG_TRACING */
-
+#endif /* !defined(__ASSEMBLY__) && !defined(__GENKSYMS__) */
 #endif
diff --git a/kernel/trace/Kconfig b/kernel/trace/Kconfig
index e130da35808f..e305b27ca029 100644
--- a/kernel/trace/Kconfig
+++ b/kernel/trace/Kconfig
@@ -210,6 +210,16 @@ menuconfig FTRACE
 
 if FTRACE
 
+config TRACE_PRINTK_DEBUGGING
+	bool "Debug with trace_printk()"
+	help
+	  If you need to debug with trace_printk(), instead of adding
+	  include <linux/trace_printk.h> to every file you add a trace_printk
+	  to, select this option and it will add trace_printk.h to all code
+	  to allow tracing with trace_printk() with.
+
+	  If in doubt, select N
+
 config TRACEFS_AUTOMOUNT_DEPRECATED
 	bool "Automount tracefs on debugfs [DEPRECATED]"
 	depends on TRACING


-- Steve

Re: [GIT PULL] tracing: Updates for 7.2

[Linus Torvalds]

On Sat, 20 Jun 2026 at 15:20, Steven Rostedt <rostedt@goodmis.org> wrote:
>
> I totally agree that trace_printk.h should not be added to kernel.h for
> debugging if printk.h is also not included for that purpose. But you
> are stating that it is included because everything "uses" it. But
> really, if we are worried about getting rid of adding headers that are
> not used, I think it would be a worth while exercise to remove printk.h
> from kernel.h to have it included only where it is used.

Feel free to try to come up with such a patch.

But honestly, before you do, what is the *advantage* of such a thing?

Because you seem to think that "trace_printk()" and "printk()" are the same.

They really aren't.

You don't seem to be willing or able to acknowledge that one is very
common, and the other is not.

If I did my quick grep and math right, we have this situation:

'printk()' and the basic 'pr_err()/pr_warn()' calls exported by
<linux/printk.h> exist in 19.2% of all C files in the kernel (7104
files):

    git grep -E -wl '(printk)|(pr_err)|(pr_warn)' -- '*.c' | wc -l

While 'trace_printk()' exists in 0.054% of all C files in the kernel
(twenty files total):

    git grep -E -wl '(trace_printk)' -- '*.c' | wc -l

and I think you'll get a few more hits if you look for 'tracing_on'
etc, so maybe it will hit a tenth of a percent overall.

But from a quick look, even *that* is actually mostly sample code and
tools/perf/, not in actual kernel code.

So be honest now: one of them makes sense to have as a pretty standard
baseline for kernel programming.

The other DOES NOT.

If you want to find other header files to split out, by all means, be my guest.

As mentioned, the more headers we can split off, the better.

But when one header file is used by about a fifth of all C files,
that's a pretty important header, wouldn't you say? Certainly not the
*first* header you should try to split off.

So I think you should look for *other* headers to split off, instead
of going "ok, printk and trace_printk are the same".

Because they damn well are not even remotely the same, and when you
try to equate the two, you just look incompetent.

           Linus

Re: [GIT PULL] tracing: Updates for 7.2

[Steven Rostedt]

On Sat, 20 Jun 2026 15:39:25 -0700
Linus Torvalds <torvalds@linux-foundation.org> wrote:

> Feel free to try to come up with such a patch.
> 
> But honestly, before you do, what is the *advantage* of such a thing?

For debugging it is really useful.

> 
> Because you seem to think that "trace_printk()" and "printk()" are the same.
> 
> They really aren't.

I do not in any way think they are the same. To me, printk() is used
for information to the console for various production messages, whereas
trace_printk() is used when you have that nasty bug that you don't know
exactly where it is. trace_printk() should never be used in a
production environment.

When I (and many others) use trace_printk() to debug, we basically use
the "shotgun" approach. That is, we add trace_printk() all over the
place to see where the bug occurs. This could be for 10s of files.
Having to add an include to each one of these files is a burden and
adds to the frustration when you are debugging something that doesn't
work. You just want to add trace_printk() in places to see where the
bug triggers.

A lot of times, all I add is:

	trace_printk("%s:%d\n", __func__, __LINE__):

and cut and paste that in several locations in several files between if
statements. I may even add:

	if (bad_condition())
		tracing_off();

Which will disable tracing when the bad condition is detected. Then I
can look at the trace to see all the prints up to the bug. This is
*really* useful!!!

I really want to avoid having to add an include for trace_printk when
I'm focusing on just finding were the bug happens.

I'm pretty sure others on the Cc list have the same use case.

I totally sympathize with getting rid of junk out of kernel.h (and
possibly getting rid of kernel.h altogether) but I also want to keep
this debugging ability around.

-- Steve

Re: [GIT PULL] tracing: Updates for 7.2

[Linus Torvalds]

On Sat, 20 Jun 2026 at 16:44, Steven Rostedt <steven@rostedt.org> wrote:
>
> For debugging it is really useful.

According to that logic, we should put everything in one header file
and call it a day.

Because no, trace_printk() isn't special.

Christ, the only reason *you* think it's so special is because it's
your code and you use it, and the people you interact with use it.

But that's not because it's special - the common issue is *you*, not
trace_printk.

And yes, we're *all* special. Just like our mothers told us. But
because we're *all* special, none of us are.

So no. Stop making arguments based on that "I'm special" argument.

If you can't back it up with numbers - like I did - then you're not
special, you're just a snowflake.

Honestly, the "I'm using it for debugging" is not an argument.

You can damn well use it for debugging and add the header then.

But I personally think that debugging should be done by looking at and
understanding the code. To me, "debugging using printk's" - fancy
printk's or not - is a failure mode, and soemthing you should be
quietly ashamed of, instead of using it as some kind of argument for
making everybody elses kernel worse.

By the time you resort to debugging with printk, adding the
appropriate include just isn't the issue any more.

We're done here. That header gets removed from kernel.h.

                    Linus

Re: [GIT PULL] tracing: Updates for 7.2

[Steven Rostedt]

On Sat, 20 Jun 2026 17:18:26 -0700
Linus Torvalds <torvalds@linux-foundation.org> wrote:

> 
> Because no, trace_printk() isn't special.

When you have a bug that triggers once every 11 days, due to some
strange race, placing trace_printk()s all over to help you narrow down
*where* to look is super important. trace_printk() has an extremely low
overhead, like 150 ns which keeps the chances of it affecting the races
low. Note, the above scenario was real story another kernel developer
told me about, but that's besides the point.

> 
> Christ, the only reason *you* think it's so special is because it's
> your code and you use it, and the people you interact with use it.

Do you know who the people I interact with are? 100s if not 1000s of
kernel developers! I get my ideas from going to conferences and talking
with people. This isn't me in some silo helping out an internal
corporation with their secret sauce. I was unemployed from April 19th
to June 15th and in that time I paid my own way to go to LSF/MM/BPF to
find out more ways the tracing subsystem can be useful to others!

Hang out more at conferences and talk to your kernel's developers and
get a better idea of what people use. You interface mostly with top
level maintainers. How much are you interfacing with the 1000s of
contributors we have? Tracing has been extremely helpful for new
developers. I've mentored several. How many new developers have you
mentored in the last 5 years? That's a serious question. I don't know
if you mentor new developers or not. It would be great if you did.

Just last week, someone asked me how I got involved with the Linux
kernel. They said, "It's so big and complex, how did you start?". I
started looking at the kernel code in 1998. It was much simpler back
then. Learning most of the kernel wasn't that difficult compared to
starting development today. I only had to learn things as they came
into existence. Learning about new features that appear over the years
is much easier than coming in fresh and everything is a new feature to
learn.

> 
> But that's not because it's special - the common issue is *you*, not
> trace_printk.
> 

You already had others tell you that they use trace_printk() for
debugging too. It's *not* just me! Ask people. There's been several
times I've had people come up to me at conferences and thank me for
trace_printk().

Heck, trace_prink() was important enough to be included in eBPF for
debugging they programs.

> 
> We're done here. That header gets removed from kernel.h.

I told you I'm 100% fine with removing trace_printk.h from kernel.h if
you are OK with the Makefile solution I posted. Having a
TRACE_PRINTK_DEBUGGING config would not cause any issue for you or
others that don't care about it. I believe it's a proper compromise.

You seem to act like I'm arguing to keep trace_printk.h in kernel.h
after I came up with an alternate solution. Let me add back the
question you asked to the very first line of my reply that you went off
on:

> On Sat, 20 Jun 2026 15:39:25 -0700
> Linus Torvalds <torvalds@linux-foundation.org> wrote:
> 
> > Feel free to try to come up with such a patch.
> > 
> > But honestly, before you do, what is the *advantage* of such a thing?  
> 
> For debugging it is really useful.

You seem to think that comment was about why to keep trace_printk.h in
kernel.h and bother everyone that doesn't use it. No, you asked me what
the advantage of having the Makefile solution was, and that was my answer.

-- Steve

Re: [GIT PULL] tracing: Updates for 7.2

[Steven Rostedt]

On Sat, 20 Jun 2026 17:18:26 -0700
Linus Torvalds <torvalds@linux-foundation.org> wrote:

> But I personally think that debugging should be done by looking at and
> understanding the code. To me, "debugging using printk's" - fancy
> printk's or not - is a failure mode, and soemthing you should be
> quietly ashamed of, instead of using it as some kind of argument for
> making everybody elses kernel worse.

One last note on this. I don't think I ever used "debugging using
printk's" for a design bug. It's always ended up being something like an
off by one issue. That is, looking at the code can easily miss it.

-- Steve