* Re: [GIT PULL] tracing: Updates for 7.2
2026-06-19 14:35 ` Linus Torvalds
@ 2026-06-19 15:40 ` Sebastian Andrzej Siewior
2026-06-19 15:43 ` Linus Torvalds
2026-06-19 15:54 ` Steven Rostedt
` (2 subsequent siblings)
3 siblings, 1 reply; 30+ messages in thread
From: Sebastian Andrzej Siewior @ 2026-06-19 15:40 UTC (permalink / raw)
To: Linus Torvalds, John Ogness, Thomas Gleixner, Peter Zijlstra
Cc: Steven Rostedt, Yury Norov, LKML, Masami Hiramatsu,
Mathieu Desnoyers, Ao Sun, David Carlier, Karl Mehltretter,
Martin Kaiser, Pengpeng Hou, Qian-Yu Lin, Rik van Riel,
Rosen Penev, Shuvam Pandey, Vineeth Pillai, Yash Suthar, Yu Peng
On 2026-06-19 07:35:15 [-0700], Linus Torvalds wrote:
> On Fri, 19 Jun 2026 at 05:15, Steven Rostedt <rostedt@goodmis.org> wrote:
> >
> > Many developers use trace_printk() over printk().
>
> You're on some bad drugs, Steven.
>
> By "many developers" you clearly must have meant "almost nobody".
I do use it for debugging all the time. The alternative is printk but it
is "slower" if it throws everything to the UART which I prefer to limit
to BUG/WARN kind of messages. Also it does not record the context where
this comes from which is often helpful.
Afaik it is the only to debug printk.
> > Requiring people to add that include for every use case would
> > severely limit debugging of the kernel. Would you be OK if we did
> > the same for printk?
>
> Stop making inane and silly arguments that have nothing to do with
> reality.
>
> The two are not the same:
>
> $ git grep -wl printk | wc -l 3212 $ git grep -wl trace_printk | wc
> -l 50
>
> and making them sound remotely similar is just making you look dishonst.
>
> Look, the attached patch makes my defconfig build cleanly, and makes a
>
> $ touch include/linux/trace_printk.h
>
> cause my recompile to build just a handful of files. While touching
> linux/kernel.h causes thousands of files to be re-build.
>
> That's a whopping
>
> 6 files changed, 6 insertions(+), 1 deletion(-)
>
> to make my tree build without including <linux/trace_printk.h> everywhere.
>
> Now, that's _just_ my defconfig, and the full build clearly has a
> handful of other cases. From a quick look, mostly in the 'samples'
> directory.
>
> But it looks like a *handful* of other cases.
>
> Now, if the same was true for printk(), I'd happily split that into
> some other header too.
>
> But it isn't.
>
> Because we live in reality.
>
> Linus
Sebastian
^ permalink raw reply [flat|nested] 30+ messages in thread* Re: [GIT PULL] tracing: Updates for 7.2
2026-06-19 15:40 ` Sebastian Andrzej Siewior
@ 2026-06-19 15:43 ` Linus Torvalds
2026-06-19 18:30 ` Sebastian Andrzej Siewior
0 siblings, 1 reply; 30+ messages in thread
From: Linus Torvalds @ 2026-06-19 15:43 UTC (permalink / raw)
To: Sebastian Andrzej Siewior
Cc: John Ogness, Thomas Gleixner, Peter Zijlstra, Steven Rostedt,
Yury Norov, LKML, Masami Hiramatsu, Mathieu Desnoyers, Ao Sun,
David Carlier, Karl Mehltretter, Martin Kaiser, Pengpeng Hou,
Qian-Yu Lin, Rik van Riel, Rosen Penev, Shuvam Pandey,
Vineeth Pillai, Yash Suthar, Yu Peng
On Fri, 19 Jun 2026 at 08:40, Sebastian Andrzej Siewior
<bigeasy@linutronix.de> wrote:
>
> I do use it for debugging all the time. The alternative is printk but it
> is "slower" if it throws everything to the UART which I prefer to limit
> to BUG/WARN kind of messages. Also it does not record the context where
> this comes from which is often helpful.
Go ahead and use it. But don't make everybody else hurt just because
you do. If you add local hacks, add the local hack header.
Don't say "I'M SO IMPORTANT THAT I CAN'T BE BOTHERED, SO I'LL BOTHER
EVERYBODY ELSE BY MAKING THEIR BUILDS POINTLESSLY SLOWER"
Do you really want to be that person? Do you really want to make that argument?
Linus
^ permalink raw reply [flat|nested] 30+ messages in thread* Re: [GIT PULL] tracing: Updates for 7.2
2026-06-19 15:43 ` Linus Torvalds
@ 2026-06-19 18:30 ` Sebastian Andrzej Siewior
2026-06-19 19:07 ` Linus Torvalds
0 siblings, 1 reply; 30+ messages in thread
From: Sebastian Andrzej Siewior @ 2026-06-19 18:30 UTC (permalink / raw)
To: Linus Torvalds
Cc: John Ogness, Thomas Gleixner, Peter Zijlstra, Steven Rostedt,
Yury Norov, LKML, Masami Hiramatsu, Mathieu Desnoyers, Ao Sun,
David Carlier, Karl Mehltretter, Martin Kaiser, Pengpeng Hou,
Qian-Yu Lin, Rik van Riel, Rosen Penev, Shuvam Pandey,
Vineeth Pillai, Yash Suthar, Yu Peng
On 2026-06-19 08:43:14 [-0700], Linus Torvalds wrote:
> Go ahead and use it. But don't make everybody else hurt just because
> you do. If you add local hacks, add the local hack header.
>
> Don't say "I'M SO IMPORTANT THAT I CAN'T BE BOTHERED, SO I'LL BOTHER
> EVERYBODY ELSE BY MAKING THEIR BUILDS POINTLESSLY SLOWER"
>
> Do you really want to be that person? Do you really want to make that argument?
Well, no.
This started as "Steven you must be crazy to think someone is using
trace_printk". That is not the case. There should be no in-tree users as
nothing should use it - there are trace events for regular users.
So if this is just about splitting trace_printk away from kernel.h,
okay. Even container_of() moved out and is included explicitly which is
used a lot. If this next so be it.
> Linus
Sebastian
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [GIT PULL] tracing: Updates for 7.2
2026-06-19 18:30 ` Sebastian Andrzej Siewior
@ 2026-06-19 19:07 ` Linus Torvalds
2026-06-19 20:28 ` Thomas Gleixner
2026-06-19 22:28 ` Linus Torvalds
0 siblings, 2 replies; 30+ messages in thread
From: Linus Torvalds @ 2026-06-19 19:07 UTC (permalink / raw)
To: Sebastian Andrzej Siewior
Cc: John Ogness, Thomas Gleixner, Peter Zijlstra, Steven Rostedt,
Yury Norov, LKML, Masami Hiramatsu, Mathieu Desnoyers, Ao Sun,
David Carlier, Karl Mehltretter, Martin Kaiser, Pengpeng Hou,
Qian-Yu Lin, Rik van Riel, Rosen Penev, Shuvam Pandey,
Vineeth Pillai, Yash Suthar, Yu Peng
On Fri, 19 Jun 2026 at 11:30, Sebastian Andrzej Siewior
<bigeasy@linutronix.de> wrote:
>
> So if this is just about splitting trace_printk away from kernel.h,
> okay. Even container_of() moved out and is included explicitly which is
> used a lot. If this next so be it.
Absolutely.
We should aim to never have the insanity that is a typical C++ build
system, where you have headers that include the world because of how
the C++ method and object models work.
That way lies insanity. It's not only "random small header changes
rebuild everything", it's also simply "everything is slower because
the compiler has to parse way too much irrelevant garbage".
Now, we're not exactly doing great in this area, I'm afraid. To pick a
random small core kernel file, we have this kind of disgusting
situation:
$ make kernel/pid.i
$ wc -c kernel/pid.c kernel/pid.i
24460 kernel/pid.c
5271779 kernel/pid.i
ie a relatively tiny sub-1kloc file balloons by a factor of 215x and
becomes 5MB worth of noise to parse for the compiler because of header
inclusion.
And the build cost is pretty much directly related to that kind of
expansion (admittedly linking is bad too, mainly because our linking
phases tend to be more serialized than building most C files)
Now, I picked a small file on purpose and this is with 'allmodconfig'
which makes a bad situation even worse by about 20%. That small file
expands to "only" 4.2MB in a more sedate config.
But even our *biggest* C file in the kernel subdirectory sees a 20x
expansion due to pre-processing.
And yes, <kernel/trace_printk.c> is part of that, but certainly not a
very big part - it's only 6kB and 202 lines in that mess.
A pittance. Hardly noticeable.
But "it's a small part of the problem" doesn't mean that it shouldn't
be fixed when it looks like it's *trivial* to fix.
Now, the biggest single cause of horrible expansion to be some of our
more complex nested macros. So it's not always about the size of the
header and how many other headers it includes, we have some other
structural issues that make it worse.
We had very real build performance issues due to the deep nesting of
the min/max macros to the point that they even caused build failures
on machines with limited memory.
But we mostly fixed the worst part of that mess, and the type checking
in those macros at least add real value to normal code.
But in general, if the question is "should we split this header file
up if it's straightforward and easy", the answer is *ALWAYS* a
resounding "YES".
Most of the time it's just a pain to unravel all the type
dependencies. This case look strivial compared to some we've done.
One of the things I at one point hoped sparse would do was a "you're
including this header for no good reason" tooling.
But it's non-trivial to do, particularly with lots of different
architectures that aren't very consistent in where they declare or use
things.
Linus
^ permalink raw reply [flat|nested] 30+ messages in thread* Re: [GIT PULL] tracing: Updates for 7.2
2026-06-19 19:07 ` Linus Torvalds
@ 2026-06-19 20:28 ` Thomas Gleixner
2026-06-19 20:55 ` Linus Torvalds
2026-06-19 22:28 ` Linus Torvalds
1 sibling, 1 reply; 30+ messages in thread
From: Thomas Gleixner @ 2026-06-19 20:28 UTC (permalink / raw)
To: Linus Torvalds, Sebastian Andrzej Siewior
Cc: John Ogness, Peter Zijlstra, Steven Rostedt, Yury Norov, LKML,
Masami Hiramatsu, Mathieu Desnoyers, Ao Sun, David Carlier,
Karl Mehltretter, Martin Kaiser, Pengpeng Hou, Qian-Yu Lin,
Rik van Riel, Rosen Penev, Shuvam Pandey, Vineeth Pillai,
Yash Suthar, Yu Peng
On Fri, Jun 19 2026 at 12:07, Linus Torvalds wrote:
> On Fri, 19 Jun 2026 at 11:30, Sebastian Andrzej Siewior
> <bigeasy@linutronix.de> wrote:
> But in general, if the question is "should we split this header file
> up if it's straightforward and easy", the answer is *ALWAYS* a
> resounding "YES".
Full ACK.
> Most of the time it's just a pain to unravel all the type
> dependencies. This case look strivial compared to some we've done.
>
> One of the things I at one point hoped sparse would do was a "you're
> including this header for no good reason" tooling.
>
> But it's non-trivial to do, particularly with lots of different
> architectures that aren't very consistent in where they declare or use
> things.
That's only one part of the overall problem.
People are really careless when they add new things. It took me
literally weeks to handle all the 0-day compile fallout when I tried to
disentangle the __percpu mess so that a header using a __percpu
annotation in a struct declaration is not required to include the world
and running into recursive header hell.
See c06cd66387da ("percpu: Sanitize __percpu_qual include hell")
I'm still fighting to resolve the last remaining issue to get rid of the
asm/xtime.h dependency in some other cleanup project. That header is so
deeply ingrained in some of our header chains for the very wrong
reasons.
I've tried to utilize AI for analyzing it, but that ended up in an even
bigger disaster.
One thing I'm seeing is that the kernel is patently bad in separating
data type declarations from actual APIs, where the APIs usually just
need a forward declaration of the pointer type. Ditto for struct
declarations with pointer types.
After doing quite some of these disentanglements manually and trying to
utilize all kinds of tooling, I still have no good answer to this
unfortunately.
Thanks,
tglx
^ permalink raw reply [flat|nested] 30+ messages in thread* Re: [GIT PULL] tracing: Updates for 7.2
2026-06-19 20:28 ` Thomas Gleixner
@ 2026-06-19 20:55 ` Linus Torvalds
2026-06-20 9:22 ` Willy Tarreau
0 siblings, 1 reply; 30+ messages in thread
From: Linus Torvalds @ 2026-06-19 20:55 UTC (permalink / raw)
To: Thomas Gleixner
Cc: Sebastian Andrzej Siewior, John Ogness, Peter Zijlstra,
Steven Rostedt, Yury Norov, LKML, Masami Hiramatsu,
Mathieu Desnoyers, Ao Sun, David Carlier, Karl Mehltretter,
Martin Kaiser, Pengpeng Hou, Qian-Yu Lin, Rik van Riel,
Rosen Penev, Shuvam Pandey, Vineeth Pillai, Yash Suthar, Yu Peng
On Fri, 19 Jun 2026 at 13:28, Thomas Gleixner <tglx@kernel.org> wrote:
>
> One thing I'm seeing is that the kernel is patently bad in separating
> data type declarations from actual APIs, where the APIs usually just
> need a forward declaration of the pointer type. Ditto for struct
> declarations with pointer types.
We tend to do a lot of inline functions, and that often makes it
impossible to do the simple "just forward-declare the type".
And often there are fairly good reasons for the inline functions, in
that they tend to be much safer than macros, both for type checking
and for the whole "use argument once" reason (they also avoid compiler
warnings when the argument _isn't_ used).
But yes, it's often really nasty to try to split things up, even when
it *looks* straightforward.
Linus
^ permalink raw reply [flat|nested] 30+ messages in thread* Re: [GIT PULL] tracing: Updates for 7.2
2026-06-19 20:55 ` Linus Torvalds
@ 2026-06-20 9:22 ` Willy Tarreau
0 siblings, 0 replies; 30+ messages in thread
From: Willy Tarreau @ 2026-06-20 9:22 UTC (permalink / raw)
To: Linus Torvalds
Cc: Thomas Gleixner, Sebastian Andrzej Siewior, John Ogness,
Peter Zijlstra, Steven Rostedt, Yury Norov, LKML,
Masami Hiramatsu, Mathieu Desnoyers, Ao Sun, David Carlier,
Karl Mehltretter, Martin Kaiser, Pengpeng Hou, Qian-Yu Lin,
Rik van Riel, Rosen Penev, Shuvam Pandey, Vineeth Pillai,
Yash Suthar, Yu Peng
On Fri, Jun 19, 2026 at 01:55:15PM -0700, Linus Torvalds wrote:
> On Fri, 19 Jun 2026 at 13:28, Thomas Gleixner <tglx@kernel.org> wrote:
> >
> > One thing I'm seeing is that the kernel is patently bad in separating
> > data type declarations from actual APIs, where the APIs usually just
> > need a forward declaration of the pointer type. Ditto for struct
> > declarations with pointer types.
>
> We tend to do a lot of inline functions, and that often makes it
> impossible to do the simple "just forward-declare the type".
>
> And often there are fairly good reasons for the inline functions, in
> that they tend to be much safer than macros, both for type checking
> and for the whole "use argument once" reason (they also avoid compiler
> warnings when the argument _isn't_ used).
In haproxy we've had a hard time dealing with circular dependencies and
build time issue for a while and a few years ago we found what looks
like a mostly optimal arrangement. Each include file exists in two
versions:
- one suffixed with "-t" which contains only type declarations
(#define as well as enums, structs etc)
- the main one without "-t" that includes the first one and contains
declarations (extern variables & functions as well as inline)
And all this with a strict rule: no -t file may ever include any of the
other ones. It now works pretty well, and we could significantly reduce
the depth of includes and no longer need to resolve broken inline-to-inline
references. And we discovered that many C files don't even need the main
file and only need types definitions, further reducing build time. Also
we figured that certain -t files only included another one for a single
struct definition that appeared as a pointer, so these were sometimes
replaced with just a forward-declaration before use and that was done.
We made mistakes too, most of our macros that replaced inline functions
tend to be in the main files instead of the -t one as the rule implies,
encouraging to place further macro definitions at the wrong place. But
moving them is no big deal.
If I would change something, it would probably be around long macro
definitions (e.g. series of flags) that are often not needed by other
-t files and only by functions. It's just difficult for me to express
that distinction in a way that's clear enough for eveyrone to respect,
but this could definitely further reduce the preprocessing time of the
types-only "-t" files.
Willy
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [GIT PULL] tracing: Updates for 7.2
2026-06-19 19:07 ` Linus Torvalds
2026-06-19 20:28 ` Thomas Gleixner
@ 2026-06-19 22:28 ` Linus Torvalds
1 sibling, 0 replies; 30+ messages in thread
From: Linus Torvalds @ 2026-06-19 22:28 UTC (permalink / raw)
To: Sebastian Andrzej Siewior, Luis Chamberlain, Petr Pavlu,
Daniel Gomez, Sami Tolvanen
Cc: John Ogness, Thomas Gleixner, Peter Zijlstra, Steven Rostedt,
Yury Norov, LKML, Masami Hiramatsu, Mathieu Desnoyers, Ao Sun,
David Carlier, Karl Mehltretter, Martin Kaiser, Pengpeng Hou,
Qian-Yu Lin, Rik van Riel, Rosen Penev, Shuvam Pandey,
Vineeth Pillai, Yash Suthar, Yu Peng
On Fri, 19 Jun 2026 at 12:07, Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
> Now, we're not exactly doing great in this area, I'm afraid. To pick a
> random small core kernel file, we have this kind of disgusting
> situation:
Actually, we have much worse cases.
Doing a "make allmodconfig" will do all those '*.mod.c" files, which
are tiny generated files that just have some mod-info structure etc
for linking into the module.
They're like ten lines of code. MODULE_INFO() and a 'struct module'
that gets linked in with the actual code. Auto-generated by
scropts/mod/modpost.c.
And then they have
#include <linux/module.h>
#include <linux/export-internal.h>
#include <linux/compiler.h>
and those three lines of include make them balloon into 3MB of code,
so building those *tiny* things takes forever.
(Where "forever" is obviously relative - but it's disgustingly long
for something that should be instant).
In fact, just a single
#include <linux/module.h>
and absolutely nothing else, will give you 2.8MB of output.
(In contrast, the two other headers we include - export-internal.h and
compiler.h - result in 13kB of output. Yesh, it's 550 lines of pretty
random stuff, but it's not really all that bad. It's mostly just
really core type declarations, so it's all quite understandable)
That module.h header situation is beyond disgusting - and yes, it's
very much noticeable in my build time. Adding some module maintainer
people, because this is bad, and I think it needs to be fixed.
Linus
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [GIT PULL] tracing: Updates for 7.2
2026-06-19 14:35 ` Linus Torvalds
2026-06-19 15:40 ` Sebastian Andrzej Siewior
@ 2026-06-19 15:54 ` Steven Rostedt
2026-06-19 16:29 ` Linus Torvalds
2026-06-21 20:26 ` Agatha Isabelle Moreira
2026-06-20 20:24 ` Julia Lawall
2026-06-20 22:19 ` Steven Rostedt
3 siblings, 2 replies; 30+ messages in thread
From: Steven Rostedt @ 2026-06-19 15:54 UTC (permalink / raw)
To: Linus Torvalds
Cc: Yury Norov, LKML, Masami Hiramatsu, Mathieu Desnoyers, Ao Sun,
David Carlier, Karl Mehltretter, Martin Kaiser, Pengpeng Hou,
Qian-Yu Lin, Rik van Riel, Rosen Penev, Shuvam Pandey,
Vineeth Pillai, Yash Suthar, Yu Peng
On Fri, 19 Jun 2026 07:35:15 -0700
Linus Torvalds <torvalds@linux-foundation.org> wrote:
>
> Now, if the same was true for printk(), I'd happily split that into
> some other header too.
>
> But it isn't.
>
> Because we live in reality.
I'm talking about using it for debugging and not to print info. In
fact, I try hard to *prevent* people keeping trace_printk() *in* the
kernel code. If you add it, it triggers that nasty banner.
The point of having it in kernel.h is for debugging purposes. That's
probably why printk.h is also there too. Many others use the "shotgun"
approach where they add trace_printk()s all over the place to track down
a bug.
Your patch now requires developers from remembering to add a
#include <linux/trace_printk.h> in every file the touch. Or they may
have to touch kernel.h triggering a full build as well. It moves burden
from people who build the kernel and have to wait a little longer to
developers that now need to remember to add includes that they didn't
need to before!
trace_printk() hardly ever changes, so it should not trigger these
builds on changing kernels. The only reason it was changed this time
was because the compilers now have more warnings about printk formats
that required annotation to trace_printk(). As well as a clean up to
remove a variable from a macro.
When I said people use trace_printk() over printk() I meant I know
several (including myself) the seldom ever use printk for debugging the
kernel. If I need to see what's happening, I use trace_printk().
Because it can be used in any context without worrying about slowing
down the system, and less likely to cause heissenbugs.
-- Steve
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [GIT PULL] tracing: Updates for 7.2
2026-06-19 15:54 ` Steven Rostedt
@ 2026-06-19 16:29 ` Linus Torvalds
2026-06-21 20:26 ` Agatha Isabelle Moreira
1 sibling, 0 replies; 30+ messages in thread
From: Linus Torvalds @ 2026-06-19 16:29 UTC (permalink / raw)
To: Steven Rostedt
Cc: Yury Norov, LKML, Masami Hiramatsu, Mathieu Desnoyers, Ao Sun,
David Carlier, Karl Mehltretter, Martin Kaiser, Pengpeng Hou,
Qian-Yu Lin, Rik van Riel, Rosen Penev, Shuvam Pandey,
Vineeth Pillai, Yash Suthar, Yu Peng
On Fri, 19 Jun 2026 at 08:54, Steven Rostedt <rostedt@goodmis.org> wrote:
>
> The point of having it in kernel.h is for debugging purposes. That's
> probably why printk.h is also there too.
No, Steven.
printk.h is there because EVERYTHING USES IT.
trace_printk.h is getting removed from there because ALMOST NOTHING USES IT.
See the subtle but important difference? I'm trying to highlight it here.
Linus
^ permalink raw reply [flat|nested] 30+ messages in thread* Re: [GIT PULL] tracing: Updates for 7.2
2026-06-19 15:54 ` Steven Rostedt
2026-06-19 16:29 ` Linus Torvalds
@ 2026-06-21 20:26 ` Agatha Isabelle Moreira
2026-06-21 20:51 ` Linus Torvalds
1 sibling, 1 reply; 30+ messages in thread
From: Agatha Isabelle Moreira @ 2026-06-21 20:26 UTC (permalink / raw)
To: Steven Rostedt
Cc: Linus Torvalds, Yury Norov, LKML, Masami Hiramatsu,
Mathieu Desnoyers, Ao Sun, David Carlier, Karl Mehltretter,
Martin Kaiser, Pengpeng Hou, Qian-Yu Lin, Rik van Riel,
Rosen Penev, Shuvam Pandey, Vineeth Pillai, Yash Suthar, Yu Peng
On Fri, Jun 19, 2026 at 11:54:27AM -0400, Steven Rostedt wrote:
> On Fri, 19 Jun 2026 07:35:15 -0700
> Linus Torvalds <torvalds@linux-foundation.org> wrote:
>
> >
> > Now, if the same was true for printk(), I'd happily split that into
> > some other header too.
> >
> > But it isn't.
> >
> > Because we live in reality.
>
>
> Your patch now requires developers from remembering to add a
> #include <linux/trace_printk.h> in every file the touch. Or they may
> have to touch kernel.h triggering a full build as well. It moves burden
> from people who build the kernel and have to wait a little longer to
> developers that now need to remember to add includes that they didn't
> need to before!
Not only *add* a #include <linux/trace_printk.h>, but also remember to
not commit them later. If people eventually forget (of course nobody
ever forgets to do something in programming) and commit it and it goes
unnoticed in the review, we might end up with it included in several
places again, defeating the very purpose of this patch.
That would me easily solved with an #ifdef that would not only reduce
the #include bloat but also keep debugging comfortable.
--
Agatha Isabelle Moreira
Systems Engineer | C
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [GIT PULL] tracing: Updates for 7.2
2026-06-21 20:26 ` Agatha Isabelle Moreira
@ 2026-06-21 20:51 ` Linus Torvalds
2026-06-21 21:19 ` Steven Rostedt
0 siblings, 1 reply; 30+ messages in thread
From: Linus Torvalds @ 2026-06-21 20:51 UTC (permalink / raw)
To: Agatha Isabelle Moreira
Cc: Steven Rostedt, Yury Norov, LKML, Masami Hiramatsu,
Mathieu Desnoyers, Ao Sun, David Carlier, Karl Mehltretter,
Martin Kaiser, Pengpeng Hou, Qian-Yu Lin, Rik van Riel,
Rosen Penev, Shuvam Pandey, Vineeth Pillai, Yash Suthar, Yu Peng
On Sun, 21 Jun 2026 at 13:26, Agatha Isabelle Moreira <code@agatha.dev> wrote:
>
> Not only *add* a #include <linux/trace_printk.h>, but also remember to
> not commit them later.
Stop this idiocy already.
You guys are literally making sh*t up now.
First you say that it's for a debugging patch that adds the
trace_printk() locally and never makes it into the tree, which
explains why nobody else wants to see that thing.
And now you say the problem is that you have to remember to not commit
the header you added.
JUST LIKE THAT WHOLE trace_printk() that was apparently all just local
and never committed either.
So in your magical world, adding the trace_printk() isn't a problem.
Spending effort debugging things that are hard to reproduce isn't a
problem. And remembering to not commit said debug code is apparently
also a non-issue.
But that extra #include is suddenly a huge problem both to add in the
first place - oh woe, how it hurts to add that one line to make it
possible to do those week-long debug sessions - and then to not
commit.
This thread has gone from just plain dumb to ridiculous, and I have to
mute this conversation in order to not get dumber by association.
Linus
^ permalink raw reply [flat|nested] 30+ messages in thread* Re: [GIT PULL] tracing: Updates for 7.2
2026-06-21 20:51 ` Linus Torvalds
@ 2026-06-21 21:19 ` Steven Rostedt
2026-06-21 23:16 ` Linus Torvalds
0 siblings, 1 reply; 30+ messages in thread
From: Steven Rostedt @ 2026-06-21 21:19 UTC (permalink / raw)
To: Linus Torvalds
Cc: Agatha Isabelle Moreira, Yury Norov, LKML, Masami Hiramatsu,
Mathieu Desnoyers, Ao Sun, David Carlier, Karl Mehltretter,
Martin Kaiser, Pengpeng Hou, Qian-Yu Lin, Rik van Riel,
Rosen Penev, Shuvam Pandey, Vineeth Pillai, Yash Suthar, Yu Peng
On Sun, 21 Jun 2026 13:51:30 -0700
Linus Torvalds <torvalds@linux-foundation.org> wrote:
>
> First you say that it's for a debugging patch that adds the
> trace_printk() locally and never makes it into the tree, which
> explains why nobody else wants to see that thing.
Who is that nobody? I know you don't use trace_printk() at all, but I
would love to have a survey of *all* kernel developers to find out the
percentage that find it useful.
You seem to be of the mind set "I don't think it's useful, so nobody
should think it's useful". Just like you did with Link tags that point
to patches.
>
> And now you say the problem is that you have to remember to not commit
> the header you added.
It's not the main problem, it's just one of many things that can
happen. But yeah, it's not a big deal.
>
> JUST LIKE THAT WHOLE trace_printk() that was apparently all just local
> and never committed either.
There's been cases of trace_printk() ending up accidentally in
production. The reason to not allow it in production is because it
writes to the main tracing buffer and it's always on. If it was
allowed, every subsystem would use it and then when you want to debug
something you have to fight the noise of everyone else's trace_printk()
when you want to see yours. It would also make trace events useless.
Thus, a common workflow (and this is what others have told me they do,
I personally don't do this) is to use trace_printk() to find out what
is useful to track for debugging. Then, if important data was traced,
they would turn it into a trace_event. Trace events are basically
trace_printk()s but in a proper form.
On my Fedora laptop:
$ sudo cat /sys/kernel/tracing/available_events | wc -l
2719
Thus trace_events are very commonly used, which are just
trace_printk()s that you can individually enable and disable.
>
> So in your magical world, adding the trace_printk() isn't a problem.
> Spending effort debugging things that are hard to reproduce isn't a
> problem. And remembering to not commit said debug code is apparently
> also a non-issue.
>
> But that extra #include is suddenly a huge problem both to add in the
> first place - oh woe, how it hurts to add that one line to make it
> possible to do those week-long debug sessions - and then to not
> commit.
It is just frustrating when you are focusing on debugging something,
and you go to compile and it errors with "prototype not found". It's
just one more thing added to frustration of fixing code that doesn't
work. I thought we want to make developers lives easier. But since
*you* don't use it, you think it's not an issue.
>
> This thread has gone from just plain dumb to ridiculous, and I have to
> mute this conversation in order to not get dumber by association.
I 100% agree with getting rid of things in kernel.h. But the little
Makefile change and the config can keep people using trace_printk()
like they have been doing since 2009. Would you at least be OK with
that? It shouldn't make any difference to your workflow, but would make
things easier for others, even if you don't think it would. At least
humor us ;-)
-- Steve
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [GIT PULL] tracing: Updates for 7.2
2026-06-21 21:19 ` Steven Rostedt
@ 2026-06-21 23:16 ` Linus Torvalds
0 siblings, 0 replies; 30+ messages in thread
From: Linus Torvalds @ 2026-06-21 23:16 UTC (permalink / raw)
To: Steven Rostedt
Cc: Agatha Isabelle Moreira, Yury Norov, LKML, Masami Hiramatsu,
Mathieu Desnoyers, Ao Sun, David Carlier, Karl Mehltretter,
Martin Kaiser, Pengpeng Hou, Qian-Yu Lin, Rik van Riel,
Rosen Penev, Shuvam Pandey, Vineeth Pillai, Yash Suthar, Yu Peng
On Sun, 21 Jun 2026 at 14:19, Steven Rostedt <rostedt@goodmis.org> wrote:
>
> I 100% agree with getting rid of things in kernel.h. But the little
> Makefile change and the config
No. We're not adding stupid config options for this.
Christ. Why do you want to make the tree worse? Too many inscrutable
Kconfig options is another one of my pet peeves. Stop making life
harder for others for some made-up convenience for you.
Linus
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [GIT PULL] tracing: Updates for 7.2
2026-06-19 14:35 ` Linus Torvalds
2026-06-19 15:40 ` Sebastian Andrzej Siewior
2026-06-19 15:54 ` Steven Rostedt
@ 2026-06-20 20:24 ` Julia Lawall
2026-06-21 20:34 ` Agatha Isabelle Moreira
2026-06-20 22:19 ` Steven Rostedt
3 siblings, 1 reply; 30+ messages in thread
From: Julia Lawall @ 2026-06-20 20:24 UTC (permalink / raw)
To: Linus Torvalds
Cc: Steven Rostedt, Yury Norov, LKML, Masami Hiramatsu,
Mathieu Desnoyers, Ao Sun, David Carlier, Karl Mehltretter,
Martin Kaiser, Pengpeng Hou, Qian-Yu Lin, Rik van Riel,
Rosen Penev, Shuvam Pandey, Vineeth Pillai, Yash Suthar, Yu Peng
On Fri, 19 Jun 2026, Linus Torvalds wrote:
> On Fri, 19 Jun 2026 at 05:15, Steven Rostedt <rostedt@goodmis.org> wrote:
> >
> > Many developers use trace_printk() over printk().
>
> You're on some bad drugs, Steven.
>
> By "many developers" you clearly must have meant "almost nobody".
I use trace_printk on a daily basis. Tracepoints are useful to get an
overview of what is going on, and then trace_printk is useful for
interleaving that with information relevant to the specific problem at
hand.
julia
>
> > Requiring people to
> > add that include for every use case would severely limit debugging of
> > the kernel. Would you be OK if we did the same for printk?
>
> Stop making inane and silly arguments that have nothing to do with reality.
>
> The two are not the same:
>
> $ git grep -wl printk | wc -l
> 3212
> $ git grep -wl trace_printk | wc -l
> 50
>
> and making them sound remotely similar is just making you look dishonst.
>
> Look, the attached patch makes my defconfig build cleanly, and makes a
>
> $ touch include/linux/trace_printk.h
>
> cause my recompile to build just a handful of files. While touching
> linux/kernel.h causes thousands of files to be re-build.
>
> That's a whopping
>
> 6 files changed, 6 insertions(+), 1 deletion(-)
>
> to make my tree build without including <linux/trace_printk.h> everywhere.
>
> Now, that's _just_ my defconfig, and the full build clearly has a
> handful of other cases. From a quick look, mostly in the 'samples'
> directory.
>
> But it looks like a *handful* of other cases.
>
> Now, if the same was true for printk(), I'd happily split that into
> some other header too.
>
> But it isn't.
>
> Because we live in reality.
>
> Linus
>
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [GIT PULL] tracing: Updates for 7.2
2026-06-20 20:24 ` Julia Lawall
@ 2026-06-21 20:34 ` Agatha Isabelle Moreira
0 siblings, 0 replies; 30+ messages in thread
From: Agatha Isabelle Moreira @ 2026-06-21 20:34 UTC (permalink / raw)
To: Julia Lawall
Cc: Linus Torvalds, Steven Rostedt, Yury Norov, LKML,
Masami Hiramatsu, Mathieu Desnoyers, Ao Sun, David Carlier,
Karl Mehltretter, Martin Kaiser, Pengpeng Hou, Qian-Yu Lin,
Rik van Riel, Rosen Penev, Shuvam Pandey, Vineeth Pillai,
Yash Suthar, Yu Peng
On Sat, Jun 20, 2026 at 10:24:14PM +0200, Julia Lawall wrote:
>
>
> On Fri, 19 Jun 2026, Linus Torvalds wrote:
>
> > On Fri, 19 Jun 2026 at 05:15, Steven Rostedt <rostedt@goodmis.org> wrote:
> > >
> > > Many developers use trace_printk() over printk().
> >
> > You're on some bad drugs, Steven.
> >
> > By "many developers" you clearly must have meant "almost nobody".
>
> I use trace_printk on a daily basis. Tracepoints are useful to get an
> overview of what is going on, and then trace_printk is useful for
> interleaving that with information relevant to the specific problem at
> hand.
I think the problem is that "use" here is being counted by greping
lines upstream. But there's a missed spot on this ount, most of
trace_printk use on a daily basis is not commited.
--
Agatha Isabelle Moreira
Systems Engineer | C
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [GIT PULL] tracing: Updates for 7.2
2026-06-19 14:35 ` Linus Torvalds
` (2 preceding siblings ...)
2026-06-20 20:24 ` Julia Lawall
@ 2026-06-20 22:19 ` Steven Rostedt
2026-06-20 22:39 ` Linus Torvalds
3 siblings, 1 reply; 30+ messages in thread
From: Steven Rostedt @ 2026-06-20 22:19 UTC (permalink / raw)
To: Linus Torvalds
Cc: Yury Norov, LKML, Masami Hiramatsu, Mathieu Desnoyers, Ao Sun,
David Carlier, Karl Mehltretter, Martin Kaiser, Pengpeng Hou,
Qian-Yu Lin, Rik van Riel, Rosen Penev, Shuvam Pandey,
Vineeth Pillai, Yash Suthar, Yu Peng, Sebastian Andrzej Siewior,
Thomas Gleixner, Peter Zijlstra
On Fri, 19 Jun 2026 07:35:15 -0700
Linus Torvalds <torvalds@linux-foundation.org> wrote:
> The two are not the same:
>
> $ git grep -wl printk | wc -l
> 3212
> $ git grep -wl trace_printk | wc -l
> 50
>
> and making them sound remotely similar is just making you look dishonst.
>
> Look, the attached patch makes my defconfig build cleanly, and makes a
>
> $ touch include/linux/trace_printk.h
>
> cause my recompile to build just a handful of files. While touching
> linux/kernel.h causes thousands of files to be re-build.
>
> That's a whopping
>
> 6 files changed, 6 insertions(+), 1 deletion(-)
>
> to make my tree build without including <linux/trace_printk.h> everywhere.
>
> Now, that's _just_ my defconfig, and the full build clearly has a
> handful of other cases. From a quick look, mostly in the 'samples'
> directory.
>
> But it looks like a *handful* of other cases.
>
> Now, if the same was true for printk(), I'd happily split that into
> some other header too.
I totally agree that trace_printk.h should not be added to kernel.h for
debugging if printk.h is also not included for that purpose. But you
are stating that it is included because everything "uses" it. But
really, if we are worried about getting rid of adding headers that are
not used, I think it would be a worth while exercise to remove printk.h
from kernel.h to have it included only where it is used.
That said, I'm also happy to get rid of trace_printk.h from kernel.h if
there's an easy way to have it included for debugging purposes.
Are you OK with the following approach? Basically, it adds
"CONFIG_TRACE_PRINTK_DEBUGGING" config to be set by those that want to
use it for debugging. When set, it will add to KBUILD_CFLAGS:
-include $(srctree)/include/linux/trace_printk.h
which is basically the same as having it in kernel.h but without adding
the "burden" to those that don't want it (like yourself).
This would allow the best of both words:
diff --git a/Makefile b/Makefile
index d33a7cadd237..84236890a806 100644
--- a/Makefile
+++ b/Makefile
@@ -839,6 +839,10 @@ ifdef CONFIG_FUNCTION_TRACER
CC_FLAGS_FTRACE := -pg
endif
+ifdef CONFIG_TRACE_PRINTK_DEBUGGING
+ LINUXINCLUDE += -include $(srctree)/include/linux/trace_printk.h
+endif
+
ifdef CONFIG_TRACEPOINTS
# To check for unused tracepoints (tracepoints that are defined but never
# called), run with:
diff --git a/include/linux/trace_printk.h b/include/linux/trace_printk.h
index 2670ec7f4262..3fe5fc5c6a2a 100644
--- a/include/linux/trace_printk.h
+++ b/include/linux/trace_printk.h
@@ -1,11 +1,12 @@
/* SPDX-License-Identifier: GPL-2.0 */
#ifndef _LINUX_TRACE_PRINTK_H
#define _LINUX_TRACE_PRINTK_H
+#if !defined(__ASSEMBLY__) && !defined(__GENKSYMS__) && !defined(BUILD_VDSO)
-#include <linux/compiler_attributes.h>
#include <linux/instruction_pointer.h>
#include <linux/stddef.h>
#include <linux/stringify.h>
+#include <linux/stdarg.h>
/*
* General tracing related utility functions - trace_printk(),
@@ -199,5 +200,5 @@ ftrace_vprintk(const char *fmt, va_list ap)
}
static inline void ftrace_dump(enum ftrace_dump_mode oops_dump_mode) { }
#endif /* CONFIG_TRACING */
-
+#endif /* !defined(__ASSEMBLY__) && !defined(__GENKSYMS__) */
#endif
diff --git a/kernel/trace/Kconfig b/kernel/trace/Kconfig
index e130da35808f..e305b27ca029 100644
--- a/kernel/trace/Kconfig
+++ b/kernel/trace/Kconfig
@@ -210,6 +210,16 @@ menuconfig FTRACE
if FTRACE
+config TRACE_PRINTK_DEBUGGING
+ bool "Debug with trace_printk()"
+ help
+ If you need to debug with trace_printk(), instead of adding
+ include <linux/trace_printk.h> to every file you add a trace_printk
+ to, select this option and it will add trace_printk.h to all code
+ to allow tracing with trace_printk() with.
+
+ If in doubt, select N
+
config TRACEFS_AUTOMOUNT_DEPRECATED
bool "Automount tracefs on debugfs [DEPRECATED]"
depends on TRACING
-- Steve
^ permalink raw reply related [flat|nested] 30+ messages in thread* Re: [GIT PULL] tracing: Updates for 7.2
2026-06-20 22:19 ` Steven Rostedt
@ 2026-06-20 22:39 ` Linus Torvalds
2026-06-20 23:43 ` Steven Rostedt
0 siblings, 1 reply; 30+ messages in thread
From: Linus Torvalds @ 2026-06-20 22:39 UTC (permalink / raw)
To: Steven Rostedt
Cc: Yury Norov, LKML, Masami Hiramatsu, Mathieu Desnoyers, Ao Sun,
David Carlier, Karl Mehltretter, Martin Kaiser, Pengpeng Hou,
Qian-Yu Lin, Rik van Riel, Rosen Penev, Shuvam Pandey,
Vineeth Pillai, Yash Suthar, Yu Peng, Sebastian Andrzej Siewior,
Thomas Gleixner, Peter Zijlstra
On Sat, 20 Jun 2026 at 15:20, Steven Rostedt <rostedt@goodmis.org> wrote:
>
> I totally agree that trace_printk.h should not be added to kernel.h for
> debugging if printk.h is also not included for that purpose. But you
> are stating that it is included because everything "uses" it. But
> really, if we are worried about getting rid of adding headers that are
> not used, I think it would be a worth while exercise to remove printk.h
> from kernel.h to have it included only where it is used.
Feel free to try to come up with such a patch.
But honestly, before you do, what is the *advantage* of such a thing?
Because you seem to think that "trace_printk()" and "printk()" are the same.
They really aren't.
You don't seem to be willing or able to acknowledge that one is very
common, and the other is not.
If I did my quick grep and math right, we have this situation:
'printk()' and the basic 'pr_err()/pr_warn()' calls exported by
<linux/printk.h> exist in 19.2% of all C files in the kernel (7104
files):
git grep -E -wl '(printk)|(pr_err)|(pr_warn)' -- '*.c' | wc -l
While 'trace_printk()' exists in 0.054% of all C files in the kernel
(twenty files total):
git grep -E -wl '(trace_printk)' -- '*.c' | wc -l
and I think you'll get a few more hits if you look for 'tracing_on'
etc, so maybe it will hit a tenth of a percent overall.
But from a quick look, even *that* is actually mostly sample code and
tools/perf/, not in actual kernel code.
So be honest now: one of them makes sense to have as a pretty standard
baseline for kernel programming.
The other DOES NOT.
If you want to find other header files to split out, by all means, be my guest.
As mentioned, the more headers we can split off, the better.
But when one header file is used by about a fifth of all C files,
that's a pretty important header, wouldn't you say? Certainly not the
*first* header you should try to split off.
So I think you should look for *other* headers to split off, instead
of going "ok, printk and trace_printk are the same".
Because they damn well are not even remotely the same, and when you
try to equate the two, you just look incompetent.
Linus
^ permalink raw reply [flat|nested] 30+ messages in thread* Re: [GIT PULL] tracing: Updates for 7.2
2026-06-20 22:39 ` Linus Torvalds
@ 2026-06-20 23:43 ` Steven Rostedt
2026-06-21 0:18 ` Linus Torvalds
0 siblings, 1 reply; 30+ messages in thread
From: Steven Rostedt @ 2026-06-20 23:43 UTC (permalink / raw)
To: Linus Torvalds
Cc: Yury Norov, LKML, Masami Hiramatsu, Mathieu Desnoyers, Ao Sun,
David Carlier, Karl Mehltretter, Martin Kaiser, Pengpeng Hou,
Qian-Yu Lin, Rik van Riel, Rosen Penev, Shuvam Pandey,
Vineeth Pillai, Yash Suthar, Yu Peng, Sebastian Andrzej Siewior,
Thomas Gleixner, Peter Zijlstra
On Sat, 20 Jun 2026 15:39:25 -0700
Linus Torvalds <torvalds@linux-foundation.org> wrote:
> Feel free to try to come up with such a patch.
>
> But honestly, before you do, what is the *advantage* of such a thing?
For debugging it is really useful.
>
> Because you seem to think that "trace_printk()" and "printk()" are the same.
>
> They really aren't.
I do not in any way think they are the same. To me, printk() is used
for information to the console for various production messages, whereas
trace_printk() is used when you have that nasty bug that you don't know
exactly where it is. trace_printk() should never be used in a
production environment.
When I (and many others) use trace_printk() to debug, we basically use
the "shotgun" approach. That is, we add trace_printk() all over the
place to see where the bug occurs. This could be for 10s of files.
Having to add an include to each one of these files is a burden and
adds to the frustration when you are debugging something that doesn't
work. You just want to add trace_printk() in places to see where the
bug triggers.
A lot of times, all I add is:
trace_printk("%s:%d\n", __func__, __LINE__):
and cut and paste that in several locations in several files between if
statements. I may even add:
if (bad_condition())
tracing_off();
Which will disable tracing when the bad condition is detected. Then I
can look at the trace to see all the prints up to the bug. This is
*really* useful!!!
I really want to avoid having to add an include for trace_printk when
I'm focusing on just finding were the bug happens.
I'm pretty sure others on the Cc list have the same use case.
I totally sympathize with getting rid of junk out of kernel.h (and
possibly getting rid of kernel.h altogether) but I also want to keep
this debugging ability around.
-- Steve
^ permalink raw reply [flat|nested] 30+ messages in thread* Re: [GIT PULL] tracing: Updates for 7.2
2026-06-20 23:43 ` Steven Rostedt
@ 2026-06-21 0:18 ` Linus Torvalds
2026-06-21 6:34 ` Steven Rostedt
2026-06-21 7:10 ` Steven Rostedt
0 siblings, 2 replies; 30+ messages in thread
From: Linus Torvalds @ 2026-06-21 0:18 UTC (permalink / raw)
To: Steven Rostedt
Cc: Yury Norov, LKML, Masami Hiramatsu, Mathieu Desnoyers, Ao Sun,
David Carlier, Karl Mehltretter, Martin Kaiser, Pengpeng Hou,
Qian-Yu Lin, Rik van Riel, Rosen Penev, Shuvam Pandey,
Vineeth Pillai, Yash Suthar, Yu Peng, Sebastian Andrzej Siewior,
Thomas Gleixner, Peter Zijlstra
On Sat, 20 Jun 2026 at 16:44, Steven Rostedt <steven@rostedt.org> wrote:
>
> For debugging it is really useful.
According to that logic, we should put everything in one header file
and call it a day.
Because no, trace_printk() isn't special.
Christ, the only reason *you* think it's so special is because it's
your code and you use it, and the people you interact with use it.
But that's not because it's special - the common issue is *you*, not
trace_printk.
And yes, we're *all* special. Just like our mothers told us. But
because we're *all* special, none of us are.
So no. Stop making arguments based on that "I'm special" argument.
If you can't back it up with numbers - like I did - then you're not
special, you're just a snowflake.
Honestly, the "I'm using it for debugging" is not an argument.
You can damn well use it for debugging and add the header then.
But I personally think that debugging should be done by looking at and
understanding the code. To me, "debugging using printk's" - fancy
printk's or not - is a failure mode, and soemthing you should be
quietly ashamed of, instead of using it as some kind of argument for
making everybody elses kernel worse.
By the time you resort to debugging with printk, adding the
appropriate include just isn't the issue any more.
We're done here. That header gets removed from kernel.h.
Linus
^ permalink raw reply [flat|nested] 30+ messages in thread* Re: [GIT PULL] tracing: Updates for 7.2
2026-06-21 0:18 ` Linus Torvalds
@ 2026-06-21 6:34 ` Steven Rostedt
2026-06-21 7:10 ` Steven Rostedt
1 sibling, 0 replies; 30+ messages in thread
From: Steven Rostedt @ 2026-06-21 6:34 UTC (permalink / raw)
To: Linus Torvalds
Cc: Yury Norov, LKML, Masami Hiramatsu, Mathieu Desnoyers, Ao Sun,
David Carlier, Karl Mehltretter, Martin Kaiser, Pengpeng Hou,
Qian-Yu Lin, Rik van Riel, Rosen Penev, Shuvam Pandey,
Vineeth Pillai, Yash Suthar, Yu Peng, Sebastian Andrzej Siewior,
Thomas Gleixner, Peter Zijlstra
On Sat, 20 Jun 2026 17:18:26 -0700
Linus Torvalds <torvalds@linux-foundation.org> wrote:
>
> Because no, trace_printk() isn't special.
When you have a bug that triggers once every 11 days, due to some
strange race, placing trace_printk()s all over to help you narrow down
*where* to look is super important. trace_printk() has an extremely low
overhead, like 150 ns which keeps the chances of it affecting the races
low. Note, the above scenario was real story another kernel developer
told me about, but that's besides the point.
>
> Christ, the only reason *you* think it's so special is because it's
> your code and you use it, and the people you interact with use it.
Do you know who the people I interact with are? 100s if not 1000s of
kernel developers! I get my ideas from going to conferences and talking
with people. This isn't me in some silo helping out an internal
corporation with their secret sauce. I was unemployed from April 19th
to June 15th and in that time I paid my own way to go to LSF/MM/BPF to
find out more ways the tracing subsystem can be useful to others!
Hang out more at conferences and talk to your kernel's developers and
get a better idea of what people use. You interface mostly with top
level maintainers. How much are you interfacing with the 1000s of
contributors we have? Tracing has been extremely helpful for new
developers. I've mentored several. How many new developers have you
mentored in the last 5 years? That's a serious question. I don't know
if you mentor new developers or not. It would be great if you did.
Just last week, someone asked me how I got involved with the Linux
kernel. They said, "It's so big and complex, how did you start?". I
started looking at the kernel code in 1998. It was much simpler back
then. Learning most of the kernel wasn't that difficult compared to
starting development today. I only had to learn things as they came
into existence. Learning about new features that appear over the years
is much easier than coming in fresh and everything is a new feature to
learn.
>
> But that's not because it's special - the common issue is *you*, not
> trace_printk.
>
You already had others tell you that they use trace_printk() for
debugging too. It's *not* just me! Ask people. There's been several
times I've had people come up to me at conferences and thank me for
trace_printk().
Heck, trace_prink() was important enough to be included in eBPF for
debugging they programs.
>
> We're done here. That header gets removed from kernel.h.
I told you I'm 100% fine with removing trace_printk.h from kernel.h if
you are OK with the Makefile solution I posted. Having a
TRACE_PRINTK_DEBUGGING config would not cause any issue for you or
others that don't care about it. I believe it's a proper compromise.
You seem to act like I'm arguing to keep trace_printk.h in kernel.h
after I came up with an alternate solution. Let me add back the
question you asked to the very first line of my reply that you went off
on:
> On Sat, 20 Jun 2026 15:39:25 -0700
> Linus Torvalds <torvalds@linux-foundation.org> wrote:
>
> > Feel free to try to come up with such a patch.
> >
> > But honestly, before you do, what is the *advantage* of such a thing?
>
> For debugging it is really useful.
You seem to think that comment was about why to keep trace_printk.h in
kernel.h and bother everyone that doesn't use it. No, you asked me what
the advantage of having the Makefile solution was, and that was my answer.
-- Steve
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [GIT PULL] tracing: Updates for 7.2
2026-06-21 0:18 ` Linus Torvalds
2026-06-21 6:34 ` Steven Rostedt
@ 2026-06-21 7:10 ` Steven Rostedt
1 sibling, 0 replies; 30+ messages in thread
From: Steven Rostedt @ 2026-06-21 7:10 UTC (permalink / raw)
To: Linus Torvalds
Cc: Yury Norov, LKML, Masami Hiramatsu, Mathieu Desnoyers, Ao Sun,
David Carlier, Karl Mehltretter, Martin Kaiser, Pengpeng Hou,
Qian-Yu Lin, Rik van Riel, Rosen Penev, Shuvam Pandey,
Vineeth Pillai, Yash Suthar, Yu Peng, Sebastian Andrzej Siewior,
Thomas Gleixner, Peter Zijlstra
On Sat, 20 Jun 2026 17:18:26 -0700
Linus Torvalds <torvalds@linux-foundation.org> wrote:
> But I personally think that debugging should be done by looking at and
> understanding the code. To me, "debugging using printk's" - fancy
> printk's or not - is a failure mode, and soemthing you should be
> quietly ashamed of, instead of using it as some kind of argument for
> making everybody elses kernel worse.
One last note on this. I don't think I ever used "debugging using
printk's" for a design bug. It's always ended up being something like an
off by one issue. That is, looking at the code can easily miss it.
-- Steve
^ permalink raw reply [flat|nested] 30+ messages in thread