[RFC] LTTng upstreaming next steps

[RFC] LTTng upstreaming next steps

[Mathieu Desnoyers]

Hi,

It was suggested that the LTTng kernel tracer be upstreamed and become
its own kernel subsystem in a recent LKML discussion [1]. This would
allow this 20 years old project to move its development from kernel
modules within an out-of-tree git repository into the Linux kernel tree.

The LTTng kernel tracer repository can be found here:

   https://git.lttng.org/lttng-modules.git

Considering that the most recent attempts to upstream it were not
successful due to the size of this project (~75k LOC) and lack of time
from tracing maintainers to review it, I'm sending this RFC to ask what
is the best way forward.

The LTTng project has been maintained for 20 years, being a kernel patch
series from 2005 to 2010, and an out-of-tree kernel module from 2010
until now.

Over those 20 years, we have tried to be good citizens by actively
contributing to Linux kernel within the following area:

- Tracepoints,
- Restartable Sequences (rseq) system call,
- Membarrier system call,
- RCU review,
- Tracing review,

LTTng is available in most Linux distributions [2]. We have also
contributed to help the tracing users and developers meet and discuss by
organizing 14 Tracing Summit events over the last 17 years [3]. For the
past two years, we have hosted monthly HPC tracing collaboration
meetings with the HPC community.

Past LTTng upstreaming attempts
===============================

Please keep in mind that this is only my personal recollection of the
LTTng upstreaming attempts.

- 2005 to 2010:
   - Several attempts at upstreaming LTTng 0.x were made between 2005 and
     2010. The original LTTng was a patch series, and it predates both
     ftrace and perf.
     In 2007 and 2008, I contributed the Kernel Markers and then the
     Tracepoint instrumentation APIs to the Linux kernel, so make static
     instrumentation available to all tracers. Ftrace and perf were quick
     to adopt it, then making it harder for LTTng to justify its
     integration upstream.
   - What made Tracepoint accepted by kernel maintainers was the ongoing
     work on asm goto (used for jump labels). I have been actively
     involved in its genesis through redaction of a desiderata document
     and various discussions with both kernel and compiler people. This
     led to the existence of asm goto in gcc.

- 2010:
   - The upstreaming plan was presented at the Collaboration Summit 2010,
     discussed with the community.
     - https://www.efficios.com/pub/lfcs2010/desnoyers-presentation-lfcs2010.pdf
   - I attempted to modify TRACE_EVENT macros to suit the needs of LTTng.
     My changes rejected because not immediately useful to in-kernel tracers.
   - After this first failure, I moved the focus on libringbuffer:
     - libringbuffer was discussed and planned with the community before
        beginning the effort, initially aimed at replacing ftrace ring buffer,
        creating a common ring buffer infrastructure for high-troughput tracers
        (ftrace and lttng). It was rejected upon submission upstream by the
        perf maintainers, moving the goal post to cover perf as well, which
        is significantly different because it focuses on sampling use-cases.
   - After those two unsuccessful attempts at upstreaming and running out
     of funding to work on kernel upstreaming, the focus of the
     lttng-modules project changed and aimed to become a standalone OOT
     kernel module. It was officially announced 2 years later:
     https://events.static.linuxfound.org/images/stories/pdf/eeus2012_desnoyers.pdf

- 2012:
   - I attempted to merge LTTng through the staging tree. Ingo Molnar
     opposed wanting LTTng to be integrated into Perf instead.
     https://lore.kernel.org/lkml/20111219104915.GA19861@elte.hu/
   - I just did not have the available resources to undertake the
     endeavor proposed by Ingo, which was to rebuild LTTng within
     ftrace and perf to make its code base common with tools which have
     only a partial requirement overlap.

- 2020:
   - Steven Rostedt showed interest in helping me upstream LTTng.
     After discussion with him, I prepared a "v5.7-rc1-lttng-upstreaming"
     branch. The initial plan was that he would provide a few rounds of
     feedback privately before I would publish it publically.
     He never had time to review it and provide feedback, only mentioned
     that it was more code than he initially expected. This effort did
     not lead anywhere.
     https://github.com/compudj/linux-dev/tree/v5.7-rc1-lttng-upstreaming

Open questions on how to proceed with upstreaming
=================================================

My goal is to move the development of the "lttng-modules" project into
the upstream kernel.

There are a few possible approaches we can envision, just to list a few
choices we have:

A) Bulk upstreaming

    Upsides:
    - The upstreamed LTTng kernel tracer becomes a drop-in replacement
      for the lttng-modules project. Allows to quickly phase out the OOT
      development model.
    - Preserves the existing LTTng-modules userspace ABI.
    - This moves the status-quo from "out of tree" to "in tree".

    Downsides:
    - Not developed from the ground up with the help of the kernel
      community. This rather becomes an after-the-fact review of a large
      code base.
    - Brings in backward compatibility macros and ifdefs to support
      older kernels which are pointless upstream. This can be easily
      mitigated by doing a clean up removing this compatibility code
      as part of the upstreaming exercise.

B) Piecewise upstreaming, basically creating a "new" LTTng kernel tracer
    by reimplementing the current tracer differently based on feedback.

    Upsides:
    - Allows the kernel community to be involved in the
      "re-implementation" of LTTng within the kernel tree.

    Downsides:
    - Large commitment. Not sure if funding this effort is feasible.
    - Uncertainty about the result: after all this effort, there is no
      guarantee that any upstream maintainer will be able to justify the
      time needed for review.
    - Requires to still maintain lttng-modules OOT for the forseeable
      future, as the upstream work is not a drop-in replacement ABI-wise.
    - This preserves the status-quo of LTTng-modules as "out of tree".

C) If we go for the bulk upstreaming, we can either:

   - Try to keep the development history of LTTng-modules. This may be
     tricky because the lttng-modules git tree is separate from the Linux
     kernel. AFAIR this has been done in the past when integrating Btrfs
     into the kernel tree:
       commit 4b48d9d44ebe ("Merge git://git.kernel.org/pub/scm/linux/kernel/git/mason/btrfs-unstable")

   - Import LTTng-modules as a set of new kernel commits, e.g. one commit
     grouping a set of files which implement a given facet of the tracer.

D) Should we integrate LTTng through the staging tree ?

    LTTng could end up into kernel/lttng/, or could go through the staging
    tree while we remove the code which is not relevant in the context of
    the upstream kernel (e.g. older kernel support).

E) Keep existing ioctl ABIs or move to new set of ABIs ?

    If we choose to go for a piecewise approach where we want to redefine
    the userspace ABIs, we would have to use new ioctl numbers to make
    sure we do not break the existing LTTng-modules ABI guarantees.

F) Integrate the LTTng-modules "core", initially leaving the tracepoint
    and system call instrumentation out-of-tree.

    Considering that the LTTng-modules instrumentation code for
    tracepoints and system calls represents a significant portion of
    the SLOC, we could keep those out of the initial upstreaming and
    focus on the LTTng core first.

LTTng Kernel Tracer Overview
============================

SLOC    path
----------------------------------------
11231   include/instrumentation/events
25667   include/instrumentation/syscalls

5090    include/lttng
1767    include/ringbuffer
1246    include/wrapper
396     include/counter
11      include/blacklist

23334   src/*.[ch]
4645    src/lib
865     src/probes
273     src/wrapper
207     src/tests

The largest piece of code is include/instrumentation/syscalls
(25667 SLOC), which contains the generated instrumentation code for system
calls which allow fetching the system call ABI fields from userspace for
the following architectures: arm 32/64, mips 32/64, powerpc 32/64, x86
32/64.

Those are included from src/lttng-syscalls* (1687 SLOC) to generate the
syscall tables for instrumentation.

A large chunk is also the tracepoint instrumentation
include/instrumentation/events (11231 SLOC), which reimplements a
portion of the upstream kernel include/trace/events in ways that are
compatible with LTTng filtering. Those are included from "probe" files
in src/probes/* to generate probe modules (865 SLOC).  Note that a large
portion of those instrumentation headers consists of backward
compatibility for older kernels with kernel version #ifdefs, and very
fortunately be removed as part of the upstreaming process.

The "wrapper" include subdir is mostly for backward compatibility with
the supported older kernels, which can go away as a cleanup as part of
the upstreaming benefits.

The LTTng ring buffer sits in include/ringbuffer and src/lib/ringbuffer,
for a total of 3589 SLOC. This is a generic ring buffer which was
originally created to cover LTTng-UST, LTTng modules and Ftrace. The
LTTng ring buffer "clients" (templated C) wiring up the ring buffer
library for LTTng can be found in src/lttng-ring-buffer-* (1455 SLOC).

The other significant portion is src/*.[ch], which contains the ABI that
exposes LTTng-modules to userspace:

lttng-abi.c (4190 SLOC) implements the userspace ABI (LTTng ioctls).

Note that the other relevant portion of the ABI is in the ringbuffer
code:

src/lib/ringbuffer/ring_buffer_vfs.c (381 SLOC)

All the src/lttng-context*.c implement the LTTng contexts, stuff like
pid, uid, perf counters, which can be optionally enabled by users.
(2947 SLOC)

The LTTng dynamic filter is implemented under src/lttng-bytecode*
(5028 SLOC).

The LTTng support for CTF 1.8 and CTF 2 is implemented under
src/metadata-ctf* (2082 SLOC).

We also have src/lttng-events.c (3742 SLOC) which handles all the
session/channel/event object model of LTTng.

Also relevant is src/lttng-statedump-impl.c (601 SLOC) which implement
the LTTng state dump, dumping kernel state at trace start and when
requested by the end user (e.g. list processes, interrupts, FDs,
namespaces, network interfaces, cpu topology), block devices.

Note that src/tests (207 SLOC) is fairly small. Our testing is mostly
done from the lttng-tools project [4].

Your feedback would be most appreciated,

Thanks,

Mathieu

[1] https://lore.kernel.org/lkml/20250711142834.73b2864c@batman.local.home/
[2] https://lttng.org/download/
[3] https://tracingsummit.org/
[4] https://ci.lttng.org/

-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com

Re: [RFC] LTTng upstreaming next steps

[Linus Torvalds]

On Mon, 14 Jul 2025 at 12:41, Mathieu Desnoyers
<mathieu.desnoyers@efficios.com> wrote>
> A) Bulk upstreaming

Honestly, I don't see the point.

The reason the current tracting infrastructure got merged was that
people were willing to do it incrementally.

I was hoping that there would be some kind of eventual merging of the
different ring buffers etc. That was discussed as a hopeful end goal
originally, but here were are, decades later, and it never happened.

And honestly, I am NOT interested in just having two different tracing models.

If people need two tracing models, then the other one will be out of
tree. It's that simple.

Because if people haven't been able to agree on common models in the
decades past, I really don't see the point in maintaining two models
indefinitely side-by-side in the upstream kernel.

So as far as I'm concerned, this discussion is not a discussion.
Either there's a way to merge things incrementally with SHARED
infrastructure, or there isn't.

No "two different and disjoint trace buffers".

No "two different and disjoint trace interfaces".

And very clearly - based on history - that unification will never happen.

            Linus

Re: [RFC] LTTng upstreaming next steps

[Steven Rostedt]

On Mon, 14 Jul 2025 13:00:27 -0700
Linus Torvalds <torvalds@linux-foundation.org> wrote:

> On Mon, 14 Jul 2025 at 12:41, Mathieu Desnoyers
> <mathieu.desnoyers@efficios.com> wrote>
> > A) Bulk upstreaming  
> 
> Honestly, I don't see the point.
> 
> The reason the current tracting infrastructure got merged was that
> people were willing to do it incrementally.
> 
> I was hoping that there would be some kind of eventual merging of the
> different ring buffers etc. That was discussed as a hopeful end goal
> originally, but here were are, decades later, and it never happened.

I'll note that one of the reasons that we couldn't use the LTTng ring
buffer is because the format of the tracing ring buffer and the perf ring
buffer were already exposed to user space and would break tooling if they
were changed. LTTng actually has more features than either the existing in
kernel buffers so it couldn't use the existing buffers without breaking its
own tooling.

> 
> And honestly, I am NOT interested in just having two different tracing models.
> 
> If people need two tracing models, then the other one will be out of
> tree. It's that simple.

It's not that anyone needs two models. It's that there's one model that one
set of people use, and there's another model that another set of people use.
The in-tree tracing (aka ftrace) is designed to be used for kernel
developers. It's quick and easy to use to find issues and is mostly focused
around kernel development. The tracefs directory interface has been a
godsend for embedded developers since it only needs busybox to be useful.

LTTng is more focused toward user admins and is around monitoring how the
system is behaving. It has a different set of tooling and interfaces.

> 
> Because if people haven't been able to agree on common models in the
> decades past, I really don't see the point in maintaining two models
> indefinitely side-by-side in the upstream kernel.
> 
> So as far as I'm concerned, this discussion is not a discussion.
> Either there's a way to merge things incrementally with SHARED
> infrastructure, or there isn't.
> 
> No "two different and disjoint trace buffers".
> 
> No "two different and disjoint trace interfaces".

Although they do have a separate set of buffers and interfaces, the actual
infrastructure (tracepoints, function hooks, etc) is shared.

> 
> And very clearly - based on history - that unification will never happen.

Unfortunately, I think you're right. :-(

-- Steve

Re: [RFC] LTTng upstreaming next steps

[Steven Rostedt]

On Mon, 14 Jul 2025 16:27:50 -0400
Steven Rostedt <rostedt@goodmis.org> wrote:

> > And very clearly - based on history - that unification will never happen.  
> 
> Unfortunately, I think you're right. :-(

Actually, let me take that back. I think you are right as long as one is
out of tree. Because there's no incentive to merge them.

If LTTng were to be in tree, it would be much easier to work on a strategy
to merge the infrastructure as the maintainers of both would have better
access to each other's code.

The Linux kernel has had a long history of two or more ways of doing
something that eventually was merged into a single shared infrastructure.

But as long as one is out of tree, it will never happen. That's because it
is much harder to merge infrastructure together when they are separated
like that.

-- Steve

Re: [RFC] LTTng upstreaming next steps

[Linus Torvalds]

On Mon, 14 Jul 2025 at 13:37, Steven Rostedt <rostedt@goodmis.org> wrote:
>
> If LTTng were to be in tree, it would be much easier to work on a strategy
> to merge the infrastructure as the maintainers of both would have better
> access to each other's code.

That's not a bet I'd take.

If people haven't unified this in the last two decades, I'm noty going
to take the argument of "hey, merge it because *then* it will be
unified".

Because honestly, that sounds like a total fairy tale to me: "the
princess came along and kissed the toad, and he turned into a
beautiful price, and they lived happily ever after".

So no. I don't believe in fairy tales. Not when we have two decades of
"that didn't happen".

If people can unify this and merge it incrementally, that's one thing.

Until then, you're just making stuff up.

"Show me the code", in other words.

              Linus

Re: [RFC] LTTng upstreaming next steps

[Steven Rostedt]

On Mon, 14 Jul 2025 14:04:38 -0700
Linus Torvalds <torvalds@linux-foundation.org> wrote:

> On Mon, 14 Jul 2025 at 13:37, Steven Rostedt <rostedt@goodmis.org> wrote:
> >
> > If LTTng were to be in tree, it would be much easier to work on a strategy
> > to merge the infrastructure as the maintainers of both would have better
> > access to each other's code.  
> 
> That's not a bet I'd take.
> 
> If people haven't unified this in the last two decades, I'm noty going
> to take the argument of "hey, merge it because *then* it will be
> unified".
> 
> Because honestly, that sounds like a total fairy tale to me: "the
> princess came along and kissed the toad, and he turned into a
> beautiful price, and they lived happily ever after".
> 
> So no. I don't believe in fairy tales. Not when we have two decades of
> "that didn't happen".
> 
> If people can unify this and merge it incrementally, that's one thing.
> 
> Until then, you're just making stuff up.
> 
> "Show me the code", in other words.

We'd love to, but how is one to merge code in incremental steps when they
are not allowed to use the current infrastructure?

LTTng being out of tree means it is only allowed to use EXPORT_SYMBOL_GPL()
functions as it must be a module. Perf, ftrace and BPF are all 100% in-tree
and has no modules so they do not require any EXPORT_SYMBOL_GPL() to use
the current infrastructure.

This is the main difference between an external tracer and an external
driver. That's because other in-tree drivers and file systems all have a
module component. If ftrace, perf and BPF were modules, it would require the
infrastructure they use to all be defined as EXPORT_SYMBOL_GPL(). This
makes LTTng have a huge disadvantage as those are not modules and the
interface they use are not exported.

Christoph and Greg keep saying "this is no different than any other out of
tree code", but I'm saying due to this limitation it is. When they talk
about out of tree drivers and file systems that need to do the same, that
code has a huge advantage because it is competing with other code that are
modules and they can use the same infrastructure. Even when they are out of
tree.

Case in point, Mathieu is helping with the sframe work so that it can share
the same code that perf and ftrace will use. But as soon as we want to
allow it to use that code, it is shot down because it requires a
EXPORT_SYMBOL_GPL() to access it. Because perf an ftrace have no "module"
selection, there will be no in-tree module that would use that interface,
thus LTTng is out of luck.

This forces LTTng to come up with a different solution for everything that
perf and ftrace do.

This is a huge catch-22 for LTTng. You want unification before LTTng can
become part of the kernel. But because Christoph and Greg refuse to allow
LTTng to have access to anything that perf and ftrace use, there's no path
for LTTng to share the code.

This discussion is going full circle:

1. Mathieu helps with new infrastructure that it could share with perf and
   ftrace.

2. We ask to add EXPORT_SYMBOL_GPL() so that LTTng can share this code.

3. It is denied because LTTng is out of tree, and we are told that if LTTng
   wants to use this it must first be in tree.

4. We send an RFC on how to get LTTng in tree.

5. We are told that LTTng can only get in tree through incremental steps.

6. The only way to do incremental steps is for LTTng to have access to the
   same functions as perf and ftrace have, but is not allow to because it's
   out of tree!

When I said being out of tree makes it much more difficult to merge code,
this is the reason why.

Breaking up LTTng is a huge task to do it in incremental states. If it can't
use in kernel infrastructure while it is out of tree, it will mean the out
of tree LTTng cannot use the code being submitted. Thus, the code going
upstream will be unused or will be a non-functional LTTng. If it takes several
years to complete, it is forcing Mathieu to have a broken LTTng for several
years before it is allowed to use code upstream.

So which is it?

Either LTTng can use the same infrastructure as perf and ftrace, meaning it
requires EXPORT_SYMBOL_GPL() on those interfaces, or it gets pulled in as
is, so that it can incrementally start to share the code.

-- Steve

Re: [RFC] LTTng upstreaming next steps

[James Bottomley]

On Tue, 2025-07-15 at 05:24 -0400, Steven Rostedt wrote:
> On Mon, 14 Jul 2025 14:04:38 -0700
> Linus Torvalds <torvalds@linux-foundation.org> wrote:
[...]
> > If people can unify this and merge it incrementally, that's one
> > thing.
> > 
> > Until then, you're just making stuff up.
> > 
> > "Show me the code", in other words.
> 
> We'd love to, but how is one to merge code in incremental steps when
> they are not allowed to use the current infrastructure?
> 
> LTTng being out of tree means it is only allowed to use
> EXPORT_SYMBOL_GPL() functions as it must be a module. Perf, ftrace
> and BPF are all 100% in-tree and has no modules so they do not
> require any EXPORT_SYMBOL_GPL() to use the current infrastructure.
> 
> This is the main difference between an external tracer and an
> external driver. That's because other in-tree drivers and file
> systems all have a module component. If ftrace, perf and BPF were
> modules, it would require the infrastructure they use to all be
> defined as EXPORT_SYMBOL_GPL().  This makes LTTng have a huge
> disadvantage as those are not modules and the interface they use are
> not exported.  This makes LTTng have a huge disadvantage as those are
> not modules and the interface they use are not exported.

What's wrong with doing this the other way around?  i.e. making ftrace
and perf modules?  That way you could legitimately export the symbol
you're asking about and there would be way less out-of-tree
disadvantage to LTTng?  I know a lot of cloud people who would be
really happy if the tracing infrastructure were modular because it
would save us from having to boot different kernels to do in depth
problem analysis for otherwise locked down environments, so they might
be willing to invest in upstream development to help you achieve this.

Regards,

James

Re: [RFC] LTTng upstreaming next steps

[Steven Rostedt]

On Tue, 15 Jul 2025 08:50:08 -0400
James Bottomley <James.Bottomley@HansenPartnership.com> wrote:

> What's wrong with doing this the other way around?  i.e. making ftrace
> and perf modules?  That way you could legitimately export the symbol
> you're asking about and there would be way less out-of-tree
> disadvantage to LTTng?  I know a lot of cloud people who would be
> really happy if the tracing infrastructure were modular because it
> would save us from having to boot different kernels to do in depth
> problem analysis for otherwise locked down environments, so they might
> be willing to invest in upstream development to help you achieve this.

Well, if you can get those cloud people to invest in that work without
causing any regressions, go for it.

But I doubt it would be acceptable to make the ftrace tracing
infrastructure into a module for the sole purpose of allowing LTTng to have
EXPORT_SYMBOL_GPL().

-- Steve

Re: [RFC] LTTng upstreaming next steps

[James Bottomley]

On Tue, 2025-07-15 at 09:16 -0400, Steven Rostedt wrote:
> On Tue, 15 Jul 2025 08:50:08 -0400
> James Bottomley <James.Bottomley@HansenPartnership.com> wrote:
> > What's wrong with doing this the other way around?  i.e. making
> > ftrace and perf modules?  That way you could legitimately export
> > the symbol you're asking about and there would be way less out-of-
> > tree disadvantage to LTTng?  I know a lot of cloud people who would
> > be really happy if the tracing infrastructure were modular because
> > it would save us from having to boot different kernels to do in
> > depth problem analysis for otherwise locked down environments, so
> > they might be willing to invest in upstream development to help you
> > achieve this.
> 
> Well, if you can get those cloud people to invest in that work
> without causing any regressions, go for it.

I think you know as well as I no investment happens without some
indication of upstream being in favour, particularly for large changes.
So they're not going to invest in doing this on spec because it would
be unmaintainable out of tree and would be way more hassle than simply
having customers reboot as they do today.

> But I doubt it would be acceptable to make the ftrace tracing
> infrastructure into a module for the sole purpose of allowing LTTng
> to have EXPORT_SYMBOL_GPL().

I don't believe I said that: purpose is not monolithic in open source
because people do things for wildly different reasons which a clever
leader can stitch together into something more synergistically useful.
The cloud vendors would be invested solely for the purpose of being
able to load tracing infrastructure on demand (with the permission of
the tenant) into a running kernel.  They wouldn't care at all about the
symbol export problems of LTTng.  However, working with the cloud
vendors on what they want (and could be persuaded to invest in) would
give you what you wanted: an in-tree consumer for these symbols.

Regards,

James

Re: [RFC] LTTng upstreaming next steps

[Steven Rostedt]

On Tue, 15 Jul 2025 10:38:34 -0400
James Bottomley <James.Bottomley@HansenPartnership.com> wrote:

> > Well, if you can get those cloud people to invest in that work
> > without causing any regressions, go for it.  
> 
> I think you know as well as I no investment happens without some
> indication of upstream being in favour, particularly for large changes.

I think you just pointed out why Mathieu doesn't want to do this huge
update to merge upstream.


> So they're not going to invest in doing this on spec because it would
> be unmaintainable out of tree and would be way more hassle than simply
> having customers reboot as they do today.
> 
> > But I doubt it would be acceptable to make the ftrace tracing
> > infrastructure into a module for the sole purpose of allowing LTTng
> > to have EXPORT_SYMBOL_GPL().  
> 
> I don't believe I said that: purpose is not monolithic in open source

No you didn't say that, but I figured I'd mention it as it was likely
a thought someone may have had when reading your reply ;-)

> because people do things for wildly different reasons which a clever
> leader can stitch together into something more synergistically useful.
> The cloud vendors would be invested solely for the purpose of being
> able to load tracing infrastructure on demand (with the permission of
> the tenant) into a running kernel.  They wouldn't care at all about the
> symbol export problems of LTTng.  However, working with the cloud
> vendors on what they want (and could be persuaded to invest in) would
> give you what you wanted: an in-tree consumer for these symbols.

I have no vested interest in it. Mathieu has just been very helpful for
the last several years with reviewing and improving the in-tree tracing
infrastructure (which LTTng received no benefit from), that I'm just
paying back the favor in trying to help him out.

-- Steve

Re: [RFC] LTTng upstreaming next steps

[James Bottomley]

On Tue, 2025-07-15 at 10:50 -0400, Steven Rostedt wrote:
> On Tue, 15 Jul 2025 10:38:34 -0400
> James Bottomley <James.Bottomley@HansenPartnership.com> wrote:
> 
> > > Well, if you can get those cloud people to invest in that work
> > > without causing any regressions, go for it.  
> > 
> > I think you know as well as I no investment happens without some
> > indication of upstream being in favour, particularly for large
> > changes.
> 
> I think you just pointed out why Mathieu doesn't want to do this huge
> update to merge upstream.

I didn't say he would ... I said you might be able to persuade the
cloud companies to do it.  That's not to say he (and you) couldn't help
incentivise them that their investment might pay off.

> > So they're not going to invest in doing this on spec because it
> > would be unmaintainable out of tree and would be way more hassle
> > than simply having customers reboot as they do today.
> > 
> > > But I doubt it would be acceptable to make the ftrace tracing
> > > infrastructure into a module for the sole purpose of allowing
> > > LTTng to have EXPORT_SYMBOL_GPL().  
> > 
> > I don't believe I said that: purpose is not monolithic in open
> > source
> 
> No you didn't say that, but I figured I'd mention it as it was likely
> a thought someone may have had when reading your reply ;-)
> 
> > because people do things for wildly different reasons which a
> > clever leader can stitch together into something more
> > synergistically useful. The cloud vendors would be invested solely
> > for the purpose of being able to load tracing infrastructure on
> > demand (with the permission of the tenant) into a running kernel. 
> > They wouldn't care at all about the symbol export problems of
> > LTTng.  However, working with the cloud vendors on what they want
> > (and could be persuaded to invest in) would give you what you
> > wanted: an in-tree consumer for these symbols.
> 
> I have no vested interest in it. Mathieu has just been very helpful
> for the last several years with reviewing and improving the in-tree
> tracing infrastructure (which LTTng received no benefit from), that
> I'm just paying back the favor in trying to help him out.

Well, if you've no interest in thinking outside the box, by all means
continue banging your head against it until you think your payback has
been achieved ...  I'll go and get the popcorn.

Regards,

James

Re: [RFC] LTTng upstreaming next steps

[Steven Rostedt]

On Tue, 15 Jul 2025 05:24:59 -0400
Steven Rostedt <rostedt@goodmis.org> wrote:

> Either LTTng can use the same infrastructure as perf and ftrace, meaning it
> requires EXPORT_SYMBOL_GPL() on those interfaces, or it gets pulled in as
> is, so that it can incrementally start to share the code.

I believe that if we allowed LTTng to have access to the perf and
ftrace infrastructure, it would encourage more collaboration. It would
force Mathieu into working to get perf and ftrace and LTTng working
under one infrastructure so that it could finally get rid of its out of
tree module. Whereas, if we pulled it in as-is, there wouldn't be
really any incentive for it to change.

Now the question is, how do we allow LTTng's out of tree module to have
access to the internal infrastructure without breaking the
"EXPORT_SYMBOL_GPL() is only for in-tree modules" rule?

-- Steve

Re: [RFC] LTTng upstreaming next steps

[Steven Rostedt]

On Tue, 15 Jul 2025 11:10:43 -0400
James Bottomley <James.Bottomley@HansenPartnership.com> wrote:

> Well, if you've no interest in thinking outside the box, by all means
> continue banging your head against it until you think your payback has
> been achieved ...  I'll go and get the popcorn.

I didn't say that. I have no time to give on this. I don't have the
time to finish the things I am responsible for :-p

-- Steve