libtracecmd API extensions

libtracecmd API extensions

[MOESSBAUER, Felix]

Dear Tracers,

while developing a kernel ftrace to Common Trace Format (CTF) converter
[1], I noticed a couple of aspects of the libtracecmd API I would like
to discuss. In general, I am VERY happy with the libtracefs,
libtraceevent and libtracecmd API. These are great libraries which
really helped to get this implemented easily.

Trace clocks:

- Information about the used trace clock is available in the trace.dat,
however there is no public API to extract it
- A trace clock (e.g. mono) to world-clock snapshot could be added to
the trace.dat to more easily align the trace with other traces

Metadata:

- some metadata (like uname) is already included in the trace.dat,
however the APIs to retrieve it are private: tracecmd_get_uname
- further metadata like a full "uname -a" output and /proc/cmdline
would be helpful

Event time ranges:

- it would be good to have an API to get the first and last event
timestamp per CPU (I'm currently fully walking the events to get that
data)

PS:
We are also looking into writing a converter in the opposite direction
(i.e. CTF -> ftrace binary format / trace.dat), but it looks like the
libtraceevent is a pure parsing library and cannot be used for
synthesizing. Is this true, or did I miss something?

[1] https://github.com/siemens/bt2-ftrace-to-ctf

Best regards,
Felix Moessbauer

-- 
Siemens AG
Linux Expert Center
Friedrich-Ludwig-Bauer-Str. 3
85748 Garching, Germany

Re: libtracecmd API extensions

[Steven Rostedt]

On Thu, 20 Nov 2025 12:24:18 +0000
"MOESSBAUER, Felix" <felix.moessbauer@siemens.com> wrote:

> Dear Tracers,

Hello!

> 
> while developing a kernel ftrace to Common Trace Format (CTF) converter
> [1], I noticed a couple of aspects of the libtracecmd API I would like
> to discuss. In general, I am VERY happy with the libtracefs,
> libtraceevent and libtracecmd API. These are great libraries which
> really helped to get this implemented easily.
> 
> Trace clocks:
> 
> - Information about the used trace clock is available in the trace.dat,
> however there is no public API to extract it
> - A trace clock (e.g. mono) to world-clock snapshot could be added to
> the trace.dat to more easily align the trace with other traces
> 
> Metadata:
> 
> - some metadata (like uname) is already included in the trace.dat,
> however the APIs to retrieve it are private: tracecmd_get_uname
> - further metadata like a full "uname -a" output and /proc/cmdline
> would be helpful

So, libtracecmd is still very much a work in progress, of something I never
seem to have the time to work on :-p  I don't want to make the mistake I
made with libtraceevent, and have APIs exposed before they are ready. I hate
most of the libtraceevent APIs, but that's because it was ported over to
perf before it was complete, and we got stuck with what we have.

libtracefs was much more planned, and has a better API because of that.

I have lots of ideas for libtracecmd. In fact, the goal is to make
trace-cmd a simple shell of the library, and have every feature of
trace-cmd exposed to other tools. Unfortunately, I just don't have the time
to work on it.

Hence, if you want to add these APIs, I'm more than happy to review any
patches you send my way!

> 
> Event time ranges:
> 
> - it would be good to have an API to get the first and last event
> timestamp per CPU (I'm currently fully walking the events to get that
> data)

Hmm, I could have sworn I implemented something like this in the past.
Perhaps I only did it to my local tree. I have lots of updates to trace-cmd
that haven't made it out to public (simply because they are still in debug
form).

There is a way to iterate backwards:

  tracecmd_iterate_events_reverse()

That would give you the last event without walking all events.

> 
> PS:
> We are also looking into writing a converter in the opposite direction
> (i.e. CTF -> ftrace binary format / trace.dat), but it looks like the
> libtraceevent is a pure parsing library and cannot be used for
> synthesizing. Is this true, or did I miss something?
> 

Yeah, libtraceevent was made to be purely parsing. In fact, there was an
effort to convert it over to flex/bison. But again, time ran out.

That said, if you want to add a feature to create formats, that could
definitely be added.

Cheers,

-- Steve

Re: libtracecmd API extensions

[MOESSBAUER, Felix]

On Thu, 2025-11-20 at 13:08 -0500, Steven Rostedt wrote:
> On Thu, 20 Nov 2025 12:24:18 +0000
> "MOESSBAUER, Felix" <felix.moessbauer@siemens.com> wrote:
> 
> > Dear Tracers,
> 
> Hello!
> 
> > 
> > while developing a kernel ftrace to Common Trace Format (CTF) converter
> > [1], I noticed a couple of aspects of the libtracecmd API I would like
> > to discuss. In general, I am VERY happy with the libtracefs,
> > libtraceevent and libtracecmd API. These are great libraries which
> > really helped to get this implemented easily.
> > 
> > Trace clocks:
> > 
> > - Information about the used trace clock is available in the trace.dat,
> > however there is no public API to extract it
> > - A trace clock (e.g. mono) to world-clock snapshot could be added to
> > the trace.dat to more easily align the trace with other traces
> > 
> > Metadata:
> > 
> > - some metadata (like uname) is already included in the trace.dat,
> > however the APIs to retrieve it are private: tracecmd_get_uname
> > - further metadata like a full "uname -a" output and /proc/cmdline
> > would be helpful
> 
> So, libtracecmd is still very much a work in progress, of something I never
> seem to have the time to work on :-p  I don't want to make the mistake I
> made with libtraceevent, and have APIs exposed before they are ready. I hate
> most of the libtraceevent APIs, but that's because it was ported over to
> perf before it was complete, and we got stuck with what we have.
> 
> libtracefs was much more planned, and has a better API because of that.
> 
> I have lots of ideas for libtracecmd. In fact, the goal is to make
> trace-cmd a simple shell of the library, and have every feature of
> trace-cmd exposed to other tools. Unfortunately, I just don't have the time
> to work on it.

Yes, that would be exactly what we need.

> 
> Hence, if you want to add these APIs, I'm more than happy to review any
> patches you send my way!

I'll send patches to the ML.

> 
> > 
> > Event time ranges:
> > 
> > - it would be good to have an API to get the first and last event
> > timestamp per CPU (I'm currently fully walking the events to get that
> > data)
> 
> Hmm, I could have sworn I implemented something like this in the past.
> Perhaps I only did it to my local tree. I have lots of updates to trace-cmd
> that haven't made it out to public (simply because they are still in debug
> form).
> 
> There is a way to iterate backwards:
> 
>   tracecmd_iterate_events_reverse()
> 
> That would give you the last event without walking all events.

I just tried it and in principle it should work, just two things:

1. the interface was just recently added (3.3, so that's too new for
bookworm)
2. we leak one record per CPU when not fully iterating (i.e. when
returning != 0 from the callback). This leak unfortunately generates
output which I must not have.

I still don't fully understand why we leak, but it is somewhere around
this location:
https://github.com/rostedt/trace-cmd/blame/364faec474e4e4e0236c21af8927186e0e214e7f/lib/trace-cmd/trace-input.c#L3060

The corresponding valgrind callstack looks like this:

-snip-
==1497610== 1,024 bytes in 16 blocks are definitely lost in loss record
241 of 249
==1497610==    at 0x4883BA3: calloc (vg_replace_malloc.c:1675)
==1497610==    by 0x5D139F4: allocate_page (trace-input.c:1574)
==1497610==    by 0x5D14C8A: get_page (trace-input.c:1891)
==1497610==    by 0x5D17882: load_records (trace-input.c:2897)
==1497610==    by 0x5D19CD6: initialize_last_events (trace-
input.c:2957)
==1497610==    by 0x5D19CD6: tracecmd_iterate_events_reverse (trace-
input.c:3036)
-snip-

> 
> > 
> > PS:
> > We are also looking into writing a converter in the opposite direction
> > (i.e. CTF -> ftrace binary format / trace.dat), but it looks like the
> > libtraceevent is a pure parsing library and cannot be used for
> > synthesizing. Is this true, or did I miss something?
> > 
> 
> Yeah, libtraceevent was made to be purely parsing. In fact, there was an
> effort to convert it over to flex/bison. But again, time ran out.

That's reasonable. Having the CTF to ftrace way is also not of highest
priority for us, but it would help our Xenomai users to view combined
traces in the kernelshark, as they already have custom plugins for the
visualization.

> 
> That said, if you want to add a feature to create formats, that could
> definitely be added.

Good to know. I'll have a deeper look what exactly we need.

Cheers!
Felix

> 
> Cheers,
> 
> -- Steve

-- 
Siemens AG
Linux Expert Center
Friedrich-Ludwig-Bauer-Str. 3
85748 Garching, Germany