libtraceevent + Rust

libtraceevent + Rust

[Douglas Raillard]

Hi everyone,

The idea of having some Rust interop for libtraceevent was floated during the
Tracing Summit that took place this week.

The overall work has two scopes:
     1. libtraceevent: low-level helpers to decode event formats and binary buffer pages.
     2. high level API that e.g. can consume a trace.dat file and expose an iterator API etc.

The current discussion is only about #1. The 2nd part of the work has an
entirely different set of constraints as it would aim at integrating in the
Rust ecosystem as well as possible without trying to replace anything
pre-existent.


* Rust bindings for C code:
     Pros:
         * Faster to implement
         * By construction, any current covered use case stays covered.

     Cons:
         * Less straightforward to build and distribute on crates.io. Current
           best practice seems to simply expect the C lib to already be
           compiled/installed on the host system and find it with pkg-config.
           Otherwise, the lib sources can be bundled and a build recipe
           included. At any rate, it will bring back the daemons of C package
           management world where Rust typically only requires "cargo build" to
           obtain a working binary.

         * More unsafe code overall when combined with higher level Rust code.


* Rust implementation:
     Pros:
         * Occasion to fix unsatisfactory points in the current API:
             * Some inefficiencies due to looking up fields by name mentioned
               by Steven (correct that if I did not get it right).

             * Some unsatisfactory naming convention (Steven).

             * Event format is hard to parse as lots of event strongly
               depends on the format string to specify the actual type. E.g. The
               print event uses unsigned long for "ip" and then casts to (void*)
               in the format string. sched_switch or mm_page_alloc require even
               more advanced C parsing power to extract info necessary to
               correctly/usefully represent the state type in most languages.
               This might be fixable with more codified best practices and
               fixing existing trace events, but we would still need to deal
               with the existing situation for a few more years at best.  A new
               API could merge the type info from the parsed print format with
               the other "ABI info" to expose a richer representation to
               consumers.

         * Can be used by higher level Rust code, preserving soundness
           guarantees end to end.

         * Can open new use cases such as:
             * Likely easier to maintain Python bindings using e.g. PyO3
             * Good WASM support is something the Rust community cares about, so
               that might open an avenue for new trace visualization tools.

     Cons:
         * More work than just making bindings for existing C code.
         * Portability: is there any architecture not currenlty supported by
           Rust we care about ?
         * New toolchain required to build the code. rustup solves that issue
           very well though.

     Possible cons requiring a prototype to check:
         * If we aim at 100% API/ABI compatibility with current libtraceevent,
           this may force Rust code into very non-idiomatic code that may affect
           the benefits of the language.

         * While it can still be exposed to C code, some soundness
           guarantees will likely be lost in passing: Rust ref to C pointer
           conversion is statically lossy. Having a func implemented in Rust
           does not magically make consuming C compilers smarter and able to
           check for memory safety. The improvement would only be inside the
           implementation itself, but if we expose relatively simple low level
           helpers the advantage is decreased.


I'm sure I missed some things, so let's use this thread to get a better view of
where to head next.

Steven also made a twitter poll on the matter:
https://twitter.com/srostedt/status/1580589911706521600


Thanks,

Douglas

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Re: libtraceevent + Rust

[Hidenori Kobayashi]

Hi Douglas,

On Fri, Oct 14, 2022 at 04:16:33PM +0100, Douglas Raillard wrote:
> Hi everyone,
> 
> The idea of having some Rust interop for libtraceevent was floated during the
> Tracing Summit that took place this week.
> 
> The overall work has two scopes:
>     1. libtraceevent: low-level helpers to decode event formats and binary buffer pages.
>     2. high level API that e.g. can consume a trace.dat file and expose an iterator API etc.
> 
> The current discussion is only about #1. The 2nd part of the work has an
> entirely different set of constraints as it would aim at integrating in the
> Rust ecosystem as well as possible without trying to replace anything
> pre-existent.
 
Has anyone actually started working on these?

I have a Rust tool that parses the output from trace-cmd to infer the
internal states of a kernel module. While it serves our needs for now,
Steven kindly pointed me to this thread and suggested interacting
directly with the tracefs and using the tracefs libraries for parsing.

So I'd like to know:
- if this discussion reached a consensus
- if there is any publicly available work already

 
> * Rust bindings for C code:
>     Pros:
>         * Faster to implement
>         * By construction, any current covered use case stays covered.
> 
>     Cons:
>         * Less straightforward to build and distribute on crates.io. Current
>           best practice seems to simply expect the C lib to already be
>           compiled/installed on the host system and find it with pkg-config.
>           Otherwise, the lib sources can be bundled and a build recipe
>           included. At any rate, it will bring back the daemons of C package
>           management world where Rust typically only requires "cargo build" to
>           obtain a working binary.
> 
>         * More unsafe code overall when combined with higher level Rust code.
> 
> 
> * Rust implementation:
>     Pros:
>         * Occasion to fix unsatisfactory points in the current API:
>             * Some inefficiencies due to looking up fields by name mentioned
>               by Steven (correct that if I did not get it right).
> 
>             * Some unsatisfactory naming convention (Steven).
> 
>             * Event format is hard to parse as lots of event strongly
>               depends on the format string to specify the actual type. E.g. The
>               print event uses unsigned long for "ip" and then casts to (void*)
>               in the format string. sched_switch or mm_page_alloc require even
>               more advanced C parsing power to extract info necessary to
>               correctly/usefully represent the state type in most languages.
>               This might be fixable with more codified best practices and
>               fixing existing trace events, but we would still need to deal
>               with the existing situation for a few more years at best.  A new
>               API could merge the type info from the parsed print format with
>               the other "ABI info" to expose a richer representation to
>               consumers.
> 
>         * Can be used by higher level Rust code, preserving soundness
>           guarantees end to end.
> 
>         * Can open new use cases such as:
>             * Likely easier to maintain Python bindings using e.g. PyO3
>             * Good WASM support is something the Rust community cares about, so
>               that might open an avenue for new trace visualization tools.
> 
>     Cons:
>         * More work than just making bindings for existing C code.
>         * Portability: is there any architecture not currenlty supported by
>           Rust we care about ?
>         * New toolchain required to build the code. rustup solves that issue
>           very well though.
> 
>     Possible cons requiring a prototype to check:
>         * If we aim at 100% API/ABI compatibility with current libtraceevent,
>           this may force Rust code into very non-idiomatic code that may affect
>           the benefits of the language.
> 
>         * While it can still be exposed to C code, some soundness
>           guarantees will likely be lost in passing: Rust ref to C pointer
>           conversion is statically lossy. Having a func implemented in Rust
>           does not magically make consuming C compilers smarter and able to
>           check for memory safety. The improvement would only be inside the
>           implementation itself, but if we expose relatively simple low level
>           helpers the advantage is decreased.
> 
> 
> I'm sure I missed some things, so let's use this thread to get a better view of
> where to head next.
> 
> Steven also made a twitter poll on the matter:
> https://twitter.com/srostedt/status/1580589911706521600
> 
> 
> Thanks,
> 
> Douglas
> 
> IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
> 

Thanks,
Hidenori

Re: libtraceevent + Rust

[Douglas Raillard]

Hi Hidenori,

> On 13-02-2023 05:17, Hidenori Kobayashi wrote:
> Hi Douglas,
> 
> On Fri, Oct 14, 2022 at 04:16:33PM +0100, Douglas Raillard wrote:
>> Hi everyone,
>>
>> The idea of having some Rust interop for libtraceevent was floated during the
>> Tracing Summit that took place this week.
>>
>> The overall work has two scopes:
>>      1. libtraceevent: low-level helpers to decode event formats and binary buffer pages.
>>      2. high level API that e.g. can consume a trace.dat file and expose an iterator API etc.
>>
>> The current discussion is only about #1. The 2nd part of the work has an
>> entirely different set of constraints as it would aim at integrating in the
>> Rust ecosystem as well as possible without trying to replace anything
>> pre-existent.
>   
> Has anyone actually started working on these?

We have, this has been my main focus since around November.

> 
> I have a Rust tool that parses the output from trace-cmd to infer the
> internal states of a kernel module. While it serves our needs for now,
> Steven kindly pointed me to this thread and suggested interacting
> directly with the tracefs and using the tracefs libraries for parsing.
> 
> So I'd like to know:
> - if this discussion reached a consensus

You are the first one to answer on that thread but the twitter poll from Steven was encouraging. At any
rate, we need it for LISA [1] so I'm making it happen.

> - if there is any publicly available work already

Yes and no: I'm currently developing it as part of LISA but in its own "traceevent" crate,
as added by that PR under tools/analysis/traceevent/:
https://github.com/ARM-software/lisa/pull/1843

This is very much WIP, so far I got:
* A parser for v6 header up to and including the event formats.
   Since that requires a C parser, it was a large chunk of the total work.
* A C interpreter as (somewhat) required for array lengths and mostly for the print fmt args.
* a zero-copy I/O layer that can load data from:
	* existing in-memory buffer
	* Any type implementing Read + Seek
	* mmap

Adding support for v7 header should be straightforward except for the I/O that might require adjustment
depending on how compression is implemented.


My current roadmap is:
1. Get to a state where I can share a trace-cmd report equivalent static tool for people to play with on
    their own data and report issues.
2. Make a serde backend hooked to it since we need it for LISA (assuming performance is good enough)
3. Figure out exactly how it sits in the libtraceevent ecosystem and maintains it in the
    long run, libtraceevent-style C bindings etc.

Note that if and when a serde backend is available, that should bring a conversion to JSON for basically free.

Unless I'm sidetracked by urgent things, 1. and 2. should become available in a few months. I expect 3. to happen
when I have something concrete to show to the world.

[1] LISA: https://github.com/ARM-software/lisa
> 
> Thanks,
> Hidenori

Thanks,

Douglas

Re: libtraceevent + Rust

[Hidenori Kobayashi]

Hi Douglas,

On Mon, Feb 13, 2023 at 12:04:36PM +0000, Douglas Raillard wrote:
> Hi Hidenori,
> 
> > On 13-02-2023 05:17, Hidenori Kobayashi wrote:
> > Hi Douglas,
> > 
> > On Fri, Oct 14, 2022 at 04:16:33PM +0100, Douglas Raillard wrote:
> > > Hi everyone,
> > > 
> > > The idea of having some Rust interop for libtraceevent was floated during the
> > > Tracing Summit that took place this week.
> > > 
> > > The overall work has two scopes:
> > >      1. libtraceevent: low-level helpers to decode event formats and binary buffer pages.
> > >      2. high level API that e.g. can consume a trace.dat file and expose an iterator API etc.
> > > 
> > > The current discussion is only about #1. The 2nd part of the work has an
> > > entirely different set of constraints as it would aim at integrating in the
> > > Rust ecosystem as well as possible without trying to replace anything
> > > pre-existent.
> > Has anyone actually started working on these?
> 
> We have, this has been my main focus since around November.
> 
> > 
> > I have a Rust tool that parses the output from trace-cmd to infer the
> > internal states of a kernel module. While it serves our needs for now,
> > Steven kindly pointed me to this thread and suggested interacting
> > directly with the tracefs and using the tracefs libraries for parsing.
> > 
> > So I'd like to know:
> > - if this discussion reached a consensus
> 
> You are the first one to answer on that thread but the twitter poll from Steven was encouraging. At any
> rate, we need it for LISA [1] so I'm making it happen.
> 
> > - if there is any publicly available work already
> 
> Yes and no: I'm currently developing it as part of LISA but in its own "traceevent" crate,
> as added by that PR under tools/analysis/traceevent/:
> https://github.com/ARM-software/lisa/pull/1843

Thanks! Let me take a look.

Do you intend to keep this inside the LISA project repository?

The LISA project per se is interesting, but I feel your work is also
valuable for other projects. Is there any chance that this becomes a
standalone crate?

> This is very much WIP, so far I got:
> * A parser for v6 header up to and including the event formats.
>   Since that requires a C parser, it was a large chunk of the total work.
> * A C interpreter as (somewhat) required for array lengths and mostly for the print fmt args.
> * a zero-copy I/O layer that can load data from:
> 	* existing in-memory buffer
> 	* Any type implementing Read + Seek
> 	* mmap
> 
> Adding support for v7 header should be straightforward except for the I/O that might require adjustment
> depending on how compression is implemented.
> 
> 
> My current roadmap is:
> 1. Get to a state where I can share a trace-cmd report equivalent static tool for people to play with on
>    their own data and report issues.
> 2. Make a serde backend hooked to it since we need it for LISA (assuming performance is good enough)
> 3. Figure out exactly how it sits in the libtraceevent ecosystem and maintains it in the
>    long run, libtraceevent-style C bindings etc.
> 
> Note that if and when a serde backend is available, that should bring a conversion to JSON for basically free.
> 
> Unless I'm sidetracked by urgent things, 1. and 2. should become available in a few months. I expect 3. to happen
> when I have something concrete to show to the world.
> 
> [1] LISA: https://github.com/ARM-software/lisa
> > 
> > Thanks,
> > Hidenori
> 
> Thanks,
> 
> Douglas

Thanks,
Hidenori

Re: libtraceevent + Rust

[Douglas Raillard]

On 14-02-2023 03:56, Hidenori Kobayashi wrote:
> Hi Douglas,
> 
> On Mon, Feb 13, 2023 at 12:04:36PM +0000, Douglas Raillard wrote:
>> Hi Hidenori,
>>
>>> On 13-02-2023 05:17, Hidenori Kobayashi wrote:
>>> Hi Douglas,
>>>
>>> On Fri, Oct 14, 2022 at 04:16:33PM +0100, Douglas Raillard wrote:
>>>> Hi everyone,
>>>>
>>>> The idea of having some Rust interop for libtraceevent was floated during the
>>>> Tracing Summit that took place this week.
>>>>
>>>> The overall work has two scopes:
>>>>       1. libtraceevent: low-level helpers to decode event formats and binary buffer pages.
>>>>       2. high level API that e.g. can consume a trace.dat file and expose an iterator API etc.
>>>>
>>>> The current discussion is only about #1. The 2nd part of the work has an
>>>> entirely different set of constraints as it would aim at integrating in the
>>>> Rust ecosystem as well as possible without trying to replace anything
>>>> pre-existent.
>>> Has anyone actually started working on these?
>>
>> We have, this has been my main focus since around November.
>>
>>>
>>> I have a Rust tool that parses the output from trace-cmd to infer the
>>> internal states of a kernel module. While it serves our needs for now,
>>> Steven kindly pointed me to this thread and suggested interacting
>>> directly with the tracefs and using the tracefs libraries for parsing.
>>>
>>> So I'd like to know:
>>> - if this discussion reached a consensus
>>
>> You are the first one to answer on that thread but the twitter poll from Steven was encouraging. At any
>> rate, we need it for LISA [1] so I'm making it happen.
>>
>>> - if there is any publicly available work already
>>
>> Yes and no: I'm currently developing it as part of LISA but in its own "traceevent" crate,
>> as added by that PR under tools/analysis/traceevent/:
>> https://github.com/ARM-software/lisa/pull/1843
> 
> Thanks! Let me take a look.
> 
> Do you intend to keep this inside the LISA project repository?
> 
> The LISA project per se is interesting, but I feel your work is also
> valuable for other projects. Is there any chance that this becomes a
> standalone crate?

Yes, LISA is not necessarily the final location of that piece of code. I've made it
as a separate crate from the other Rust bits so it can be split effortlessly when the time comes.

> 
>> This is very much WIP, so far I got:
>> * A parser for v6 header up to and including the event formats.
>>    Since that requires a C parser, it was a large chunk of the total work.
>> * A C interpreter as (somewhat) required for array lengths and mostly for the print fmt args.
>> * a zero-copy I/O layer that can load data from:
>> 	* existing in-memory buffer
>> 	* Any type implementing Read + Seek
>> 	* mmap
>>
>> Adding support for v7 header should be straightforward except for the I/O that might require adjustment
>> depending on how compression is implemented.
>>
>>
>> My current roadmap is:
>> 1. Get to a state where I can share a trace-cmd report equivalent static tool for people to play with on
>>     their own data and report issues.
>> 2. Make a serde backend hooked to it since we need it for LISA (assuming performance is good enough)
>> 3. Figure out exactly how it sits in the libtraceevent ecosystem and maintains it in the
>>     long run, libtraceevent-style C bindings etc.
>>
>> Note that if and when a serde backend is available, that should bring a conversion to JSON for basically free.
>>
>> Unless I'm sidetracked by urgent things, 1. and 2. should become available in a few months. I expect 3. to happen
>> when I have something concrete to show to the world.
>>
>> [1] LISA: https://github.com/ARM-software/lisa
>>>
>>> Thanks,
>>> Hidenori
>>
>> Thanks,
>>
>> Douglas
> 
> Thanks,
> Hidenori

Re: libtraceevent + Rust

[Hidenori Kobayashi]

Hi Douglas,

On Tue, Feb 14, 2023 at 10:38:52AM +0000, Douglas Raillard wrote:
> On 14-02-2023 03:56, Hidenori Kobayashi wrote:
> > Hi Douglas,
> > 
> > On Mon, Feb 13, 2023 at 12:04:36PM +0000, Douglas Raillard wrote:
> > > Hi Hidenori,
> > > 
> > > > On 13-02-2023 05:17, Hidenori Kobayashi wrote:
> > > > Hi Douglas,
> > > > 
> > > > On Fri, Oct 14, 2022 at 04:16:33PM +0100, Douglas Raillard wrote:
> > > > > Hi everyone,
> > > > > 
> > > > > The idea of having some Rust interop for libtraceevent was floated during the
> > > > > Tracing Summit that took place this week.
> > > > > 
> > > > > The overall work has two scopes:
> > > > >       1. libtraceevent: low-level helpers to decode event formats and binary buffer pages.
> > > > >       2. high level API that e.g. can consume a trace.dat file and expose an iterator API etc.
> > > > > 
> > > > > The current discussion is only about #1. The 2nd part of the work has an
> > > > > entirely different set of constraints as it would aim at integrating in the
> > > > > Rust ecosystem as well as possible without trying to replace anything
> > > > > pre-existent.
> > > > Has anyone actually started working on these?
> > > 
> > > We have, this has been my main focus since around November.
> > > 
> > > > 
> > > > I have a Rust tool that parses the output from trace-cmd to infer the
> > > > internal states of a kernel module. While it serves our needs for now,
> > > > Steven kindly pointed me to this thread and suggested interacting
> > > > directly with the tracefs and using the tracefs libraries for parsing.
> > > > 
> > > > So I'd like to know:
> > > > - if this discussion reached a consensus
> > > 
> > > You are the first one to answer on that thread but the twitter poll from Steven was encouraging. At any
> > > rate, we need it for LISA [1] so I'm making it happen.
> > > 
> > > > - if there is any publicly available work already
> > > 
> > > Yes and no: I'm currently developing it as part of LISA but in its own "traceevent" crate,
> > > as added by that PR under tools/analysis/traceevent/:
> > > https://github.com/ARM-software/lisa/pull/1843
> > 
> > Thanks! Let me take a look.
> > 
> > Do you intend to keep this inside the LISA project repository?
> > 
> > The LISA project per se is interesting, but I feel your work is also
> > valuable for other projects. Is there any chance that this becomes a
> > standalone crate?
> 
> Yes, LISA is not necessarily the final location of that piece of code. I've made it
> as a separate crate from the other Rust bits so it can be split effortlessly when the time comes.
 
Looking forward to the upcoming announcement!

> > 
> > > This is very much WIP, so far I got:
> > > * A parser for v6 header up to and including the event formats.
> > >    Since that requires a C parser, it was a large chunk of the total work.
> > > * A C interpreter as (somewhat) required for array lengths and mostly for the print fmt args.
> > > * a zero-copy I/O layer that can load data from:
> > > 	* existing in-memory buffer
> > > 	* Any type implementing Read + Seek
> > > 	* mmap
> > > 
> > > Adding support for v7 header should be straightforward except for the I/O that might require adjustment
> > > depending on how compression is implemented.
> > > 
> > > 
> > > My current roadmap is:
> > > 1. Get to a state where I can share a trace-cmd report equivalent static tool for people to play with on
> > >     their own data and report issues.
> > > 2. Make a serde backend hooked to it since we need it for LISA (assuming performance is good enough)
> > > 3. Figure out exactly how it sits in the libtraceevent ecosystem and maintains it in the
> > >     long run, libtraceevent-style C bindings etc.
> > > 
> > > Note that if and when a serde backend is available, that should bring a conversion to JSON for basically free.
> > > 
> > > Unless I'm sidetracked by urgent things, 1. and 2. should become available in a few months. I expect 3. to happen
> > > when I have something concrete to show to the world.
> > > 
> > > [1] LISA: https://github.com/ARM-software/lisa
> > > > 
> > > > Thanks,
> > > > Hidenori
> > > 
> > > Thanks,
> > > 
> > > Douglas
> > 
> > Thanks,
> > Hidenori
> 

Re: libtraceevent + Rust

[Douglas Raillard]

Hi Hidenori,

On 15-02-2023 00:36, Hidenori Kobayashi wrote:

> Looking forward to the upcoming announcement!

I've just sent it to trace-devel if you are still interested in these matters.

Thanks,

Douglas