[lttng-dev] C API message iterator example for libbabeltrace2

[lttng-dev] C API message iterator example for libbabeltrace2

[Maksim Khmelevskiy via lttng-dev]

[-- Attachment #1.1: Type: text/plain, Size: 558 bytes --]

Hi,

there is a nice py message iterator example
<https://babeltrace.org/docs/v2.0/python/bt2/examples.html> but for C API
only plugins are covered with examples,
do you think it would make sense to create an example of a standalone
application which simply uses `source.ctf.fs` as source and iterates over
all messages? It would be nice hint for those who want to see an example of
graph creation with all the code in one file.
I'm interested in that example because I want to transform CTF file into
list of C structures representing messages.

Thank you!

[-- Attachment #1.2: Type: text/html, Size: 665 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] C API message iterator example for libbabeltrace2

[Simon Marchi via lttng-dev]

On 8/12/22 09:19, Maksim Khmelevskiy via lttng-dev wrote:
> Hi,
>

> there is a nice py message iterator example
> <https://babeltrace.org/docs/v2.0/python/bt2/examples.html> but for C
> API only plugins are covered with examples, do you think it would make
> sense to create an example of a standalone application which simply
> uses `source.ctf.fs` as source and iterates over all messages? It
> would be nice hint for those who want to see an example of graph
> creation with all the code in one file.

Hi Maksim,

I'm not sure which example you are referring to exactly.  But in Python,
we have the high-level TraceCollectionMessageIterator object, which does
roughly:

 - Instantiate source and filter components according to the provided
   specs, including automatic source discovery
 - Instantiate a flt.utils.muxer component to merge the streams from all
   sources ports
 - Instantiate a sink component that presents the events as a Python
   iterable
 - Connect the ports of all these components
 - More things that are irrelevant here

There is no such high-level object in the C interface, so you have
to do all this by hand, it will necessarily be much more verbose.  It
would be nice to have the equivalent of TraceCollectionMessageIterator
in the C API, it is just not done yet due to lack of resources.

I did search in the documentation for an example program that uses the C
API to create and run a graph, and I haven't found one.  I agree that
adding one would be nice.  I'll look into writing one.

Regarding your use case:

> I'm interested in that example because I want to transform CTF file
> into list of C structures representing messages.

I have some questions:

 - Is the data you want to convert found in the metadata of the traces
   (description of event types) of in the payload of events?
 - Why do you want to write this in C instead of Python?  It sounds like
   it would be much faster to write in Python (with
   TraceCollectionMessageIterator), and it doesn't sound like something
   where the performance is critical (but of course I don't have the
   full context).
 - Why do you need to write an application where you create and run the
   graph yourself?  Could you instead just write your sink component
   class (which reads the messages and writes your output files),
   packaged in a plugin and use it through the babeltrace2 command-line:

     $ babeltrace2 /path/to/ctf/trace -c sink.foo.bar -p 'output="out.h"'

   ?  This way, you just have to care about writing your component
   class, which does the conversion you need.

Simon
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] C API message iterator example for libbabeltrace2

[Maksim Khmelevskiy via lttng-dev]

[-- Attachment #1.1: Type: text/plain, Size: 3705 bytes --]

Hi Simon,
thanks for the reply!
What I came up with for now is this example
<https://github.com/ImMax/babeltrace/commit/2082a0f4d77d2edeec6fb95c308a79fb313f7a02>.
It's probably has a lot of design mistakes but it's at least runnable.
I also see that examples for C API aren't build, but maybe I'm wrong.
Another thing, if you have lack of resources I would be happy to help, I
could make a PR with your guidance and review.

Answers to the questions:
1) Not sure that I completely understand the question, I wanted to parse
events(name, fields), not the metadata file aligned with the CTF trace file.
2) Because I wanted to get C-structs directly from the CTF traces. I'm sure
that it's very niche requirement, sane people would not need it.
3) I would do something like that, but I have a requirement of providing C
structs. I guess to apply filters or do something else with traces (I'm not
sure, not my idea, but I also find it weird)


On Fri, 19 Aug 2022 at 17:00, Simon Marchi <simark@simark.ca> wrote:

> On 8/12/22 09:19, Maksim Khmelevskiy via lttng-dev wrote:
> > Hi,
> >
>
> > there is a nice py message iterator example
> > <https://babeltrace.org/docs/v2.0/python/bt2/examples.html> but for C
> > API only plugins are covered with examples, do you think it would make
> > sense to create an example of a standalone application which simply
> > uses `source.ctf.fs` as source and iterates over all messages? It
> > would be nice hint for those who want to see an example of graph
> > creation with all the code in one file.
>
> Hi Maksim,
>
> I'm not sure which example you are referring to exactly.  But in Python,
> we have the high-level TraceCollectionMessageIterator object, which does
> roughly:
>
>  - Instantiate source and filter components according to the provided
>    specs, including automatic source discovery
>  - Instantiate a flt.utils.muxer component to merge the streams from all
>    sources ports
>  - Instantiate a sink component that presents the events as a Python
>    iterable
>  - Connect the ports of all these components
>  - More things that are irrelevant here
>
> There is no such high-level object in the C interface, so you have
> to do all this by hand, it will necessarily be much more verbose.  It
> would be nice to have the equivalent of TraceCollectionMessageIterator
> in the C API, it is just not done yet due to lack of resources.
>
> I did search in the documentation for an example program that uses the C
> API to create and run a graph, and I haven't found one.  I agree that
> adding one would be nice.  I'll look into writing one.
>
> Regarding your use case:
>
> > I'm interested in that example because I want to transform CTF file
> > into list of C structures representing messages.
>
> I have some questions:
>
>  - Is the data you want to convert found in the metadata of the traces
>    (description of event types) of in the payload of events?
>  - Why do you want to write this in C instead of Python?  It sounds like
>    it would be much faster to write in Python (with
>    TraceCollectionMessageIterator), and it doesn't sound like something
>    where the performance is critical (but of course I don't have the
>    full context).
>  - Why do you need to write an application where you create and run the
>    graph yourself?  Could you instead just write your sink component
>    class (which reads the messages and writes your output files),
>    packaged in a plugin and use it through the babeltrace2 command-line:
>
>      $ babeltrace2 /path/to/ctf/trace -c sink.foo.bar -p 'output="out.h"'
>
>    ?  This way, you just have to care about writing your component
>    class, which does the conversion you need.
>
> Simon
>

[-- Attachment #1.2: Type: text/html, Size: 4740 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] C API message iterator example for libbabeltrace2

[Simon Marchi via lttng-dev]

On 8/20/22 12:14, Maksim Khmelevskiy via lttng-dev wrote:
> Hi Simon,
> thanks for the reply!

> What I came up with for now is this example
> <https://github.com/ImMax/babeltrace/commit/2082a0f4d77d2edeec6fb95c308a79fb313f7a02>.
> It's probably has a lot of design mistakes but it's at least runnable.

Looks like a good prototype, it seems right.  Some hopefully
constructive comments:

 - You can probably use bt_plugin_find to find a plugin by name instead
   of your code that does it by hand
 - When you instantiate a src.ctf.fs component, it may have more than
   one port (there is one port per data stream), in which case you want
   to connect them all to a flt.utils.muxer component (which is what the
   CLI and the Python TraceCollectionMessageIterator class do) to funnel
   all the messages to a single output port.  Of course, if you know
   the CTF traces you deal with have a single data stream, that is not
   necessary.

> I also see that examples for C API aren't build, but maybe I'm wrong.

C API examples in the documentation indeed aren't built.

> Another thing, if you have lack of resources I would be happy to help, I could make a PR with your guidance and review.
>

> Answers to the questions:
> 1) Not sure that I completely understand the question, I wanted to parse events(name, fields), not the metadata file aligned with the CTF trace file.
> 2) Because I wanted to get C-structs directly from the CTF traces. I'm sure that it's very niche requirement, sane people would not need it.
> 3) I would do something like that, but I have a requirement of providing C structs. I guess to apply filters or do something else with traces (I'm not sure, not my idea, but I also find it weird)

Ok, I understand.  When you said you need to write "C structs", I
thought you meant that you needed to output some generated C code, for
instance convert event types to structures, for use in a program later.
But that's not the case, you want to take event payload and fill out
some structures at runtime, from what I can see.  That explains why you
want to use the C API and why you want to create and run a graph in your
own application.

Following your questions, I have written an example of creating and
running a graph, which I just submitted for review here:

  https://review.lttng.org/c/babeltrace/+/8741

It is roughly equivalent to this Python example:

  https://babeltrace.org/docs/v2.0/python/bt2/examples.html#build-and-run-a-trace-processing-graph

Simon
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev