[diamon-discuss] Diamon CTF Roadmap?

[diamon-discuss] Diamon CTF Roadmap?

[Wang Nan]

Hi everyone,

Looks like my post will be the second thread in this list. That's great.

I'll be the technical contact person representing Huawei for Diamon.
I'm glad to work with this new workgroup to create "de-facto, widely
used open source trace technologies".

Beyond sharing new ideas and technologies, our main concern on Diamon
now are CTF tracing format and related things for analysing traces from
different sources in a unified manner. I have read Jiri Olsa's slides in
TracingSummit 2014, and also begin to read his code in
https://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git. However,
it looks like that there's only a trace generating tool is under
developing. What about upper trace parsing and analysing tools? Is there
any roadmap on it? How can we participate in the project?

Thanks.

Re: [diamon-discuss] Diamon CTF Roadmap?

[Dominique Toupin]

Hi Nan,

Many tools are able to handle CTF today, e.g.:
- GDB Tracepoint https://sourceware.org/gdb/onlinedocs/gdb/Trace-Files.html#Trace-Files
- LTTng kernel and LTTng user space C/C++, Java, Python, etc. https://lttng.org/download
- babel trace, http://lttng.org/viewers/ 
- Trace Compass http://projects.eclipse.org/projects/tools.tracecompass 
- CTF tracer for bare metal https://github.com/efficios/barectf/blob/master/README.md

You will also find many converter to CTF, e.g. NEXUS HW Trace, perf, 
And I am most likely missing a few.

If you can provide a description of your needs it would help to give you more info related to your use case.
e.g. is it more related to cloud, embedded data base, real time, or do you have a specific view or analysis you want to see in the trace viewer, etc.

Thanks for the second post!

Dominique

-----Original Message-----
From: Wang Nan [mailto:wangnan0@huawei.com] 
Sent: November-25-14 8:04 AM
To: diamon-discuss@lists.linuxfoundation.org; Dominique Toupin; Jiri Olsa
Cc: genghui 00204690; Xinwei Hu
Subject: Diamon CTF Roadmap?

Hi everyone,

Looks like my post will be the second thread in this list. That's great.

I'll be the technical contact person representing Huawei for Diamon.
I'm glad to work with this new workgroup to create "de-facto, widely used open source trace technologies".

Beyond sharing new ideas and technologies, our main concern on Diamon now are CTF tracing format and related things for analysing traces from different sources in a unified manner. I have read Jiri Olsa's slides in TracingSummit 2014, and also begin to read his code in https://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git. However, it looks like that there's only a trace generating tool is under developing. What about upper trace parsing and analysing tools? Is there any roadmap on it? How can we participate in the project?

Thanks.

Re: [diamon-discuss] Diamon CTF Roadmap?

[Wang Nan]

On 2014/11/25 21:42, Dominique Toupin wrote:
> Hi Nan,
> 
> Many tools are able to handle CTF today, e.g.:
> - GDB Tracepoint https://sourceware.org/gdb/onlinedocs/gdb/Trace-Files.html#Trace-Files
> - LTTng kernel and LTTng user space C/C++, Java, Python, etc. https://lttng.org/download
> - babel trace, http://lttng.org/viewers/ 
> - Trace Compass http://projects.eclipse.org/projects/tools.tracecompass 
> - CTF tracer for bare metal https://github.com/efficios/barectf/blob/master/README.md
> 
> You will also find many converter to CTF, e.g. NEXUS HW Trace, perf, 
> And I am most likely missing a few.
> 
> If you can provide a description of your needs it would help to give you more info related to your use case.
> e.g. is it more related to cloud, embedded data base, real time, or do you have a specific view or analysis you want to see in the trace viewer, etc.
> 

Our use cases are diverse, both embedded and server scene have requirements on profiling,
and methodologies are also different.

We have two starting points now:

1. On ARM (32/64) platform, extracing information (instruction footprints) from CoreSight
   using perf and use CTF as trace format, to make it possible to analysis with trace from
   upper layer. Intel has similar hardware too, but currently only on specification.

2. On server field, providing some interactive, graphical user interfaces for some generic
   problems. Android's SysTrace is a good example. However, our the real goal is to create
   a tool for dealing with problems not so generic. Profiling using a language like SQL maybe,
   like Splunk, to synthesize traces from different layers and components. So the underlying
   traces are better to be in a uniform format.

We also concern the upstreaming plan about kernel stuffs in DiaMon.

> Thanks for the second post!
> 
> Dominique
> 
> -----Original Message-----
> From: Wang Nan [mailto:wangnan0@huawei.com] 
> Sent: November-25-14 8:04 AM
> To: diamon-discuss@lists.linuxfoundation.org; Dominique Toupin; Jiri Olsa
> Cc: genghui 00204690; Xinwei Hu
> Subject: Diamon CTF Roadmap?
> 
> Hi everyone,
> 
> Looks like my post will be the second thread in this list. That's great.
> 
> I'll be the technical contact person representing Huawei for Diamon.
> I'm glad to work with this new workgroup to create "de-facto, widely used open source trace technologies".
> 
> Beyond sharing new ideas and technologies, our main concern on Diamon now are CTF tracing format and related things for analysing traces from different sources in a unified manner. I have read Jiri Olsa's slides in TracingSummit 2014, and also begin to read his code in https://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git. However, it looks like that there's only a trace generating tool is under developing. What about upper trace parsing and analysing tools? Is there any roadmap on it? How can we participate in the project?
> 
> Thanks.
> 

Re: [diamon-discuss] Diamon CTF Roadmap?

[Dominique Toupin]

You can see my answer below preceded by [Dominique Toupin]

-----Original Message-----
From: Wang Nan [mailto:wangnan0@huawei.com] 
Sent: November-25-14 9:08 PM
To: Dominique Toupin; diamon-discuss@lists.linuxfoundation.org; Jiri Olsa
Cc: genghui 00204690; Xinwei Hu
Subject: Re: Diamon CTF Roadmap?

On 2014/11/25 21:42, Dominique Toupin wrote:
> Hi Nan,
> 
> Many tools are able to handle CTF today, e.g.:
> - GDB Tracepoint 
> https://sourceware.org/gdb/onlinedocs/gdb/Trace-Files.html#Trace-Files
> - LTTng kernel and LTTng user space C/C++, Java, Python, etc. 
> https://lttng.org/download
> - babel trace, http://lttng.org/viewers/
> - Trace Compass 
> http://projects.eclipse.org/projects/tools.tracecompass
> - CTF tracer for bare metal 
> https://github.com/efficios/barectf/blob/master/README.md
> 
> You will also find many converter to CTF, e.g. NEXUS HW Trace, perf, 
> And I am most likely missing a few.
> 
> If you can provide a description of your needs it would help to give you more info related to your use case.
> e.g. is it more related to cloud, embedded data base, real time, or do you have a specific view or analysis you want to see in the trace viewer, etc.
> 

Our use cases are diverse, both embedded and server scene have requirements on profiling, and methodologies are also different.

We have two starting points now:

1. On ARM (32/64) platform, extracing information (instruction footprints) from CoreSight
   using perf and use CTF as trace format, to make it possible to analysis with trace from
   upper layer. Intel has similar hardware too, but currently only on specification.

[Dominique Toupin] I believe they are able to get perf data in CTF and if you also have 
CTF trace from upper layer then you can use Trace Compass (python should also be an option) 
to visualize, correlate, analyze trace data from different layers, you can also open text logs in 
Trace Compass. In the upper layer, you can take a similar approach as the Linux kernel where 
static tracepoints represent the wisdom of developers who are most familiar with the code and 
the rest of the world (testers, system administrators, field engineers, developers) can use them 
to extract a great deal of useful information without having to know the code. 

2. On server field, providing some interactive, graphical user interfaces for some generic
   problems. Android's SysTrace is a good example. However, our the real goal is to create
   a tool for dealing with problems not so generic. Profiling using a language like SQL maybe,
   like Splunk, to synthesize traces from different layers and components. So the underlying
   traces are better to be in a uniform format.

[Dominique Toupin] SQL can be nice but it requires a lot more space, if I remember 
correctly, the size is sometimes multiplied by 70 compared to CTF. CTF is like a specialized 
trace data base, in Trace Compass they have efficient query to navigate in the trace, they
 add a state system, etc. You can do all this with SQL but it require a lot more space and it is slower.
If you see things missing in CTF compared to what you want to do with SQL, let us know we can 
investigate how to add those features directly in CTF.


We also concern the upstreaming plan about kernel stuffs in DiaMon.

> Thanks for the second post!
> 
> Dominique
> 
> -----Original Message-----
> From: Wang Nan [mailto:wangnan0@huawei.com]
> Sent: November-25-14 8:04 AM
> To: diamon-discuss@lists.linuxfoundation.org; Dominique Toupin; Jiri 
> Olsa
> Cc: genghui 00204690; Xinwei Hu
> Subject: Diamon CTF Roadmap?
> 
> Hi everyone,
> 
> Looks like my post will be the second thread in this list. That's great.
> 
> I'll be the technical contact person representing Huawei for Diamon.
> I'm glad to work with this new workgroup to create "de-facto, widely used open source trace technologies".
> 
> Beyond sharing new ideas and technologies, our main concern on Diamon now are CTF tracing format and related things for analysing traces from different sources in a unified manner. I have read Jiri Olsa's slides in TracingSummit 2014, and also begin to read his code in https://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git. However, it looks like that there's only a trace generating tool is under developing. What about upper trace parsing and analysing tools? Is there any roadmap on it? How can we participate in the project?
> 
> Thanks.
> 

Re: [diamon-discuss] Diamon CTF Roadmap?

[Wang Nan]

On 2014/11/27 3:43, Dominique Toupin wrote:
> You can see my answer below preceded by [Dominique Toupin]
> 
> -----Original Message-----
> From: Wang Nan [mailto:wangnan0@huawei.com] 
> Sent: November-25-14 9:08 PM
> To: Dominique Toupin; diamon-discuss@lists.linuxfoundation.org; Jiri Olsa
> Cc: genghui 00204690; Xinwei Hu
> Subject: Re: Diamon CTF Roadmap?
> 
> On 2014/11/25 21:42, Dominique Toupin wrote:
>> Hi Nan,
>>
>> Many tools are able to handle CTF today, e.g.:
>> - GDB Tracepoint 
>> https://sourceware.org/gdb/onlinedocs/gdb/Trace-Files.html#Trace-Files
>> - LTTng kernel and LTTng user space C/C++, Java, Python, etc. 
>> https://lttng.org/download
>> - babel trace, http://lttng.org/viewers/
>> - Trace Compass 
>> http://projects.eclipse.org/projects/tools.tracecompass
>> - CTF tracer for bare metal 
>> https://github.com/efficios/barectf/blob/master/README.md
>>
>> You will also find many converter to CTF, e.g. NEXUS HW Trace, perf, 
>> And I am most likely missing a few.
>>
>> If you can provide a description of your needs it would help to give you more info related to your use case.
>> e.g. is it more related to cloud, embedded data base, real time, or do you have a specific view or analysis you want to see in the trace viewer, etc.
>>
> 
> Our use cases are diverse, both embedded and server scene have requirements on profiling, and methodologies are also different.
> 
> We have two starting points now:
> 
> 1. On ARM (32/64) platform, extracing information (instruction footprints) from CoreSight
>    using perf and use CTF as trace format, to make it possible to analysis with trace from
>    upper layer. Intel has similar hardware too, but currently only on specification.
> 
> [Dominique Toupin] I believe they are able to get perf data in CTF and if you also have 
> CTF trace from upper layer then you can use Trace Compass (python should also be an option) 
> to visualize, correlate, analyze trace data from different layers, you can also open text logs in 
> Trace Compass. In the upper layer, you can take a similar approach as the Linux kernel where 
> static tracepoints represent the wisdom of developers who are most familiar with the code and 
> the rest of the world (testers, system administrators, field engineers, developers) can use them 
> to extract a great deal of useful information without having to know the code. 
> 
> 2. On server field, providing some interactive, graphical user interfaces for some generic
>    problems. Android's SysTrace is a good example. However, our the real goal is to create
>    a tool for dealing with problems not so generic. Profiling using a language like SQL maybe,
>    like Splunk, to synthesize traces from different layers and components. So the underlying
>    traces are better to be in a uniform format.
> 
> [Dominique Toupin] SQL can be nice but it requires a lot more space, if I remember 
> correctly, the size is sometimes multiplied by 70 compared to CTF. CTF is like a specialized 
> trace data base, in Trace Compass they have efficient query to navigate in the trace, they
>  add a state system, etc. You can do all this with SQL but it require a lot more space and it is slower.
> If you see things missing in CTF compared to what you want to do with SQL, let us know we can 
> investigate how to add those features directly in CTF.
> 

Thanks for your reply. I'll try Trace Compass in our situation.

In addition, is there any description on the organization and policy of DiaMon?

> 
> We also concern the upstreaming plan about kernel stuffs in DiaMon.
> 
>> Thanks for the second post!
>>
>> Dominique
>>
>> -----Original Message-----
>> From: Wang Nan [mailto:wangnan0@huawei.com]
>> Sent: November-25-14 8:04 AM
>> To: diamon-discuss@lists.linuxfoundation.org; Dominique Toupin; Jiri 
>> Olsa
>> Cc: genghui 00204690; Xinwei Hu
>> Subject: Diamon CTF Roadmap?
>>
>> Hi everyone,
>>
>> Looks like my post will be the second thread in this list. That's great.
>>
>> I'll be the technical contact person representing Huawei for Diamon.
>> I'm glad to work with this new workgroup to create "de-facto, widely used open source trace technologies".
>>
>> Beyond sharing new ideas and technologies, our main concern on Diamon now are CTF tracing format and related things for analysing traces from different sources in a unified manner. I have read Jiri Olsa's slides in TracingSummit 2014, and also begin to read his code in https://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git. However, it looks like that there's only a trace generating tool is under developing. What about upper trace parsing and analysing tools? Is there any roadmap on it? How can we participate in the project?
>>
>> Thanks.
>>
> 
> 

Re: [diamon-discuss] Diamon CTF Roadmap?

[Dominique Toupin]

You can try Trace Compass and also the cmd line/python scripts as they can even be used in some context on the target, e.g. in a cloud 

Regarding the governance of DiaMon it is the simplest form of a workgroup at the Linux foundation, you can find it at http://diamon.org/projects/governance/
In summary anyone can participate and optionally if you want your company logo on diamon.org you can send the request to Mike Dolan <mdolan@linuxfoundation.org>

If you have specific layer that you want to get instrumented with static tracepoint, let us know we should be able to help.


-----Original Message-----
From: Wang Nan [mailto:wangnan0@huawei.com] 
Sent: November-26-14 8:52 PM
To: Dominique Toupin; diamon-discuss@lists.linuxfoundation.org; Jiri Olsa
Cc: genghui 00204690; Xinwei Hu
Subject: Re: Diamon CTF Roadmap?




On 2014/11/27 3:43, Dominique Toupin wrote:
> You can see my answer below preceded by [Dominique Toupin]
> 
> -----Original Message-----
> From: Wang Nan [mailto:wangnan0@huawei.com]
> Sent: November-25-14 9:08 PM
> To: Dominique Toupin; diamon-discuss@lists.linuxfoundation.org; Jiri 
> Olsa
> Cc: genghui 00204690; Xinwei Hu
> Subject: Re: Diamon CTF Roadmap?
> 
> On 2014/11/25 21:42, Dominique Toupin wrote:
>> Hi Nan,
>>
>> Many tools are able to handle CTF today, e.g.:
>> - GDB Tracepoint
>> https://sourceware.org/gdb/onlinedocs/gdb/Trace-Files.html#Trace-File
>> s
>> - LTTng kernel and LTTng user space C/C++, Java, Python, etc. 
>> https://lttng.org/download
>> - babel trace, http://lttng.org/viewers/
>> - Trace Compass
>> http://projects.eclipse.org/projects/tools.tracecompass
>> - CTF tracer for bare metal
>> https://github.com/efficios/barectf/blob/master/README.md
>>
>> You will also find many converter to CTF, e.g. NEXUS HW Trace, perf, 
>> And I am most likely missing a few.
>>
>> If you can provide a description of your needs it would help to give you more info related to your use case.
>> e.g. is it more related to cloud, embedded data base, real time, or do you have a specific view or analysis you want to see in the trace viewer, etc.
>>
> 
> Our use cases are diverse, both embedded and server scene have requirements on profiling, and methodologies are also different.
> 
> We have two starting points now:
> 
> 1. On ARM (32/64) platform, extracing information (instruction footprints) from CoreSight
>    using perf and use CTF as trace format, to make it possible to analysis with trace from
>    upper layer. Intel has similar hardware too, but currently only on specification.
> 
> [Dominique Toupin] I believe they are able to get perf data in CTF and 
> if you also have CTF trace from upper layer then you can use Trace 
> Compass (python should also be an option) to visualize, correlate, 
> analyze trace data from different layers, you can also open text logs 
> in Trace Compass. In the upper layer, you can take a similar approach 
> as the Linux kernel where static tracepoints represent the wisdom of 
> developers who are most familiar with the code and the rest of the world (testers, system administrators, field engineers, developers) can use them to extract a great deal of useful information without having to know the code.
> 
> 2. On server field, providing some interactive, graphical user interfaces for some generic
>    problems. Android's SysTrace is a good example. However, our the real goal is to create
>    a tool for dealing with problems not so generic. Profiling using a language like SQL maybe,
>    like Splunk, to synthesize traces from different layers and components. So the underlying
>    traces are better to be in a uniform format.
> 
> [Dominique Toupin] SQL can be nice but it requires a lot more space, 
> if I remember correctly, the size is sometimes multiplied by 70 
> compared to CTF. CTF is like a specialized trace data base, in Trace 
> Compass they have efficient query to navigate in the trace, they  add a state system, etc. You can do all this with SQL but it require a lot more space and it is slower.
> If you see things missing in CTF compared to what you want to do with 
> SQL, let us know we can investigate how to add those features directly in CTF.
> 

Thanks for your reply. I'll try Trace Compass in our situation.

In addition, is there any description on the organization and policy of DiaMon?

> 
> We also concern the upstreaming plan about kernel stuffs in DiaMon.
> 
>> Thanks for the second post!
>>
>> Dominique
>>
>> -----Original Message-----
>> From: Wang Nan [mailto:wangnan0@huawei.com]
>> Sent: November-25-14 8:04 AM
>> To: diamon-discuss@lists.linuxfoundation.org; Dominique Toupin; Jiri 
>> Olsa
>> Cc: genghui 00204690; Xinwei Hu
>> Subject: Diamon CTF Roadmap?
>>
>> Hi everyone,
>>
>> Looks like my post will be the second thread in this list. That's great.
>>
>> I'll be the technical contact person representing Huawei for Diamon.
>> I'm glad to work with this new workgroup to create "de-facto, widely used open source trace technologies".
>>
>> Beyond sharing new ideas and technologies, our main concern on Diamon now are CTF tracing format and related things for analysing traces from different sources in a unified manner. I have read Jiri Olsa's slides in TracingSummit 2014, and also begin to read his code in https://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git. However, it looks like that there's only a trace generating tool is under developing. What about upper trace parsing and analysing tools? Is there any roadmap on it? How can we participate in the project?
>>
>> Thanks.
>>
> 
>