* [diamon-discuss] Diamon CTF Roadmap? @ 2014-11-25 13:03 Wang Nan 2014-11-25 13:42 ` Dominique Toupin 0 siblings, 1 reply; 6+ messages in thread From: Wang Nan @ 2014-11-25 13:03 UTC (permalink / raw) To: diamon-discuss, dominique.toupin, Jiri Olsa; +Cc: Xinwei Hu, genghui 00204690 Hi everyone, Looks like my post will be the second thread in this list. That's great. I'll be the technical contact person representing Huawei for Diamon. I'm glad to work with this new workgroup to create "de-facto, widely used open source trace technologies". Beyond sharing new ideas and technologies, our main concern on Diamon now are CTF tracing format and related things for analysing traces from different sources in a unified manner. I have read Jiri Olsa's slides in TracingSummit 2014, and also begin to read his code in https://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git. However, it looks like that there's only a trace generating tool is under developing. What about upper trace parsing and analysing tools? Is there any roadmap on it? How can we participate in the project? Thanks. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [diamon-discuss] Diamon CTF Roadmap? 2014-11-25 13:03 [diamon-discuss] Diamon CTF Roadmap? Wang Nan @ 2014-11-25 13:42 ` Dominique Toupin 2014-11-26 2:08 ` Wang Nan 0 siblings, 1 reply; 6+ messages in thread From: Dominique Toupin @ 2014-11-25 13:42 UTC (permalink / raw) To: Wang Nan, diamon-discuss@lists.linuxfoundation.org, Jiri Olsa Cc: Xinwei Hu, genghui 00204690 Hi Nan, Many tools are able to handle CTF today, e.g.: - GDB Tracepoint https://sourceware.org/gdb/onlinedocs/gdb/Trace-Files.html#Trace-Files - LTTng kernel and LTTng user space C/C++, Java, Python, etc. https://lttng.org/download - babel trace, http://lttng.org/viewers/ - Trace Compass http://projects.eclipse.org/projects/tools.tracecompass - CTF tracer for bare metal https://github.com/efficios/barectf/blob/master/README.md You will also find many converter to CTF, e.g. NEXUS HW Trace, perf, And I am most likely missing a few. If you can provide a description of your needs it would help to give you more info related to your use case. e.g. is it more related to cloud, embedded data base, real time, or do you have a specific view or analysis you want to see in the trace viewer, etc. Thanks for the second post! Dominique -----Original Message----- From: Wang Nan [mailto:wangnan0@huawei.com] Sent: November-25-14 8:04 AM To: diamon-discuss@lists.linuxfoundation.org; Dominique Toupin; Jiri Olsa Cc: genghui 00204690; Xinwei Hu Subject: Diamon CTF Roadmap? Hi everyone, Looks like my post will be the second thread in this list. That's great. I'll be the technical contact person representing Huawei for Diamon. I'm glad to work with this new workgroup to create "de-facto, widely used open source trace technologies". Beyond sharing new ideas and technologies, our main concern on Diamon now are CTF tracing format and related things for analysing traces from different sources in a unified manner. I have read Jiri Olsa's slides in TracingSummit 2014, and also begin to read his code in https://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git. However, it looks like that there's only a trace generating tool is under developing. What about upper trace parsing and analysing tools? Is there any roadmap on it? How can we participate in the project? Thanks. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [diamon-discuss] Diamon CTF Roadmap? 2014-11-25 13:42 ` Dominique Toupin @ 2014-11-26 2:08 ` Wang Nan 2014-11-26 19:43 ` Dominique Toupin 0 siblings, 1 reply; 6+ messages in thread From: Wang Nan @ 2014-11-26 2:08 UTC (permalink / raw) To: Dominique Toupin, diamon-discuss@lists.linuxfoundation.org, Jiri Olsa Cc: Xinwei Hu, genghui 00204690 On 2014/11/25 21:42, Dominique Toupin wrote: > Hi Nan, > > Many tools are able to handle CTF today, e.g.: > - GDB Tracepoint https://sourceware.org/gdb/onlinedocs/gdb/Trace-Files.html#Trace-Files > - LTTng kernel and LTTng user space C/C++, Java, Python, etc. https://lttng.org/download > - babel trace, http://lttng.org/viewers/ > - Trace Compass http://projects.eclipse.org/projects/tools.tracecompass > - CTF tracer for bare metal https://github.com/efficios/barectf/blob/master/README.md > > You will also find many converter to CTF, e.g. NEXUS HW Trace, perf, > And I am most likely missing a few. > > If you can provide a description of your needs it would help to give you more info related to your use case. > e.g. is it more related to cloud, embedded data base, real time, or do you have a specific view or analysis you want to see in the trace viewer, etc. > Our use cases are diverse, both embedded and server scene have requirements on profiling, and methodologies are also different. We have two starting points now: 1. On ARM (32/64) platform, extracing information (instruction footprints) from CoreSight using perf and use CTF as trace format, to make it possible to analysis with trace from upper layer. Intel has similar hardware too, but currently only on specification. 2. On server field, providing some interactive, graphical user interfaces for some generic problems. Android's SysTrace is a good example. However, our the real goal is to create a tool for dealing with problems not so generic. Profiling using a language like SQL maybe, like Splunk, to synthesize traces from different layers and components. So the underlying traces are better to be in a uniform format. We also concern the upstreaming plan about kernel stuffs in DiaMon. > Thanks for the second post! > > Dominique > > -----Original Message----- > From: Wang Nan [mailto:wangnan0@huawei.com] > Sent: November-25-14 8:04 AM > To: diamon-discuss@lists.linuxfoundation.org; Dominique Toupin; Jiri Olsa > Cc: genghui 00204690; Xinwei Hu > Subject: Diamon CTF Roadmap? > > Hi everyone, > > Looks like my post will be the second thread in this list. That's great. > > I'll be the technical contact person representing Huawei for Diamon. > I'm glad to work with this new workgroup to create "de-facto, widely used open source trace technologies". > > Beyond sharing new ideas and technologies, our main concern on Diamon now are CTF tracing format and related things for analysing traces from different sources in a unified manner. I have read Jiri Olsa's slides in TracingSummit 2014, and also begin to read his code in https://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git. However, it looks like that there's only a trace generating tool is under developing. What about upper trace parsing and analysing tools? Is there any roadmap on it? How can we participate in the project? > > Thanks. > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [diamon-discuss] Diamon CTF Roadmap? 2014-11-26 2:08 ` Wang Nan @ 2014-11-26 19:43 ` Dominique Toupin 2014-11-27 1:51 ` Wang Nan 0 siblings, 1 reply; 6+ messages in thread From: Dominique Toupin @ 2014-11-26 19:43 UTC (permalink / raw) To: Wang Nan, diamon-discuss@lists.linuxfoundation.org, Jiri Olsa Cc: Xinwei Hu, genghui 00204690 You can see my answer below preceded by [Dominique Toupin] -----Original Message----- From: Wang Nan [mailto:wangnan0@huawei.com] Sent: November-25-14 9:08 PM To: Dominique Toupin; diamon-discuss@lists.linuxfoundation.org; Jiri Olsa Cc: genghui 00204690; Xinwei Hu Subject: Re: Diamon CTF Roadmap? On 2014/11/25 21:42, Dominique Toupin wrote: > Hi Nan, > > Many tools are able to handle CTF today, e.g.: > - GDB Tracepoint > https://sourceware.org/gdb/onlinedocs/gdb/Trace-Files.html#Trace-Files > - LTTng kernel and LTTng user space C/C++, Java, Python, etc. > https://lttng.org/download > - babel trace, http://lttng.org/viewers/ > - Trace Compass > http://projects.eclipse.org/projects/tools.tracecompass > - CTF tracer for bare metal > https://github.com/efficios/barectf/blob/master/README.md > > You will also find many converter to CTF, e.g. NEXUS HW Trace, perf, > And I am most likely missing a few. > > If you can provide a description of your needs it would help to give you more info related to your use case. > e.g. is it more related to cloud, embedded data base, real time, or do you have a specific view or analysis you want to see in the trace viewer, etc. > Our use cases are diverse, both embedded and server scene have requirements on profiling, and methodologies are also different. We have two starting points now: 1. On ARM (32/64) platform, extracing information (instruction footprints) from CoreSight using perf and use CTF as trace format, to make it possible to analysis with trace from upper layer. Intel has similar hardware too, but currently only on specification. [Dominique Toupin] I believe they are able to get perf data in CTF and if you also have CTF trace from upper layer then you can use Trace Compass (python should also be an option) to visualize, correlate, analyze trace data from different layers, you can also open text logs in Trace Compass. In the upper layer, you can take a similar approach as the Linux kernel where static tracepoints represent the wisdom of developers who are most familiar with the code and the rest of the world (testers, system administrators, field engineers, developers) can use them to extract a great deal of useful information without having to know the code. 2. On server field, providing some interactive, graphical user interfaces for some generic problems. Android's SysTrace is a good example. However, our the real goal is to create a tool for dealing with problems not so generic. Profiling using a language like SQL maybe, like Splunk, to synthesize traces from different layers and components. So the underlying traces are better to be in a uniform format. [Dominique Toupin] SQL can be nice but it requires a lot more space, if I remember correctly, the size is sometimes multiplied by 70 compared to CTF. CTF is like a specialized trace data base, in Trace Compass they have efficient query to navigate in the trace, they add a state system, etc. You can do all this with SQL but it require a lot more space and it is slower. If you see things missing in CTF compared to what you want to do with SQL, let us know we can investigate how to add those features directly in CTF. We also concern the upstreaming plan about kernel stuffs in DiaMon. > Thanks for the second post! > > Dominique > > -----Original Message----- > From: Wang Nan [mailto:wangnan0@huawei.com] > Sent: November-25-14 8:04 AM > To: diamon-discuss@lists.linuxfoundation.org; Dominique Toupin; Jiri > Olsa > Cc: genghui 00204690; Xinwei Hu > Subject: Diamon CTF Roadmap? > > Hi everyone, > > Looks like my post will be the second thread in this list. That's great. > > I'll be the technical contact person representing Huawei for Diamon. > I'm glad to work with this new workgroup to create "de-facto, widely used open source trace technologies". > > Beyond sharing new ideas and technologies, our main concern on Diamon now are CTF tracing format and related things for analysing traces from different sources in a unified manner. I have read Jiri Olsa's slides in TracingSummit 2014, and also begin to read his code in https://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git. However, it looks like that there's only a trace generating tool is under developing. What about upper trace parsing and analysing tools? Is there any roadmap on it? How can we participate in the project? > > Thanks. > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [diamon-discuss] Diamon CTF Roadmap? 2014-11-26 19:43 ` Dominique Toupin @ 2014-11-27 1:51 ` Wang Nan 2014-11-27 12:41 ` Dominique Toupin 0 siblings, 1 reply; 6+ messages in thread From: Wang Nan @ 2014-11-27 1:51 UTC (permalink / raw) To: Dominique Toupin, diamon-discuss@lists.linuxfoundation.org, Jiri Olsa Cc: Xinwei Hu, genghui 00204690 On 2014/11/27 3:43, Dominique Toupin wrote: > You can see my answer below preceded by [Dominique Toupin] > > -----Original Message----- > From: Wang Nan [mailto:wangnan0@huawei.com] > Sent: November-25-14 9:08 PM > To: Dominique Toupin; diamon-discuss@lists.linuxfoundation.org; Jiri Olsa > Cc: genghui 00204690; Xinwei Hu > Subject: Re: Diamon CTF Roadmap? > > On 2014/11/25 21:42, Dominique Toupin wrote: >> Hi Nan, >> >> Many tools are able to handle CTF today, e.g.: >> - GDB Tracepoint >> https://sourceware.org/gdb/onlinedocs/gdb/Trace-Files.html#Trace-Files >> - LTTng kernel and LTTng user space C/C++, Java, Python, etc. >> https://lttng.org/download >> - babel trace, http://lttng.org/viewers/ >> - Trace Compass >> http://projects.eclipse.org/projects/tools.tracecompass >> - CTF tracer for bare metal >> https://github.com/efficios/barectf/blob/master/README.md >> >> You will also find many converter to CTF, e.g. NEXUS HW Trace, perf, >> And I am most likely missing a few. >> >> If you can provide a description of your needs it would help to give you more info related to your use case. >> e.g. is it more related to cloud, embedded data base, real time, or do you have a specific view or analysis you want to see in the trace viewer, etc. >> > > Our use cases are diverse, both embedded and server scene have requirements on profiling, and methodologies are also different. > > We have two starting points now: > > 1. On ARM (32/64) platform, extracing information (instruction footprints) from CoreSight > using perf and use CTF as trace format, to make it possible to analysis with trace from > upper layer. Intel has similar hardware too, but currently only on specification. > > [Dominique Toupin] I believe they are able to get perf data in CTF and if you also have > CTF trace from upper layer then you can use Trace Compass (python should also be an option) > to visualize, correlate, analyze trace data from different layers, you can also open text logs in > Trace Compass. In the upper layer, you can take a similar approach as the Linux kernel where > static tracepoints represent the wisdom of developers who are most familiar with the code and > the rest of the world (testers, system administrators, field engineers, developers) can use them > to extract a great deal of useful information without having to know the code. > > 2. On server field, providing some interactive, graphical user interfaces for some generic > problems. Android's SysTrace is a good example. However, our the real goal is to create > a tool for dealing with problems not so generic. Profiling using a language like SQL maybe, > like Splunk, to synthesize traces from different layers and components. So the underlying > traces are better to be in a uniform format. > > [Dominique Toupin] SQL can be nice but it requires a lot more space, if I remember > correctly, the size is sometimes multiplied by 70 compared to CTF. CTF is like a specialized > trace data base, in Trace Compass they have efficient query to navigate in the trace, they > add a state system, etc. You can do all this with SQL but it require a lot more space and it is slower. > If you see things missing in CTF compared to what you want to do with SQL, let us know we can > investigate how to add those features directly in CTF. > Thanks for your reply. I'll try Trace Compass in our situation. In addition, is there any description on the organization and policy of DiaMon? > > We also concern the upstreaming plan about kernel stuffs in DiaMon. > >> Thanks for the second post! >> >> Dominique >> >> -----Original Message----- >> From: Wang Nan [mailto:wangnan0@huawei.com] >> Sent: November-25-14 8:04 AM >> To: diamon-discuss@lists.linuxfoundation.org; Dominique Toupin; Jiri >> Olsa >> Cc: genghui 00204690; Xinwei Hu >> Subject: Diamon CTF Roadmap? >> >> Hi everyone, >> >> Looks like my post will be the second thread in this list. That's great. >> >> I'll be the technical contact person representing Huawei for Diamon. >> I'm glad to work with this new workgroup to create "de-facto, widely used open source trace technologies". >> >> Beyond sharing new ideas and technologies, our main concern on Diamon now are CTF tracing format and related things for analysing traces from different sources in a unified manner. I have read Jiri Olsa's slides in TracingSummit 2014, and also begin to read his code in https://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git. However, it looks like that there's only a trace generating tool is under developing. What about upper trace parsing and analysing tools? Is there any roadmap on it? How can we participate in the project? >> >> Thanks. >> > > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [diamon-discuss] Diamon CTF Roadmap? 2014-11-27 1:51 ` Wang Nan @ 2014-11-27 12:41 ` Dominique Toupin 0 siblings, 0 replies; 6+ messages in thread From: Dominique Toupin @ 2014-11-27 12:41 UTC (permalink / raw) To: Wang Nan, diamon-discuss@lists.linuxfoundation.org, Jiri Olsa Cc: Xinwei Hu, genghui 00204690 You can try Trace Compass and also the cmd line/python scripts as they can even be used in some context on the target, e.g. in a cloud Regarding the governance of DiaMon it is the simplest form of a workgroup at the Linux foundation, you can find it at http://diamon.org/projects/governance/ In summary anyone can participate and optionally if you want your company logo on diamon.org you can send the request to Mike Dolan <mdolan@linuxfoundation.org> If you have specific layer that you want to get instrumented with static tracepoint, let us know we should be able to help. -----Original Message----- From: Wang Nan [mailto:wangnan0@huawei.com] Sent: November-26-14 8:52 PM To: Dominique Toupin; diamon-discuss@lists.linuxfoundation.org; Jiri Olsa Cc: genghui 00204690; Xinwei Hu Subject: Re: Diamon CTF Roadmap? On 2014/11/27 3:43, Dominique Toupin wrote: > You can see my answer below preceded by [Dominique Toupin] > > -----Original Message----- > From: Wang Nan [mailto:wangnan0@huawei.com] > Sent: November-25-14 9:08 PM > To: Dominique Toupin; diamon-discuss@lists.linuxfoundation.org; Jiri > Olsa > Cc: genghui 00204690; Xinwei Hu > Subject: Re: Diamon CTF Roadmap? > > On 2014/11/25 21:42, Dominique Toupin wrote: >> Hi Nan, >> >> Many tools are able to handle CTF today, e.g.: >> - GDB Tracepoint >> https://sourceware.org/gdb/onlinedocs/gdb/Trace-Files.html#Trace-File >> s >> - LTTng kernel and LTTng user space C/C++, Java, Python, etc. >> https://lttng.org/download >> - babel trace, http://lttng.org/viewers/ >> - Trace Compass >> http://projects.eclipse.org/projects/tools.tracecompass >> - CTF tracer for bare metal >> https://github.com/efficios/barectf/blob/master/README.md >> >> You will also find many converter to CTF, e.g. NEXUS HW Trace, perf, >> And I am most likely missing a few. >> >> If you can provide a description of your needs it would help to give you more info related to your use case. >> e.g. is it more related to cloud, embedded data base, real time, or do you have a specific view or analysis you want to see in the trace viewer, etc. >> > > Our use cases are diverse, both embedded and server scene have requirements on profiling, and methodologies are also different. > > We have two starting points now: > > 1. On ARM (32/64) platform, extracing information (instruction footprints) from CoreSight > using perf and use CTF as trace format, to make it possible to analysis with trace from > upper layer. Intel has similar hardware too, but currently only on specification. > > [Dominique Toupin] I believe they are able to get perf data in CTF and > if you also have CTF trace from upper layer then you can use Trace > Compass (python should also be an option) to visualize, correlate, > analyze trace data from different layers, you can also open text logs > in Trace Compass. In the upper layer, you can take a similar approach > as the Linux kernel where static tracepoints represent the wisdom of > developers who are most familiar with the code and the rest of the world (testers, system administrators, field engineers, developers) can use them to extract a great deal of useful information without having to know the code. > > 2. On server field, providing some interactive, graphical user interfaces for some generic > problems. Android's SysTrace is a good example. However, our the real goal is to create > a tool for dealing with problems not so generic. Profiling using a language like SQL maybe, > like Splunk, to synthesize traces from different layers and components. So the underlying > traces are better to be in a uniform format. > > [Dominique Toupin] SQL can be nice but it requires a lot more space, > if I remember correctly, the size is sometimes multiplied by 70 > compared to CTF. CTF is like a specialized trace data base, in Trace > Compass they have efficient query to navigate in the trace, they add a state system, etc. You can do all this with SQL but it require a lot more space and it is slower. > If you see things missing in CTF compared to what you want to do with > SQL, let us know we can investigate how to add those features directly in CTF. > Thanks for your reply. I'll try Trace Compass in our situation. In addition, is there any description on the organization and policy of DiaMon? > > We also concern the upstreaming plan about kernel stuffs in DiaMon. > >> Thanks for the second post! >> >> Dominique >> >> -----Original Message----- >> From: Wang Nan [mailto:wangnan0@huawei.com] >> Sent: November-25-14 8:04 AM >> To: diamon-discuss@lists.linuxfoundation.org; Dominique Toupin; Jiri >> Olsa >> Cc: genghui 00204690; Xinwei Hu >> Subject: Diamon CTF Roadmap? >> >> Hi everyone, >> >> Looks like my post will be the second thread in this list. That's great. >> >> I'll be the technical contact person representing Huawei for Diamon. >> I'm glad to work with this new workgroup to create "de-facto, widely used open source trace technologies". >> >> Beyond sharing new ideas and technologies, our main concern on Diamon now are CTF tracing format and related things for analysing traces from different sources in a unified manner. I have read Jiri Olsa's slides in TracingSummit 2014, and also begin to read his code in https://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git. However, it looks like that there's only a trace generating tool is under developing. What about upper trace parsing and analysing tools? Is there any roadmap on it? How can we participate in the project? >> >> Thanks. >> > > ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2014-11-27 12:41 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2014-11-25 13:03 [diamon-discuss] Diamon CTF Roadmap? Wang Nan 2014-11-25 13:42 ` Dominique Toupin 2014-11-26 2:08 ` Wang Nan 2014-11-26 19:43 ` Dominique Toupin 2014-11-27 1:51 ` Wang Nan 2014-11-27 12:41 ` Dominique Toupin
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.