Re: [diamon-discuss] Diamon workgroup kickoff meeting

Re: [diamon-discuss] Diamon workgroup kickoff meeting

[Mathieu Desnoyers]

Hi everyone,

Here are the minutes taken during the Diagnostic and Monitoring
Workgroup kickoff meeting (http://diamon.org) on November 12, 2014.

Please note that the future discussion and meeting notes will be sent to
the Diamon mailing list only (no bcc). If you are interested, please
register to the mailing list at the following address:

https://lists.linuxfoundation.org/mailman/listinfo/diamon-discuss

Were present:

Dominique Toupin
Agustin Vega-Frias
Mathieu Desnoyers
Christian Babeux
Masami Hiramatsu
Christoph Lameter
Thomas Gleixner
Petru Lauric
Ed Martinez
(and others we for which we missed the name)

* Diamon Introduction

  - Dominique introduce the reason why the workgroup was created.
  - Current diamon draft is largely based on Linux in telecom companies
    (Ericsson), other requirements welcomed.
  - Overview of website, share information with other companies and user.

* Overview of mini-coredumps
  - Linutronix
  - Trigger memory dump when application crash.
  - Dump only a small portion of memory instead of the whole memory.
  - Configurable, setup watches, per application fine tuning.
  - Libminicoredump, apps can register data structures to be dumped.
  - Capture state of application
  - Snapshot in time, asynchronous dump to debug
  - Plan is to host the mini core dump at diamon.org

How to integrate mini-coredump with tracing? Snapshot?
-> Application registration mechanism with mini-coredump

How to integrate those tools? Distribution to users and packaging not ready.

* Overview of TraceCompass (UI for Linux tracing)

- Import text logs, libpcap traces, LTTng traces
- Show information from mini-coredump
- Standalone application similar to Wireshark

How to integrate this standalone tool in another toolkit?
-> Java based toolkit should be easy

- Used by Intel for trace tool, Sourcery Analyser from Mentor, etc.

* Overview of the reason for the creation of the workgroup

  - In tracing, everyone is scratching their own itch, workgroup a
    reason to share and collaborate together
  - LTTng project is focusing on trace analysis, usable by non-expert
    (e.g. non-kernel developers)
  - CTF, a trace format to be able to correlate multiple tracing data
    sources

Christoph Lameter: Finding latencies in their own applications.
Multiple tools, difficult to use.
Get user feedback on what they would like to see in those tools.

* Interest from perf/ftrace to CTF?

Linutronix + Red Hat, perf to CTF module, work is on-going.
Post-processing perf output file to convert to CTF.
Mostly functional, few rough-edges, still need to modify Eclipse viewer
to adapt to the event semantic exposed by Perf.

People asking for the presentation from Jiri Olsa at this year tracing summit.
http://www.tracingsummit.org/w/images/9/98/TracingSummit2014-Perf-CTF.pdf

Steven Rostedt from ftrace interested in looking into CTF.

Masami: Where can the CTF specification be found ?
CTF specification available at http://www.efficios.com/ctf Eventually, the plan is to move the specification to the Diamon workgroup.
Babeltrace and libbabeltrace can be used to convert/read CTF.

* Work on Common Trace Clocks
Thomas Gleixner:
- Provide a NMI-safe accessor to the clock monotonic. Provides
  a clock that can be correlated with user-space monotonic clock.
- Correlated timestamp across the network possible.
- Merged in 3.17

How to use this clock ?
- extern u64 ktime_get_mono_fast_ns(void);
- Can change the clock in ftrace, via debugfs.
- Perf still has to figure out how to expose this.
- use clock monotonic: give correlation across network,
  which is harder to do when using directly TSC.
- Correlation between clock monotonic and clock realtime
  can be done with a single correlation point, since both
  clocks are adjusted in the same way by NTP.

* HW assisted tracing

Freescale:
- Trace from the cpu, hw port, ETM/PTM like trace.
- HW trace form SoC, DDR controllers
- At tracing summit, presented tool for soc trace, for network developers
- Issue about having a common clock, traces from multiple hw,
  different hw clocks
- How to correlate with hardware and software traces

- Hw trace tend to be proprietary, need for common generic format.
- Consuming CTF file, analysis in TraceCompass
- DMTF format to CTF?
  - Could probably use babeltrace to convert. Semantic about the clocks
    could be missing from DMTF though.

The diamon workgroup is a lean workgroup, straightforward governance.
Do not need to be a member of LF to join,
If you want your company logo on diamon.org (this is optional), send the
request to Mike Dolan <mdolan@linuxfoundation.org>

How to correspond with diamon? "Diamon discuss" mailing list.
https://lists.linuxfoundation.org/mailman/listinfo/diamon-discuss

Next meeting: Contribution from other users to see what they would like
to be improved in tooling.

Christoph Lameter: will bring someone from his team to present
Masami Hiramatsu: Interested in cloud logging/tracing. Integration of
app logging with tracing.

Thank you!

Mathieu

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com