Re: [diamon-discuss] Common Trace Format 1.9 planning

[Mathieu Desnoyers <mathieu.desnoyers@efficios.com>]

----- Original Message -----
> This is great, so v1.9 is a superset of 1.8.

No. A superset would be a completely backward compatible 1.9.
This is not the case here. We plan to do incompatible changes
within 1.9, hence the 1.8 and 1.9 parsers needed. Since the
version is self-described, it should not be an issue to detect
the CTF version from the trace metadata.

> 
> Would hints for time format be interesting or overspecialized?

I would keep that for presentation, either an appendix to CTF or
a separate spec.

After much discussion, we came to the conclusion that keeping CTF
as a binary trace layout description language, and sticking to just
that, seems to be a good way to keep the CTF specification minimal.

This can be seen as splitting presentation from a web page content,
where the presentation is in CSS, and the content in HTML.

We can then optionally add "attributes" associated to types within the
CTF metadata, but the semantic of those attributes would not be
defined within CTF. If a trace reader does not understand or care about
those attributes, they should simply be ignored.

> 
> Would hardware description in the metadata in a standardized form be a
> good thing to have?

This is not needed to describe the binary trace layout, so this
should be covered by another spec, or by a CTF appendix.

> 
> Would including a state system specification in the tsdl be an
> interesting thing to do? Basically hints on how to process the events.

We could have a separate spec for this. We could then map those
descriptions to the events. Again, this is not needed to decode
the binary trace.

The basic idea is that whatever is specified by CTF needs to be
fully implemented by _all_ CTF readers, and completely covered,
hence my intent to keep the optional stuff outside of this core
spec.

Thoughts ?

Thanks,

Mathieu

> 
> Thanks!
>  
> Matthew
> 
> On 15-03-25 02:39 PM, Mathieu Desnoyers wrote:
> > Hi,
> >
> > As discussed on the workgroup call today, I am posting
> > the list of changes I have gathered for CTF 1.9. Feedback
> > is welcome!
> >
> > * Add new set of features to Common Trace Format and to the Babeltrace
> > reference implementation.
> >
> >
> > 1. Handle transition from CTF 1.8 to 1.9, including compatibility and
> > upgrade path for users.
> >
> > We must take great care when extending the Common Trace Format
> > specification in non-fully backward compatible way, so that users do not
> > suffer from this transition. We also want to minimize the frequency at
> > which we do those non-compatible changes, by bundling those changes all
> > within one release, and by ensuring, for the future, that the format can
> > be extended with "features" that are simply ignored by a parser
> > implementation that does not know about them.
> >
> > Since the description format contains version numbering, we can keep CTF
> > 1.8 and CTF 1.9 parsers side-by-side in the trace reader, so users
> > gathering CTF 1.8 traces can use the new tools to view them. Only users
> > generating new trace format (CTF 1.9) will need the new tools to view
> > them.
> >
> >
> > 2. Add base address and symbol information support.
> >
> > The Perf developers expressed strong interest for add base address and
> > symbol information support within the Common Trace Format at the Trace
> > Summit 2014 [3]. This requirement seems to be interesting for most of the
> > community.
> >
> > In the spirit of keeping the Common Trace Format dedicated as much as
> > possible to describe the layout of the binary trace, we intend to keep the
> > base address and symbol information separate from the CTF metadata file.
> > This can be achieved by creating a stream that contains the following
> > object load and unload events:
> >
> > [ object load, timestamp, address space, address, mapping length, object
> > path ]
> > [ object unload, timestamp, address space, address ]
> >
> > This should allow describing load/unload events for the kernel, and for
> > each user-space process.
> >
> > Initially, we can introduce those events into LTTng and Perf-to-CTF in the
> > same way through discussion.
> >
> >
> > 3. Add event versioning.
> >
> > Experience with the Linux kernel Tracepoints shows us that relying on event
> > and field names is not sufficient for binding a trace analysis on the
> > semantic of the kernel or application. Sometimes, changes in the software
> > keep the same event and field name, but change their semantic.
> >
> > A good example of such a change is the wakeup delivery within the Linux
> > kernel which has moved from the context of the thread performing the wake
> > up to an inter-processor interrupt performing the wake-up on a remote CPU.
> > This is an issue for critical path analysis, and it needs to know about
> > this semantic change to the PID field, although the event and field names
> > are unchanged.
> >
> > Therefore, introduce an optional versioning property for the event, which
> > allow trace analysis and models to track which semantic it is tracking.
> >
> >
> > 4. Add field name prefix '$' to eliminate conflicts with reserved keywords.
> >
> > Currently, the Common Trace Format metadata uses a '_' prefix to mitigate
> > conflicts with reserved keywords. However, it was a bad idea, because '_'
> > is a legitimate character part of the identifiers. This creates some
> > confusion in the cases where an event name is indeed
> > prefixed by '_' already: the underscore is either removed when it should
> > not be removed, or '__' is then necessary.
> >
> > Fix this confusion by introducing the '$' optional prefix for identifiers.
> >  
> $context actually looks better than _context
> >
> > 5. Add attributes for type and field reference.
> >
> > Turn the integer and floating point values description into attributes
> > attached to the type:
> >
> > integer ( attribute-list )
> > floating_point ( attribute-list )
> >
> > This will allow using the same concept of attributes on compound types,
> > e.g.:
> >
> > struct { ... } ( attribute-list )
> > variant <tag> { ... } ( attribute-list )
> > int32_t [10] ( attribute-list )
> >
> > attribute-list is a comma-separated list of:
> >
> >   identifier = expression
> >
> > For instance, in the case of event pretty-printing, this can be used as:
> >
> >   format = "content of format string"
> >
> > Where the content of the format string follows what has been proposed in
> > the report on layout description.
> >
> >
> > 6. Add pretty-printing "hints", specified outside of CTF, within either
> > another specification, or an appendix. This will allow splitting data
> > layout (CTF) from formatting.
> >
> > The goal here is to allow the tracer implementation (e.g. LTTng-UST) to
> > specify additional attributes attached to a CTF type. The two attributes
> > we will implement in LTTng-UST and Babeltrace are:
> >
> >   * format = "{field1}: {field3:02x} ({event.context.allo})"
> >
> > The format string formatting would follow the Python format string syntax
> > [4]. The choice of this syntax over printf-alike is due to the ability to
> > refer to specific fields by name within the format string, and thus using
> > just a string as an attribute (rather than a function call which would
> > have a large impact on the CTF TSDL grammar). The Python format strings
> > are very similar to its printf counterpart: they share a very similar
> > format specification.
> >
> > A few examples of printf vs Python format string syntax similarity:
> >
> > Event payload layout:
> >
> > struct {
> >         int myint;
> >         string mystring;
> >         double mydouble;
> > };
> >
> > Printf: printf("abc %05d %s %g", myint, mystring, mydouble)
> > Python: "abc {:05} {} {}"
> >
> > Printf: printf("%s: %x", mystring, myint)
> > Python: "{mystring}: {myint:x}"
> >
> >
> >   * printers = [ "hexdump", "..." ]
> >
> > The tracer and trace viewer implementations can agree on a "printers"
> > attribute, which describes pretty-printing plugins that shall be used to
> > render the type, if such plugins are available.
> >
> > This includes implementing in the Babeltrace reference implementation
> > support for such pretty-printing plugins, which can register to
> > Babeltrace, and express which type or types they expect as input.
> >
> > A reference implementation of plugin will be implemented, which prints a
> > sequence of bytes as:
> >
> > [...]
> > 000015F0 5F 73 70 72 69 6E 74 66 5F 63 68 6B 00 5F 5F 78 _sprintf_chk.__x
> > 00001600 73 74 61 74 00 6D 65 6D 6D 6F 76 65 00 5F 6F 62 stat.memmove._ob
> > 00001610 73 74 61 63 6B 5F 62 65 67 69 6E 00 62 69 6E 64 stack_begin.bind
> > [...]
> >
> 
> _______________________________________________
> diamon-discuss mailing list
> diamon-discuss@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/diamon-discuss
> 

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com