From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Date: Wed, 26 Oct 2016 15:32:37 +0000 (UTC) From: Mathieu Desnoyers Message-ID: <1016012603.5816.1477495957751.JavaMail.zimbra@efficios.com> In-Reply-To: <1589847494.4423.1477419969037.JavaMail.zimbra@efficios.com> References: <1589847494.4423.1477419969037.JavaMail.zimbra@efficios.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Subject: Re: [diamon-discuss] [lttng-dev] CTF2-PROP-1.0: Proposal for a major revision of the Common Trace Format, version 1.8 List-Id: DiaMon diagnostic and monitoring workgroup general discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Philippe Proulx , Julien Desfossez Cc: diamon-discuss , lttng-dev , Trace Compass Developer Discussions , Etienne Bergeron , =?utf-8?Q?Fran=C3=A7ois?= Doray ----- On Oct 25, 2016, at 2:26 PM, Philippe Proulx pproulx@efficios.com wro= te: > Hello fellow trace format enthusiasts. >=20 > This is a proposal for a major revision of CTF v1.8 (to CTF v2). How about we present it over a diamon.org conference call ? We could setup a call for the week of November 1st. Philippe could prepare a few slides to present it, and it would be mostly oriented towards gathering feedback. Thanks, Mathieu >=20 > I strongly suggest that you read the HTML version at: >=20 > http://diamon.org/ctf/files/CTF2-PROP-1.0.html >=20 > since the text below is an AsciiDoc source. You should, however, > inline-comment if you have something to say about a specific section of > the document, but _please_ keep only a few lines of context (what's > necessary) above and below your comment, because the text is about 4500 > lines long. >=20 > Other emails will follow with other CTF 2 documents. We decided to > decouple some features and optional parts of CTF 2 in different > documents so that each one is really focused on its own subject. Then > producers and consumers may comply with this or that document. For > example, as long as a consumer can decode a CTF 2 trace (following the > specification itself), it's not the end of the world if it doesn't know > that a given integer field type prefers to be displayed in base 16. >=20 > The other documents are: >=20 > * http://diamon.org/ctf/files/CTF2-DOCID-1.0.html > * http://diamon.org/ctf/files/CTF2-BASICATTRS-1.0.html > * http://diamon.org/ctf/files/CTF2-PMETA-1.0.html > * http://diamon.org/ctf/files/CTF2-FS-1.0.html >=20 > Feel free to question this proposal! >=20 > A few things that still annoy me: >=20 > * Should a boolean field type inherit the properties of an integer field > type instead of a simple bit array field type? In other words, should > a boolean field type have a signedness property? >=20 > Since the interesting values of a boolean field are really _true_ and > _false_, in my opinion we should not care about any signedness here. > If you need this, you can use the new union field type and match a > boolean field type of size X with a (signed, for example) integer > field type of size X. >=20 > * Do we really need to support other bases than base 10 in the constant > integer JSON object? AFAIK, other bases are not required to encode and > decode any integer value. They're only there to ease human reading of > the metadata stream... however it's pretty much the only place where > such a human-friendly entity is defined, so is it really needed? >=20 > Keep in mind that keeping the support for bases 2, 8, and 16 requires > each single CTF 2 consumer to be able to convert those strings to > integers. >=20 > * There's a clear relation between some field types that, the way it's > written now, have no common parent. >=20 > For example, a variable-length integer field type describes fields > that, once decoded, provide integer values, just like the integer > field type does. However, they have no relation. Even though they both > share a `signed` property, the variable-length integer field type does > not need a `size` property, which is inherited from the bit array > field type. >=20 > Same thing for the text array field type vs. the array field type > (former does not need an `element-field-type` property because it's > implicit). >=20 > Array field type and sequence field type could also be related by > their common `element-field-type` property, but they are not as of > this version. >=20 > Do you have any idea how to bring them into relation with one another > without making the text too heavy? I'm thinking about some kind of > property mixin (or trait?) which could be applied over a field type. > For example, the "integer field type mixin" could define a single > `signed` property and both the integer field type and the > variable-length integer field type could claim to "implement" this > mixin. "Mixin" is probably not the right term. This could simplify > some parts of the text where a field providing an integer value is > needed: the text could read something like "a field type with the > integer field type mixin applied is needed here". >=20 > * I'm not impressed by the clock field tags, in that we define in an > upper layer an m-map which can be inserted _within_ an m-map which was > defined in a _lower_ layer of the specification. >=20 > However I believe it's important that all field tags which target a > specific scope field be in the same fragment that defines the type of > this scope field. For example, all field tags which target the event > record header scope should be part of the data stream class fragment > where this event record header field type is defined. >=20 > Any idea? >=20 > Thanks for your comments! >=20 > Philippe Proulx > EfficiOS Inc. > http://www.efficios.com/ >=20 > - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - >=20 > =3D CTF2-PROP-1.0: Proposal for a major revision of the Common Trace Form= at, > version 1.8 > Philippe Proulx > v1.0, 21 October 2016 > :toc: > :toclevels: 5 > :ieee754: IEEE 754-2008's binary interchange format >=20 > This document is an informal proposal for the next major revision of the > **Common Trace Format** (CTF): version 2.0 (hereafter named > _CTF{nbsp}2_). >=20 > This is _not_ a formal reference. Some parts of this document, however, > may be formal enough to be elligible for a specification document. >=20 > .RFC 2119 > NOTE: The key words _must_, _must not_, _required_, _shall_, _shall > not_, _should_, _should not_, _recommended_, _may_, and _optional_ in > this document, when emphasized, are to be interpreted as described in > https://www.ietf.org/rfc/rfc2119.txt[RFC 2119]. >=20 >=20 > =3D=3D A new workgroup responsible for the publication of CTF documents >=20 > The http://diamon.org/[DiaMon Workgroup], a > https://www.linuxfoundation.org/[Linux Foundation] workgroup which > creates de-facto standards and tools for tracing, monitoring, and > diagnostics, is now responsible for the publication of the official > documents about CTF. >=20 > The DiaMon Workgroup is also responsible for making available a platform > where interested parties can comment on the proposals related to CTF. >=20 >=20 > =3D=3D A new method for identifying CTF{nbsp}2 documents >=20 > We suggest that all documents related to CTF{nbsp}2 bear a unique > **document identifier** (ID) having the following format: >=20 > CTF2--.[r] >=20 > [options=3D"header"] > .Descriptions and roles of CTF{nbsp}2 document ID parts >|=3D=3D=3D >|Part |Description |Bump _may_ introduce new concepts, procedures, and for= mats? >||Bump _may_ remove or change existing concepts, procedures, and formats? >=20 >|++ >|The capitalized short name of the document, > unique amongst all the CTF{nbsp}2 documents. >|N/A >|N/A >=20 >|`` >|The major version number of the document. >|Yes >|Yes >=20 >|`` >|The minor version number of the document. >|Yes >|No >=20 >|`` >| > The revision letter of the document (from `A` to `Z`). >=20 > Document revisions are used to add examples, clarify existing concepts, > fix grammar or content mistakes, or reword existing parts, for example. >|No >|No >|=3D=3D=3D >=20 > For example, the short name of this document is `PROP`, for _proposal_, > and its full document ID is `CTF2-PROP-1.0`. The next revision would be > `CTF2-PROP-1.0rA`, and the following would be `CTF2-PROP-1.0rB`. >=20 > In any CTF{nbsp}2 document, another CTF{nbsp}2 document _must_ be > referred to by using only its ID. For example: _This concept is further > explained in did:CTF2-SOMEID-1.2_. There is no need to refer to a > specific revision: the reference always targets the latest document's > revision. >=20 > We suggest the following IDs for the initial documents: >=20 > * did:CTF2-DOCID-1.0: _CTF{nbsp}2 document identifier format_ > * did:CTF2-SPEC-2.0: _The Common Trace Format (CTF), version 2.0_ > * did:CTF2-BASICATTRS-1.0: _Basic CTF{nbsp}2 user attributes_ > * did:CTF2-FS-1.0: _Layout of a CTF{nbsp}2 trace stored on a file system_ > * did:CTF2-PMETA-1.0: _CTF{nbsp}2 metadata stream packet format_ >=20 >=20 > =3D=3D Why CTF{nbsp}2? >=20 > Why do we need a major version bump of the CTF specification? >=20 > A major version bump is never an easy choice when it comes to revisiting > a software library, a communication protocol, a file format, or anything > that serves as a contract between a producer and a consumer. When such a > decision is taken, it must be justified by solid arguments, since it > makes it impossible for old consumers to consume the product of new > producers. >=20 > In this proposal, for instance, CTF{nbsp}2 traces are not backward > compatible with CTF{nbsp}1 traces. Although the binary format is not > changed, the metadata stream is written in a different language. >=20 > CTF{nbsp}1 has been used and tested for many years now, by different > producers and consumers. Over that time, we have noted a few gaps in the > trace format, gaps that prevent us from extending CTF{nbsp}1 as much as > we would like, amongst other things. CTF{nbsp}2 is designed to overcome > those gaps, as far as we know them, and to be flexible enough to > gracefully accept future additions while avoiding another major version > bump in the following years. >=20 >=20 > [[design-goals]] > =3D=3D=3D Design goals >=20 > The design goals of CTF{nbsp}2 are as follows, in order, beginning with > the most important: >=20 > . **CTF{nbsp}2 data streams _must_ be backward compatible with > CTF{nbsp}1 data streams.** > + > Many applications are already programmed to write valid CTF{nbsp}1 > packets. Modifying the code of those applications to produce different > binary packets can be cumbersome, and sometimes impossible if the > application passed acceptance tests, for example. > + > Making sure that applications producing CTF{nbsp}1 traces can also > produce CTF{nbsp}2 traces only by changing the metadata stream is an > absolute necessity. >=20 > . **The CTF{nbsp}2 streams _must_ be as efficient as possible to produce > by a tracer.** > + > This design goal was also one of the major ones which motivated the > design of CTF{nbsp}1. > + > In other words, a small embedded system _must_ be able to produce > CTF{nbsp}2 streams natively. Moreover, the tracer _must_ be able to copy > binary data to the packet buffer of a data stream without altering it. >=20 > . **CTF{nbsp}2's model _should_ be backward compatible > with CTF{nbsp}1's.** > + > Some APIs are already written to deal with CTF{nbsp}1 ``objects'', or > concepts (event record classes, event records, field types, and data > stream clock classes, to name a few). The model of CTF{nbsp}2 _should_ > be compatible with the model of CTF{nbsp}1, so that those existing APIs > can operate on CTF{nbsp}2 objects too without requiring huge > refactorings. >=20 > . **The size of any CTF{nbsp}2 static field, that is, any field with a > non-dynamic type, _must_ always be the same in data streams to speed > up the validation in some situations.** > + > In other words, a field described by a fixed-size type, or by a compound > type containing only fixed-size types, _should_ always have the same > size, no matter its offset in the data stream. > + > This guarantee allows, in certain situations, to greatly speed up the > validation process by having a great part of it done at the metadata > stream level. >=20 > . **A CTF{nbsp}2 trace _should_ be as easy as possible to consume.** > + > CTF{nbsp}1 focuses on being easy to be produced, which is a good idea > since producers are tracers in this context, and a minimal tracer > _should_ be able to produce a correct CTF trace with minimal code > (design goal 2). > + > However, because CTF{nbsp}1's metadata stream is written in TSDL, a > custom, declarative, C-like DSL designed for CTF, writing a minimal > consumer of CTF{nbsp}1 is not an easy task. TSDL is an intricate > language, with many special cases, many of which are borrowed from the C > language, which cannot be ignored when writing a consumer supporting all > its features. TSDL was developed to ease the _manual_ (human) production > of metadata streams. > + > Over time, we realized that, while producing traces is important, > consuming them to solve problems by analysing the event records is just > as important, if not more. > + > This is why CTF{nbsp}2 _should_ encourage the development of CTF > consumers in any programming language by reducing the number of special > cases, as well as by using a very simple, yet well-known grammar for the > metadata stream. > + > CTF{nbsp}1 tries to accomodate other trace formats, which can be > converted to CTF without changing the data streams by writing the > matching metadata stream. This is also a source of special cases. > CTF{nbsp}2 _should_ build on binary trace conversion (from another, > non-CTF trace format to CTF{nbsp}2) rather than trying to accomodate > other formats. >=20 > . **A CTF{nbsp}2 metadata stream _must_ be extensible by users and by > future minor revisions of the specification (forward compatibility).** > + > CTF{nbsp}1's TSDL grammar is pretty restrictive when it comes to > customizing existing blocks with user-defined attributes. > + > Many protocols and declarative languages support custom user data in > their payload. For example, HTML5 allows any element to have user > attributes by prefixing their names with `data-`. > + > A producer of CTF{nbsp}2 _should_ be able to add custom attributes to > almost any object defined by the specification. This allows standard > consumers to read any CTF{nbsp}2 trace and ignore unknown user > attributes, providing a ``bland'', yet complete view of the trace > fields, while special consumers can be written (or existing consumers > can be extended) to interpret specific user attributes and use them to > present a meaningful visualization. > + > It is also possible for the DiaMon Workgroup to publish a proposal for > new properties for a next minor revision of the specification, but test > them as temporary user attributes for some time in order to collect > comments before updating the specification itself. > + > This design goal also means that an ``old'' CTF{nbsp}2 consumer _should_ > be able to decode a ``new'' CTF{nbsp}2 trace, possibly with missing > field semantics if field types are added in minor revisions. >=20 > . **CTF{nbsp}2's specification _must_ focus on how to encode and decode > data streams.** > + > CTF{nbsp}1 has a few base properties in its metadata stream that are not > strictly needed to encode or decode data streams. For example, the > `base` property of the `integer` field type is only useful to visualize > the decoded integer fields: the decoding process does not depend on a > preferred radix. Also, the name of an event record class is not needed > to decode the event records it describes: only its numeric ID is needed > to select the appropriate context and payload field types to use for the > encoding/decoding process. > + > All the object properties defined by the CTF{nbsp}2 specification > document _should_ exist only because they have a role in the > encoding/decoding process. Everything else _should_ be delegated to > other documents which define optional extension layers using the > mechanisms designed as a response to design goal 6. >=20 > . **CTF{nbsp}2's specification _must not_ specify how to transport a > trace, nor how a trace should be stored.** > + > In the CTF{nbsp}2 specification, a CTF{nbsp}2 trace _should_ be defined > as a set of bit streams, without specifying how those streams are > transported or stored. Other official documents published by the DiaMon > Workgroup can define standard ways to support and transport CTF{nbsp}2 > streams for specific use cases. Trace producers and consumers can choose > to implement one or more transport/storage mechanisms by following the > other documents. >=20 >=20 > =3D=3D Changes since CTF{nbsp}1 >=20 > Here is a brief summary of the changes, from CTF{nbsp}1 to CTF{nbsp}2, > introduced by this proposal. >=20 > * The terminology of the specification and the binary layouts of the > data streams are **completely detached from the C language** and from > the behaviour of any C compiler. > + > CTF{nbsp}2 is a programming language-agnostic trace format. >=20 > * **Terminology update**: > ** _Binary stream_ =E2=86=92 _data stream_. > ** _Trace_ =E2=86=92 _trace class_, when it names a block of > metadata which describes traces. > ** _Stream_ =E2=86=92 _data stream class_, when it names a > block of metadata which describes data streams. > ** _Stream ID_ =E2=86=92 _data stream class ID_. > ** _Stream instance ID_ =E2=86=92 _data stream ID_. > ** _Event_ =E2=86=92 _event record class_, when it names a > block of metadata which describes event records. > ** _Event ID_ =E2=86=92 _event record class ID_. > ** _Event_ =E2=86=92 _event record_, when it names an actual > recorded event contained in a packet. > ** _Declaration_ =E2=86=92 _field type_. > ** _Type alias_ =E2=86=92 _field type alias_. > ** _Clock_ =E2=86=92 _data stream clock class_, when it names a > block of metadata which describes actual data stream clocks. > ** _Clock_ =E2=86=92 _data stream clock_, when it names a > per-data stream instance of a specific data stream clock class. > ** _Native byte order_ =E2=86=92 _default byte order_. > ** _Stream packet context_ =E2=86=92 _data stream packet context_. > ** _Stream event header_ =E2=86=92 _data stream event record header_. > ** _Stream event context_ =E2=86=92 _data stream event record context_. > ** _Event context_ =E2=86=92 _event record context_. > ** _Event fields_ =E2=86=92 _event record payload_. > ** _Events discarded_ =E2=86=92 _discarded event record count_. > ** _Packet size_ =E2=86=92 _packet's total size_. > ** _Content size_ =E2=86=92 _packet's content size_. > * **The <> is written in JSON.** JSON > values are used to represent <>, the > internal type system of the CTF{nbsp}2 metadata. > * The CTF{nbsp}2 specification **does not specify a ``packetized'' > metadata stream format**. This is a back-end-specific way of wrapping > a <> as defined in this document. > Another document specifies the packetized metadata stream format, > which can be used by standard storage and transport documents if > needed. > * Most objects (called mdt:map values) of the CTF{nbsp}2 metadata model > _may_ contain **<>**. > * **Properties that are not strictly needed to _encode_ or _decode_ the > data streams are defined as user attributes in other documents.** > This includes, but is not limited to, the trace's environment, > the event record class's name and log level, and the integer field > type's preferred display base (CTF{nbsp}1's `base` property). > * **New field types**: > ** A <> describes the most > primitive fields of a data stream actually holding values. > + > An integer field type is now defined as a bit array field type with a > signedness property. > + > A floating point number field type is now defined as a bit array field > type: its size property is enough to determine its encoding under > {ieee754}. >=20 > ** A <>, which represents > nonexistent, or missing fields. The intended use of this field type > is as one of the possible types of a variant field type, especially > when representing the types of a dynamically-typed language using a > variant field type. > + > This field type _should_ be used to represent Python's `None`, Java's > and JavaScript's `null`, and Ruby's `nil`, for example. >=20 > ** A <>, which is a bit array > field type with a special meaning. When all the bits of a boolean > field are cleared, the field's value is said to be _false_. > Otherwise, the field's value is said to be _true_. > ** A < type>>. Each byte of a field having this type has its most > significant bit set if one more byte must be encoded. The 7 low-order > bits of each byte are concatenated in a specific way to form the > final, equivalent bit array. > ** A <>, a > <>, and a > <>, > which use the encoding mechanism of the variable-length bit array > field type. > ** A <> and a > <>, which are > specialized versions of the array field type and sequence field type > with a byte as the element field type. Text array and sequence fields > hold possibly null-terminated UTF-8 string values. > ** A <>, which provides a list of > one or more field types, each of which is an alternative > representation of the same binary field. > + > This mechanism _may_ be used to add field types in minor revisions of > CTF{nbsp}2, while still ensuring the forward compatibility of ``old'' > consumers. >=20 > * **Modified field types**: > ** All CTF{nbsp}2 field types have an **alignment** property. Some field > types, however, impose alignment constraints to match the constraints > of CTF{nbsp}1's field types and to make consumers easier to develop. > ** The default alignment of an <> > is{nbsp}1. It used to be{nbsp}8 if the size is a multiple of{nbsp}8, > and{nbsp}1 otherwise, in CTF{nbsp}1. > ** The **`base` and `encoding` properties are removed** from the > <>. The `encoding` property is > used in CTF{nbsp}1 to indicate that a byte is a UTF-8 character, for > example, but since some UTF-8 characters are encoded on more than one > byte, this property makes no sense here. The new > <> and the > <> _may_ be used to > achieve the same result instead. > ** The **`exp` and `mant` properties are removed** from the > <>. As mentioned > above, a floating point number field type is now defined as a bit > array field type, which has a `size` property to indicate its total, > fixed size, in bits. A floating point number field encoded following > {ieee754} can be decoded knowing only this parameter, the storage > width of the bit array, from which other parameters can be deduced > according to the standard. > + > In other words, as far as CTF{nbsp}1's floating point number field type > is following {ieee754} for encoding its fields, only specific pairs of > `exp` and `mant` properties are valid: 8 and 24, 11 and 53, 15 and 113, > 20 and 237, and so on. >=20 > ** The **`encoding` property** of the < type>> **is removed**. CTF{nbsp}2 string fields always contain a > sequence of bytes which encode a string with UTF-8. If a string is > encoded with the `ascii` encoding in a CTF{nbsp}1 data stream, it's > still valid in CTF{nbsp}2 since an ASCII string is a UTF-8 string. >=20 > * The mechanism by which a relative variant field type's tag or sequence > field type's length is searched for in preceding scopes (trace packet > size, data stream packet context, event record context, event record > payload, etc.) when it's not found in the current scope is removed. It > has not be shown to be useful and it adds complexity to consumers. It > is always possible to convert a CTF{nbsp}1 metadata stream using this > behaviour to a CTF{nbsp}2 metadata stream not using it. > * **``Special'' fields**, such as the magic number field, the data > stream class ID field, and the packet's total size field, **can have > any name**: they are _tagged_ with specified tag names by the producer > for the consumer to know that they have a special meaning without > relying on reserved names like CTF{nbsp}1's `magic`, `stream_id`, and > `packet_size`. > + > With this mechanism, it's possible for a field to be tagged multiple > times, with different meanings. >=20 > * The frequency of a < clock class>> no longer defaults to the arbitrary 1{nbsp}GHz (it's a > mandatory property). > * There is no more lexical scopes to limit the scope of field type > aliases and other definitions. < aliases>> must have unique names within the whole metadata stream. >=20 >=20 > =3D=3D CTF{nbsp}2 actors >=20 > There are two main _actors_ when it comes to tracing: >=20 > Producer:: > A software or hardware system which produces (writes) the streams of a > trace. > + > A trace producer is often called a _tracer_. > + > A producer is only concerned with how to write the metadata stream and > how to encode supported values as CTF{nbsp}2 data fields and serialize > them to one or more data streams. >=20 > Consumer:: > A software or hardware system which consumes (reads) the streams of a > trace. > + > A trace consumer is often called a _trace viewer_ or a _trace analyzer_. > + > A consumer is only concerned with how to read and interpret the metadata > stream, and how to deserialize CTF{nbsp}2 data fields from data streams > and decode them to retrieve the values they represent. >=20 > Note that a piece of software can be both a consumer and a producer. > This is the case of a trace converter, for example. >=20 >=20 > =3D=3D What is a CTF{nbsp}2 trace? >=20 > A _CTF{nbsp}2 trace_ is a set of zero or more < streams>> and exactly one <>. >=20 > The data streams contain actual packets of event records, while the > metadata stream contains information on how to interpret the data > streams of the same trace. >=20 > The support or transport of data streams and the metadata stream is not > specified. A stream _may_ be serialized as a single file on a file > system, or it _may_ be sent over the network using TCP, to name a few > examples. The mechanism to identify the metadata stream amongst the > streams of a trace is also not specified. >=20 >=20 > [[metadata-types]] > =3D=3D=3D Metadata types >=20 > Many of the following sections of this document describe the _required_ > and _optional_ properties of _metadata mdt:map values_. All the metadata > mdt:map values can be represented using a defined set of types. The > values allowed by those types have no specific textual or binary > representation. >=20 > To avoid any confusion with field types and JSON types, the `m-` prefix > is used before the names of the metadata types. An **m-value** is any > value allowed by any of the following types. >=20 > [NOTE] > .Not to be confused with _field types_! > =3D=3D=3D=3D > The metadata types define the possible values that can be used to define > the metadata mdt:map values, for example: >=20 > * Integer field type mdt:map > * Structure field type mdt:map > * Event record class mdt:map > * Data stream class mdt:map >=20 > As an example, here's a JSON representation of a possible integer field > type mdt:map: >=20 > [source,json] > ---- > { > "field-type": "int", > "size": 23, > "alignment": 16 > } > ---- >=20 > Here: >=20 > * The whole JSON object represents an mdt:map value. > * `"field-type"` represents an mdt:string value, which is an mdt:map key > here. > * `23` represents an mdt:number value. >=20 > The whole mdt:map value represents an integer field type. This integer > field type can be used to encode and decode integer values to and from > binary data fields, depending on where exactly this mdt:map is placed > within the whole <>. > =3D=3D=3D=3D >=20 > The metadata types are: >=20 > mdt:null:: > Nullable type: the only possible value of this type is the _null_ > value. >=20 > mdt:bool:: > Boolean type, that is, the following set of values: > + > -- > * _True_ > * _False_ > -- >=20 > mdt:int:: > Integer type, that is, the set of all the negative and positive > integer values. >=20 > mdt:number:: > Number type, that is, the set of all the rational numbers that can be > represented with the > https://en.wikipedia.org/wiki/Decimal_representation[decimal > representation]. >=20 > mdt:string:: > String type, that is, the set of all the possible finite sequences of > Unicode characters, including the zero-length sequence. >=20 > mdt:array:: > Array type, that is, the set of all the possible finite sequences of > any m-values, including the zero-length sequence. >=20 > mdt:map:: > Unordered map type, that is, the set of all the possible sets of > (mdt:string value, m-value) pairs, including the zero-length set. The > mdt:string value in a given pair is called the _key_ of the > association. An association _may_ also be called a _property_. >=20 > NOTE: For reasons of brevity and style, the word _value_ after a > metadata type name is sometimes discarded in this text. For example, > _you can use an mdt:int to..._ means _you can use an mdt:int value > to..._. >=20 >=20 > [[json]] > =3D=3D=3D Bidirectional association between metadata types and JSON value= s >=20 > The <> can be bidirectionally mapped to > http://json.org/[JSON] values as follows: >=20 > .Bidirectional association between metadata types and JSON values > [options=3D"header"] >|=3D=3D=3D >|Metadata type |JSON values >=20 >|mdt:null >| > The _null_ m-value is mapped to the JSON `null` value. >=20 >|mdt:bool >| > The _true_ and _false_ m-values are mapped to the `true` and `false` > JSON values respectively. >=20 >|mdt:int >| > Any allowed m-value is mapped to a JSON number without a > fractional part or to a <>. >=20 >|mdt:number >| > Any allowed m-value is mapped to a JSON number or to > a <>. >=20 >|mdt:string >| > Any allowed value is mapped to a JSON string. >=20 >|mdt:array >| > Any finite sequence is mapped to a JSON array, where each > m-value is mapped to a JSON value using this table. >=20 >|mdt:map >| > Any set of associations is mapped to a JSON object, where each pair's > mdt:string value is mapped to a key (JSON string, using this table) in > this JSON object, and its associated m-value is mapped to a JSON value > using this table and associated to this key. >|=3D=3D=3D >=20 > All the examples of m-values in this document use this mapping to show > textual representations. >=20 >=20 > [[const-int]] > =3D=3D=3D=3D Constant integer JSON object >=20 > Unfortunately, JSON does not support binary, octal, or hexadecimal > constant integers. Also, it is known that some JSON parsers have a > limited support for big integers (generally, integer values which do not > fit a 64-bit representation). A constant integer JSON object can > represent an mdt:int value or an mdt:number value without a fractional > part. >=20 > It is _recommended_ to use a constant integer JSON object instead of a > JSON number when the mdt:int or mdt:number value to represent is lesser > than -9223372036854775808 or greater than 9223372036854775807 (signed > 64-bit range). >=20 > .Constant integer JSON object properties > [options=3D"header"] >|=3D=3D=3D >|Name |Allowed JSON values |Description |Required? |Default value >=20 >|`base` >|`2`, `8`, `10`, and `16` >|Radix of the number. >|Optional >|`10` >=20 >|`value` >|JSON string >| > Integer's value using the digits allowed by the base, _without_ the > typical `0b`/`0`/`0x` prefix. The string _may_ be prefixed with `-` > (U+002D) if the value is negative. >|Required >| >|=3D=3D=3D >=20 > .Constant integer JSON object: positive decimal integer > =3D=3D=3D=3D > Equivalent to 2876321721982327: >=20 > [source,json] > {"value": "2876321721982327"} > =3D=3D=3D=3D >=20 > .Constant integer JSON object: negative binary integer > =3D=3D=3D=3D > Equivalent to -253339: >=20 > [source,json] > {"base": 2, "value": "-111101110110011011"} > =3D=3D=3D=3D >=20 > .Constant integer JSON object: positive octal integer > =3D=3D=3D=3D > Equivalent to 420: >=20 > [source,json] > {"base": 8, "value": "644"} > =3D=3D=3D=3D >=20 > .Constant integer JSON object: positive hexadecimal integer > =3D=3D=3D=3D > Equivalent to 3735928559: >=20 > [source,json] > {"base": 16, "value": "deadbeef"} > =3D=3D=3D=3D >=20 > .Constant integer JSON object: negative decimal integer > =3D=3D=3D=3D > Equivalent to -2317: >=20 > [source,json] > {"value": "-2317"} > =3D=3D=3D=3D >=20 >=20 > [[metadata-array]] > =3D=3D=3D Metadata mdt:array >=20 > The metadata mdt:array is an mdt:array of **fragments** which contains > all the metadata information of a given trace. >=20 > A fragment is an m-value. >=20 > A fragment is either: >=20 > * The version fragment, that is, an mdt:string which is always `CTF 2`. > * One of the other allowed fragments, which are described in the upper > layers of CTF{nbsp}2. >=20 > The first fragment in the metadata mdt:array is always the version > fragment. It is followed by one or more fragments, as described by the > upper layers of CTF{nbsp}2. >=20 >=20 > [[metadata-stream]] > =3D=3D=3D Metadata stream >=20 > A _metadata stream_ is the <> of a > <>, that is, a UTF-8 JSON array which > is written by the producer to describe the data streams of the same > trace. >=20 > The rationale for choosing JSON over another representation, for example > TSDL (CTF{nbsp}1's metadata language), is as follows: >=20 > . JSON can represent all the possible m-values using the > <>. > . JSON is a simple language to consume. A very basic JSON parser > can be written in a few hundred lines of C code. Moreover, tested > and documented JSON parsers exist for all the major programming > languages. > . JSON is a very simple language to produce. > . JSON strings support Unicode. >=20 > One of the <> of CTF{nbsp}2 is to make > consumption as easy as possible. Relieving the burden of implementing a > custom TSDL parser is a substantial part of how this goal is achieved. >=20 > Keep in mind that this JSON metadata is expected to be generated by > machines, thus shortcuts that would save time to human beings are > avoided in favor of easier consumption, but without compromising easy > and fast machine generation. >=20 >=20 > [[data-streams]] > =3D=3D=3D Data streams >=20 > A CTF{nbsp}2 _data stream_ is a sequence of packets. >=20 > Each packet starts with an _optional_ header field followed by an > _optional_ context field, after which is a sequence of event records. >=20 > An event record starts with an _optional_ header, followed by an > _optional_ context field defined at the data stream class level, > followed by an _optional_ context field defined at the event record > class level, followed by an _optional_ payload field. An event record's > total binary size _must_ be greater than 0. >=20 >=20 > =3D=3D=3D Summary >=20 > A _CTF{nbsp}2 trace_ is a set of: >=20 > * One <>, which is the UTF-8 JSON > representation of a <>. A metadata > mdt:array is an mdt:array containing fragments. Fragments are m-values > which describe properties of the trace and the binary layouts of its > various parts. > * Zero or more <>. A data stream contains > packets. A packet contains event records. The layout of packet headers > and contexts, and of event record headers, contexts, and payloads, are > described by the metadata stream of the same trace. >=20 >=20 > =3D=3D Structure of the CTF{nbsp}2 specification >=20 > We suggest that the concepts of CTF{nbsp}2 be presented in the > specification document as **three layers**: >=20 > . The first layer, named the <> > (CTFFER), shows **how to encode common programming language values as > binary data fields according to the their descriptions, field types**. > This is a serialization protocol. This is the foundation of > CTF{nbsp}2, in that the other layers need data fields to have any > meaning. This layer is independent of the tracing domain, in that it > can be used to encode any self-described bit stream for any > application. >=20 > . The <> adds the concept of **packets** and > **event records**, and how different layouts of packet header and > context fields, and of event record header, context, and payload > fields, _may_ exist within different data streams and event records of > the same trace thanks to **data stream class fragments** and **event > record class fragments** in the metadata stream. > + > This layer also introduces the field type alias fragment and the trace > class fragment. >=20 > . The <> adds the concept of **time** (clocks). > Clocks are essential data stream variables in a CTF{nbsp}2 trace > because they associate event records and packets with one or more > points in time. Data stream clocks are sampled by the producer when > writing specific, regular data fields. They are updated by the > consumer when reading the corresponding fields. Data stream clocks are > described by **data stream clock class fragments** in the metadata > stream. >=20 > CTF{nbsp}2 is designed so that each layer _may_ be implemented in its > own software package. This structure separates the concepts of > CTF{nbsp}2 into different sections of the text, making it more easy to > read. This structure should also make testing more easy. >=20 > Each layer depends on the previous one. >=20 >=20 > =3D=3D=3D Compliance >=20 > A _CTF{nbsp}2 producer_, either a piece of software or a machine, > _must_ implement the first two layers of did:CTF2-SPEC-2.0. >=20 > A _CTF{nbsp}2 consumer_, either a piece of software or a machine, > _must_ implement all three layers of did:CTF2-SPEC-2.0. >=20 >=20 > [[ctffer]] > =3D=3D Layer 1: CTF{nbsp}2 field encoding rules (CTFFER) >=20 > The _CTF{nbsp}2 field encoding rules_, or CTFFER, dictate how to > serialize a _value_ to a binary data field by using the properties of a > _field type_, that is, an mdt:map which describes a set of possible > binary data field values. The field type can later be used to > deserialize a data field back to a value. >=20 > The representation of a _value_ depends on the programming language in > which the CTF{nbsp}2 producer or consumer is written here. For example, > a producer written in C _may_ serialize an `int` variable as a > CTF{nbsp}2 integer field described by an < type>> having the appropriate size, alignment, byte order, and > signedness properties to accommodate any value that an `int` variable > could hold for a specific architecture (32-bit for IA-32 and 16-bit for > AVR, for example). However, since a Python{nbsp}3 `int` object can hold > any integer value, a better choice for a Python{nbsp}3 producer would be > to serialize such an object as a CTF{nbsp}2 variable-length integer > field described by a < type>>. >=20 > The following subsections only describe how to encode values as binary > fields by using field types. The specification does not suggest specific > field type configurations. It is up to the producer side to choose > appropriate field type properties depending on its environment. The > procedures to encode values presented in the following subsections are > very generic: they take no account of optimizations that would be > possible in specific situations. For example, it is often possible, in > programming languages which are aware of their memory layout, to encode > a whole complex structure of values by a simple memory copy to the data > stream, as far as appropriate field types are used to describe this > exact memory layout for further correct decoding. It is in order to > satisfy those situations, which support faster producers, that field > types are flexible: a produder can encode some value in various ways by > choosing the resulting field's alignment, byte order, size, and the rest > within the data stream destination. >=20 > The values which can be encoded are: >=20 > Null values:: > A value with no size, which usually represents missing or unknown > data. >=20 > Bit array values:: > A finite sequence of contiguous bits without a specific meaning. The > required size, in bits, to represent a bit array value can be known > statically or dynamically. >=20 > Boolean values:: > A bit array value which is either _false_ (all bits are cleared) or > _true_ (anything else). >=20 > Integer values:: > A bit array value which represents an integer (signed or not). >=20 > Number values:: > A real number which can be represented with {ieee754}. >=20 > Enumeration values:: > An integer value with an associated label (known by its type). >=20 > String values:: > A finite sequence, with a length known dynamically, of Unicode > characters. >=20 > Structure values:: > A finite sequence of values of different types. >=20 > Array values:: > A finite sequence, with a length known statically, of values sharing > the same type. >=20 > Sequence values:: > A finite sequence, with a length known dynamically, of values sharing > the same type. >=20 > The CTFFER also support _variant_ and _union_ fields. A variant field is > an encoded value of a given type amongst many possible types. This type > is dynamically chosen by a tag (a previously encoded enumeration value). > A union field is a data field which represents, at the same time, > different values of different types. >=20 >=20 > [[byte-order]] > =3D=3D=3D Byte order mdt:string >=20 > A _byte order mdt:string_ is one of the following values: >=20 > `default`:: > Use default byte order. >=20 > `be`:: > Big-endian. >=20 > `le`:: > Little-endian. >=20 > CTF{nbsp}2 does not support middle or mixed endianness. >=20 >=20 > [[user-attrs]] > =3D=3D=3D User attributes mdt:map >=20 > Many metadata mdt:map values described in this document may have a > `user-attrs` property, which _must_ be set to an mdt:map, if set at all. >=20 > Each key of the user attributes mdt:map is a **namespace**. The value of > a given key is the custom user attribute within this namespace (any > m-value is valid). >=20 > The format of a namespace is not specified. It is _recommended_ to use a > URI, or at least to include a domain name owned by the organization > defining the attributes nested under this namespace. A UUID is also a > rational option. >=20 > What to do with those user attributes from a consumer's standpoint is > not specified by this document. The values of those attributes are _not_ > needed to decode the CTF data streams, and _may_ be safely ignored by > any CTF{nbsp}2 consumer. >=20 > It is expected that ``industrial standards'' defining sets of useful > attributes within given namespaces will emerge naturally over time. > Producers and consumers supporting the same attributes can enhance the > experience of the whole tracing ecosystem. >=20 > It is expected that user attributes usually fall into one of the > following categories: >=20 > * **Model**: Information about the application data model of an object. > * **Textual style**: Style attributes/hints that could be applied to a > textual rendering of the object (color, font attributes, print format, > etc.). > * **Graphical style**: Style attributes/hints that could be applied to a > graphical output of the object. >=20 > .User attributes mdt:map with namespace `diamon.org/ctf/ns/std` > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > { > "diamon.org/ctf/ns/std": { > "base": 16 > } > } > =3D=3D=3D=3D >=20 > .User attributes mdt:map with different namespaces > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > { > "diamon.org/ctf/ns/std": { > "name": "sched_switch", > "ns": "lttng.org/ctf-ns/modules/2.9" > }, > "lttng.org/ctf-ns": { > "tmp-event": true, > "ignore-ip": true > } > } > =3D=3D=3D=3D >=20 > .User attributes mdt:map with namespace `mytracer.org/ctf-ns/hints` > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > { > "mytracer.org/ctf-ns/hints": { > "format-string": "{src} sent {size} bytes to {dst} at {addr}" > } > } > =3D=3D=3D=3D >=20 > Although _not recommended_, an empty mdt:string is a valid namespace: >=20 > .User attributes mdt:map with empty namespace > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > { > "": { > "my-option": 23, > "include": ["this", "and", "that"] > } > } > =3D=3D=3D=3D >=20 > The value of user attributes for a given namespace need not be an > mdt:map: >=20 > .User attributes mdt:map with an mdt:number attribute > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > { > "my namespace": -17.22 > } > =3D=3D=3D=3D >=20 >=20 > [[scope]] > =3D=3D=3D Scope >=20 > A _scope_ is a specific field within a data stream. The exact location > of a scope within a data stream depends on the current encoding context. > The upper layer defines which scopes are available in its context and > how to find them by name (mdt:string). >=20 >=20 > [[field-path]] > =3D=3D=3D Field path m-value >=20 > A _field path m-value_, used by <> and > <> field types, is a path leading to a > previously encoded data field by ``digging'' into structure and union > fields. It can be either _relative_ (starting from a known field), or > _absolute_ (starting from a user-specified scope field). >=20 >=20 > [[abs-field-path]] > =3D=3D=3D=3D Absolute field path mdt:map >=20 > An _absolute field path mdt:map_ defines a field path from a specific > scope. >=20 > .Absolute field path mdt:map properties > [options=3D"header"] >|=3D=3D=3D >|Name |Type |Description |Required? |Default m-value >=20 >|`scope` >|mdt:string >|Name of the scope. >|Required >| >=20 >|`path` >|mdt:array of mdt:string values >|Field names, from the scope's root field, to follow to reach the > desired field (last element of this path). >|Required >| >|=3D=3D=3D >=20 > The `path` property _may_ be empty: this targets the scope field itself. >=20 > .Absolute field path mdt:map > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > { > "scope": "data-stream-packet-context", > "path": ["path", "to", "cpu_id"] > } > =3D=3D=3D=3D >=20 >=20 > =3D=3D=3D=3D Relative field path mdt:array >=20 > A _relative field path mdt:array_ is an mdt:array of field names > (mdt:string values) to follow, starting from the sequence/variant field > using the field path. >=20 > .Relative field path array > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > ---- > ["path", "to", "cpu_id"] > ---- > =3D=3D=3D=3D >=20 >=20 > =3D=3D=3D=3D Field lookup mechanism >=20 > Field path elements are names of _structure_ or _union_ fields. If one > of those fields is a variant field, then the lookup _must_ recursively > find the variant's current field. >=20 > For example, let's say we have the following scope named `my-scope` > (_FT_ means _field type_): >=20 > a: int FT > b: struct FT > v: variant FT > choice1: int FT > choice2: int FT > choice3: struct FT > a: int FT > b: float FT > choice4: enum FT > i: int FT >=20 > All the following field path m-values are valid sequence field type > lengths here: >=20 > * Trivial: > + > -- > [source,json] > { > "scope": "my-scope", > "path": ["a"] > } > -- > + > and > + > -- > [source,json] > { > "scope": "my-scope", > "path": ["b", "i"] > } > -- >=20 > * If `choice1`, `choice2`, or `choice4` is selected > in `v` when performing the lookup: > + > -- > [source,json] > { > "scope": "my-scope", > "path": ["b", "v"] > } > -- >=20 > * If `choice3` is selected in `v` when performing the lookup: > + > -- > [source,json] > { > "scope": "my-scope", > "path": ["b", "v", "a"] > } > -- >=20 > Relative field paths are looked up by going back into the current > structure/union field, and then back into the current structure/union > field's parent structure/union field, and so on. For example: >=20 > z: int FT <------------------------------------. > y: struct FT | > a: int FT | > b: struct FT | > c: int FT | > d: string FT | > e: struct FT | > f: int FT <----------------------------. | > g: int FT | | > h: int FT | | > i: struct FT | | > j: int FT | | > k: sequence FT, length: ["b", "e", "f"] -' | > x: sequence FT, length: ["z"] -----------------' >=20 >=20 > [[enc-ctx]] > =3D=3D=3D Encoding context >=20 > We define a _current encoding head_, an integer variable which is > initialized by the upper layer before encoding a value to a data field. > This variable is the current position of the writing ``head''. >=20 > When a bit array is written, the current encoding head is updated by > adding the written size to it. >=20 > Before encoding a value as a data field, the current encoding head > _must_ be **aligned** to respect the alignment requirements of said > field (given by its field type). The following operation can be used to > update the current encoding head _p_ to the beginning of a field with an > effective alignment of _a_ (bits): >=20 > p =3D (p + a - 1) & -a >=20 > For example, if the current encoding head is 37 and the alignment of the > next field to write is 8, then the current encoding head _must_ be > updated to 40 before writing the field. If the current encoding head is > 48 and the alignment of the next field to write is 8, then the current > head is already aligned. >=20 >=20 > [[field-type-m-value]] > =3D=3D=3D Field type m-value >=20 > A _field type m-value_ can be either an mdt:string value or an mdt:map > value: >=20 > * If it's an mdt:string value, it is the name of a field type alias. The > upper layer has a mapping of field type alias names to complete field > types (mdt:map values). > + > -- > .Field type alias > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > "my alias" > =3D=3D=3D=3D > -- >=20 > * [[field-type-m-map]]If it's an mdt:map value, it has the following > base properties: > + > -- > .Field type mdt:map properties > [options=3D"header"] >|=3D=3D=3D >|Name |Type |Description |Required? |Default m-value >=20 >|`field-type` >|mdt:string >|Field type's class. >|Required >| >=20 >|`user-attrs` >|<> >|User attributes. >|Optional >|Empty mdt:map value. >=20 >|`alignment` >|mdt:int >|Field's alignment (_must_ be a power of two, greater than 0). >|Optional >|1 >|=3D=3D=3D > -- > + > The following table summarizes the available field types: > + > -- > .Available field types > [options=3D"header"] >|=3D=3D=3D >|`field-type` property (mdt:string) |Name |Inherits properties from >=20 >|`null` >|<>. >|<>. >=20 >|`bitarray` >|<>. >|<>. >=20 >|`bool` >|<>. >|<>. >=20 >|`int` >|<>. >|<>. >=20 >|`float` >|<>. >|<>. >=20 >|`enum` >|<>. >|<>. >=20 >|`varbitarray` >|<>. >|<>. >=20 >|`varbool` >|<>. >|<>. >=20 >|`varint` >|<>. >|<>. >=20 >|`varenum` >|<>. >|<>. >=20 >|`string` >|Null-terminated UTF-8 <>. >|<>. >=20 >|`struct` >|<>. >|<>. >=20 >|`array` >|<>. >|<>. >=20 >|`textarray` >|<>. >|<>. >=20 >|`sequence` >|<>. >|<>. >=20 >|`textsequence` >|<>. >|<>. >=20 >|`variant` >|<>. >|<>. >=20 >|`union` >|<>. >|<>. >|=3D=3D=3D > -- > + > Note that, on the decoding side, any unknown field type must be ignored, > and any unknown field type mdt:map property must also be ignored. >=20 >=20 > [[null-field-type]] > =3D=3D=3D=3D Null field type >=20 > A _null field type_ describes null fields. >=20 > A null value usually represents missing or unknown data. >=20 > .Null field type mdt:map properties > [options=3D"header"] >|=3D=3D=3D >|Name |Type |Description |Required? |Default m-value >=20 >|`field-type` >|mdt:string >|_Must_ be set to `null`. >|Required >| >=20 >|`user-attrs` >|<> >|User attributes. >|Optional >|Empty mdt:map value. >=20 >|`alignment` >|mdt:int >|Field's alignment (_must_ be a power of two, greater than 0). >|Optional >|1 >|=3D=3D=3D >=20 > .Null field type > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > { > "user-attrs": { > "my-namespace": { > "my-attr": "desc" > } > }, > "field-type": "null", > "alignment": 4 > } > =3D=3D=3D=3D >=20 > .Minimal null field type > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > {"field-type": "null"} > =3D=3D=3D=3D >=20 >=20 > =3D=3D=3D=3D=3D Encode a null value as a null field >=20 > To encode a null value using a null field type: >=20 > * <> the current encoding head using the `alignment` > property of the null field type. >=20 >=20 > [[bitarray-field-type]] > =3D=3D=3D=3D Bit array field type >=20 > A _bit array field type_ describes bit array fields. >=20 > A bit array value is a simple array of bits. It is not an integer > value (it has no signedness). >=20 > .Bit array field type mdt:map properties > [options=3D"header"] >|=3D=3D=3D >|Name |Type |Description |Required? |Default m-value >=20 >|`field-type` >|mdt:string >|_Must_ be set to `bitarray`. >|Required >| >=20 >|`user-attrs` >|<> >|User attributes. >|Optional >|Empty mdt:map value. >=20 >|`alignment` >|mdt:int >|Field's alignment (_must_ be a power of two, greater than 0). >|Optional >|1 >=20 >|`size` >|mdt:int >|Field's size, in bits (_must_ be greater than 0). >|Required >| >=20 >|`byte-order` >|<> >|Field's byte order. >|Optional >|`default` >|=3D=3D=3D >=20 > .Bit array field type > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > { > "field-type": "bitarray", > "alignment": 16, > "size": 5, > "byte-order": "le", > "user-attrs": { > "my-namespace": { > "my-attr": "desc" > } > } > } > =3D=3D=3D=3D >=20 > .Minimal bit array field type > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > { > "field-type": "bitarray", > "size": 32 > } > =3D=3D=3D=3D >=20 >=20 >=20 > [[enc-bit-array-field]] > =3D=3D=3D=3D=3D Encode a bit array value as a bit array field >=20 > To encode a bit array value using a bit array field type: >=20 > . <> the current encoding head using the `alignment` > property of the bit array field type. > . Follow the rules of _Common Trace Format v1.8.2_, section 4.1.5, to > encode the bit array value according to its `byte-order` and `size` > properties. > . Add the value of the `size` property to the current encoding head. >=20 >=20 > [[bool-field-type]] > =3D=3D=3D=3D Boolean field type >=20 > A _boolean field type_ describes boolean fields. >=20 > A boolean value is a bit array value which, when all its bits are > cleared, is said to be _false_, and otherwise is said to be _true_. >=20 > .Boolean field type mdt:map properties > [options=3D"header"] >|=3D=3D=3D >|Name |Type |Description |Required? |Default m-value >=20 >|`field-type` >|mdt:string >|_Must_ be set to `bool`. >|Required >| >=20 >|`user-attrs` >|<> >|User attributes. >|Optional >|Empty mdt:map value. >=20 >|`alignment` >|mdt:int >|Field's alignment (_must_ be a power of two, greater than 0). >|Optional >|1 >=20 >|`size` >|mdt:int >|Field's size, in bits (_must_ be greater than 0). >|Required >| >=20 >|`byte-order` >|<> >|Field's byte order. >|Optional >|`default` >|=3D=3D=3D >=20 > .Boolean field type > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > { > "field-type": "bool", > "size": 8, > "byte-order": "be", > "user-attrs": { > "my-namespace": { > "my-attr": "desc" > } > } > } > =3D=3D=3D=3D >=20 > .Minimal boolean field type > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > { > "field-type": "bool", > "size": 8 > } > =3D=3D=3D=3D >=20 >=20 > [[enc-bool-field]] > =3D=3D=3D=3D=3D Encode a boolean value as a boolean field >=20 > To encode a boolean value using a boolean field type: >=20 > . Encode the boolean value as a bit array value. This process is > platform-dependent. > . Follow the rules of how to < value as a bit array field>>. >=20 >=20 > [[int-field-type]] > =3D=3D=3D=3D Integer field type >=20 > An _integer field type_ describes integer fields. >=20 > .Integer field type mdt:map properties > [options=3D"header"] >|=3D=3D=3D >|Name |Type |Description |Required? |Default m-value >=20 >|`field-type` >|mdt:string >|_Must_ be set to `int`. >|Required >| >=20 >|`user-attrs` >|<> >|User attributes. >|Optional >|Empty mdt:map value. >=20 >|`alignment` >|mdt:int >|Field's alignment (_must_ be a power of two, greater than 0). >|Optional >|1 >=20 >|`size` >|mdt:int >|Field's size, in bits (_must_ be greater than 0). >|Required >| >=20 >|`byte-order` >|<> >|Field's byte order. >|Optional >|`default` >=20 >|`signed` >|mdt:bool >|_True_ if the integer field represents a signed integer value, in > which case the field has the two's complement format. >|Optional >|_False_ >|=3D=3D=3D >=20 > .Integer field type > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > { > "field-type": "int", > "alignment": 16, > "size": 5, > "byte-order": "le", > "signed": true, > "user-attrs": { > "my-namespace": { > "my-attr": "desc" > } > } > } > =3D=3D=3D=3D >=20 > .Minimal integer field type > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > { > "field-type": "int", > "size": 32 > } > =3D=3D=3D=3D >=20 >=20 > [[enc-int-field]] > =3D=3D=3D=3D=3D Encode an integer value as an integer field >=20 > To encode an integer value using an integer field type: >=20 > . Encode the integer value as a bit array value. Follow the two's > complement representation if the integer value is signed. This > process is platform-dependent. > . Follow the rules of how to < value as a bit array field>>. >=20 >=20 >=20 > [[float-field-type]] > =3D=3D=3D=3D Floating point number field type >=20 > A _floating point number field type_ describes floating point number > fields encoded with {ieee754}. >=20 > A number value (real number) can be encoded as a floating point number > field provided it is representable with one of the versions of > {ieee754}. >=20 > .Floating point number field type mdt:map properties > [options=3D"header"] >|=3D=3D=3D >|Name |Type |Description |Required? |Default m-value >=20 >|`field-type` >|mdt:string >|_Must_ be set to `float`. >|Required >| >=20 >|`user-attrs` >|<> >|User attributes. >|Optional >|Empty mdt:map value. >=20 >|`alignment` >|mdt:int >|Field's alignment (_must_ be a power of two, greater than 0). >|Optional >|1 >=20 >|`size` >|mdt:int >|Field's size, in bits (_must_ be greater than 0). >|Required >| >=20 >|`byte-order` >|<> >|Field's byte order. >|Optional >|`default` >|=3D=3D=3D >=20 > The value of the `size` property corresponds to the value of the > parameter _k_ (storage width in bits) in Table 3.5, _Binary interchange > format parameters_, of the IEEE Std 754-2008 document. All the other > parameters of the format needed to encode and decode the floating point > number value can be deduced from the value of _k_. >=20 > .Floating point number field type describing the basic binary64 format > =3D=3D=3D=3D > This floating point number field type describes fields encoded with the > parameters of the basic binary64 format, which is the encoding used by > the ``double'' type of most programming languages. >=20 > JSON representation: >=20 > [source,json] > { > "field-type": "float", > "alignment": 8, > "size": 64, > "user-attrs": { > "my-namespace": { > "my-attr": "desc" > } > } > } > =3D=3D=3D=3D >=20 > .Minimal floating point number field type > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > { > "field-type": "float", > "size": 32 > } > =3D=3D=3D=3D >=20 >=20 > [[enc-float-field]] > =3D=3D=3D=3D=3D Encode a number value as a floating point number field >=20 > To encode a number value using a floating point number field type: >=20 > . Encode the number value as a bit array value following {ieee754}. This > process is platform-dependent. > . Follow the rules of how to < value as a bit array field>>. >=20 >=20 > [[enum-field-type-member]] > =3D=3D=3D=3D Enumeration field type member m-value >=20 > An _enumeration field type member m-value_ represents the range of > values mapped to the label of an < type>> member. >=20 > An enumeration field type member m-value is either: >=20 > * An mdt:int value, in which case the member's label is mapped to this > value. For example (JSON representation): > + > [source,json] > ---- > 28 > ---- >=20 > * An mdt:map with the `lower` and `upper` properties (mdt:int values) > which indicate the lower (inclusive) and upper (inclusive) limits of a > range to which the member's label is mapped. For example (JSON > representation): > + > [source,json] > ---- > {"lower": -3, "upper": 17} > ---- >=20 >=20 > [[enum-field-type]] > =3D=3D=3D=3D Enumeration field type >=20 > An _enumeration field type_ describes enumeration fields. >=20 > An enumeration value is an integer value mapped to a label. >=20 > .Enumeration field type mdt:map properties > [options=3D"header"] >|=3D=3D=3D >|Name |Type |Description |Required? |Default m-value >=20 >|`field-type` >|mdt:string >|_Must_ be set to `enum`. >|Required >| >=20 >|`user-attrs` >|<> >|User attributes. >|Optional >|Empty mdt:map value. >=20 >|`alignment` >|mdt:int >|Field's alignment (_must_ be a power of two, greater than 0). >|Optional >|1 >=20 >|`size` >|mdt:int >|Field's size, in bits (_must_ be greater than 0). >|Required >| >=20 >|`byte-order` >|<> >|Field's byte order. >|Optional >|`default` >=20 >|`signed` >|mdt:bool >|_True_ if the integer field is signed, in which case the field has > the two's complement format. >|Optional >|_False_ >=20 >|`members` >|mdt:map which maps label names (mdt:string) to mdt:array values of > <>. >|Enumeration field type's members. >|Required >| >|=3D=3D=3D >=20 > .Enumeration field type > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > { > "field-type": "enum", > "alignment": 16, > "size": 32, > "signed": true, > "members": { > "NEW": [0], > "TERMINATED": [-1], > "READY": [2, 17], > "RUNNING": [-3], > "WAITING": [ > {"lower": 19, "upper": 199}, > 1000 > ], > "RESTARTING": [ > {"base": 8, "value": "126674015"}, > {"lower": -155, "upper": -98} > ] > }, > "user-attrs": { > "my-namespace": { > "my-attr": "desc" > } > } > } >=20 > With this enumeration field type, the following enumeration fields would > have the following associated labels: >=20 > * -1: `TERMINATED` > * 17: `READY` > * -101: `RESTARTING` > * 1000: `WAITING` > * 22771725: `RESTARTING` > * 2: `READY` > * 50: `WAITING` > =3D=3D=3D=3D >=20 > .Minimal enumeration field type > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > { > "field-type": "enum", > "members": { > "": [0] > } > } > =3D=3D=3D=3D >=20 >=20 > [[enc-enum-field]] > =3D=3D=3D=3D=3D Encode an enumeration value as an enumeration field >=20 > To encode an enumeration value (which is an integer value) using an > enumeration field type: >=20 > * Follow the rules of how to < an integer field>>. >=20 >=20 > [[string-field-type]] > =3D=3D=3D=3D String field type >=20 > A _string field type_ describes string fields. >=20 > A string value is a finite sequence of Unicode characters. >=20 > .String field type mdt:map properties > [options=3D"header"] >|=3D=3D=3D >|Name |Type |Description |Required? |Default m-value >=20 >|`user-attrs` >|<> >|User attributes. >|Optional >|Empty mdt:map value. >=20 >|`field-type` >|mdt:string >|_Must_ be set to `string`. >|Required >| >=20 >|`alignment` >|mdt:int >|Field's alignment (_must_ be a power of two, > greater than or equal to 8). >|Optional >|8 >|=3D=3D=3D >=20 > .String field type > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > { > "field-type": "string", > "alignment": 16, > "user-attrs": { > "my-namespace": { > "my-attr": "desc" > } > } > } > =3D=3D=3D=3D >=20 > .Minimal string field type > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > {"field-type": "string"} > =3D=3D=3D=3D >=20 >=20 > [[enc-string-field]] > =3D=3D=3D=3D=3D Encode a string value as a string field >=20 > To encode a string value using a string field type: >=20 > . <> the current encoding head using the `alignment` > property of the string field type. > . For each Unicode character of the string value: > .. Encode this Unicode character as a sequence of 8-bit bit arrays > (bytes) following UTF-8. This process is platform-dependent. > .. For each resulting UTF-8 byte of this character: > *** Follow the rules of how to < array value as a bit array field>>. > . Follow the rules of how to < an integer field>> to encode the UTF-8 null character (U+0000). >=20 >=20 > [[varbitarray-field-type]] > =3D=3D=3D=3D Variable-length bit array field type >=20 > A _variable-length bit array field type_ describes variable-length bit > array fields. >=20 > A bit array value of any size that is a multiple of 7{nbsp}bits, and at > least 7{nbsp}bits, can be dynamically encoded as a variable-length bit > array field. >=20 > .Variable-length bit array field type mdt:map > [options=3D"header"] >|=3D=3D=3D >|Name |Type |Description |Required? |Default m-value >=20 >|`field-type` >|mdt:string >|_Must_ be set to `varbitarray`. >|Required >| >=20 >|`user-attrs` >|<> >|User attributes. >|Optional >|Empty mdt:map value. >=20 >|`alignment` >|mdt:int >|Field's alignment (_must_ be a power of two, > greater than or equal to 8). >|Optional >|8 >|=3D=3D=3D >=20 > .Variable-length bit array field type > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > { > "field-type": "varbitarray", > "user-attrs": { > "my-namespace": { > "my-attr": "desc" > } > } > } > =3D=3D=3D=3D >=20 > .Minimal variable-length bit array field type > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > {"field-type": "varbitarray"} > =3D=3D=3D=3D >=20 >=20 > [[enc-varbitarray-field]] > =3D=3D=3D=3D=3D Encode a bit array value as a variable-length bit array f= ield >=20 > To encode a bit array value using a variable-length bit array field type: >=20 > . <> the current encoding head using the `alignment` > property of the variable-length bit array field type. > . Encode the bit array value as a bit array field following the unsigned > https://en.wikipedia.org/wiki/LEB128[LEB128] format. > . Add the encoded variable-length bit array field's size (_not_ the > original bit array value's size) to the current encoding head. >=20 >=20 > [[varbool-field-type]] > =3D=3D=3D=3D Variable-length boolean field type >=20 > A _variable-length boolean field type_ describes variable-length boolean > fields. >=20 > A boolean value is a bit array value which, when all its bits are > cleared, is said to be _false_, and otherwise is said to be _true_. >=20 > A boolean value of any size that is a multiple of 7{nbsp}bits, and at > least 7{nbsp}bits, can be dynamically encoded as a variable-length bit > array field. >=20 > .Variable-length boolean field type mdt:map properties > [options=3D"header"] >|=3D=3D=3D >|Name |Type |Description |Required? |Default m-value >=20 >|`field-type` >|mdt:string >|_Must_ be set to `varbool`. >|Required >| >=20 >|`user-attrs` >|<> >|User attributes. >|Optional >|Empty mdt:map value. >=20 >|`alignment` >|mdt:int >|Field's alignment (_must_ be a power of two, > greater than or equal to 8). >|Optional >|8 >|=3D=3D=3D >=20 > .Variable-length boolean field type > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > { > "field-type": "varbool", > "alignment": 32, > "user-attrs": { > "my-namespace": { > "my-attr": "desc" > } > } > } > =3D=3D=3D=3D >=20 > .Minimal variable-length boolean field type > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > {"field-type": "varbool"} > =3D=3D=3D=3D >=20 >=20 > [[enc-varbool-field]] > =3D=3D=3D=3D=3D Encode a boolean value as a variable-length boolean field >=20 > To encode a boolean value using a variable-length boolean field type: >=20 > . Encode the boolean value as a bit array value. This process is > platform-dependent. > . Follow the rules of how to < value as a variable-length bit array field>>. >=20 >=20 > [[varint-field-type]] > =3D=3D=3D=3D Variable-length integer field type >=20 > A _variable-length integer field type_ describes variable-length integer > fields. >=20 > An integer value of any size can be encoded dynamically as a > variable-length integer field. >=20 > .Variable-length integer field type mdt:map properties > [options=3D"header"] >|=3D=3D=3D >|Name |Type |Description |Required? |Default m-value >=20 >|`field-type` >|mdt:string >|_Must_ be set to `varint`. >|Required >| >=20 >|`user-attrs` >|<> >|User attributes. >|Optional >|Empty mdt:map value. >=20 >|`alignment` >|mdt:int >|Field's alignment (_must_ be a power of two, > greater than or equal to 8). >|Optional >|8 >=20 >|`signed` >|mdt:bool >|_True_ if the integer field is signed, in which case the field has > the two's complement format. >|Optional >|_False_ >|=3D=3D=3D >=20 > .Variable-length integer field type > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > { > "field-type": "varint", > "alignment": 16, > "signed": true, > "user-attrs": { > "my-namespace": { > "my-attr": "desc" > } > } > } > =3D=3D=3D=3D >=20 > .Minimal variable-length integer field type > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > {"field-type": "varint"} > =3D=3D=3D=3D >=20 >=20 > [[enc-varint-field]] > =3D=3D=3D=3D=3D Encode an integer value as a variable-length integer fiel= d >=20 > To encode an integer value using a variable-length integer field type: >=20 > . Encode the integer value as a bit array value. Follow the two's > complement representation if the integer value is signed. Sign-extend > the bit array to the next multiple of 7{nbsp}bits. This process is > platform-dependent. > . Follow the rules of how to < value as a variable-length bit array field>>. >=20 >=20 > [[varenum-field-type]] > =3D=3D=3D=3D Variable-length enumeration field type >=20 > A _variable-length enumeration field type_ describes variable-length > enumeration fields. >=20 > An enumeration value is an integer value mapped to a label. >=20 > An enumeration value of any size can be encoded dynamically as a > variable-length enumeration field. >=20 > .Variable-length enumeration field type mdt:map properties > [options=3D"header"] >|=3D=3D=3D >|Name |Type |Description |Required? |Default m-value >=20 >|`field-type` >|mdt:string >|_Must_ be set to `varint`. >|Required >| >=20 >|`user-attrs` >|<> >|User attributes. >|Optional >|Empty mdt:map value. >=20 >|`alignment` >|mdt:int >|Field's alignment (_must_ be a power of two, > greater than or equal to 8). >|Optional >|8 >=20 >|`signed` >|mdt:bool >|_True_ if the integer field is signed, in which case the field has > the two's complement format. >|Optional >|_False_ >=20 >|`members` >|mdt:map which maps label names (mdt:string) to mdt:array values of > <>. >|Enumeration field type's members. >|Required >| >|=3D=3D=3D >=20 > .Variable-length enumeration field type > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > { > "field-type": "varenum", > "signed": true, > "members": { > "NEW": [0], > "TERMINATED": [-1], > "READY": [2, 17], > "RUNNING": [-3], > "WAITING": [ > {"lower": 19, "upper": 199}, > 1000 > ], > "RESTARTING": [ > {"base": 8, "value": "126674015"}, > {"lower": -155, "upper": -98} > ] > }, > "user-attrs": { > "my-namespace": { > "my-attr": "desc" > } > } > } >=20 > With this enumeration field type, the following enumeration fields would > have the following associated labels: >=20 > * -1: `TERMINATED` > * 17: `READY` > * -101: `RESTARTING` > * 1000: `WAITING` > * 22771725: `RESTARTING` > * 2: `READY` > * 50: `WAITING` > =3D=3D=3D=3D >=20 > .Minimal variable-length enumeration field type > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > { > "field-type": "varenum", > "members": { > "": [0] > } > } > =3D=3D=3D=3D >=20 >=20 > [[enc-varenum-field]] > =3D=3D=3D=3D=3D Encode an enumeration value as a variable-length integer = field >=20 > To encode an enumeration value (which is an integer value) using a > variable-length enumeration field type: >=20 > . Encode the integer value as a bit array value. Follow the two's > complement representation if the integer value is signed. Sign-extend > the bit array to the next multiple of 7{nbsp}bits. This process is > platform-dependent. > . Follow the rules of how to < value as a variable-length bit array field>>. >=20 >=20 > [[struct-union-variant-field]] > =3D=3D=3D=3D Structure/union/variant field type field mdt:map >=20 > A _structure/union/variant field type field mdt:map_ represents one > field of a <> or of a > <>, or one choice of a > <>. >=20 > .Structure/union/variant field type field mdt:map properties > [options=3D"header"] >|=3D=3D=3D >|Name |Type |Description |Required? |Default m-value >=20 >|`user-attrs` >|<> >|User attributes. >|Optional >|Empty mdt:map value. >=20 >|`name` >|mdt:string >|Structure/union/variant field type field's name. >|Required >| >=20 >|`field-type` >|<> >|Structure/union/variant field type field's type. >|Required >| >|=3D=3D=3D >=20 > .Structure/union/variant field type field mdt:map > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > { > "name": "src_addr", > "field-type": { > "field-type": "array", > "length": 4, > "element-field-type": "uint8" > }, > "user-attrs": { > "my-namespace": { > "my-attr": "desc" > } > } > } > =3D=3D=3D=3D >=20 >=20 > [[struct-field-type]] > =3D=3D=3D=3D Structure field type >=20 > A _structure field type_ describes structure fields. >=20 > A structure value is a finite sequence of values of different types. > This is sometimes also called a _record_. >=20 > .Structure field type mdt:map properties > [options=3D"header"] >|=3D=3D=3D >|Name |Type |Description |Required? |Default m-value >=20 >|`field-type` >|mdt:string >|_Must_ be set to `struct`. >|Required >| >=20 >|`user-attrs` >|<> >|User attributes. >|Optional >|Empty mdt:map value. >=20 >|`alignment` >|mdt:int >|Field's minimal alignment (_must_ be a power of two, greater than 0). >|Optional >|1 >=20 >|`fields` >|mdt:array of < field type field mdt:map values>> >|Fields of the structure field type. >|Optional >|Empty mdt:array value. >|=3D=3D=3D >=20 > The `alignment` property indicates the _minimal_ alignment of the > structure fields encoded with this field type. The _automatic_ alignment > of the structure field type is the greatest value amongst all the > effective alignments of the field type's fields. The _effective_ > alignment of the structure field type is the greatest value amongst the > field type's minimal and automatic alignments. >=20 > .Structure field type > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > { > "field-type": "struct", > "alignment": 8, > "fields": [ > { > "name": "timestamp_begin", > "field-type": "uint64" > }, > { > "name": "timestamp_end", > "field-type": "uint64" > }, > { > "name": "packet_size", > "field-type": "uint32" > }, > { > "name": "content_size", > "field-type": "uint32" > }, > { > "name": "core location", > "field-type": { > "field-type": "struct", > "fields": [ > { > "name": "x", > "field-type": { > "field-type": "int", > "size": 3 > } > }, > { > "name": "y", > "field-type": { > "field-type": "int", > "size": 5 > } > } > ] > } > } > ], > "user-attrs": { > "my-namespace": { > "my-attr": "desc" > } > } > } >=20 > In this example, assuming that the effective alignment of the > `timestamp_end` field is 64, and that this is the greatest alignment > amongst all the alignments of the structure field type's fields, then > the structure field type's effective alignment is also 64. > =3D=3D=3D=3D >=20 > .Minimal (empty) structure field type > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > {"field-type": "struct"} > =3D=3D=3D=3D >=20 >=20 > [[enc-struct-field]] > =3D=3D=3D=3D=3D Encode a structure value as a structure field >=20 > To encode a structure value using a structure field type: >=20 > . <> the current encoding head using the _effective_ > alignment of the structure field type. > . For each field of the structure value: > ** Encode the field's value using the field type of the > structure/union/variant field type field mdt:map at the corresponding > position in the `fields` mdt:array of the structure field type. >=20 >=20 > [[array-field-type]] > =3D=3D=3D=3D Array field type >=20 > An _array field type_ describes array fields. >=20 > An array value is a finite sequence of values. >=20 > The length of all the possible array fields represented by a given array > field type is known statically (when producing the array field type). >=20 > .Array field type mdt:map properties > [options=3D"header"] >|=3D=3D=3D >|Name |Type |Description |Required? |Default m-value >=20 >|`field-type` >|mdt:string >|_Must_ be set to `array`. >|Required >| >=20 >|`user-attrs` >|<> >|User attributes. >|Optional >|Empty mdt:map value. >=20 >|`alignment` >|mdt:int >|Field's alignment (_must_ be a power of two, greater than 0). >|Optional >|1 >=20 >|`length` >|mdt:int >|Number of elements contained in array fields described by this > field type. >|Required >| >=20 >|`element-field-type` >|<> >|Field type of the elements contained in array fields described by > this field type. >|Required >| >|=3D=3D=3D >=20 > .Array field type > =3D=3D=3D=3D > [source,json] > { > "field-type": "array", > "alignment": 64, > "length": 72, > "element-field-type": { > "field-type": "float", > "size": 64, > "byte-order": "be" > }, > "user-attrs": { > "my-namespace": { > "my-attr": "desc" > } > } > } > =3D=3D=3D=3D >=20 > .Array field type describing UUID fields > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > { > "field-type": "array", > "alignment": 8, > "length": 16, > "element-field-type": { > "field-type": "int", > "size": 8 > }, > "user-attrs": { > "my-namespace": { > "my-attr": "desc" > } > } > } > =3D=3D=3D=3D >=20 > .Minimal array field type > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > { > "field-type": "array", > "length": 32, > "element-field-type": "elem-ft" > } > =3D=3D=3D=3D >=20 >=20 > [[enc-array-field]] > =3D=3D=3D=3D=3D Encode an array value as an array field >=20 > To encode an array value using an array field type: >=20 > . <> the current encoding head using the `alignment` > property of the array field type. > . For each element of the array value: > ** Encode the value using the `element-field-type` property of the array > field type. >=20 >=20 > [[textarray-field-type]] > =3D=3D=3D=3D Text array field type >=20 > A _text array field type_ describes text array fields. It is a > specialized version of the <>. >=20 > A text array value is a finite sequence of bytes which form a UTF-8 > string. The length of all the possible text array fields represented by > a given text array field type is known statically (when producing the > text array field type). The text array value's length _may_ be greater > than the number of effective UTF-8 bytes, as long as the string is > null-terminated. In this case, the padding bytes after the UTF-8 null > character can have any value. >=20 > .Text array field type mdt:map properties > [options=3D"header"] >|=3D=3D=3D >|Name |Type |Description |Required? |Default m-value >=20 >|`field-type` >|mdt:string >|_Must_ be set to `textarray`. >|Required >| >=20 >|`user-attrs` >|<> >|User attributes. >|Optional >|Empty mdt:map value. >=20 >|`alignment` >|mdt:int >|Field's alignment (_must_ be a power of two, greater than 0). >|Optional >|1 >=20 >|`length` >|mdt:int >|Number of bytes contained in text array fields described by this > field type. >|Required >| >|=3D=3D=3D >=20 > .Text array field type > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > { > "field-type": "textarray", > "alignment": 32, > "length": 16, > "user-attrs": { > "my-namespace": { > "my-attr": "desc" > } > } > } > =3D=3D=3D=3D >=20 > .Minimal text array field type > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > { > "field-type": "textarray", > "length": 32 > } > =3D=3D=3D=3D >=20 >=20 > [[enc-textarray-field]] > =3D=3D=3D=3D=3D Encode a text array value as a text array field >=20 > A text array value is an array value with bytes as elements. >=20 > To encode a text array value using a text array field type: >=20 > * Follow the rules of how to < an array field>>. >=20 >=20 > [[sequence-field-type]] > =3D=3D=3D=3D Sequence field type >=20 > A _sequence field type_ describes sequence fields. >=20 > A sequence value is a finite sequence of values. Its length is known > dynamically. >=20 > .Sequence field type mdt:map properties > [options=3D"header"] >|=3D=3D=3D >|Name |Type |Description |Required? |Default m-value >=20 >|`field-type` >|mdt:string >|_Must_ be set to `sequence`. >|Required >| >=20 >|`user-attrs` >|<> >|User attributes. >|Optional >|Empty mdt:map value. >=20 >|`alignment` >|mdt:int >|Field's alignment (_must_ be a power of two, greater than 0). >|Optional >|1 >=20 >|`length` >|<> >|Path to a previously encoded integer, enumeration, variable-length > integer, or variable-length enumeration field which > indicates the number of elements contained in the sequence field. >|Required >| >=20 >|`element-field-type` >|<> >|Field type of the elements contained in sequence fields described by > this field type. >|Required >| >|=3D=3D=3D >=20 > .Sequence field type > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > { > "field-type": "sequence", > "alignment": 32, > "length": ["msg", "info", "count"], > "element-field-type": { > "field-type": "string" > }, > "user-attrs": { > "my-namespace": { > "my-attr": "desc" > } > } > } > =3D=3D=3D=3D >=20 > .Minimal sequence field type > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > { > "field-type": "sequence", > "length": ["len"], > "element-field-type": "elem-ft" > } > =3D=3D=3D=3D >=20 >=20 > [[enc-sequence-field]] > =3D=3D=3D=3D=3D Encode a sequence value as a sequence field >=20 > To encode a sequence value using a sequence field type: >=20 > . <> the current encoding head using the `alignment` > property of the sequence field type. > . For each element of the sequence value (the count given by a > previously encoded integer or enumeration field, located thanks to the > `length` property): > ** Encode the value using the `element-field-type` property of the > sequence field type. >=20 >=20 > [[textsequence-field-type]] > =3D=3D=3D=3D Text sequence field type >=20 > A _text sequence field type_ describes text sequence fields. It is a > specialized version of the <>. >=20 > A text sequence value is a finite sequence of bytes which form a UTF-8 > string. Its length is known dynamically. The text sequence value's > length _may_ be greater than the number of effective UTF-8 bytes, as long > as the string is null-terminated. In this case the padding bytes after > the UTF-8 null character can have any value. >=20 > .Text sequence field type mdt:map properties > [options=3D"header"] >|=3D=3D=3D >|Name |Type |Description |Required? |Default m-value >=20 >|`field-type` >|mdt:string >|_Must_ be set to `textsequence`. >|Required >| >=20 >|`user-attrs` >|<> >|User attributes. >|Optional >|Empty mdt:map value. >=20 >|`alignment` >|mdt:int >|Field's alignment (_must_ be a power of two, greater than 0). >|Optional >|1 >=20 >|`length` >|<> >|Path to a previously encoded integer, enumeration, variable-length > integer, or variable-length enumeration field which > indicates the number of bytes contained in the text sequence field. >|Required >| >|=3D=3D=3D >=20 > .Text sequence field type > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > { > "field-type": "textsequence", > "alignment": 32, > "length": { > "scope": "my-scope", > "path": "cmd-len" > }, > "user-attrs": { > "my-namespace": { > "my-attr": "desc" > } > } > } > =3D=3D=3D=3D >=20 > .Minimal text sequence field type > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > { > "field-type": "textsequence", > "length": ["len"] > } > =3D=3D=3D=3D >=20 >=20 > [[enc-textsequence-field]] > =3D=3D=3D=3D=3D Encode a text sequence value as a text sequence field >=20 > A text sequence value is a sequence value with bytes as elements. >=20 > To encode a text sequence value using a text sequence field type: >=20 > * Follow the rules of how to < value as a sequence field>>. >=20 >=20 > [[variant-field-type]] > =3D=3D=3D=3D Variant field type >=20 > A _variant field type_ describes variant fields. >=20 > A variant value is a value of some type amongst many possible types. The > exact type of the value is indicated dynamically by a previously encoded > tag field (an enumeration or variable-length enumeration field). >=20 > .Variant field type mdt:map properties > [options=3D"header"] >|=3D=3D=3D >|Name |Type |Description |Required? |Default m-value >=20 >|`field-type` >|mdt:string >|_Must_ be set to `textsequence`. >|Required >| >=20 >|`user-attrs` >|<> >|User attributes. >|Optional >|Empty mdt:map value. >=20 >|`alignment` >|mdt:int >|Field's alignment (_must_ be a power of two, greater than 0). >|Optional >|1 >=20 >|`tag` >|<> >|Path to a previously encoded enumeration or variable-length > enumeration field which indicates the type, by label name to choice name > association, of the variant field. >|Required >| >=20 >|`choices` >|mdt:array of < field type field mdt:map values>> >|Choices of the variant field type. >|Required >| >|=3D=3D=3D >=20 > The `name` property of the all the structure/union/variant field type > field mdt:map values listed in the `choices` property _must_ exist as a > member's label name in the tag's enumeration field type. >=20 > .Variant field type > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > { > "field-type": "variant", > "alignment": 16, > "tag": ["path", "to", "tag"], > "choices": [ > { > "user-attrs": { > "ns": { > "split": 5 > } > }, > "name": "ID", > "field-type": { > "field-type": "int", > "size": 35, > "signed": true, > "alignment": 32 > } > }, > { > "name": "NAME", > "field-type": { > "field-type": "string" > } > } > ], > "user-attrs": { > "my-namespace": { > "my-attr": "desc" > } > } > } > =3D=3D=3D=3D >=20 > .Minimal variant field type > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > { > "field-type": "variant", > "tag": ["tag"], > "choices": [ > { > "name": "", > "field-type": "some-ft" > } > ] > } >=20 > Note that such a variant field type, with a single choice, is useless > because this choice could be used directly instead. > =3D=3D=3D=3D >=20 >=20 > [[enc-variant-field]] > =3D=3D=3D=3D=3D Encode a variant value as a variant field >=20 > To encode a variant value using a variant field type: >=20 > . <> the current encoding head using the `alignment` > property of the variant field type. > . Follow the encoding rules of the field type, amongst the field types > of the choices listed in the `choices` property, currently selected by > the previously encoded `tag` field. >=20 >=20 > [[union-field-type]] > =3D=3D=3D=3D Union field type >=20 > A _union field type_ describes union fields. >=20 > A union field is a binary field which, once encoded using some field > type, can be decoded using other field types. >=20 > .Union field type mdt:map properties > [options=3D"header"] >|=3D=3D=3D >|Name |Type |Description |Required? |Default m-value >=20 >|`field-type` >|mdt:string >|_Must_ be set to `struct`. >|Required >| >=20 >|`user-attrs` >|<> >|User attributes. >|Optional >|Empty mdt:map value. >=20 >|`alignment` >|mdt:int >|Field's minimal alignment (_must_ be a power of two, greater than 0). >|Optional >|1 >=20 >|`fields` >|mdt:array of at least one > < field mdt:map value>> >|Fields of the union field type. >|Required >| >|=3D=3D=3D >=20 > .Union field type > =3D=3D=3D=3D > In this example, the union fields represented by this field type can > _always_ be decoded as either string fields, or as unsigned 64-bit > integer fields. This means that the producer ensures that, when writing > a string field, its length is always 64 bits (including the terminating > null character). >=20 > JSON representation: >=20 > [source,json] > { > "field-type": "union", > "alignment": 8, > "fields": [ > { > "name": "as string", > "field-type": { > "field-type": "string" > } > }, > { > "name": "as int", > "field-type": { > "field-type": "int", > "size": 64 > } > } > ], > "user-attrs": { > "my-namespace": { > "my-attr": "desc" > } > } > } > =3D=3D=3D=3D >=20 > .Minimal union field type > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > { > "field-type": "union", > "length": 32, > "fields": [ > { > "name": "", > "field-type": "some-ft" > } > ] > } >=20 > Note that such a union field type, with a single field, is useless > because this field could be used directly instead. > =3D=3D=3D=3D >=20 >=20 > [[enc-variant-field]] > =3D=3D=3D=3D=3D Encode a value as a union field >=20 > To encode a value using a union field type: >=20 > . <> the current encoding head using the `alignment` > property of the union field type. > . Follow the encoding rules of the chosen field type, amongst the field > types of the fields listed in the `fields` property, corresponding to > the type of the value to encode. >=20 > An important condition when encoding a union field is that, whichever > field type is chosen to decode the field afterwards, the current > decoding head _must_ always be the same after the process. >=20 >=20 > [[layer-2]] > =3D=3D Layer 2: Data streams, packets, and event records >=20 > This layer adds the following concepts to the CTF{nbsp}2 specification: >=20 > * Field type alias fragment. > * Field tag mdt:map. > * A _trace_ contains zero or more _data streams_. A data stream is a > sequence of zero or more _packets_. A packet contains zero or more > _event records_. > * The concepts and layouts of data streams, packets, and event records. > * Trace class, data stream class, and event record class fragments. >=20 >=20 > [[data-stream]] > =3D=3D=3D Data stream >=20 > A CTF{nbsp}2 _data stream_ is defined as a sequence of > <>. >=20 > [packet] > [packet] > [packet] > [packet] > [packet] > ... >=20 > A <> describes, > in the <>, a class of data streams. > For a single data stream class, there can be zero or more data streams > in the trace. >=20 > There are two ways to distinguish individual data streams: >=20 > * Rely on the storage or transport back-end to separate individual data > streams. > + > For example, a trace stored on a file system could contain one data > stream per file. If a trace is sent over the network, the wrapping > network protocol could assign a unique ID to each data stream. >=20 > * Tag a field of the trace packet header field with the > <>. > + > This is the _recommended_ way to isolate a data stream: it is more > robust and more portable. With this method, each packet holds the unique > ID of the data stream to which it logically belongs. >=20 >=20 > [[packet]] > =3D=3D=3D Packet >=20 > A _packet_ contains two _optional_, contiguous scope fields, named the > _trace packet header_ and the _data stream packet context_ fields, > followed by zero or more event records, and finally by _optional_ > padding bits to honor its total size: >=20 > [trace packet header field] > [data stream packet context field] > [event record] > [event record] > [event record] > ... > [padding] >=20 > Before the first bit of a packet is written, the current encoding head > of the <> is set to 0. This current > encoding head keeps on incrementing as fields are encoded following the > <> until the beginning of the next packet in the same > <>, where it is reset to 0 again. This means > that the reference to align any binary field within a packet is the > beginning of a packet, _not_ the beginning of a data stream. >=20 > To simplify matters for CTF{nbsp}2 consumers, the size of a packet, in > bits, _must_ be greater than 8, and _must_ be a multiple of 8. >=20 > The first packet's padding bit is reached when the current encoding head > is equal to the value of the packet's content size field found in the > data stream packet context field (if any). >=20 > If there is more than one data stream class in the trace, the trace > packet header field contains a data stream class ID field which > indicates which data stream class describes the data stream packet > context field and the event records. >=20 >=20 > [[event-record]] > =3D=3D=3D Event record >=20 > An _event record_ contains four _optional_, contiguous scope fields, > named the _data stream event record header_, the _data stream event > record context_, the _event record context_, and the _event record > payload_ fields. >=20 > [data stream event record header field] > [data stream event record context field] > [event record context] > [event record payload] >=20 > If there is more than one event record class in a given data stream > class, the data stream event record header field contains an event > record class ID field which indicates which event record class describes > the event record context and event record payload fields. >=20 > The encoded size of any event record must be at least 1{nbsp}bit. >=20 > [[field-type-alias-fragment]] > =3D=3D=3D Field type alias fragment >=20 > A _field type alias fragment_ is a <> which > associates a name (mdt:string) to a < m-value>> (complete field type mdt:map, or previously defined field type > alias's name). >=20 > The name of a field type alias can be used by field types which are > written after the field type alias fragment in the metadata mdt:array. >=20 > .Field type alias fragment > [options=3D"header"] >|=3D=3D=3D >|Name |Type |Description |Required? |Default m-value >=20 >|`fragment` >|mdt:string >|_Must_ be set to `field-type-alias`. >|Required >| >=20 >|`user-attrs` >|<> >|User attributes. >|Optional >|Empty mdt:map value. >=20 >|`name` >|mdt:string >|Name of the field type alias. >|Required >| >=20 >|`field-type` >|<> >|Field type of the field type alias. >|Required >| >|=3D=3D=3D >=20 > Within a given <>, two field type > alias fragments cannot have the same `name` property. >=20 > .Field type alias fragment > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > { > "fragment": "field-type-alias", > "name": "uint8", > "field-type": { > "field-type": "integer", > "size": 8, > "alignment": 8 > } > } > =3D=3D=3D=3D >=20 > .Field type alias fragment giving another name to a previously defined al= ias > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > { > "fragment": "field-type-alias", > "name": "uint8_t", > "field-type": "uint8" > } > =3D=3D=3D=3D >=20 >=20 > [[field-tag]] > =3D=3D=3D Field tag mdt:map >=20 > A _field tag mdt:map_ ``tags'' a scope's field with a special meaning. >=20 > The following sections define specific field tag mdt:map values which > can be used in specific contexts. >=20 > .Field tag mdt:map base properties > [options=3D"header"] >|=3D=3D=3D >|Name |Type |Description |Required? |Default m-value >=20 >|`tag` >|mdt:string >|Name of the tag (fragment-dependent). >|Required >| >=20 >|`path` >|<> >|Absolute path to the tagged field. >|Required >| >|=3D=3D=3D >=20 >=20 > [[discarded-event-record-count-field-tag]] > =3D=3D=3D Discarded event record count field tag mdt:map >=20 > A _discarded event record count field tag mdt:map_ is a > <> which tags a field as being a counter of > discarded event records. >=20 > Event records can be discarded for multiple reasons from the producer's > perspective. This document specifies the available reasons, when event > records are lost for those reasons, and how to compute the total number > of discarded event records so far for a given < stream>>. >=20 > .Discarded event record count field tag mdt:map properties > [options=3D"header"] >|=3D=3D=3D >|Name |Type |Description |Required? |Default m-value >=20 >|`tag` >|mdt:string >|_Must_ be set to `discarded-event-record-count`. >|Required >| >=20 >|`path` >|<> >|Absolute path to the tagged field. >|Required >| >=20 >|`reason` >|mdt:string >|_Must_ be set to `legacy` as of this version. >|Required >| >|=3D=3D=3D >=20 > A field tagged with the `legacy` reason has the same behaviour as the > `events_discarded` field of the stream packet context of CTF v1.8.2. >=20 > The type of the field located using the `path` property _must_ be an > unsigned <>, an unsigned > <>, an unsigned > <>, or an unsigned > <>. >=20 > .Discarded event record count field tag > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > { > "tag": "discarded-event-record-count", > "path": { > "scope": "data-stream-packet-context", > "path": ["number of discarded events"] > }, > "reason": "legacy" > } > =3D=3D=3D=3D >=20 >=20 > [[trace-class-fragment]] > =3D=3D=3D Trace class fragment >=20 > A _trace class fragment_ is a <> which defines > properties that are common to the whole trace, that is, to all the > <>. >=20 > Exactly one trace class fragment _must_ exist in a given > <>, and it _must_ precede any > <>. >=20 > .Trace class fragment mdt:map properties > [options=3D"header"] >|=3D=3D=3D >|Name |Type |Description |Required? |Default m-value >=20 >|`fragment` >|mdt:string >|_Must_ be set to `trace-class`. >|Required >| >=20 >|`user-attrs` >|<> >|User attributes. >|Optional >|Empty mdt:map value. >=20 >|`default-byte-order` >|<> (excluding `default`) >|Trace's default byte order. >|Optional >|No trace's default byte order. >=20 > Note that, if this property is missing, no field types in the whole > <> can have a > `byte-order` property set to `default`. >=20 >|`uuid` >|mdt:string >|https://en.wikipedia.org/wiki/Universally_unique_identifier[UUID] > (canonical form) of the trace. >|Optional >|No trace UUID. >=20 >|`packet-header-field-type` >|<> >|Field type of the trace packet header field. >=20 > The name of this scope is `trace-packet-header`, to locate it > with an <>. >|Optional >|1-bit aligned <>. >=20 >|`tags` >|mdt:array of <>. >|Field tags of this trace class. See the allowed field tags below. >=20 > Upper layers may also define specific field tags that are allowed here. >|Optional >|Empty mdt:array value. >|=3D=3D=3D >=20 > .Trace class fragment > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > { > "fragment": "trace-class", > "default-byte-order": "le", > "uuid": "1255cddc-5afe-4b4a-92b2-17aa5dde2ea6", > "packet-header-field-type": { > "field-type": "struct", > "fields": [ > { > "name": "the magic", > "field-type": "uint32" > }, > { > "name": "the uuid", > "field-type": { > "field-type": "array", > "element-field-type": "uint8" > } > }, > { > "name": "the data stream class ID", > "field-type": "uint8" > } > ] > }, > "tags": [ > { > "tag": "magic", > "path": { > "scope": "trace-packet-header", > "path": ["the magic"] > } > }, > { > "tag": "uuid", > "path": { > "scope": "trace-packet-header", > "path": ["the uuid"] > } > }, > { > "tag": "data-stream-class-id", > "path": { > "scope": "trace-packet-header", > "path": ["the data stream class ID"] > } > } > ], > "user-attrs": { > "my ns": "yes" > } > } > =3D=3D=3D=3D >=20 >=20 > =3D=3D=3D=3D Allowed field tags targetting trace packet header fields >=20 > In addition to the tags below, > < field tags>> are allowed, and any field tag defined by the upper layers > of CTF{nbsp}2. >=20 > .Trace class fragment's allowed tags for the trace packet header fields > [options=3D"header"] >|=3D=3D=3D >|Tag name |Meaning |Tagged field constraints >=20 >|`magic` >|**Magic number field**. >=20 > This field indicates the CTF{nbsp}2 data stream magic number. >=20 >|This field _must_ be the first field of the trace packet header field > type. >=20 > This field's type _must_ be a 32-bit unsigned < field type>>. >=20 > The value of this field in all the data streams _must_ be 3254525889 > (0xc1fc1fc1). >=20 >|`uuid` >|**Trace's UUID field**. >=20 > This field indicates the UUID of the trace. >=20 >|This field's type _must_ be an <> > of length 16, with an 8-bit aligned, 8-bit > <> as its element. >=20 > The value of this field in all the data streams _must_ be equal to the > binary equivalent of the trace class fragment's `uuid` property. >=20 >|`data-stream-class-id` >|**Data stream class ID field**. >=20 > This field indicates the numeric ID of the > <> used to encode the rest > of the packet. >=20 > If this tag is not specified, the ID of the data stream class used to > encode the rest of the packet is implicitly 0. >=20 > If more than one field are tagged with this tag, the _last_ one to be > encoded in the entire trace packet header field is the effective ID of > the data stream class used to encode the rest of the <>. >=20 >|This field's type _must_ be an unsigned > <>, an unsigned > <>, an unsigned > <>, or an > unsigned <>. >=20 >|[[data-stream-id]]`data-stream-id` >|**Data stream ID field**. >=20 > This field indicates the unique numeric ID of the < stream>> to which the packet belongs. >=20 > If this tag is not specified, the data stream to which the packet > belongs is identified using the storage or transport back-end of the > trace. >=20 > If more than one field are tagged with this tag, the _last_ one to be > encoded in the entire trace packet header field is the effective unique > data stream ID. >=20 >|This field's type _must_ be an unsigned > <>, an unsigned > <>, an unsigned > <>, or an > unsigned <>. >|=3D=3D=3D >=20 >=20 > [[data-stream-class-fragment]] > =3D=3D=3D Data stream class fragment >=20 > A _data stream class fragment_ is a <> which > defines properties that are common to one or more < streams>>. >=20 > More than one data stream class fragment may exist in a given > <>, but they must come after the > <>. >=20 > Any <> which is > a child of a given data stream class fragment must come after the latter > in the metadata mdt:array. >=20 > .Data stream class fragment mdt:map properties > [options=3D"header"] >|=3D=3D=3D >|Name |Type |Description |Required? |Default m-value >=20 >|`fragment` >|mdt:string >|_Must_ be set to `data-stream-class`. >|Required >| >=20 >|`user-attrs` >|<> >|User attributes. >|Optional >|Empty mdt:map value. >=20 >|`id` >|mdt:int >|Data stream class's numeric ID. >|Optional >|0 >=20 >|`packet-context-field-type` >|<> >|Field type of the data stream packet context field. >=20 > The name of this scope is `data-stream-packet-context`, to locate it > with an <>. >|Optional >|1-bit aligned <>. >=20 >|`event-record-context-field-type` >|<> >|Field type of the data stream event record context field. >=20 > The name of this scope is `data-stream-event-record-context`, to refer > to it with an <>. >|Optional >|1-bit aligned <>. >=20 >|`event-record-header-field-type` >|<> >|Field type of the data stream event record header field. >=20 > The name of this scope is `data-stream-event-record-header`, to refer > to it with an <>. >|Optional >|1-bit aligned <>. >=20 >|`tags` >|mdt:array of <>. >|Field tags of this data stream class. See the allowed field tags below. >=20 > Upper layers may also define specific field tags that are allowed here. >|Optional >|Empty mdt:array value. >|=3D=3D=3D >=20 > Two data stream class fragments with the same `id` property cannot exist > in the same <>. >=20 > .Data stream class fragment > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > { > "fragment": "data-stream-class", > "id": 3, > "packet-context-field-type": { > "field-type": "struct", > "fields": [ > { > "name": "packet size", > "field-type": "uint32" > }, > { > "name": "content size", > "field-type": "uint32" > }, > { > "name": "discarded events", > "field-type": "uint32" > }, > { > "name": "timestamp begin", > "field-type": "uint64" > }, > { > "name": "timestamp end", > "field-type": "uint64" > }, > { > "name": "cpu ID", > "field-type": "uint8" > } > ] > }, > "event-record-header-field-type": { > "field-type": "struct", > "fields": [ > { > "name": "ID", > "field-type": { > "field-type": "enum", > "size": 5, > "members": { > "compact": [{"lower": 0, "upper": 30}], > "extended": [31] > } > } > }, > { > "name": "compact or extended", > "field-type": { > "field-type": "variant", > "tag": ["ID"], > "choices": [ > { > "name": "compact", > "field-type": { > "field-type": "struct", > "fields": [ > { > "name": "timestamp", > "field-type": "uint27" > } > ] > } > }, > { > "name": "extended", > "field-type": { > "field-type": "struct", > "fields": [ > { > "name": "ID", > "field-type": "uint32" > }, > { > "name": "timestamp", > "field-type": "uint64" > } > ] > } > } > ] > } > } > ] > }, > "tags": [ > { > "tag": "packet-total-size", > "path": { > "scope": "data-stream-packet-context", > "path": ["packet size"] > } > }, > { > "tag": "packet-content-size", > "path": { > "scope": "data-stream-packet-context", > "path": ["content size"] > } > }, > { > "tag": "discarded-event-record-count", > "path": { > "scope": "data-stream-packet-context", > "path": ["discarded events"] > }, > "reason": "legacy" > }, > { > "tag": "event-record-class-id", > "path": { > "scope": "data-stream-event-record-header", > "path": ["ID"] > } > }, > { > "tag": "event-record-class-id", > "path": { > "scope": "data-stream-event-record-header", > "path": ["compact or extended", "ID"] > } > } > ], > "user-attrs": { > "my ns": "yes" > } > } > =3D=3D=3D=3D >=20 >=20 > =3D=3D=3D=3D Allowed field tags targetting data stream packet context fie= lds >=20 > In addition to the tags below, > < field tags>> are allowed, and any field tag defined by the upper layers > of CTF{nbsp}2. >=20 > .Data stream class fragment's allowed tags for the data stream packet con= text > fields > [options=3D"header"] >|=3D=3D=3D >|Tag name |Meaning |Tagged field constraints >=20 >|`packet-total-size` >|**Packet's total size field**. >=20 > This field indicates the size, in bits, of the _whole_ <> > in which this data stream packet context field is encoded. This size > includes the padding bits after the packet's content, if any. >=20 > If this tag is not specified, this is the only packet of its data > stream, that is, it ends where the <> ends. >=20 >|This field's type _must_ be an unsigned > <>, an unsigned > <>, an unsigned > <>, or an > unsigned <>. >=20 >|`packet-content-size` >|**Packet's content size field**. >=20 > This field indicates the content size, in bits, of the <> > in which this data stream packet context field is encoded. The packet's > content size is the number of bits from the packet's first bit to the > last bit of the last event record (included). The difference between the > packet's total size and the packet's content size is the padding size. >=20 > If this tag is not specified, the packet has no padding bits, thus its > content size is the same as its total size. >=20 >|This field's type _must_ be an unsigned > <>, an unsigned > <>, an unsigned > <>, or an > unsigned <>. >=20 >|`packet-sequence-number` >|**Packet's sequence number field**. >=20 > This field indicates the sequence number of the packet in which this > data stream packet context field is encoded. This is the zero-based > index of the packet within its <>. >=20 >|This field's type _must_ be an unsigned > <>, an unsigned > <>, an unsigned > <>, or an > unsigned <>. >|=3D=3D=3D >=20 >=20 > =3D=3D=3D=3D Allowed field tags targetting data stream event record heade= r fields >=20 > In addition to the tags below, any field tag defined by the upper layers > of CTF{nbsp}2 is allowed. >=20 > .Data stream class fragment's allowed tags for the data stream event reco= rd > header fields > [options=3D"header"] >|=3D=3D=3D >|Tag name |Meaning |Tagged field constraints >=20 >|`event-record-class-id` >|**Event record class ID field**. >=20 > This field indicates the numeric ID of the > <> used to encode the > rest of the <>. >=20 > If this tag is not specified, the ID of the event record class used to > encode the rest of the event record is implicitly 0. >=20 > If more than one field are tagged with this tag, the _last_ one to be > encoded in the entire data stream event record header field is the > effective ID of the event record class used to encode the rest of the > event record. >=20 >|This field's type _must_ be an unsigned > <>, an unsigned > <>, an unsigned > <>, or an > unsigned <>. >|=3D=3D=3D >=20 >=20 > [[event-record-class-fragment]] > =3D=3D=3D Event record class fragment >=20 > An _event record class fragment_ is a <> which > defines properties that are common to one or more event records. >=20 > More than one event record class fragment may exist in a given > <>, but they must come after their > parent <>. >=20 > .Event record class fragment mdt:map properties > [options=3D"header"] >|=3D=3D=3D >|Name |Type |Description |Required? |Default m-value >=20 >|`fragment` >|mdt:string >|_Must_ be set to `event-record-class`. >|Required >| >=20 >|`user-attrs` >|<> >|User attributes. >|Optional >|Empty mdt:map value. >=20 >|`id` >|mdt:int >|Event record class's numeric ID. >|Optional >|0 >=20 >|`parent-data-stream-class-id` >|mdt:int >|Numeric ID of parent <>. >|Optional >|0 >=20 >|`context-field-type` >|<> >|Field type of the event record context field. >=20 > The name of this scope is `event-record-context`, to locate it > with an <>. >|Optional >|1-bit aligned <>. >=20 >|`payload-field-type` >|<> >|Field type of the event record payload field. >=20 > The name of this scope is `event-record-payload`, to locate it > with an <>. >|Optional >|1-bit aligned <>. >=20 >|`tags` >|mdt:array of <>. >|Field tags of this event record class. >=20 > Upper layers may also define specific field tags that are allowed here. >|Optional >|Empty mdt:array value. >|=3D=3D=3D >=20 > Two event record class fragments with the same `id` property, and with > the same `parent-data-stream-class-id` property, cannot exist in the > same <>. >=20 > .Event record class fragment > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > { > "fragment": "event-record-class", > "id": 45, > "parent-data-stream-class-id": 3, > "payload-field-type": { > "field-type": "struct", > "fields": [ > { > "name": "prev_comm", > "field-type": { > "field-type": "textarray", > "length": 16 > } > }, > {"name": "prev_tid", "field-type": "uint32"}, > {"name": "prev_prio", "field-type": "uint32"}, > {"name": "prev_state", "field-type": "uint64"}, > { > "name": "next_comm", > "field-type": { > "field-type": "textarray", > "length": 16 > } > }, > {"name": "next_tid", "field-type": "uint32"}, > {"name": "next_prio", "field-type": "uint32"} > ] > }, > "user-attrs": { > "my ns": "yes" > } > } > =3D=3D=3D=3D >=20 >=20 > [[layer-3]] > =3D=3D Layer 3: Timekeeping >=20 > This layer adds concepts of time to the CTF{nbsp}2 specification. With > this layer, timestamps _may_ be associated to packet and event record > fields. >=20 >=20 > [[data-stream-clock-class-fragment]] > =3D=3D=3D Data stream clock class fragment >=20 > A _data stream clock class fragment_ is a <> > which defines properties that are common to one or more data stream > clocks. >=20 > .Data stream clock class fragment mdt:map properties > [options=3D"header"] >|=3D=3D=3D >|Name |Type |Description |Required? |Default m-value >=20 >|`fragment` >|mdt:string >|_Must_ be set to `data-stream-clock-class`. >|Required >| >=20 >|`user-attrs` >|<> >|User attributes. >|Optional >|Empty mdt:map value. >=20 >|`name` >|mdt:string >|Name of this data stream clock class. >|Required >| >=20 >|`freq` >|mdt:int >|Frequency (Hz). >|Required >| >=20 >|`uuid` >|mdt:string >|https://en.wikipedia.org/wiki/Universally_unique_identifier[UUID] > (canonical form) of this data stream clock class. >|Optional >|No UUID. >=20 >|`error-cycles` >|mdt:int >|Error in cycles. >|Optional >|0 >=20 >|`offset-seconds` >|mdt:int >|Offset in seconds. >|Optional >|0 >=20 >|`offset-cycles` >|mdt:int >|Offset in cycles. >|Optional >|0 >=20 >|`is-absolute` >|mdt:bool >|_True_ if this data stream clock class defines absolute data stream > clocks, that is, clocks that are considered to be global references, > even across different UUIDs. >|Optional >|_False_ >|=3D=3D=3D >=20 > Two data stream clock class fragments with the same `name` property > cannot exist in the same <>. >=20 > See _Common Trace Format v1.8.2_, section 8, for more information about > data stream clock classes (equivalent to TSDL's `clock` block). >=20 > .Data stream clock class fragment > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > { > "fragment": "data-stream-clock-class", > "name": "monotonic", > "uuid": "96a25753-91af-4602-a71b-d53b7c3dde45", > "freq": 1000000000, > "error-cycles": 1000, > "offset-seconds": 1326476837, > "offset-cycles": 897235420, > "is-absolute": true, > "user-attrs": { > "my ns": "yes" > } > } > =3D=3D=3D=3D >=20 >=20 > [[data-stream-clock]] > =3D=3D=3D Data stream clock >=20 > A _data stream clock_ is an instance of a > <>. >=20 > Each <> has one such clock instance for each > known data stream clock class. >=20 > A data stream clock is an integer variable initialized to 0 before > decoding the <> to which it is attached. This > variable holds the current value, in cycles, for this specific data > stream, of its data stream clock class. >=20 > When any field tagged with one of the data stream clock update tags is > decoded, its associated data stream clock can be updated. >=20 >=20 > [[update-data-stream-clock-now-field-tag]] > =3D=3D=3D Update data stream clock now field tag mdt:map >=20 > An _update data stream clock now field tag mdt:map_ is a > <> which tags a field to update a data stream clock > with its value when it is decoded. >=20 > .Update data stream clock now field tag mdt:map properties > [options=3D"header"] >|=3D=3D=3D >|Name |Type |Description |Required? |Default m-value >=20 >|`path` >|<> >|Absolute path to the tagged field. >|Required >| >=20 >|`tag` >|mdt:string >|_Must_ be set to `update-data-stream-clock-now`. >|Required >| >=20 >|`data-stream-clock-class-name` >|mdt:string >|Name of the data stream clock class of the data stream clock to > update. >|Required >| >|=3D=3D=3D >=20 > This tag _may_ be used in the `tags` property of a > <>, a > <>, or an > <>. >=20 > If this tag is used, a matching < stream clock class fragment>> _must_ exist in the > <> before the fragment in which it is > used. >=20 > The tagged field's type _must_ be an unsigned < field type>>, an unsigned <>, an > unsigned <>, or an > unsigned <>. >=20 > .Data stream class fragment using this field tag > =3D=3D=3D=3D > JSON representation: >=20 > [source,json] > { > "fragment": "data-stream-class", > "id": 2, > "event-record-header-field-type": { > "field-type": "struct", > "fields": [ > { > "name": "ts", > "field-type": "uint32" > } > ] > }, > "tags": [ > { > "tag": "update-data-stream-clock-now", > "data-stream-clock-class-name": "monotonic", > "path": { > "scope": "data-stream-event-record-header", > "path": ["ts"] > } > } > ] > } > =3D=3D=3D=3D >=20 >=20 > [[update-data-stream-clock-after-packet-field-tag]] > =3D=3D=3D Update data stream clock after packet field tag mdt:map >=20 > An _update data stream clock after packet field tag mdt:map_ is a > <> which tags a field to update a data stream clock > with its value after the whole packet is decoded (delayed update). >=20 > .Update data stream clock after packet field tag mdt:map properties > [options=3D"header"] >|=3D=3D=3D >|Name |Type |Description |Required? |Default m-value >=20 >|`path` >|<> >|Absolute path to the tagged field. >|Required >| >=20 >|`tag` >|mdt:string >|_Must_ be set to `update-data-stream-clock-after-packet`. >|Required >| >=20 >|`data-stream-clock-class-name` >|mdt:string >|Name of the data stream clock class of the data stream clock to > update. >|Required >| >|=3D=3D=3D >=20 > This tag is only allowed in the `tags` property of a > <>. It > _must_ target the `data-stream-packet-context` scope. >=20 > If this tag is used, a matching < stream clock class fragment>> > _must_ exist in the <> > before the fragment in which it is used. >=20 > The tagged field's type _must_ be an unsigned < field type>>, an unsigned <>, an > unsigned <>, or an > unsigned <>. >=20 >=20 > =3D=3D Complete metadata stream example >=20 > Here's an example of a complete, valid < stream>>. It contains one < fragment>> with two < fragments>> as its children. >=20 > [source,json] > ---- > [ > "CTF 2", > { > "fragment": "field-type-alias", > "name": "uint8", > "field-type": { > "field-type": "int", > "size": 8, > "align": 8 > } > }, > { > "fragment": "field-type-alias", > "name": "uint16", > "field-type": { > "field-type": "int", > "size": 16, > "align": 8 > } > }, > { > "fragment": "field-type-alias", > "name": "uint32", > "field-type": { > "field-type": "int", > "size": 32, > "align": 8 > } > }, > { > "fragment": "field-type-alias", > "name": "uint64", > "field-type": { > "field-type": "int", > "size": 64, > "align": 8 > } > }, > { > "fragment": "field-type-alias", > "name": "int64", > "field-type": { > "field-type": "int", > "size": 64, > "align": 8, > "signed": true > } > }, > { > "fragment": "field-type-alias", > "name": "uint27", > "field-type": { > "field-type": "int", > "size": 27 > } > }, > { > "fragment": "field-type-alias", > "name": "uuid", > "field-type": { > "field-type": "array", > "alignment": 8, > "length": 16, > "field-type": "uint8" > } > }, > { > "fragment": "trace-class", > "default-byte-order": "le", > "uuid": "908d7a0d-2bbc-4584-ba3b-a9c73a62f52e", > "packet-header-field-type": { > "field-type": "struct", > "fields": [ > { > "name": "the magic", > "field-type": "uint32" > }, > { > "name": "the UUID", > "field-type": "uuid" > }, > { > "name": "ids", > "field-type": { > "field-type": "struct", > "fields": [ > { > "name": "data stream class", > "field-type": "uint8" > }, > { > "name": "data stream", > "field-type": "uint8" > } > ] > } > } > ] > }, > "tags": [ > { > "tag": "magic", > "path": { > "scope": "trace-packet-header", > "path": ["the magic"] > } > }, > { > "tag": "uuid", > "path": { > "scope": "trace-packet-header", > "path": ["the UUID"] > } > }, > { > "tag": "data-stream-class-id", > "path": { > "scope": "trace-packet-header", > "path": ["ids", "data stream class"] > } > }, > { > "tag": "data-stream-id", > "path": { > "scope": "trace-packet-header", > "path": ["ids", "data stream"] > } > } > ] > }, > { > "fragment": "data-stream-clock-class", > "name": "clock src", > "uuid": "96a25753-91af-4602-a71b-d53b7c3dde45", > "freq": 1000000000, > "error-cycles": 1000, > "offset-seconds": 1326476837, > "offset-cycles": 897235420, > "is-absolute": true, > }, > { > "fragment": "data-stream-class", > "packet-context-field-type": { > "field-type": "struct", > "fields": [ > { > "name": "packet sizes", > "field-type": { > "field-type": "struct", > "fields": [ > { > "name": "total", > "field-type": "uint32" > }, > { > "name": "content", > "field-type": "uint32" > } > ] > } > }, > { > "name": "discarded events", > "field-type": "uint32" > }, > { > "name": "sequence number", > "field-type": "uint64" > }, > { > "name": "packet begin TS", > "field-type": "uint64" > }, > { > "name": "packet end TS", > "field-type": "uint64" > }, > { > "name": "cpu ID", > "field-type": "uint8" > }, > ] > }, > "event-record-header-field-type": { > "field-type": "struct", > "fields": [ > { > "name": "ID", > "field-type": { > "field-type": "enum", > "size": 5, > "members": { > "compact": [{"lower": 0, "upper": 30}], > "extended": [31] > } > } > }, > { > "name": "compact or extended", > "field-type": { > "field-type": "variant", > "tag": ["ID"], > "choices": [ > { > "name": "compact", > "field-type": { > "field-type": "struct", > "fields": [ > { > "name": "timestamp", > "field-type": "uint27" > } > ] > } > }, > { > "name": "extended", > "field-type": { > "field-type": "struct", > "fields": [ > { > "name": "ID", > "field-type": "uint32" > }, > { > "name": "timestamp", > "field-type": "uint64" > } > ] > } > } > ] > } > } > ] > }, > "tags": [ > { > "tag": "packet-total-size", > "path": { > "scope": "data-stream-packet-context", > "path": ["packet sizes", "total"] > } > }, > { > "tag": "packet-content-size", > "path": { > "scope": "data-stream-packet-context", > "path": ["packet sizes", "content"] > } > }, > { > "tag": "discarded-event-record-count", > "path": { > "scope": "data-stream-packet-context", > "path": ["discarded events"] > } > }, > { > "tag": "packet-sequence-number", > "path": { > "scope": "data-stream-packet-context", > "path": ["sequence number"] > } > }, > { > "tag": "update-data-stream-clock-now", > "data-stream-clock-class-name": "clock src", > "path": { > "scope": "data-stream-packet-context", > "path": ["packet begin TS"] > } > }, > { > "tag": "update-data-stream-clock-after-packet", > "data-stream-clock-class-name": "clock src", > "path": { > "scope": "data-stream-packet-context", > "path": ["packet end TS"] > } > }, > { > "tag": "event-record-class-id", > "path": { > "scope": "data-stream-event-record-header", > "path": ["ID"] > } > }, > { > "tag": "event-record-class-id", > "path": { > "scope": "data-stream-event-record-header", > "path": ["compact or extended", "ID"] > } > }, > { > "tag": "update-data-stream-clock-now", > "data-stream-clock-class-name": "clock src", > "path": { > "scope": "data-stream-event-record-header", > "path": ["compact or extended", "timestamp"] > } > } > ] > }, > { > "fragment": "field-type-alias", > "name": "comm", > "field-type": { > "field-type": "textarray", > "length": 16 > } > }, > { > "fragment": "event-record-class", > "id": 0, > "payload-field-type": { > "field-type": "struct", > "fields": [ > {"name": "prev_comm", "field-type": "comm"}, > {"name": "prev_tid", "field-type": "uint32"}, > {"name": "prev_prio", "field-type": "uint32"}, > {"name": "prev_state", "field-type": "uint64"}, > {"name": "next_comm", "field-type": "comm"}, > {"name": "next_tid", "field-type": "uint32"}, > {"name": "next_prio", "field-type": "uint32"} > ] > } > }, > { > "fragment": "event-record-class", > "id": 1, > "payload-field-type": { > "field-type": "struct", > "fields": [ > {"name": "ret", "field-type": "int64"}, > {"name": "addr", "field-type": "uint64"}, > {"name": "len", "field-type": "uint64"}, > {"name": "prot", "field-type": "uint64"}, > {"name": "flags", "field-type": "uint64"}, > {"name": "fd", "field-type": "uint64"}, > {"name": "pgoff", "field-type": "uint64"} > ] > } > } > ] > ---- > _______________________________________________ > lttng-dev mailing list > lttng-dev@lists.lttng.org > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev --=20 Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com