[RFC PATCH 1/3] Generic Trace Setup and Control (GTSC) Documentation

[RFC PATCH 1/3] Generic Trace Setup and Control (GTSC) Documentation

[Tom Zanussi]

This is the documentation for the Generic Trace Setup and Control
patchset, first submitted a couple of weeks ago.  See

http://marc.info/?l=linux-kernel&m=118214274912586&w=2

for a more detailed description.

I've updated this patch to incorporate the suggestions made by Alexey
Dobriyan in that thread.

Signed-off-by: Tom Zanussi <zanussi@us.ibm.com>
Signed-off-by: David Wilder <dwilder@us.ibm.com>
---
 gtsc.txt |  247 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 247 insertions(+)

diff --git a/Documentation/gtsc.txt b/Documentation/gtsc.txt
new file mode 100644
index 0000000..470d1fc
--- /dev/null
+++ b/Documentation/gtsc.txt
@@ -0,0 +1,247 @@
+Generic Trace Setup and Control (GTSC)
+==================================
+In the kernel, GTSC provides a simple API for starting and managing
+data channels to user space.  GTSC builds on the relay interface. For a
+complete description of the relay interface, please see:
+Documentation/filesystems/relay.txt.
+
+GTSC provides one layer in a complete tracing application.  The idea of
+the GTSC is to provide a kernel API for the setup and control of tracing
+channels.  User of GTSC must provide a data layer responsible for formatting
+and writing data into the trace channels.  
+
+A layered approach to tracing
+=============================
+A complete kernel tracing application consists of a data provider and a data
+consumer.  Both provider and consumer contain three layers; each layer works
+in tandem with the corresponding layer in the opposite side.  The layers are
+represented in the following diagram.
+  
+Provider Data layer
+	Formats raw trace data and provides data-related service.
+	For example, adding timestamps used by consumer to sort data.
+
+Provider Control layer
+	Provided by GTSC. Creates trace channels and informs the data layer
+	and consumer of the current state of the trace channels.
+
+Provider Buffering layer
+	Provided by relay. This layer buffers data in the
+	kernel for consumption by the consumer's buffer
+	layer.
+
+Provider (in-kernel facility)
+-----------------------------------------------------------------------------
+Consumer (user application)
+
+
+Consumer Buffer layer
+	Reads/consumes data from the provider's data buffers.
+
+Consumer Control layer
+	Communicates to the provider's control layer to control the state
+	of the trace channels. 
+
+Consumer Data layer
+	Sorts and formats data as provided by the provider's data layer.
+
+The provider is coded as a kernel facility.  The consumer is coded as
+a user application.
+ 
+
+GTSC - Features
+==============
+The GTSC exploits services and features provided by relay.  These features are:
+- The creation and destruction of relay channels.
+- Buffer management.  Overwrite or non-overwrite modes can be selected
+  as well as global or per-CPU buffering.
+
+Overwrite mode can be called "flight recorder mode".  Flight recorder
+mode is selected by setting the TRACE_FLIGHT_CHANNEL flag when
+creating trace channels.  In flight mode when a tracing buffer is
+full, the oldest records in the buffer will be discarded to make room
+as new records arrive.  In the default non-overwrite mode, new records
+may be written only if the buffer has room.  In either case, to
+prevent data loss, a user space reader must keep the buffers
+drained. GTSC provides a means to detect the number of records that
+have been dropped due to a buffer-full condition (non-overwrite mode
+only).
+
+When per-CPU buffers are used, relay creates one debugfs file for each
+running CPU.  The user-space consumer of the data is responsible for
+reading the per-CPU buffers and collating the records presumably using
+a time stamp or sequence number included in the trace records.  The
+use of global buffers eliminates this extra work of sequencing
+records; however the provider's data layer must hold a lock when
+writing records.  The lock prevents writers running on different CPUs
+from overwriting each other's data.  However, buffering may be slower
+because write to the buffer are serialized. Global buffering is
+selected by setting the TRACE_GLOBAL_CHANNEL flag when creating trace
+channels.
+
+GTSC User Interface
+===================
+When a GTSC channel is created and tracing has been started, the following
+directories and files are created in the root of the mounted debugfs.
+
+/debug (root of the debugfs)
+        /<trace-root-dir>
+                /<trace-name>
+                        trace0 ... traceN  (Per-CPU trace data, one per CPU)
+	        	state		   (Used to  start and stop tracing)
+                        dropped            (number of records dropped due
+                                            to a full-buffer condition, only)
+                                            for non-TRACE_FLIGHT_CHANNELs)
+                        rewind             ('un-consume' channel data i.e.
+                                            start next read at the beginning
+                                            again (TRACE_FLIGHT_CHANNELS only)
+	        	nr_sub		   (Number of sub-buffers in channel)
+	        	sub_size	   (Size of sub-buffers in channnel)
+
+Trace data is gathered from the trace[0...N] files using one of the available 
+interfaces provided by relay (see Documentation/filesystems/relay.txt).
+When using the READ(2) interface, as data is read it is marked as consumed by
+the relay subsystem.  Therefore, subsequent reads will only return unconsumed
+data.
+
+GTSC Kernel API
+===============
+An overview of the GTSC Kernel API is now given. More details of the API can
+be found in linux/gtsc.h.
+
+The steps a kernel data provider takes to utilize the GTSC are:
+1) Set up a gtsc channel - trace_setup()
+2) Start the GTSC channel - trace_start()
+3) Write one or more trace records into the channel (using the relay API).
+4) Optionally stop and start tracing as desired - trace_start()/trace_stop()
+5) Destroy the GTSC channel and underlying relay channel - trace_cleanup().
+
+GTSC Example
+===========
+This small sample module creates a GTSC channel. It places a kprobe on the
+function do_fork().  The kprobe handler will write a timestamp and
+current->pid to the GTSC channel.  
+
+You can build the kernel module kprobes_gtsc.ko using the following Makefile:
+------------------------------------CUT-------------------------------------
+obj-m := kprobes_gtsc.o
+KDIR := /lib/modules/$(shell uname -r)/build
+PWD := $(shell pwd)
+default:
+	$(MAKE) -C $(KDIR) SUBDIRS=$(PWD) modules
+clean:
+	rm -f *.mod.c *.ko *.o
+----------------------------------CUT--------------------------------------
+/* kprobes_gtsc.c - An example of using GTSC in a kprobes module */
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/kprobes.h>
+#include <linux/gtsc.h>
+
+#define PROBE_POINT "do_fork"
+
+static struct kprobe kp;
+struct trace_info *kprobes_trace;
+
+#define GTSC_PRINTF_TMPBUF_SIZE (1024)
+static char gtsc_printf_tmpbuf[NR_CPUS][GTSC_PRINTF_TMPBUF_SIZE];
+
+/* This lock is needed only when using global relay buffers */
+static DEFINE_SPINLOCK(gtsc_printf_lock);
+
+void gtsc_printf(struct trace_info *trace, const char *format, ...)
+{
+	va_list args;
+	void *buf;
+	int len;
+	char *record;
+
+	if (!trace)
+		return;
+	if (!trace_running(trace))
+		return;
+
+	/* get a timestamp */
+	buf = gtsc_printf_tmpbuf[smp_processor_id()];
+	len = sprintf(buf,"[%lld] ", trace_timestamp());
+
+	/* get the data */
+	va_start(args, format);
+	len += vsnprintf(buf+len, GTSC_PRINTF_TMPBUF_SIZE, format, args);
+	va_end(args);
+
+	/*
+	 * Locking can be eliminated by specifying per-cpu buffers
+	 * when calling trace_setup().
+	 */
+	spin_lock(&gtsc_printf_lock);
+
+	/* Send everything to the relay buffer */
+	if ((record = relay_reserve(trace->rchan, len)))
+		memcpy(record, buf, len);
+
+	spin_unlock(&gtsc_printf_lock);
+}
+
+
+int handler_pre(struct kprobe *p, struct pt_regs *regs)
+{
+	gtsc_printf(kprobes_trace,"%d\n", current->pid);
+	return 0;
+}
+
+int init_module(void)
+{
+	int ret;
+	/* setup gtsc, use a global relay buffer */
+	kprobes_trace = trace_setup("gtsc_example",PROBE_POINT,
+				1024,8,
+				TRACE_GLOBAL_CHANNEL | TRACE_FLIGHT_CHANNEL);
+	if (!kprobes_trace)
+		return -1;
+
+	trace_start(kprobes_trace);
+
+	/* setup the kprobe */
+	kp.pre_handler = handler_pre;
+	kp.post_handler = NULL;
+	kp.fault_handler = NULL;
+	kp.symbol_name = PROBE_POINT;
+	if ((ret=register_kprobe(&kp)) < 0) 
+		gtsc_printf(kprobes_trace,"register_kprobe failed  %d\n", ret);
+	return 0;
+}
+
+void cleanup_module(void)
+{
+	unregister_kprobe(&kp);
+	/* close down the gtsc channel */
+	trace_stop(kprobes_trace);
+	trace_cleanup(kprobes_trace);
+}
+
+MODULE_LICENSE("GPL");
+-----------------------------CUT--------------------------------------------
+How to run the example:
+$ mount -t debugfs /debug 
+$ make
+$ insmod kprobes_gtsc.ko
+
+To view the data produced by the module:
+$ cat /debug/gtsc_example/do_fork/trace0
+
+Remove the module.
+$ rmmod kprobes_gtsc
+
+The function trace_cleanup() is called when the module
+is removed.  This will cause the GTSC channel to be destroyed and the
+corresponding files to disappear from the debug file system.
+
+Credits
+=======
+GTSC adapted from blktrace authored by Jens Axboe (axboe@suse.de).
+
+Major contributions to GTSC were made by:
+Tom Zanussi <zanussi@us.ibm.com>
+Martin Hunt <hunt@redhat.com>
+David Wilder <dwilder@us.ibm.com>

Re: [RFC PATCH 1/3] Generic Trace Setup and Control (GTSC) Documentation

[Mathieu Desnoyers]

* Tom Zanussi (zanussi@us.ibm.com) wrote:
> This is the documentation for the Generic Trace Setup and Control
> patchset, first submitted a couple of weeks ago.  See
> 
> http://marc.info/?l=linux-kernel&m=118214274912586&w=2
> 
> for a more detailed description.
> 
> I've updated this patch to incorporate the suggestions made by Alexey
> Dobriyan in that thread.
> 

It would be nice, since it claims to be "generic", to support things
brought forward by other tracers like LTTng, such as : multiple channels
(for low, medium and high event rate data, with user-selectable sizes),
hybrid trace mode (combined normal trace channels and flight recorder
channels).

Please see comments below,

> Signed-off-by: Tom Zanussi <zanussi@us.ibm.com>
> Signed-off-by: David Wilder <dwilder@us.ibm.com>
> ---
>  gtsc.txt |  247 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 247 insertions(+)
> 
> diff --git a/Documentation/gtsc.txt b/Documentation/gtsc.txt
> new file mode 100644
> index 0000000..470d1fc
> --- /dev/null
> +++ b/Documentation/gtsc.txt
> @@ -0,0 +1,247 @@
> +Generic Trace Setup and Control (GTSC)
> +==================================
> +In the kernel, GTSC provides a simple API for starting and managing
> +data channels to user space.  GTSC builds on the relay interface. For a
> +complete description of the relay interface, please see:
> +Documentation/filesystems/relay.txt.
> +
> +GTSC provides one layer in a complete tracing application.  The idea of
> +the GTSC is to provide a kernel API for the setup and control of tracing
> +channels.  User of GTSC must provide a data layer responsible for formatting
> +and writing data into the trace channels.  
> +
> +A layered approach to tracing
> +=============================
> +A complete kernel tracing application consists of a data provider and a data
> +consumer.  Both provider and consumer contain three layers; each layer works
> +in tandem with the corresponding layer in the opposite side.  The layers are
> +represented in the following diagram.
> +  
> +Provider Data layer
> +	Formats raw trace data and provides data-related service.
> +	For example, adding timestamps used by consumer to sort data.
> +
> +Provider Control layer
> +	Provided by GTSC. Creates trace channels and informs the data layer
> +	and consumer of the current state of the trace channels.
> +
> +Provider Buffering layer
> +	Provided by relay. This layer buffers data in the
> +	kernel for consumption by the consumer's buffer
> +	layer.
> +
> +Provider (in-kernel facility)
> +-----------------------------------------------------------------------------
> +Consumer (user application)
> +
> +
> +Consumer Buffer layer
> +	Reads/consumes data from the provider's data buffers.
> +
> +Consumer Control layer
> +	Communicates to the provider's control layer to control the state
> +	of the trace channels. 
> +
> +Consumer Data layer
> +	Sorts and formats data as provided by the provider's data layer.
> +
> +The provider is coded as a kernel facility.  The consumer is coded as
> +a user application.
> + 
> +
> +GTSC - Features
> +==============
> +The GTSC exploits services and features provided by relay.  These features are:
> +- The creation and destruction of relay channels.
> +- Buffer management.  Overwrite or non-overwrite modes can be selected
> +  as well as global or per-CPU buffering.
> +
> +Overwrite mode can be called "flight recorder mode".  Flight recorder
> +mode is selected by setting the TRACE_FLIGHT_CHANNEL flag when
> +creating trace channels.  In flight mode when a tracing buffer is
> +full, the oldest records in the buffer will be discarded to make room
> +as new records arrive.  In the default non-overwrite mode, new records
> +may be written only if the buffer has room.  In either case, to
> +prevent data loss, a user space reader must keep the buffers
> +drained. GTSC provides a means to detect the number of records that
> +have been dropped due to a buffer-full condition (non-overwrite mode
> +only).
> +
> +When per-CPU buffers are used, relay creates one debugfs file for each
> +running CPU.  The user-space consumer of the data is responsible for
> +reading the per-CPU buffers and collating the records presumably using
> +a time stamp or sequence number included in the trace records.  The
> +use of global buffers eliminates this extra work of sequencing
> +records; however the provider's data layer must hold a lock when
> +writing records.  The lock prevents writers running on different CPUs
> +from overwriting each other's data.  However, buffering may be slower
> +because write to the buffer are serialized. Global buffering is
> +selected by setting the TRACE_GLOBAL_CHANNEL flag when creating trace
> +channels.
> +
> +GTSC User Interface
> +===================
> +When a GTSC channel is created and tracing has been started, the following
> +directories and files are created in the root of the mounted debugfs.
> +
> +/debug (root of the debugfs)
> +        /<trace-root-dir>
> +                /<trace-name>
> +                        trace0 ... traceN  (Per-CPU trace data, one per CPU)
> +	        	state		   (Used to  start and stop tracing)
> +                        dropped            (number of records dropped due
> +                                            to a full-buffer condition, only)
> +                                            for non-TRACE_FLIGHT_CHANNELs)
> +                        rewind             ('un-consume' channel data i.e.
> +                                            start next read at the beginning
> +                                            again (TRACE_FLIGHT_CHANNELS only)
> +	        	nr_sub		   (Number of sub-buffers in channel)
> +	        	sub_size	   (Size of sub-buffers in channnel)
> +
> +Trace data is gathered from the trace[0...N] files using one of the available 
> +interfaces provided by relay (see Documentation/filesystems/relay.txt).
> +When using the READ(2) interface, as data is read it is marked as consumed by
> +the relay subsystem.  Therefore, subsequent reads will only return unconsumed
> +data.
> +
> +GTSC Kernel API
> +===============
> +An overview of the GTSC Kernel API is now given. More details of the API can
> +be found in linux/gtsc.h.
> +
> +The steps a kernel data provider takes to utilize the GTSC are:
> +1) Set up a gtsc channel - trace_setup()
> +2) Start the GTSC channel - trace_start()
> +3) Write one or more trace records into the channel (using the relay API).
> +4) Optionally stop and start tracing as desired - trace_start()/trace_stop()
> +5) Destroy the GTSC channel and underlying relay channel - trace_cleanup().
> +
> +GTSC Example
> +===========
> +This small sample module creates a GTSC channel. It places a kprobe on the
> +function do_fork().  The kprobe handler will write a timestamp and
> +current->pid to the GTSC channel.  
> +
> +You can build the kernel module kprobes_gtsc.ko using the following Makefile:
> +------------------------------------CUT-------------------------------------
> +obj-m := kprobes_gtsc.o
> +KDIR := /lib/modules/$(shell uname -r)/build
> +PWD := $(shell pwd)
> +default:
> +	$(MAKE) -C $(KDIR) SUBDIRS=$(PWD) modules
> +clean:
> +	rm -f *.mod.c *.ko *.o
> +----------------------------------CUT--------------------------------------
> +/* kprobes_gtsc.c - An example of using GTSC in a kprobes module */
> +#include <linux/kernel.h>
> +#include <linux/module.h>
> +#include <linux/kprobes.h>
> +#include <linux/gtsc.h>
> +
> +#define PROBE_POINT "do_fork"
> +
> +static struct kprobe kp;
> +struct trace_info *kprobes_trace;
> +
> +#define GTSC_PRINTF_TMPBUF_SIZE (1024)
> +static char gtsc_printf_tmpbuf[NR_CPUS][GTSC_PRINTF_TMPBUF_SIZE];
> +

Still doing an extra copy hey? :)

> +/* This lock is needed only when using global relay buffers */
> +static DEFINE_SPINLOCK(gtsc_printf_lock);
> +
> +void gtsc_printf(struct trace_info *trace, const char *format, ...)
> +{
> +	va_list args;
> +	void *buf;
> +	int len;
> +	char *record;
> +
> +	if (!trace)
> +		return;
> +	if (!trace_running(trace))
> +		return;
> +
> +	/* get a timestamp */
> +	buf = gtsc_printf_tmpbuf[smp_processor_id()];
> +	len = sprintf(buf,"[%lld] ", trace_timestamp());
> +
> +	/* get the data */
> +	va_start(args, format);
> +	len += vsnprintf(buf+len, GTSC_PRINTF_TMPBUF_SIZE, format, args);
> +	va_end(args);
> +
> +	/*
> +	 * Locking can be eliminated by specifying per-cpu buffers
> +	 * when calling trace_setup().
> +	 */
> +	spin_lock(&gtsc_printf_lock);
> +

I really hope nobody is willing to call gtsc_printf from an interrupt
handler.. :) Easy dead embrace in perspective.

> +	/* Send everything to the relay buffer */
> +	if ((record = relay_reserve(trace->rchan, len)))
> +		memcpy(record, buf, len);
> +
> +	spin_unlock(&gtsc_printf_lock);
> +}
> +
> +
> +int handler_pre(struct kprobe *p, struct pt_regs *regs)
> +{
> +	gtsc_printf(kprobes_trace,"%d\n", current->pid);
> +	return 0;
> +}
> +
> +int init_module(void)
> +{
> +	int ret;
> +	/* setup gtsc, use a global relay buffer */
> +	kprobes_trace = trace_setup("gtsc_example",PROBE_POINT,
> +				1024,8,
> +				TRACE_GLOBAL_CHANNEL | TRACE_FLIGHT_CHANNEL);
> +	if (!kprobes_trace)
> +		return -1;
> +
> +	trace_start(kprobes_trace);
> +
> +	/* setup the kprobe */
> +	kp.pre_handler = handler_pre;
> +	kp.post_handler = NULL;
> +	kp.fault_handler = NULL;
> +	kp.symbol_name = PROBE_POINT;
> +	if ((ret=register_kprobe(&kp)) < 0) 
> +		gtsc_printf(kprobes_trace,"register_kprobe failed  %d\n", ret);
> +	return 0;
> +}
> +
> +void cleanup_module(void)
> +{
> +	unregister_kprobe(&kp);
> +	/* close down the gtsc channel */
> +	trace_stop(kprobes_trace);
> +	trace_cleanup(kprobes_trace);
> +}
> +
> +MODULE_LICENSE("GPL");
> +-----------------------------CUT--------------------------------------------
> +How to run the example:
> +$ mount -t debugfs /debug 
> +$ make
> +$ insmod kprobes_gtsc.ko
> +
> +To view the data produced by the module:
> +$ cat /debug/gtsc_example/do_fork/trace0
> +
> +Remove the module.
> +$ rmmod kprobes_gtsc
> +
> +The function trace_cleanup() is called when the module
> +is removed.  This will cause the GTSC channel to be destroyed and the
> +corresponding files to disappear from the debug file system.
> +
> +Credits
> +=======
> +GTSC adapted from blktrace authored by Jens Axboe (axboe@suse.de).
> +
> +Major contributions to GTSC were made by:
> +Tom Zanussi <zanussi@us.ibm.com>
> +Martin Hunt <hunt@redhat.com>
> +David Wilder <dwilder@us.ibm.com>
> 
> 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

-- 
Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

Re: [RFC PATCH 1/3] Generic Trace Setup and Control (GTSC) Documentation

[Tom Zanussi]

On Tue, 2007-07-03 at 23:37 -0400, Mathieu Desnoyers wrote:
> * Tom Zanussi (zanussi@us.ibm.com) wrote:
> > This is the documentation for the Generic Trace Setup and Control
> > patchset, first submitted a couple of weeks ago.  See
> > 
> > http://marc.info/?l=linux-kernel&m=118214274912586&w=2
> > 
> > for a more detailed description.
> > 
> > I've updated this patch to incorporate the suggestions made by Alexey
> > Dobriyan in that thread.
> > 
> 
> It would be nice, since it claims to be "generic", to support things
> brought forward by other tracers like LTTng, such as : multiple channels
> (for low, medium and high event rate data, with user-selectable sizes),
> hybrid trace mode (combined normal trace channels and flight recorder
> channels).
> 

Those seem like useful things, but it would also be nice for LTTng to
support things brought forward by other tracers too.

GTSC comes from something I posted last year called 'UTT', for 'Unified
Trace Transport':

http://www.ecos.sourceware.org/ml/systemtap/2006-q4/msg00026.html

I wasn't comfortable with the word 'Unified' then, either, since I
thought it was presumptuous (so 'unified' was dropped, and 'generic'
used instead - basically it just means that any tracing application can
use it, not that it attempts to subsume all other tracers).

The idea was to see if the blktrace code could be generalized so that it
could be used for other tracing applications besides block layer
tracing.  The test case was to get systemtap running on top of the
blktrace transport, and it seemed to work fine.  As part of that patch
set, I also posted a couple patches that did the same thing but on top
of LTTng, which also worked fine:

http://www.ecos.sourceware.org/ml/systemtap/2006-q4/msg00028.html
http://www.ecos.sourceware.org/ml/systemtap/2006-q4/msg00027.html

But I never got any comments about it, so I assumed it wasn't something
you were interested in.  In any case, do you really think everyone who
wants to do tracing should wait around for LTTng to get into the kernel
lock, stock, and barrel?  It's been awhile since I did this experiment,
but do you think LTTng is ready today to function as the transport for
systemtap?  How hard do you think it would it be to modify blktrace to
run on top of the the LTTng tracing infrastructure?  My guess is that if
the answers were 'yes' and 'not hard', then LTTng could basically 'own'
tracing and the rest of us could move on to more interesting things.

> Please see comments below,
> 
> > +
> > +void gtsc_printf(struct trace_info *trace, const char *format, ...)
> > +{
> > +	va_list args;
> > +	void *buf;
> > +	int len;
> > +	char *record;
> > +
> > +	if (!trace)
> > +		return;
> > +	if (!trace_running(trace))
> > +		return;
> > +
> > +	/* get a timestamp */
> > +	buf = gtsc_printf_tmpbuf[smp_processor_id()];
> > +	len = sprintf(buf,"[%lld] ", trace_timestamp());
> > +
> > +	/* get the data */
> > +	va_start(args, format);
> > +	len += vsnprintf(buf+len, GTSC_PRINTF_TMPBUF_SIZE, format, args);
> > +	va_end(args);
> > +
> > +	/*
> > +	 * Locking can be eliminated by specifying per-cpu buffers
> > +	 * when calling trace_setup().
> > +	 */
> > +	spin_lock(&gtsc_printf_lock);
> > +
> 
> I really hope nobody is willing to call gtsc_printf from an interrupt
> handler.. :) Easy dead embrace in perspective.
> 

This is just example code in the documentation.  There is no actual
gtsc_printf() in the code.

Tom