[RFC PATCH 1/3] Generic Trace Setup and Control (GTSC) Documentation

[Tom Zanussi <zanussi@us.ibm.com>]

This is the documentation for the Generic Trace Setup and Control
patchset, first submitted a couple of weeks ago.  See

http://marc.info/?l=linux-kernel&m=118214274912586&w=2

for a more detailed description.

I've updated this patch to incorporate the suggestions made by Alexey
Dobriyan in that thread.

Signed-off-by: Tom Zanussi <zanussi@us.ibm.com>
Signed-off-by: David Wilder <dwilder@us.ibm.com>
---
 gtsc.txt |  247 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 247 insertions(+)

diff --git a/Documentation/gtsc.txt b/Documentation/gtsc.txt
new file mode 100644
index 0000000..470d1fc
--- /dev/null
+++ b/Documentation/gtsc.txt
@@ -0,0 +1,247 @@
+Generic Trace Setup and Control (GTSC)
+==================================
+In the kernel, GTSC provides a simple API for starting and managing
+data channels to user space.  GTSC builds on the relay interface. For a
+complete description of the relay interface, please see:
+Documentation/filesystems/relay.txt.
+
+GTSC provides one layer in a complete tracing application.  The idea of
+the GTSC is to provide a kernel API for the setup and control of tracing
+channels.  User of GTSC must provide a data layer responsible for formatting
+and writing data into the trace channels.  
+
+A layered approach to tracing
+=============================
+A complete kernel tracing application consists of a data provider and a data
+consumer.  Both provider and consumer contain three layers; each layer works
+in tandem with the corresponding layer in the opposite side.  The layers are
+represented in the following diagram.
+  
+Provider Data layer
+	Formats raw trace data and provides data-related service.
+	For example, adding timestamps used by consumer to sort data.
+
+Provider Control layer
+	Provided by GTSC. Creates trace channels and informs the data layer
+	and consumer of the current state of the trace channels.
+
+Provider Buffering layer
+	Provided by relay. This layer buffers data in the
+	kernel for consumption by the consumer's buffer
+	layer.
+
+Provider (in-kernel facility)
+-----------------------------------------------------------------------------
+Consumer (user application)
+
+
+Consumer Buffer layer
+	Reads/consumes data from the provider's data buffers.
+
+Consumer Control layer
+	Communicates to the provider's control layer to control the state
+	of the trace channels. 
+
+Consumer Data layer
+	Sorts and formats data as provided by the provider's data layer.
+
+The provider is coded as a kernel facility.  The consumer is coded as
+a user application.
+ 
+
+GTSC - Features
+==============
+The GTSC exploits services and features provided by relay.  These features are:
+- The creation and destruction of relay channels.
+- Buffer management.  Overwrite or non-overwrite modes can be selected
+  as well as global or per-CPU buffering.
+
+Overwrite mode can be called "flight recorder mode".  Flight recorder
+mode is selected by setting the TRACE_FLIGHT_CHANNEL flag when
+creating trace channels.  In flight mode when a tracing buffer is
+full, the oldest records in the buffer will be discarded to make room
+as new records arrive.  In the default non-overwrite mode, new records
+may be written only if the buffer has room.  In either case, to
+prevent data loss, a user space reader must keep the buffers
+drained. GTSC provides a means to detect the number of records that
+have been dropped due to a buffer-full condition (non-overwrite mode
+only).
+
+When per-CPU buffers are used, relay creates one debugfs file for each
+running CPU.  The user-space consumer of the data is responsible for
+reading the per-CPU buffers and collating the records presumably using
+a time stamp or sequence number included in the trace records.  The
+use of global buffers eliminates this extra work of sequencing
+records; however the provider's data layer must hold a lock when
+writing records.  The lock prevents writers running on different CPUs
+from overwriting each other's data.  However, buffering may be slower
+because write to the buffer are serialized. Global buffering is
+selected by setting the TRACE_GLOBAL_CHANNEL flag when creating trace
+channels.
+
+GTSC User Interface
+===================
+When a GTSC channel is created and tracing has been started, the following
+directories and files are created in the root of the mounted debugfs.
+
+/debug (root of the debugfs)
+        /<trace-root-dir>
+                /<trace-name>
+                        trace0 ... traceN  (Per-CPU trace data, one per CPU)
+	        	state		   (Used to  start and stop tracing)
+                        dropped            (number of records dropped due
+                                            to a full-buffer condition, only)
+                                            for non-TRACE_FLIGHT_CHANNELs)
+                        rewind             ('un-consume' channel data i.e.
+                                            start next read at the beginning
+                                            again (TRACE_FLIGHT_CHANNELS only)
+	        	nr_sub		   (Number of sub-buffers in channel)
+	        	sub_size	   (Size of sub-buffers in channnel)
+
+Trace data is gathered from the trace[0...N] files using one of the available 
+interfaces provided by relay (see Documentation/filesystems/relay.txt).
+When using the READ(2) interface, as data is read it is marked as consumed by
+the relay subsystem.  Therefore, subsequent reads will only return unconsumed
+data.
+
+GTSC Kernel API
+===============
+An overview of the GTSC Kernel API is now given. More details of the API can
+be found in linux/gtsc.h.
+
+The steps a kernel data provider takes to utilize the GTSC are:
+1) Set up a gtsc channel - trace_setup()
+2) Start the GTSC channel - trace_start()
+3) Write one or more trace records into the channel (using the relay API).
+4) Optionally stop and start tracing as desired - trace_start()/trace_stop()
+5) Destroy the GTSC channel and underlying relay channel - trace_cleanup().
+
+GTSC Example
+===========
+This small sample module creates a GTSC channel. It places a kprobe on the
+function do_fork().  The kprobe handler will write a timestamp and
+current->pid to the GTSC channel.  
+
+You can build the kernel module kprobes_gtsc.ko using the following Makefile:
+------------------------------------CUT-------------------------------------
+obj-m := kprobes_gtsc.o
+KDIR := /lib/modules/$(shell uname -r)/build
+PWD := $(shell pwd)
+default:
+	$(MAKE) -C $(KDIR) SUBDIRS=$(PWD) modules
+clean:
+	rm -f *.mod.c *.ko *.o
+----------------------------------CUT--------------------------------------
+/* kprobes_gtsc.c - An example of using GTSC in a kprobes module */
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/kprobes.h>
+#include <linux/gtsc.h>
+
+#define PROBE_POINT "do_fork"
+
+static struct kprobe kp;
+struct trace_info *kprobes_trace;
+
+#define GTSC_PRINTF_TMPBUF_SIZE (1024)
+static char gtsc_printf_tmpbuf[NR_CPUS][GTSC_PRINTF_TMPBUF_SIZE];
+
+/* This lock is needed only when using global relay buffers */
+static DEFINE_SPINLOCK(gtsc_printf_lock);
+
+void gtsc_printf(struct trace_info *trace, const char *format, ...)
+{
+	va_list args;
+	void *buf;
+	int len;
+	char *record;
+
+	if (!trace)
+		return;
+	if (!trace_running(trace))
+		return;
+
+	/* get a timestamp */
+	buf = gtsc_printf_tmpbuf[smp_processor_id()];
+	len = sprintf(buf,"[%lld] ", trace_timestamp());
+
+	/* get the data */
+	va_start(args, format);
+	len += vsnprintf(buf+len, GTSC_PRINTF_TMPBUF_SIZE, format, args);
+	va_end(args);
+
+	/*
+	 * Locking can be eliminated by specifying per-cpu buffers
+	 * when calling trace_setup().
+	 */
+	spin_lock(&gtsc_printf_lock);
+
+	/* Send everything to the relay buffer */
+	if ((record = relay_reserve(trace->rchan, len)))
+		memcpy(record, buf, len);
+
+	spin_unlock(&gtsc_printf_lock);
+}
+
+
+int handler_pre(struct kprobe *p, struct pt_regs *regs)
+{
+	gtsc_printf(kprobes_trace,"%d\n", current->pid);
+	return 0;
+}
+
+int init_module(void)
+{
+	int ret;
+	/* setup gtsc, use a global relay buffer */
+	kprobes_trace = trace_setup("gtsc_example",PROBE_POINT,
+				1024,8,
+				TRACE_GLOBAL_CHANNEL | TRACE_FLIGHT_CHANNEL);
+	if (!kprobes_trace)
+		return -1;
+
+	trace_start(kprobes_trace);
+
+	/* setup the kprobe */
+	kp.pre_handler = handler_pre;
+	kp.post_handler = NULL;
+	kp.fault_handler = NULL;
+	kp.symbol_name = PROBE_POINT;
+	if ((ret=register_kprobe(&kp)) < 0) 
+		gtsc_printf(kprobes_trace,"register_kprobe failed  %d\n", ret);
+	return 0;
+}
+
+void cleanup_module(void)
+{
+	unregister_kprobe(&kp);
+	/* close down the gtsc channel */
+	trace_stop(kprobes_trace);
+	trace_cleanup(kprobes_trace);
+}
+
+MODULE_LICENSE("GPL");
+-----------------------------CUT--------------------------------------------
+How to run the example:
+$ mount -t debugfs /debug 
+$ make
+$ insmod kprobes_gtsc.ko
+
+To view the data produced by the module:
+$ cat /debug/gtsc_example/do_fork/trace0
+
+Remove the module.
+$ rmmod kprobes_gtsc
+
+The function trace_cleanup() is called when the module
+is removed.  This will cause the GTSC channel to be destroyed and the
+corresponding files to disappear from the debug file system.
+
+Credits
+=======
+GTSC adapted from blktrace authored by Jens Axboe (axboe@suse.de).
+
+Major contributions to GTSC were made by:
+Tom Zanussi <zanussi@us.ibm.com>
+Martin Hunt <hunt@redhat.com>
+David Wilder <dwilder@us.ibm.com>