[PATCH v3 0/7] tracing: 'hist' triggers

[Tom Zanussi <tom.zanussi@linux.intel.com>]

This is v3 of the 'hist triggers' patchset, following feedback from
v2.

The functionality remains the same as v2, but this version no longer
tries to export and use bpf_maps, and more importantly removes the
associated GFP_NOTRACE/trace event hacks and kmem macros required to
work around the bpf_map implementation.

The tracing_map functionality is instead built on top of a simple
lock-free map algorithm originated by Dr. Cliff Click (see references
in the code for more details), which though too restrictive to be
general-purpose in its current form, functions nicely as a
special-purpose tracing map.

v3 also moves the hist triggers code into a separate file and puts it
all behind a new config option, CONFIG_HIST_TRIGGERS.  It also merges
in the sorting code rather than keeping it as a separate patch.

This patchset also includes a couple other new and related triggers,
enable_hist and disable_hist, very similar to the existing
enable_event/disable_event triggers used to automatically enable and
disable events based on a triggering condition, but in this case
allowing hist triggers to be enabled and disabled in the same way.

There are a couple of important bits of functionality that were
present in v1 but not yet reimplemented in v3.

The first is support for compound keys.  Currently, maps can only be
keyed on a single event field, whereas in v1 they could be keyed on
multiple keys.  With support for compound keys, you can create much
more interesting output, such as for example per-pid lists of
syscalls or read counts e.g.:

  # echo 'hist:keys=common_pid.execname,id.syscall:vals=hitcount' > \
        /sys/kernel/debug/tracing/events/raw_syscalls/sys_enter/trigger

  # cat /sys/kernel/debug/tracing/events/raw_syscalls/sys_enter/hist

  key: common_pid:bash[3112], id:sys_write		       vals: count:69
  key: common_pid:bash[3112], id:sys_rt_sigprocmask	       vals: count:218

  key: common_pid:update-notifier[3164], id:sys_poll	       vals: count:37
  key: common_pid:update-notifier[3164], id:sys_recvfrom       vals: count:118

  key: common_pid:deja-dup-monito[3194], id:sys_sendto	       vals: count:1
  key: common_pid:deja-dup-monito[3194], id:sys_read	       vals: count:4
  key: common_pid:deja-dup-monito[3194], id:sys_poll	       vals: count:8
  key: common_pid:deja-dup-monito[3194], id:sys_recvmsg	       vals: count:8
  key: common_pid:deja-dup-monito[3194], id:sys_getegid	       vals: count:8

  key: common_pid:emacs[3275], id:sys_fsync		       vals: count:1
  key: common_pid:emacs[3275], id:sys_open		       vals: count:1
  key: common_pid:emacs[3275], id:sys_symlink		       vals: count:2
  key: common_pid:emacs[3275], id:sys_poll		       vals: count:23
  key: common_pid:emacs[3275], id:sys_select		       vals: count:23
  key: common_pid:emacs[3275], id:unknown_syscall	       vals: count:34
  key: common_pid:emacs[3275], id:sys_ioctl		       vals: count:60
  key: common_pid:emacs[3275], id:sys_rt_sigprocmask	       vals: count:116

  key: common_pid:cat[3323], id:sys_munmap		       vals: count:1
  key: common_pid:cat[3323], id:sys_fadvise64		       vals: count:1

Related to that is support for sorting on multiple fields.  Currently,
you can sort using only a primary key.  Being able to sort on multiple
or at least a secondary key is indispensible for seeing trends when
displaying multiple values.

Changes from v2:
 - reimplemented tracing_map, replacing bpf_map with nmi-safe/lock-free map
 - removed GPF_NOTRACE, kmalloc/free macros and event hacks needed by bpf_maps
 - moved hist triggers from trace_events_trigger.c to trace_events_hist.c
 - added CONFIG_HIST_TRIGGERS config option
 - consolidated sorting code with main patch

Changes from v1:
 - completely rewritten on top of tracing_map (renamed and exported bpf_map)
 - added map clearing and client ops to tracing_map
 - changed the name from 'hash' triggers to 'hist' triggers
 - added new trigger 'pause' feature
 - added new enable_hist and disable_hist triggers
 - added usage for hist/enable_hist/disable hist to tracing/README
 - moved examples into Documentation/trace/event.txt
 - added ___GFP_NOTRACE, kmalloc/kfree macros, and conditional kmem tracepoints

The following changes since commit e65e0516fb5fdfe1c3138ccd333651739894197f:

  Merge branch 'for-next/ftrace/core' into trace/for-next (2015-03-31 10:06:28 -0400)

are available in the git repository at:


  git://git.yoctoproject.org/linux-yocto-contrib.git tzanussi/hist-triggers-v3
  http://git.yoctoproject.org/cgit/cgit.cgi/linux-yocto-contrib/log/?h=tzanussi/hist-triggers-v3

Tom Zanussi (7):
  tracing: Make ftrace_event_field checking functions available
  tracing: Add event record param to trigger_ops.func()
  tracing: Add get_syscall_name()
  tracing: Add a per-event-trigger 'paused' field
  tracing: Add 'hist' event trigger command
  tracing: Add enable_hist/disable_hist triggers
  tracing: Add 'hist' trigger Documentation

 Documentation/trace/events.txt      |  870 ++++++++++++++++++++
 include/linux/ftrace_event.h        |    9 +-
 kernel/trace/Kconfig                |   14 +
 kernel/trace/Makefile               |    1 +
 kernel/trace/trace.c                |   54 ++
 kernel/trace/trace.h                |   77 +-
 kernel/trace/trace_events.c         |    4 +
 kernel/trace/trace_events_filter.c  |   15 +-
 kernel/trace/trace_events_hist.c    | 1550 +++++++++++++++++++++++++++++++++++
 kernel/trace/trace_events_trigger.c |  144 ++--
 kernel/trace/trace_syscalls.c       |   11 +
 11 files changed, 2668 insertions(+), 81 deletions(-)
 create mode 100644 kernel/trace/trace_events_hist.c

-- 
1.9.3