[RFC PATCH net-next 0/4] perf tools: Support receiving output through BPF programs

[RFC PATCH net-next 0/4] perf tools: Support receiving output through BPF programs

[Wang Nan]

Alexei provided a patchset to allow BPF programs output data to ring
buffer using helper bpf_perf_event_output() [1].  and have been merged
into net-next as commit a43eec304259a6c637f4014a6d4767159b6a3aa3 (bpf:
introduce bpf_perf_event_output() helper).

This patchset introduces perf side code to utilize that helper,

This patchset only supports output data to CTF. It is enough for me
because my workflow is 'perf record' -> 'convert to CTF' -> 'python'.
However, I know some people heavily rely on 'perf script' to parse
trace in perf.data. Here I'd like discuss the way to show output data
using 'perf script'. Currently the only way to watch the output data
using perf script is to use '-D'.

[1] http://lkml.kernel.org/r/1445396556-4854-1-git-send-email-ast@kernel.org

Example:

############ BPF program #############
  struct bpf_map_def SEC("maps") map_channel = {
      .type = BPF_MAP_TYPE_PERF_EVENT_ARRAY,
      .key_size = sizeof(int),
      .value_size = sizeof(u32),
      .max_entries = __NR_CPUS__,
  };

  SEC("func_write=sys_write")
  int func_write(void *ctx)
  {
      struct {
          u64 ktime;
          int cpuid;
      } __attribute__((packed)) output_data;
      char error_data[] = "Error: failed to output\n";

      output_data.cpuid = bpf_get_smp_processor_id();
      output_data.ktime = bpf_ktime_get_ns();
      int err = perf_event_output(ctx, &map_channel,
                                  bpf_get_smp_processor_id(),
                                  &output_data, sizeof(output_data));
      if (err)
          bpf_trace_printk(error_data, sizeof(error_data));
      return 0;
  }
  char _license[] SEC("license") = "GPL";
  int _version SEC("version") = 0x40300;

############ cmdline ############
 # perf record -g -e evt=bpf-output/no-inherit/ \
                  -e ./test_bpf_output.c/maps.map_channel.event=evt/ -a ls
 # perf data convert --to-ctf ./out.ctf
 # babeltrace ./out.ctf
 [06:34:00.288671496] (+?.?????????) evt=bpf-output/no-inherit/: { cpu_id = 0 }, { perf_ip = 0xFFFFFFFF811ED5F1, perf_tid = 4469, perf_pid = 4469, perf_id = 3685, raw_len = 3, raw_data = [ [0] = 0xB163D948, [1] = 0x19E6E, [2] = 0x0 ] }
 [06:34:00.288677551] (+0.000006055) evt=bpf-output/no-inherit/: { cpu_id = 0 }, { perf_ip = 0xFFFFFFFF811ED5F1, perf_tid = 4469, perf_pid = 4469, perf_id = 3685, raw_len = 3, raw_data = [ [0] = 0xB163F289, [1] = 0x19E6E, [2] = 0x0 ] }

########### python script #######
 from babeltrace import TraceCollection

 tc = TraceCollection(
 tc.add_trace('./out.ctf', 'ctf')

 for event in tc.events:
     if not event.name.startswith('evt='):
         continue
     raw_data = event['raw_data']
     print(raw_data[0] + raw_data[1] << 32, raw_data[2]));

Wang Nan (4):
  perf tools: Enable pre-event inherit setting by config terms
  perf tools: Introduce bpf-output event
  perf data: Add u32_hex data type
  perf data: Support converting data from bpf_perf_event_output()

 tools/perf/util/data-convert-bt.c | 117 +++++++++++++++++++++++++++++++++++++-
 tools/perf/util/evsel.c           |  15 +++++
 tools/perf/util/evsel.h           |   2 +
 tools/perf/util/parse-events.c    |  18 ++++++
 tools/perf/util/parse-events.h    |   2 +
 tools/perf/util/parse-events.l    |   3 +
 6 files changed, 156 insertions(+), 1 deletion(-)

-- 
1.8.3.4

[RFC PATCH net-next 1/4] perf tools: Enable pre-event inherit setting by config terms

[Wang Nan]

This patch allows perf record setting event's attr.inherit bit by
config terms like:

 # perf record -e cycles/no-inherit/ ...
 # perf record -e cycles/inherit/ ...

So user can control inherit bit for each event separately.

In following example, a.out fork()s in main then do some complex
CPU intensive computations in both of its children.

Basic result with and without inherit:

 # perf record -e cycles -e instructions ./a.out
 [ perf record: Woken up 9 times to write data ]
 [ perf record: Captured and wrote 2.205 MB perf.data (47920 samples) ]
 # perf report --stdio
 # ...
 # Samples: 23K of event 'cycles'
 # Event count (approx.): 23641752891
 ...
 # Samples: 24K of event 'instructions'
 # Event count (approx.): 30428312415


 # perf record -i -e cycles -e instructions ./a.out
 [ perf record: Woken up 5 times to write data ]
 [ perf record: Captured and wrote 1.111 MB perf.data (24019 samples) ]
 ...
 # Samples: 12K of event 'cycles'
 # Event count (approx.): 11699501775
 ...
 # Samples: 12K of event 'instructions'
 # Event count (approx.): 15058023559

Cancel inherit for one event when globally enable:

 # perf record -e cycles/no-inherit/ -e instructions ./a.out
 [ perf record: Woken up 7 times to write data ]
 [ perf record: Captured and wrote 1.660 MB perf.data (36004 samples) ]
 ...
 # Samples: 12K of event 'cycles/no-inherit/'
 # Event count (approx.): 11895759282
 ...
 # Samples: 24K of event 'instructions'
 # Event count (approx.): 30668000441

Enable inherit for one event when globally disable:

 # perf record -i -e cycles/inherit/ -e instructions ./a.out
 [ perf record: Woken up 7 times to write data ]
 [ perf record: Captured and wrote 1.654 MB perf.data (35868 samples) ]
 ...
 # Samples: 23K of event 'cycles/inherit/'
 # Event count (approx.): 23285400229
 ...
 # Samples: 11K of event 'instructions'
 # Event count (approx.): 14969050259

Signed-off-by: Wang Nan <wangnan0@huawei.com>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Alexei Starovoitov <ast@plumgrid.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Li Zefan <lizefan@huawei.com>
Cc: pi3orama@163.com
Link: http://lkml.kernel.org/n/ebpf-eel1zncmxd8rkw7hr5u04gn0@git.kernel.org
---
 tools/perf/util/evsel.c        |  9 +++++++++
 tools/perf/util/evsel.h        |  2 ++
 tools/perf/util/parse-events.c | 14 ++++++++++++++
 tools/perf/util/parse-events.h |  2 ++
 tools/perf/util/parse-events.l |  2 ++
 5 files changed, 29 insertions(+)

diff --git a/tools/perf/util/evsel.c b/tools/perf/util/evsel.c
index f820906..397fb4e 100644
--- a/tools/perf/util/evsel.c
+++ b/tools/perf/util/evsel.c
@@ -653,6 +653,15 @@ static void apply_config_terms(struct perf_evsel *evsel,
 		case PERF_EVSEL__CONFIG_TERM_STACK_USER:
 			dump_size = term->val.stack_user;
 			break;
+		case PERF_EVSEL__CONFIG_TERM_INHERIT:
+			/*
+			 * attr->inherit should has already been set by
+			 * perf_evsel__config. If user explicitly set
+			 * inherit using config terms, override global
+			 * opt->no_inherit setting.
+			 */
+			attr->inherit = term->val.inherit ? 1 : 0;
+			break;
 		default:
 			break;
 		}
diff --git a/tools/perf/util/evsel.h b/tools/perf/util/evsel.h
index 9a95e73..e402f83 100644
--- a/tools/perf/util/evsel.h
+++ b/tools/perf/util/evsel.h
@@ -43,6 +43,7 @@ enum {
 	PERF_EVSEL__CONFIG_TERM_TIME,
 	PERF_EVSEL__CONFIG_TERM_CALLGRAPH,
 	PERF_EVSEL__CONFIG_TERM_STACK_USER,
+	PERF_EVSEL__CONFIG_TERM_INHERIT,
 	PERF_EVSEL__CONFIG_TERM_MAX,
 };
 
@@ -55,6 +56,7 @@ struct perf_evsel_config_term {
 		bool	time;
 		char	*callgraph;
 		u64	stack_user;
+		u64	inherit;
 	} val;
 };
 
diff --git a/tools/perf/util/parse-events.c b/tools/perf/util/parse-events.c
index ae973cd..b98a82c 100644
--- a/tools/perf/util/parse-events.c
+++ b/tools/perf/util/parse-events.c
@@ -841,6 +841,12 @@ do {									   \
 	case PARSE_EVENTS__TERM_TYPE_STACKSIZE:
 		CHECK_TYPE_VAL(NUM);
 		break;
+	case PARSE_EVENTS__TERM_TYPE_INHERIT:
+		CHECK_TYPE_VAL(NUM);
+		break;
+	case PARSE_EVENTS__TERM_TYPE_NOINHERIT:
+		CHECK_TYPE_VAL(NUM);
+		break;
 	case PARSE_EVENTS__TERM_TYPE_NAME:
 		CHECK_TYPE_VAL(STR);
 		break;
@@ -876,6 +882,8 @@ static int config_term_tracepoint(struct perf_event_attr *attr,
 	switch (term->type_term) {
 	case PARSE_EVENTS__TERM_TYPE_CALLGRAPH:
 	case PARSE_EVENTS__TERM_TYPE_STACKSIZE:
+	case PARSE_EVENTS__TERM_TYPE_INHERIT:
+	case PARSE_EVENTS__TERM_TYPE_NOINHERIT:
 		return config_term_common(attr, term, err);
 	default:
 		if (err) {
@@ -939,6 +947,12 @@ do {								\
 		case PARSE_EVENTS__TERM_TYPE_STACKSIZE:
 			ADD_CONFIG_TERM(STACK_USER, stack_user, term->val.num);
 			break;
+		case PARSE_EVENTS__TERM_TYPE_INHERIT:
+			ADD_CONFIG_TERM(INHERIT, inherit, term->val.num ? 1 : 0);
+			break;
+		case PARSE_EVENTS__TERM_TYPE_NOINHERIT:
+			ADD_CONFIG_TERM(INHERIT, inherit, term->val.num ? 0 : 1);
+			break;
 		default:
 			break;
 		}
diff --git a/tools/perf/util/parse-events.h b/tools/perf/util/parse-events.h
index 5ba1d3e..9d415a8 100644
--- a/tools/perf/util/parse-events.h
+++ b/tools/perf/util/parse-events.h
@@ -67,6 +67,8 @@ enum {
 	PARSE_EVENTS__TERM_TYPE_TIME,
 	PARSE_EVENTS__TERM_TYPE_CALLGRAPH,
 	PARSE_EVENTS__TERM_TYPE_STACKSIZE,
+	PARSE_EVENTS__TERM_TYPE_NOINHERIT,
+	PARSE_EVENTS__TERM_TYPE_INHERIT
 };
 
 struct parse_events_term {
diff --git a/tools/perf/util/parse-events.l b/tools/perf/util/parse-events.l
index 0d92a4d..1fa8e0b 100644
--- a/tools/perf/util/parse-events.l
+++ b/tools/perf/util/parse-events.l
@@ -200,6 +200,8 @@ branch_type		{ return term(yyscanner, PARSE_EVENTS__TERM_TYPE_BRANCH_SAMPLE_TYPE
 time			{ return term(yyscanner, PARSE_EVENTS__TERM_TYPE_TIME); }
 call-graph		{ return term(yyscanner, PARSE_EVENTS__TERM_TYPE_CALLGRAPH); }
 stack-size		{ return term(yyscanner, PARSE_EVENTS__TERM_TYPE_STACKSIZE); }
+inherit			{ return term(yyscanner, PARSE_EVENTS__TERM_TYPE_INHERIT); }
+no-inherit		{ return term(yyscanner, PARSE_EVENTS__TERM_TYPE_NOINHERIT); }
 ,			{ return ','; }
 "/"			{ BEGIN(INITIAL); return '/'; }
 {name_minus}		{ return str(yyscanner, PE_NAME); }
-- 
1.8.3.4

Re: [RFC PATCH net-next 1/4] perf tools: Enable pre-event inherit setting by config terms

[Jiri Olsa]

On Wed, Oct 28, 2015 at 10:55:02AM +0000, Wang Nan wrote:

SNIP

> diff --git a/tools/perf/util/evsel.c b/tools/perf/util/evsel.c
> index f820906..397fb4e 100644
> --- a/tools/perf/util/evsel.c
> +++ b/tools/perf/util/evsel.c
> @@ -653,6 +653,15 @@ static void apply_config_terms(struct perf_evsel *evsel,
>  		case PERF_EVSEL__CONFIG_TERM_STACK_USER:
>  			dump_size = term->val.stack_user;
>  			break;
> +		case PERF_EVSEL__CONFIG_TERM_INHERIT:
> +			/*
> +			 * attr->inherit should has already been set by
> +			 * perf_evsel__config. If user explicitly set
> +			 * inherit using config terms, override global
> +			 * opt->no_inherit setting.
> +			 */
> +			attr->inherit = term->val.inherit ? 1 : 0;
> +			break;
>  		default:
>  			break;
>  		}
> diff --git a/tools/perf/util/evsel.h b/tools/perf/util/evsel.h
> index 9a95e73..e402f83 100644
> --- a/tools/perf/util/evsel.h
> +++ b/tools/perf/util/evsel.h
> @@ -43,6 +43,7 @@ enum {
>  	PERF_EVSEL__CONFIG_TERM_TIME,
>  	PERF_EVSEL__CONFIG_TERM_CALLGRAPH,
>  	PERF_EVSEL__CONFIG_TERM_STACK_USER,
> +	PERF_EVSEL__CONFIG_TERM_INHERIT,
>  	PERF_EVSEL__CONFIG_TERM_MAX,
>  };
>  
> @@ -55,6 +56,7 @@ struct perf_evsel_config_term {
>  		bool	time;
>  		char	*callgraph;
>  		u64	stack_user;
> +		u64	inherit;

seems like bool would be enough

jirka

Re: [RFC PATCH net-next 1/4] perf tools: Enable pre-event inherit setting by config terms

[Arnaldo Carvalho de Melo]

Em Wed, Oct 28, 2015 at 02:21:26PM +0100, Jiri Olsa escreveu:
> On Wed, Oct 28, 2015 at 10:55:02AM +0000, Wang Nan wrote:
> > @@ -55,6 +56,7 @@ struct perf_evsel_config_term {
> >  		bool	time;
> >  		char	*callgraph;
> >  		u64	stack_user;
> > +		u64	inherit;
> 
> seems like bool would be enough

Ok, will change this and move it to a more suitable place member
alignment wise.

Can I, with this change, slap an Acked-by: jirka?

- Arnaldo

Re: [RFC PATCH net-next 1/4] perf tools: Enable pre-event inherit setting by config terms

[Arnaldo Carvalho de Melo]

Em Wed, Oct 28, 2015 at 10:42:13AM -0300, Arnaldo Carvalho de Melo escreveu:
> Em Wed, Oct 28, 2015 at 02:21:26PM +0100, Jiri Olsa escreveu:
> > On Wed, Oct 28, 2015 at 10:55:02AM +0000, Wang Nan wrote:
> > > @@ -55,6 +56,7 @@ struct perf_evsel_config_term {
> > >  		bool	time;
> > >  		char	*callgraph;
> > >  		u64	stack_user;
> > > +		u64	inherit;
> > 
> > seems like bool would be enough
> 
> Ok, will change this and move it to a more suitable place member
> alignment wise.

Nah, switched it to bool, but no need to move it around, that is an
union...
 
> Can I, with this change, slap an Acked-by: jirka?
> 
> - Arnaldo

Re: [RFC PATCH net-next 1/4] perf tools: Enable pre-event inherit setting by config terms

[Jiri Olsa]

On Wed, Oct 28, 2015 at 10:42:13AM -0300, Arnaldo Carvalho de Melo wrote:
> Em Wed, Oct 28, 2015 at 02:21:26PM +0100, Jiri Olsa escreveu:
> > On Wed, Oct 28, 2015 at 10:55:02AM +0000, Wang Nan wrote:
> > > @@ -55,6 +56,7 @@ struct perf_evsel_config_term {
> > >  		bool	time;
> > >  		char	*callgraph;
> > >  		u64	stack_user;
> > > +		u64	inherit;
> > 
> > seems like bool would be enough
> 
> Ok, will change this and move it to a more suitable place member
> alignment wise.
> 
> Can I, with this change, slap an Acked-by: jirka?

yep

jirka

[RFC PATCH net-next 2/4] perf tools: Introduce bpf-output event

[Wang Nan]

Commit a43eec304259a6c637f4014a6d4767159b6a3aa3 (bpf: introduce
bpf_perf_event_output() helper) add a helper to enable BPF program
output data to perf ring buffer through a new type of perf event
PERF_COUNT_SW_BPF_OUTPUT. This patch enable perf to create perf
event of that type. Now perf user can use following cmdline to
receive output data from BPF programs:

 # perf record -a -e evt=bpf-output/no-inherit/ \
                  -e ./test_bpf_output.c/maps.bpf-output.event=evt/ ls

 # perf script
	perf 12927 [004] 355971.129276:          0 evt=bpf-output/no-inherit/:  ffffffff811ed5f1 sys_write
	perf 12927 [004] 355971.129279:          0 evt=bpf-output/no-inherit/:  ffffffff811ed5f1 sys_write
	...

Signed-off-by: Wang Nan <wangnan0@huawei.com>
Cc: Alexei Starovoitov <ast@plumgrid.com>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Brendan Gregg <brendan.d.gregg@gmail.com>
Cc: David S. Miller <davem@davemloft.net>
---
 tools/perf/util/evsel.c        | 6 ++++++
 tools/perf/util/parse-events.c | 4 ++++
 tools/perf/util/parse-events.l | 1 +
 3 files changed, 11 insertions(+)

diff --git a/tools/perf/util/evsel.c b/tools/perf/util/evsel.c
index 397fb4e..f01defb 100644
--- a/tools/perf/util/evsel.c
+++ b/tools/perf/util/evsel.c
@@ -224,6 +224,12 @@ struct perf_evsel *perf_evsel__new_idx(struct perf_event_attr *attr, int idx)
 	if (evsel != NULL)
 		perf_evsel__init(evsel, attr, idx);
 
+	if ((evsel->attr.type == PERF_TYPE_SOFTWARE) &&
+	    (evsel->attr.config == PERF_COUNT_SW_BPF_OUTPUT)) {
+		evsel->attr.sample_type |= PERF_SAMPLE_RAW;
+		evsel->attr.sample_period = 1;
+	}
+
 	return evsel;
 }
 
diff --git a/tools/perf/util/parse-events.c b/tools/perf/util/parse-events.c
index b98a82c..6ac39d6 100644
--- a/tools/perf/util/parse-events.c
+++ b/tools/perf/util/parse-events.c
@@ -124,6 +124,10 @@ struct event_symbol event_symbols_sw[PERF_COUNT_SW_MAX] = {
 		.symbol = "dummy",
 		.alias  = "",
 	},
+	[PERF_COUNT_SW_BPF_OUTPUT] = {
+		.symbol = "bpf-output",
+		.alias  = "",
+	},
 };
 
 #define __PERF_EVENT_FIELD(config, name) \
diff --git a/tools/perf/util/parse-events.l b/tools/perf/util/parse-events.l
index 1fa8e0b..f4ed79b 100644
--- a/tools/perf/util/parse-events.l
+++ b/tools/perf/util/parse-events.l
@@ -249,6 +249,7 @@ cpu-migrations|migrations			{ return sym(yyscanner, PERF_TYPE_SOFTWARE, PERF_COU
 alignment-faults				{ return sym(yyscanner, PERF_TYPE_SOFTWARE, PERF_COUNT_SW_ALIGNMENT_FAULTS); }
 emulation-faults				{ return sym(yyscanner, PERF_TYPE_SOFTWARE, PERF_COUNT_SW_EMULATION_FAULTS); }
 dummy						{ return sym(yyscanner, PERF_TYPE_SOFTWARE, PERF_COUNT_SW_DUMMY); }
+bpf-output					{ return sym(yyscanner, PERF_TYPE_SOFTWARE, PERF_COUNT_SW_BPF_OUTPUT); }
 
 	/*
 	 * We have to handle the kernel PMU event cycles-ct/cycles-t/mem-loads/mem-stores separately.
-- 
1.8.3.4

Re: [RFC PATCH net-next 2/4] perf tools: Introduce bpf-output event

[Sergei Shtylyov]

Hello.

On 10/28/2015 1:55 PM, Wang Nan wrote:

> Commit a43eec304259a6c637f4014a6d4767159b6a3aa3 (bpf: introduce
> bpf_perf_event_output() helper) add a helper to enable BPF program

    You haven't run the patch thru scripts/checkpath.pl, I guess? It now 
enforces the certain style of citing a commit.

> output data to perf ring buffer through a new type of perf event
> PERF_COUNT_SW_BPF_OUTPUT. This patch enable perf to create perf
> event of that type. Now perf user can use following cmdline to
> receive output data from BPF programs:
>
>   # perf record -a -e evt=bpf-output/no-inherit/ \
>                    -e ./test_bpf_output.c/maps.bpf-output.event=evt/ ls
>
>   # perf script
> 	perf 12927 [004] 355971.129276:          0 evt=bpf-output/no-inherit/:  ffffffff811ed5f1 sys_write
> 	perf 12927 [004] 355971.129279:          0 evt=bpf-output/no-inherit/:  ffffffff811ed5f1 sys_write
> 	...
>
> Signed-off-by: Wang Nan <wangnan0@huawei.com>
> Cc: Alexei Starovoitov <ast@plumgrid.com>
> Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
> Cc: Brendan Gregg <brendan.d.gregg@gmail.com>
> Cc: David S. Miller <davem@davemloft.net>
> ---
>   tools/perf/util/evsel.c        | 6 ++++++
>   tools/perf/util/parse-events.c | 4 ++++
>   tools/perf/util/parse-events.l | 1 +
>   3 files changed, 11 insertions(+)
>
> diff --git a/tools/perf/util/evsel.c b/tools/perf/util/evsel.c
> index 397fb4e..f01defb 100644
> --- a/tools/perf/util/evsel.c
> +++ b/tools/perf/util/evsel.c
> @@ -224,6 +224,12 @@ struct perf_evsel *perf_evsel__new_idx(struct perf_event_attr *attr, int idx)
>   	if (evsel != NULL)
>   		perf_evsel__init(evsel, attr, idx);
>
> +	if ((evsel->attr.type == PERF_TYPE_SOFTWARE) &&
> +	    (evsel->attr.config == PERF_COUNT_SW_BPF_OUTPUT)) {

    Inner parens not necessary here.

[...]

MBR, Sergei

[RFC PATCH net-next 3/4] perf data: Add u32_hex data type

[Wang Nan]

Add hexdamical u32 to base data type, which is useful for raw output
because raw data are u32 aligned.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
Cc: Alexei Starovoitov <ast@plumgrid.com>
Cc: Arnaldo Carvalho de Melo <acme@kernel.org>
Cc: Brendan Gregg <brendan.d.gregg@gmail.com>
Cc: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/util/data-convert-bt.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/tools/perf/util/data-convert-bt.c b/tools/perf/util/data-convert-bt.c
index 5bfc119..34cd1e4 100644
--- a/tools/perf/util/data-convert-bt.c
+++ b/tools/perf/util/data-convert-bt.c
@@ -63,6 +63,7 @@ struct ctf_writer {
 			struct bt_ctf_field_type	*s32;
 			struct bt_ctf_field_type	*u32;
 			struct bt_ctf_field_type	*string;
+			struct bt_ctf_field_type	*u32_hex;
 			struct bt_ctf_field_type	*u64_hex;
 		};
 		struct bt_ctf_field_type *array[6];
@@ -982,6 +983,7 @@ do {							\
 	CREATE_INT_TYPE(cw->data.u64, 64, false, false);
 	CREATE_INT_TYPE(cw->data.s32, 32, true,  false);
 	CREATE_INT_TYPE(cw->data.u32, 32, false, false);
+	CREATE_INT_TYPE(cw->data.u32_hex, 32, false, true);
 	CREATE_INT_TYPE(cw->data.u64_hex, 64, false, true);
 
 	cw->data.string  = bt_ctf_field_type_string_create();
-- 
1.8.3.4

[RFC PATCH net-next 4/4] perf data: Support converting data from bpf_perf_event_output()

[Wang Nan]

bpf_perf_event_output() outputs data through sample->raw_data. This
patch adds support to convert those data into CTF. A python script
then can be used to process output data from BPF programs.

Example:

############ BPF program #############
  struct bpf_map_def SEC("maps") map_channel = {
      .type = BPF_MAP_TYPE_PERF_EVENT_ARRAY,
      .key_size = sizeof(int),
      .value_size = sizeof(u32),
      .max_entries = __NR_CPUS__,
  };

  SEC("func_write=sys_write")
  int func_write(void *ctx)
  {
      struct {
          u64 ktime;
          int cpuid;
      } __attribute__((packed)) output_data;
      char error_data[] = "Error: failed to output\n";

      output_data.cpuid = bpf_get_smp_processor_id();
      output_data.ktime = bpf_ktime_get_ns();
      int err = perf_event_output(ctx, &map_channel,
                                  bpf_get_smp_processor_id(),
                                  &output_data, sizeof(output_data));
      if (err)
          bpf_trace_printk(error_data, sizeof(error_data));
      return 0;
  }
  char _license[] SEC("license") = "GPL";
  int _version SEC("version") = 0x40300;

############ cmdline ############
 # perf record -g -e evt=bpf-output/no-inherit/ \
                  -e ./test_bpf_output.c/maps.map_channel.event=evt/ -a ls
 # perf data convert --to-ctf ./out.ctf
 # babeltrace ./out.ctf
 [06:34:00.288671496] (+?.?????????) evt=bpf-output/no-inherit/: { cpu_id = 0 }, { perf_ip = 0xFFFFFFFF811ED5F1, perf_tid = 4469, perf_pid = 4469, perf_id = 3685, raw_len = 3, raw_data = [ [0] = 0xB163D948, [1] = 0x19E6E, [2] = 0x0 ] }
 [06:34:00.288677551] (+0.000006055) evt=bpf-output/no-inherit/: { cpu_id = 0 }, { perf_ip = 0xFFFFFFFF811ED5F1, perf_tid = 4469, perf_pid = 4469, perf_id = 3685, raw_len = 3, raw_data = [ [0] = 0xB163F289, [1] = 0x19E6E, [2] = 0x0 ] }

########### python script #######
 from babeltrace import TraceCollection

 tc = TraceCollection(
 tc.add_trace('./out.ctf', 'ctf')

 for event in tc.events:
     if not event.name.startswith('evt='):
         continue
     raw_data = event['raw_data']
     print(raw_data[0] + raw_data[1] << 32, raw_data[2]));

Signed-off-by: Wang Nan <wangnan0@huawei.com>
Cc: Alexei Starovoitov <ast@plumgrid.com>
Cc: Arnaldo Carvalho de Melo <acme@kernel.org>
Cc: Brendan Gregg <brendan.d.gregg@gmail.com>
Cc: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/util/data-convert-bt.c | 115 +++++++++++++++++++++++++++++++++++++-
 1 file changed, 114 insertions(+), 1 deletion(-)

diff --git a/tools/perf/util/data-convert-bt.c b/tools/perf/util/data-convert-bt.c
index 34cd1e4..44d6e0d 100644
--- a/tools/perf/util/data-convert-bt.c
+++ b/tools/perf/util/data-convert-bt.c
@@ -352,6 +352,84 @@ static int add_tracepoint_values(struct ctf_writer *cw,
 	return ret;
 }
 
+static int
+add_bpf_output_values(struct bt_ctf_event_class *event_class,
+		      struct bt_ctf_event *event,
+		      struct perf_sample *sample)
+{
+	struct bt_ctf_field_type *len_type, *seq_type;
+	struct bt_ctf_field *len_field, *seq_field;
+	unsigned int raw_size = sample->raw_size;
+	unsigned int nr_elements = raw_size / sizeof(u32);
+	unsigned int i;
+	int ret;
+
+	if (nr_elements * sizeof(u32) != raw_size)
+		pr_warning("Incorrect raw_size (%u) in bpf output event, skip %lu bytes\n",
+			   raw_size, nr_elements * sizeof(u32) - raw_size);
+	
+	len_type = bt_ctf_event_class_get_field_by_name(event_class, "raw_len");
+	len_field = bt_ctf_field_create(len_type);
+	if (!len_field) {
+		pr_err("failed to create 'raw_len' for bpf output event\n");
+		ret = -1;
+		goto put_len_type;
+	}
+
+	ret = bt_ctf_field_unsigned_integer_set_value(len_field, nr_elements);
+	if (ret) {
+		pr_err("failed to set field value for raw_len\n");
+		goto put_len_field;
+	}
+	ret = bt_ctf_event_set_payload(event, "raw_len", len_field);
+	if (ret) {
+		pr_err("failed to set payload to raw_len\n");
+		goto put_len_field;
+	}
+
+	seq_type = bt_ctf_event_class_get_field_by_name(event_class, "raw_data");
+	seq_field = bt_ctf_field_create(seq_type);
+	if (!seq_field) {
+		pr_err("failed to create 'raw_data' for bpf output event\n");
+		ret = -1;
+		goto put_seq_type;
+	}
+
+	ret = bt_ctf_field_sequence_set_length(seq_field, len_field);
+	if (ret) {
+		pr_err("failed to set length of 'raw_data'\n");
+		goto put_seq_field;
+	}
+
+	for (i = 0; i < nr_elements; i++) {
+		struct bt_ctf_field *elem_field =
+			bt_ctf_field_sequence_get_field(seq_field, i);
+
+		ret = bt_ctf_field_unsigned_integer_set_value(elem_field,
+				((u32 *)(sample->raw_data))[i]);
+
+		bt_ctf_field_put(elem_field);
+		if (ret) {
+			pr_err("failed to set raw_data[%d]\n", i);
+			goto put_seq_field;
+		}
+	}
+
+	ret = bt_ctf_event_set_payload(event, "raw_data", seq_field);
+	if (ret)
+		pr_err("failed to set payload for raw_data\n");
+
+put_seq_field:
+	bt_ctf_field_put(seq_field);
+put_seq_type:
+	bt_ctf_field_type_put(seq_type);
+put_len_field:
+	bt_ctf_field_put(len_field);
+put_len_type:
+	bt_ctf_field_type_put(len_type);
+	return ret;
+}
+
 static int add_generic_values(struct ctf_writer *cw,
 			      struct bt_ctf_event *event,
 			      struct perf_evsel *evsel,
@@ -597,6 +675,13 @@ static int process_sample_event(struct perf_tool *tool,
 			return -1;
 	}
 
+	if ((evsel->attr.type == PERF_TYPE_SOFTWARE) &&
+	    (evsel->attr.config == PERF_COUNT_SW_BPF_OUTPUT)) {
+		ret = add_bpf_output_values(event_class, event, sample);
+		if (ret)
+			return -1;
+	}
+
 	cs = ctf_stream(cw, get_sample_cpu(cw, sample, evsel));
 	if (cs) {
 		if (is_flush_needed(cs))
@@ -744,6 +829,25 @@ static int add_tracepoint_types(struct ctf_writer *cw,
 	return ret;
 }
 
+static int add_bpf_output_types(struct ctf_writer *cw,
+				struct bt_ctf_event_class *class)
+{
+	struct bt_ctf_field_type *len_type = cw->data.u32;
+	struct bt_ctf_field_type *seq_base_type = cw->data.u32_hex;
+	struct bt_ctf_field_type *seq_type;
+	int ret;
+
+	ret = bt_ctf_event_class_add_field(class, len_type, "raw_len");
+	if (ret)
+		return ret;
+
+	seq_type = bt_ctf_field_type_sequence_create(seq_base_type, "raw_len");
+	if (!seq_type)
+		return -1;
+
+	return bt_ctf_event_class_add_field(class, seq_type, "raw_data");
+}
+
 static int add_generic_types(struct ctf_writer *cw, struct perf_evsel *evsel,
 			     struct bt_ctf_event_class *event_class)
 {
@@ -755,7 +859,8 @@ static int add_generic_types(struct ctf_writer *cw, struct perf_evsel *evsel,
 	 *                              ctf event header
 	 *   PERF_SAMPLE_READ         - TODO
 	 *   PERF_SAMPLE_CALLCHAIN    - TODO
-	 *   PERF_SAMPLE_RAW          - tracepoint fields are handled separately
+	 *   PERF_SAMPLE_RAW          - tracepoint fields and BPF output
+	 *                              are handled separately
 	 *   PERF_SAMPLE_BRANCH_STACK - TODO
 	 *   PERF_SAMPLE_REGS_USER    - TODO
 	 *   PERF_SAMPLE_STACK_USER   - TODO
@@ -824,6 +929,14 @@ static int add_event(struct ctf_writer *cw, struct perf_evsel *evsel)
 			goto err;
 	}
 
+	if ((evsel->attr.type == PERF_TYPE_SOFTWARE) &&
+	    (evsel->attr.config == PERF_COUNT_SW_BPF_OUTPUT)) {
+		ret = add_bpf_output_types(cw, event_class);
+		if (ret)
+			goto err;
+
+	}
+
 	ret = bt_ctf_stream_class_add_event_class(cw->stream_class, event_class);
 	if (ret) {
 		pr("Failed to add event class into stream.\n");
-- 
1.8.3.4

Re: [RFC PATCH net-next 0/4] perf tools: Support receiving output through BPF programs

[Wangnan (F)]

On 2015/10/28 18:55, Wang Nan wrote:
> Alexei provided a patchset to allow BPF programs output data to ring
> buffer using helper bpf_perf_event_output() [1].  and have been merged
> into net-next as commit a43eec304259a6c637f4014a6d4767159b6a3aa3 (bpf:
> introduce bpf_perf_event_output() helper).
>
> This patchset introduces perf side code to utilize that helper,
>
> This patchset only supports output data to CTF. It is enough for me
> because my workflow is 'perf record' -> 'convert to CTF' -> 'python'.
> However, I know some people heavily rely on 'perf script' to parse
> trace in perf.data. Here I'd like discuss the way to show output data
> using 'perf script'. Currently the only way to watch the output data
> using perf script is to use '-D'.
>
> [1] http://lkml.kernel.org/r/1445396556-4854-1-git-send-email-ast@kernel.org

This patchset is based on my perf ebpf support patches. Workable
code can be found from github:

https://github.com/WangNan0/linux/commits/ebpf

It should be merged after net-next merged into mainline. I post
them now for demostration and discussion.

Thank you.