[RFC PATCH 0/5] perf tools: Decode bpf output in 'perf trace'

[RFC PATCH 0/5] perf tools: Decode bpf output in 'perf trace'

[Wang Nan]

Hi Arnaldo,

  You try to record + analysis using 'perf trace' with BPF output
but failed yesterday. You need these 5 patches to make it work.

Thank you.

Wang Nan (5):
  perf trace: Apply config options in '.perfconfig'
  perf trace: Improve error message when receive non-tracepoint events
  perf tools: Only set filter for tracepoints events
  perf trace: Call bpf__apply_obj_config in 'perf trace'
  perf trace: Print content of bpf-output event

 tools/perf/builtin-trace.c | 71 ++++++++++++++++++++++++++++++++++++++++++++--
 tools/perf/util/evlist.c   |  3 ++
 2 files changed, 72 insertions(+), 2 deletions(-)

Signed-off-by: Wang Nan <wangnan0@huawei.com>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Li Zefan <lizefan@huawei.com>
Cc: pi3orama@163.com

-- 
1.8.3.4

[RFC PATCH 1/5] perf trace: Apply config options in '.perfconfig'

[Wang Nan]

'perf trace' doesn't read and apply options in ~/.perfconfig. If a BPF
script is passed to 'perf trace --ev' and clang is not in $PATH,
perf trace doesn't compile the script even clang-path is configured in
~/.perfconfig.

This patch makes 'perf trace' read and apply default config options
from that file.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Li Zefan <lizefan@huawei.com>
Cc: pi3orama@163.com
---
 tools/perf/builtin-trace.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/tools/perf/builtin-trace.c b/tools/perf/builtin-trace.c
index 20916dd..6c52f3c 100644
--- a/tools/perf/builtin-trace.c
+++ b/tools/perf/builtin-trace.c
@@ -3108,6 +3108,8 @@ int cmd_trace(int argc, const char **argv, const char *prefix __maybe_unused)
 		goto out;
 	}
 
+	perf_config(perf_default_config, NULL);
+
 	argc = parse_options_subcommand(argc, argv, trace_options, trace_subcommands,
 				 trace_usage, PARSE_OPT_STOP_AT_NON_OPTION);
 
-- 
1.8.3.4

[RFC PATCH 2/5] perf trace: Improve error message when receive non-tracepoint events

[Wang Nan]

Before this patch, strange error message is provided if passed a
non-tracepoint event to 'perf trace':

 # perf trace -a  --ev cycles sleep 1
 Failed to set filter "common_pid != 27500" on event cycles with 22 (Invalid argument)

This is because 'perf trace' accepts all valid event during cmdline
parsing, but since it need setting filter to events, actually user
can pass tracepoints with '--ev' only.

This patch validate evlist, report error earlier:

 # ./perf trace -a  --ev cycles sleep 1
 Only support tracepoint events!

Signed-off-by: Wang Nan <wangnan0@huawei.com>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Li Zefan <lizefan@huawei.com>
Cc: pi3orama@163.com
---
 tools/perf/builtin-trace.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/tools/perf/builtin-trace.c b/tools/perf/builtin-trace.c
index 6c52f3c..2deb3f0 100644
--- a/tools/perf/builtin-trace.c
+++ b/tools/perf/builtin-trace.c
@@ -2525,6 +2525,17 @@ out_enomem:
 	goto out;
 }
 
+static int validate_evlist(struct perf_evlist *evlist)
+{
+	struct perf_evsel *evsel;
+
+	evlist__for_each(evlist, evsel) {
+		if (evsel->attr.type != PERF_TYPE_TRACEPOINT)
+			return -EINVAL;
+	}
+	return 0;
+}
+
 static int trace__run(struct trace *trace, int argc, const char **argv)
 {
 	struct perf_evlist *evlist = trace->evlist;
@@ -3113,6 +3124,11 @@ int cmd_trace(int argc, const char **argv, const char *prefix __maybe_unused)
 	argc = parse_options_subcommand(argc, argv, trace_options, trace_subcommands,
 				 trace_usage, PARSE_OPT_STOP_AT_NON_OPTION);
 
+	if (validate_evlist(trace.evlist)) {
+		pr_err("Only support tracepoint events!\n");
+		return -EINVAL;
+	}
+
 	if (trace.trace_pgfaults) {
 		trace.opts.sample_address = true;
 		trace.opts.sample_time = true;
-- 
1.8.3.4

[RFC PATCH 3/5] perf tools: Only set filter for tracepoints events

[Wang Nan]

perf_evlist__set_filter() tries to set filter to every evsels linked in
the evlist. However, since filters can only be applied to tracepoints,
checking type of evsel before calling perf_evsel__set_filter() would be
better.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Li Zefan <lizefan@huawei.com>
Cc: pi3orama@163.com
---
 tools/perf/util/evlist.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/tools/perf/util/evlist.c b/tools/perf/util/evlist.c
index c42e196..86a0383 100644
--- a/tools/perf/util/evlist.c
+++ b/tools/perf/util/evlist.c
@@ -1223,6 +1223,9 @@ int perf_evlist__set_filter(struct perf_evlist *evlist, const char *filter)
 	int err = 0;
 
 	evlist__for_each(evlist, evsel) {
+		if (evsel->attr.type != PERF_TYPE_TRACEPOINT)
+			continue;
+
 		err = perf_evsel__set_filter(evsel, filter);
 		if (err)
 			break;
-- 
1.8.3.4

[RFC PATCH 4/5] perf trace: Call bpf__apply_obj_config in 'perf trace'

[Wang Nan]

Without this patch BPF map configuration is not applied.

Command like this:
 # ./perf trace --ev bpf-output/no-inherit,name=evt/ \
                --ev ./test_bpf_trace.c/map:channel.event=evt/ \
                usleep 100000

Load BPF files without error, but since map:channel.event=evt is not
applied, bpf-output event not work.

This patch allows 'perf trace' load and run BPF scripts.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Li Zefan <lizefan@huawei.com>
Cc: pi3orama@163.com
---
 tools/perf/builtin-trace.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/tools/perf/builtin-trace.c b/tools/perf/builtin-trace.c
index 2deb3f0..b7129d6 100644
--- a/tools/perf/builtin-trace.c
+++ b/tools/perf/builtin-trace.c
@@ -33,6 +33,7 @@
 #include "util/stat.h"
 #include "trace-event.h"
 #include "util/parse-events.h"
+#include "util/bpf-loader.h"
 
 #include <libaudit.h>
 #include <stdlib.h>
@@ -2597,6 +2598,16 @@ static int trace__run(struct trace *trace, int argc, const char **argv)
 	if (err < 0)
 		goto out_error_open;
 
+	err = bpf__apply_obj_config();
+	if (err) {
+		char errbuf[BUFSIZ];
+
+		bpf__strerror_apply_obj_config(err, errbuf, sizeof(errbuf));
+		pr_err("ERROR: Apply config to BPF failed: %s\n",
+			 errbuf);
+		goto out_error_open;
+	}
+
 	/*
 	 * Better not use !target__has_task() here because we need to cover the
 	 * case where no threads were specified in the command line, but a
-- 
1.8.3.4

[RFC PATCH 5/5] perf trace: Print content of bpf-output event

[Wang Nan]

With this patch the contend of BPF output event is printed by
'perf trace'. For example:

 # ./perf trace -a --ev bpf-output/no-inherit,name=evt/ \
                   --ev ./test_bpf_trace.c/map:channel.event=evt/ \
                   usleep 100000
  ...
    1.787 ( 0.004 ms): usleep/3832 nanosleep(rqtp: 0x7ffc78b18980                                        ) ...
    1.787 (         ): evt:Raise a BPF event!..)
    1.788 (         ): perf_bpf_probe:func_begin:(ffffffff810e97d0))
  ...
  101.866 (87.038 ms): gmain/1654 poll(ufds: 0x7f57a80008c0, nfds: 2, timeout_msecs: 1000               ) ...
  101.866 (         ): evt:Raise a BPF event!..)
  101.867 (         ): perf_bpf_probe:func_end:(ffffffff810e97d0 <- ffffffff81796173))
  101.869 (100.087 ms): usleep/3832  ... [continued]: nanosleep()) = 0
  ...

 (There is an extra ')' at the end of several lines. However, it is
  another problem, unrelated to this commit.)

Where test_bpf_trace.c is:
 /************************ BEGIN **************************/
 #include <uapi/linux/bpf.h>
 struct bpf_map_def {
        unsigned int type;
        unsigned int key_size;
        unsigned int value_size;
        unsigned int max_entries;
 };
 #define SEC(NAME) __attribute__((section(NAME), used))
 static u64 (*ktime_get_ns)(void) =
        (void *)BPF_FUNC_ktime_get_ns;
 static int (*trace_printk)(const char *fmt, int fmt_size, ...) =
        (void *)BPF_FUNC_trace_printk;
 static int (*get_smp_processor_id)(void) =
        (void *)BPF_FUNC_get_smp_processor_id;
 static int (*perf_event_output)(void *, struct bpf_map_def *, int, void *, unsigned long) =
        (void *)BPF_FUNC_perf_event_output;

 struct bpf_map_def SEC("maps") channel = {
        .type = BPF_MAP_TYPE_PERF_EVENT_ARRAY,
        .key_size = sizeof(int),
        .value_size = sizeof(u32),
        .max_entries = __NR_CPUS__,
 };

 static inline int __attribute__((always_inline))
 func(void *ctx, int type)
 {
	char output_str[] = "Raise a BPF event!";
	char err_str[] = "BAD %d\n";
	int err;

        err = perf_event_output(ctx, &channel, get_smp_processor_id(),
			        &output_str, sizeof(output_str));
	if (err)
		trace_printk(err_str, sizeof(err_str), err);
        return 1;
 }
 SEC("func_begin=sys_nanosleep")
 int func_begin(void *ctx) {return func(ctx, 1);}
 SEC("func_end=sys_nanosleep%return")
 int func_end(void *ctx) { return func(ctx, 2);}
 char _license[] SEC("license") = "GPL";
 int _version SEC("version") = LINUX_VERSION_CODE;
 /************************* END ***************************/

Signed-off-by: Wang Nan <wangnan0@huawei.com>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Li Zefan <lizefan@huawei.com>
Cc: pi3orama@163.com
---
 tools/perf/builtin-trace.c | 48 +++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 43 insertions(+), 5 deletions(-)

diff --git a/tools/perf/builtin-trace.c b/tools/perf/builtin-trace.c
index b7129d6..88d5ee7 100644
--- a/tools/perf/builtin-trace.c
+++ b/tools/perf/builtin-trace.c
@@ -2178,6 +2178,37 @@ out_dump:
 	return 0;
 }
 
+static void bpf_output__printer(enum binary_printer_ops op,
+				unsigned int val, void *extra)
+{
+	FILE *output = extra;
+	unsigned char ch = (unsigned char)val;
+
+	switch (op) {
+	case BINARY_PRINT_CHAR_DATA:
+		fprintf(output, "%c", isprint(ch) ? ch : '.');
+		break;
+	case BINARY_PRINT_DATA_BEGIN:
+	case BINARY_PRINT_LINE_BEGIN:
+	case BINARY_PRINT_ADDR:
+	case BINARY_PRINT_NUM_DATA:
+	case BINARY_PRINT_NUM_PAD:
+	case BINARY_PRINT_SEP:
+	case BINARY_PRINT_CHAR_PAD:
+	case BINARY_PRINT_LINE_END:
+	case BINARY_PRINT_DATA_END:
+	default:
+		break;
+	}
+}
+
+static void bpf_output__fprintf(struct trace *trace,
+				struct perf_sample *sample)
+{
+	print_binary(sample->raw_data, sample->raw_size, 8,
+		     bpf_output__printer, trace->output);
+}
+
 static int trace__event_handler(struct trace *trace, struct perf_evsel *evsel,
 				union perf_event *event __maybe_unused,
 				struct perf_sample *sample)
@@ -2190,7 +2221,9 @@ static int trace__event_handler(struct trace *trace, struct perf_evsel *evsel,
 
 	fprintf(trace->output, "%s:", evsel->name);
 
-	if (evsel->tp_format) {
+	if (perf_evsel__is_bpf_output(evsel)) {
+		bpf_output__fprintf(trace, sample);
+	} else if (evsel->tp_format) {
 		event_format__fprintf(evsel->tp_format, sample->cpu,
 				      sample->raw_data, sample->raw_size,
 				      trace->output);
@@ -2526,11 +2559,15 @@ out_enomem:
 	goto out;
 }
 
-static int validate_evlist(struct perf_evlist *evlist)
+static int validate_evlist(struct perf_evlist *evlist, bool *has_bpf_output)
 {
 	struct perf_evsel *evsel;
 
 	evlist__for_each(evlist, evsel) {
+		if (perf_evsel__is_bpf_output(evsel)) {
+			*has_bpf_output = true;
+			continue;
+		}
 		if (evsel->attr.type != PERF_TYPE_TRACEPOINT)
 			return -EINVAL;
 	}
@@ -3118,6 +3155,7 @@ int cmd_trace(int argc, const char **argv, const char *prefix __maybe_unused)
 	const char * const trace_subcommands[] = { "record", NULL };
 	int err;
 	char bf[BUFSIZ];
+	bool has_bpf_output = false;
 
 	signal(SIGSEGV, sighandler_dump_stack);
 	signal(SIGFPE, sighandler_dump_stack);
@@ -3135,12 +3173,12 @@ int cmd_trace(int argc, const char **argv, const char *prefix __maybe_unused)
 	argc = parse_options_subcommand(argc, argv, trace_options, trace_subcommands,
 				 trace_usage, PARSE_OPT_STOP_AT_NON_OPTION);
 
-	if (validate_evlist(trace.evlist)) {
-		pr_err("Only support tracepoint events!\n");
+	if (validate_evlist(trace.evlist, &has_bpf_output)) {
+		pr_err("Only support tracepoint and bpf-output events!\n");
 		return -EINVAL;
 	}
 
-	if (trace.trace_pgfaults) {
+	if (trace.trace_pgfaults || has_bpf_output) {
 		trace.opts.sample_address = true;
 		trace.opts.sample_time = true;
 	}
-- 
1.8.3.4

Re: [RFC PATCH 1/5] perf trace: Apply config options in '.perfconfig'

[Arnaldo Carvalho de Melo]

Em Thu, Feb 25, 2016 at 08:51:51AM +0000, Wang Nan escreveu:
> 'perf trace' doesn't read and apply options in ~/.perfconfig. If a BPF
> script is passed to 'perf trace --ev' and clang is not in $PATH,
> perf trace doesn't compile the script even clang-path is configured in
> ~/.perfconfig.
> 
> This patch makes 'perf trace' read and apply default config options
> from that file.

Right, I wonder if we can't move this initialization to perf's main
instead of having to call this for each subcommand...
 
> Signed-off-by: Wang Nan <wangnan0@huawei.com>
> Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
> Cc: Jiri Olsa <jolsa@kernel.org>
> Cc: Li Zefan <lizefan@huawei.com>
> Cc: pi3orama@163.com
> ---
>  tools/perf/builtin-trace.c | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/tools/perf/builtin-trace.c b/tools/perf/builtin-trace.c
> index 20916dd..6c52f3c 100644
> --- a/tools/perf/builtin-trace.c
> +++ b/tools/perf/builtin-trace.c
> @@ -3108,6 +3108,8 @@ int cmd_trace(int argc, const char **argv, const char *prefix __maybe_unused)
>  		goto out;
>  	}
>  
> +	perf_config(perf_default_config, NULL);
> +
>  	argc = parse_options_subcommand(argc, argv, trace_options, trace_subcommands,
>  				 trace_usage, PARSE_OPT_STOP_AT_NON_OPTION);
>  
> -- 
> 1.8.3.4