[PATCH v3 0/3] perf script: Add general event support to event handler of python script

[PATCH v3 0/3] perf script: Add general event support to event handler of python script

[Feng Tang]

Current python script can only handle trace point type of events, this
patch serie try to follow Robert Richter's idea in commit 37a058ea0
	"perf script: Add generic perl handler to process events"
to similarly add a python handler for general events other than
tracepoints.

Please help to revew, thanks.

- Feng


Change logs:
	Since v2:
	* Correct some __unused usage
	* Add a new parameter: event name to event handler 

	Since v1:
	* Use dictory as the paramter passwd from c to python 
	* Use "process_event" name to comply with perl

----------------
Feng Tang (3):
  perf script: Add general python handler to process non-tracepoint
    events
  perf script: Replace "struct thread" with "struct addr_location" as a
    parameter for "process_event()"
  perf script/python: Pass event/thread/dso name and symbol info to
    event handler in python

 tools/perf/builtin-script.c                        |    5 +-
 .../perf/util/scripting-engines/trace-event-perl.c |   13 ++--
 .../util/scripting-engines/trace-event-python.c    |   81 +++++++++++++++++++-
 tools/perf/util/trace-event-scripting.c            |    2 +-
 tools/perf/util/trace-event.h                      |    4 +-
 5 files changed, 93 insertions(+), 12 deletions(-)

[PATCH v3 1/3] perf script: Add general python handler to process non-tracepoint events

[Feng Tang]

This patch just follows Robert Richter's idea and the commit 37a058ea0
	"perf script: Add generic perl handler to process events"
to similarly add a python handler for general events other than tracepoints.

For non-tracepoint events, this patch will try to find a function named
"process_event" in the python script, and pass the event attribute,
perf_sample, raw_data in format of raw string. And the python script can
use "struct" module's unpack function to disasemble the needed info and process.

Signed-off-by: Feng Tang <feng.tang@intel.com>
---
 .../util/scripting-engines/trace-event-python.c    |   60 +++++++++++++++++++-
 1 files changed, 59 insertions(+), 1 deletions(-)

diff --git a/tools/perf/util/scripting-engines/trace-event-python.c b/tools/perf/util/scripting-engines/trace-event-python.c
index acb9795..ab441a4 100644
--- a/tools/perf/util/scripting-engines/trace-event-python.c
+++ b/tools/perf/util/scripting-engines/trace-event-python.c
@@ -31,6 +31,7 @@
 #include "../event.h"
 #include "../thread.h"
 #include "../trace-event.h"
+#include "../evsel.h"
 
 PyMODINIT_FUNC initperf_trace_context(void);
 
@@ -209,7 +210,7 @@ static inline struct event_format *find_cache_event(int type)
 	return event;
 }
 
-static void python_process_event(union perf_event *pevent __unused,
+static void python_process_tracepoint(union perf_event *pevent __unused,
 				 struct perf_sample *sample,
 				 struct perf_evsel *evsel __unused,
 				 struct machine *machine __unused,
@@ -328,6 +329,63 @@ static void python_process_event(union perf_event *pevent __unused,
 	Py_DECREF(t);
 }
 
+static void python_process_general_event(union perf_event *pevent __unused,
+				 struct perf_sample *sample,
+				 struct perf_evsel *evsel,
+				 struct machine *machine __unused,
+				 struct thread *thread __unused)
+{
+	PyObject *handler, *retval, *t;
+	static char handler_name[64];
+	unsigned n = 0;
+	void *data = sample->raw_data;
+
+	t = PyTuple_New(MAX_FIELDS);
+	if (!t)
+		Py_FatalError("couldn't create Python tuple");
+
+	snprintf(handler_name, sizeof(handler_name), "%s", "process_event");
+
+	handler = PyDict_GetItemString(main_dict, handler_name);
+	if (handler && !PyCallable_Check(handler)) {
+		handler = NULL;
+		goto exit;
+	}
+
+	/* Pass 3 parameters: event_attr, perf_sample, raw data */
+	PyTuple_SetItem(t, n++, PyString_FromStringAndSize(
+			(const char *)&evsel->attr, sizeof(evsel->attr)));
+	PyTuple_SetItem(t, n++, PyString_FromStringAndSize(
+			(const char *)sample, sizeof(*sample)));
+	PyTuple_SetItem(t, n++, PyString_FromStringAndSize(
+			data, sample->raw_size));
+
+	if (_PyTuple_Resize(&t, n) == -1)
+		Py_FatalError("error resizing Python tuple");
+
+	retval = PyObject_CallObject(handler, t);
+	if (retval == NULL)
+		handler_call_die(handler_name);
+exit:
+	Py_DECREF(t);
+}
+
+static void python_process_event(union perf_event *pevent,
+				 struct perf_sample *sample,
+				 struct perf_evsel *evsel,
+				 struct machine *machine,
+				 struct thread *thread)
+{
+	switch (evsel->attr.type) {
+	case PERF_TYPE_TRACEPOINT:
+		python_process_tracepoint(pevent, sample, evsel, machine, thread);
+		break;
+	/* Reserve for future process_hw/sw/raw APIs */
+	default:
+		python_process_general_event(pevent, sample, evsel, machine, thread);
+	}
+}
+
 static int run_start_sub(void)
 {
 	PyObject *handler, *retval;
-- 
1.7.1

[PATCH v3 2/3] perf script: Replace "struct thread" with "struct addr_location" as a parameter for "process_event()"

[Feng Tang]

Both perl and python script start processing events other than trace
points, and it's useful to pass the resolved symbol and the dso info
to the event handler in script for better analysis and statistics.

Struct thread is already a member of struct addr_location, using
addr_location will keep the thread info, while providing additional
symbol and dso info if exist, so that the script itself doesn't need
to bother to do the symbol resolving and dso searching work.

Signed-off-by: Feng Tang <feng.tang@intel.com>
Acked-by: David Ahern <dsahern@gmail.com>
Tested-by: David Ahern <dsahern@gmail.com>
---
 tools/perf/builtin-script.c                        |    5 +++--
 .../perf/util/scripting-engines/trace-event-perl.c |   13 +++++++------
 .../util/scripting-engines/trace-event-python.c    |   13 +++++++------
 tools/perf/util/trace-event-scripting.c            |    2 +-
 tools/perf/util/trace-event.h                      |    4 +++-
 5 files changed, 21 insertions(+), 16 deletions(-)

diff --git a/tools/perf/builtin-script.c b/tools/perf/builtin-script.c
index 8e395a5..faddacc 100644
--- a/tools/perf/builtin-script.c
+++ b/tools/perf/builtin-script.c
@@ -405,9 +405,10 @@ static void process_event(union perf_event *event __unused,
 			  struct perf_sample *sample,
 			  struct perf_evsel *evsel,
 			  struct machine *machine,
-			  struct thread *thread)
+			  struct addr_location *al)
 {
 	struct perf_event_attr *attr = &evsel->attr;
+	struct thread *thread = al->thread;
 
 	if (output[attr->type].fields == 0)
 		return;
@@ -520,7 +521,7 @@ static int process_sample_event(struct perf_tool *tool __used,
 	if (cpu_list && !test_bit(sample->cpu, cpu_bitmap))
 		return 0;
 
-	scripting_ops->process_event(event, sample, evsel, machine, thread);
+	scripting_ops->process_event(event, sample, evsel, machine, &al);
 
 	evsel->hists.stats.total_period += sample->period;
 	return 0;
diff --git a/tools/perf/util/scripting-engines/trace-event-perl.c b/tools/perf/util/scripting-engines/trace-event-perl.c
index 4c1b3d7..997ef5d 100644
--- a/tools/perf/util/scripting-engines/trace-event-perl.c
+++ b/tools/perf/util/scripting-engines/trace-event-perl.c
@@ -256,7 +256,7 @@ static void perl_process_tracepoint(union perf_event *pevent __unused,
 				    struct perf_sample *sample,
 				    struct perf_evsel *evsel,
 				    struct machine *machine __unused,
-				    struct thread *thread)
+				    struct addr_location *al)
 {
 	struct format_field *field;
 	static char handler[256];
@@ -268,6 +268,7 @@ static void perl_process_tracepoint(union perf_event *pevent __unused,
 	int cpu = sample->cpu;
 	void *data = sample->raw_data;
 	unsigned long long nsecs = sample->time;
+	struct thread *thread = al->thread;
 	char *comm = thread->comm;
 
 	dSP;
@@ -344,9 +345,9 @@ static void perl_process_tracepoint(union perf_event *pevent __unused,
 
 static void perl_process_event_generic(union perf_event *pevent __unused,
 				       struct perf_sample *sample,
-				       struct perf_evsel *evsel __unused,
+				       struct perf_evsel *evsel,
 				       struct machine *machine __unused,
-				       struct thread *thread __unused)
+				       struct addr_location *al __unused)
 {
 	dSP;
 
@@ -372,10 +373,10 @@ static void perl_process_event(union perf_event *pevent,
 			       struct perf_sample *sample,
 			       struct perf_evsel *evsel,
 			       struct machine *machine,
-			       struct thread *thread)
+			       struct addr_location *al)
 {
-	perl_process_tracepoint(pevent, sample, evsel, machine, thread);
-	perl_process_event_generic(pevent, sample, evsel, machine, thread);
+	perl_process_tracepoint(pevent, sample, evsel, machine, al);
+	perl_process_event_generic(pevent, sample, evsel, machine, al);
 }
 
 static void run_start_sub(void)
diff --git a/tools/perf/util/scripting-engines/trace-event-python.c b/tools/perf/util/scripting-engines/trace-event-python.c
index ab441a4..97f4fad 100644
--- a/tools/perf/util/scripting-engines/trace-event-python.c
+++ b/tools/perf/util/scripting-engines/trace-event-python.c
@@ -214,7 +214,7 @@ static void python_process_tracepoint(union perf_event *pevent __unused,
 				 struct perf_sample *sample,
 				 struct perf_evsel *evsel __unused,
 				 struct machine *machine __unused,
-				 struct thread *thread)
+				 struct addr_location *al)
 {
 	PyObject *handler, *retval, *context, *t, *obj, *dict = NULL;
 	static char handler_name[256];
@@ -228,6 +228,7 @@ static void python_process_tracepoint(union perf_event *pevent __unused,
 	int cpu = sample->cpu;
 	void *data = sample->raw_data;
 	unsigned long long nsecs = sample->time;
+	struct thread *thread = al->thread;
 	char *comm = thread->comm;
 
 	t = PyTuple_New(MAX_FIELDS);
@@ -333,7 +334,7 @@ static void python_process_general_event(union perf_event *pevent __unused,
 				 struct perf_sample *sample,
 				 struct perf_evsel *evsel,
 				 struct machine *machine __unused,
-				 struct thread *thread __unused)
+				 struct addr_location *al)
 {
 	PyObject *handler, *retval, *t;
 	static char handler_name[64];
@@ -352,7 +353,7 @@ static void python_process_general_event(union perf_event *pevent __unused,
 		goto exit;
 	}
 
-	/* Pass 3 parameters: event_attr, perf_sample, raw data */
+	/* Pass 4 parameters: event_attr, perf_sample, raw data, thread name */
 	PyTuple_SetItem(t, n++, PyString_FromStringAndSize(
 			(const char *)&evsel->attr, sizeof(evsel->attr)));
 	PyTuple_SetItem(t, n++, PyString_FromStringAndSize(
@@ -374,15 +375,15 @@ static void python_process_event(union perf_event *pevent,
 				 struct perf_sample *sample,
 				 struct perf_evsel *evsel,
 				 struct machine *machine,
-				 struct thread *thread)
+				 struct addr_location *al)
 {
 	switch (evsel->attr.type) {
 	case PERF_TYPE_TRACEPOINT:
-		python_process_tracepoint(pevent, sample, evsel, machine, thread);
+		python_process_tracepoint(pevent, sample, evsel, machine, al);
 		break;
 	/* Reserve for future process_hw/sw/raw APIs */
 	default:
-		python_process_general_event(pevent, sample, evsel, machine, thread);
+		python_process_general_event(pevent, sample, evsel, machine, al);
 	}
 }
 
diff --git a/tools/perf/util/trace-event-scripting.c b/tools/perf/util/trace-event-scripting.c
index 18ae6c1..b26459e 100644
--- a/tools/perf/util/trace-event-scripting.c
+++ b/tools/perf/util/trace-event-scripting.c
@@ -39,7 +39,7 @@ static void process_event_unsupported(union perf_event *event __unused,
 				      struct perf_sample *sample __unused,
 				      struct perf_evsel *evsel __unused,
 				      struct machine *machine __unused,
-				      struct thread *thread __unused)
+				      struct addr_location *al __unused)
 {
 }
 
diff --git a/tools/perf/util/trace-event.h b/tools/perf/util/trace-event.h
index 639852a..7b49397 100644
--- a/tools/perf/util/trace-event.h
+++ b/tools/perf/util/trace-event.h
@@ -72,6 +72,8 @@ struct tracing_data *tracing_data_get(struct list_head *pattrs,
 void tracing_data_put(struct tracing_data *tdata);
 
 
+struct addr_location;
+
 struct scripting_ops {
 	const char *name;
 	int (*start_script) (const char *script, int argc, const char **argv);
@@ -80,7 +82,7 @@ struct scripting_ops {
 			       struct perf_sample *sample,
 			       struct perf_evsel *evsel,
 			       struct machine *machine,
-			       struct thread *thread);
+			       struct addr_location *al);
 	int (*generate_script) (const char *outfile);
 };
 
-- 
1.7.1

[PATCH v3 3/3] perf script/python: Pass event/thread/dso name and symbol info to event handler in python

[Feng Tang]

Also as suggested by Arnaldo, pack all these parameters to a dictionary,
which is more expandable for adding new parameters while keep the
compatibility for old scripts.

Signed-off-by: Feng Tang <feng.tang@intel.com>
---
 .../util/scripting-engines/trace-event-python.c    |   32 +++++++++++++++----
 1 files changed, 25 insertions(+), 7 deletions(-)

diff --git a/tools/perf/util/scripting-engines/trace-event-python.c b/tools/perf/util/scripting-engines/trace-event-python.c
index 97f4fad..e8ac480 100644
--- a/tools/perf/util/scripting-engines/trace-event-python.c
+++ b/tools/perf/util/scripting-engines/trace-event-python.c
@@ -336,15 +336,23 @@ static void python_process_general_event(union perf_event *pevent __unused,
 				 struct machine *machine __unused,
 				 struct addr_location *al)
 {
-	PyObject *handler, *retval, *t;
+	PyObject *handler, *retval, *t, *dict;
 	static char handler_name[64];
 	unsigned n = 0;
-	void *data = sample->raw_data;
+	struct thread *thread = al->thread;
 
+	/*
+	 * Use the MAX_FIELDS to make the function expandable, though
+	 * currently there is only one itme for the tuple.
+	 */
 	t = PyTuple_New(MAX_FIELDS);
 	if (!t)
 		Py_FatalError("couldn't create Python tuple");
 
+	dict = PyDict_New();
+	if (!dict)
+		Py_FatalError("couldn't create Python dictionary");
+
 	snprintf(handler_name, sizeof(handler_name), "%s", "process_event");
 
 	handler = PyDict_GetItemString(main_dict, handler_name);
@@ -353,14 +361,23 @@ static void python_process_general_event(union perf_event *pevent __unused,
 		goto exit;
 	}
 
-	/* Pass 4 parameters: event_attr, perf_sample, raw data, thread name */
-	PyTuple_SetItem(t, n++, PyString_FromStringAndSize(
+	PyDict_SetItemString(dict, "ev_name", PyString_FromString(event_name(evsel)));
+	PyDict_SetItemString(dict, "attr", PyString_FromStringAndSize(
 			(const char *)&evsel->attr, sizeof(evsel->attr)));
-	PyTuple_SetItem(t, n++, PyString_FromStringAndSize(
+	PyDict_SetItemString(dict, "sample", PyString_FromStringAndSize(
 			(const char *)sample, sizeof(*sample)));
-	PyTuple_SetItem(t, n++, PyString_FromStringAndSize(
-			data, sample->raw_size));
+	PyDict_SetItemString(dict, "raw_buf", PyString_FromStringAndSize(
+			(const char *)sample->raw_data, sample->raw_size));
+	PyDict_SetItemString(dict, "comm",
+			PyString_FromString(thread->comm));
+	PyDict_SetItemString(dict, "dso",
+			PyString_FromString(al->map->dso->name));
+	if (al->sym) {
+		PyDict_SetItemString(dict, "symbol",
+			PyString_FromString(al->sym->name));
+	}
 
+	PyTuple_SetItem(t, n++, dict);
 	if (_PyTuple_Resize(&t, n) == -1)
 		Py_FatalError("error resizing Python tuple");
 
@@ -368,6 +385,7 @@ static void python_process_general_event(union perf_event *pevent __unused,
 	if (retval == NULL)
 		handler_call_die(handler_name);
 exit:
+	Py_DECREF(dict);
 	Py_DECREF(t);
 }
 
-- 
1.7.1

Re: [PATCH v3 3/3] perf script/python: Pass event/thread/dso name and symbol info to event handler in python

[Arnaldo Carvalho de Melo]

[-- Attachment #1: Type: text/plain, Size: 3284 bytes --]

Em Mon, Jun 18, 2012 at 02:10:39PM +0800, Feng Tang escreveu:
> Also as suggested by Arnaldo, pack all these parameters to a dictionary,
> which is more expandable for adding new parameters while keep the
> compatibility for old scripts.
> 
> Signed-off-by: Feng Tang <feng.tang@intel.com>
> ---
>  .../util/scripting-engines/trace-event-python.c    |   32 +++++++++++++++----
>  1 files changed, 25 insertions(+), 7 deletions(-)
> 
> diff --git a/tools/perf/util/scripting-engines/trace-event-python.c b/tools/perf/util/scripting-engines/trace-event-python.c
> index 97f4fad..e8ac480 100644
> --- a/tools/perf/util/scripting-engines/trace-event-python.c
> +++ b/tools/perf/util/scripting-engines/trace-event-python.c
> @@ -336,15 +336,23 @@ static void python_process_general_event(union perf_event *pevent __unused,
>  				 struct machine *machine __unused,
>  				 struct addr_location *al)
>  {
> -	PyObject *handler, *retval, *t;
> +	PyObject *handler, *retval, *t, *dict;
>  	static char handler_name[64];
>  	unsigned n = 0;
> -	void *data = sample->raw_data;
> +	struct thread *thread = al->thread;
>  
> +	/*
> +	 * Use the MAX_FIELDS to make the function expandable, though
> +	 * currently there is only one itme for the tuple.
> +	 */
>  	t = PyTuple_New(MAX_FIELDS);
>  	if (!t)
>  		Py_FatalError("couldn't create Python tuple");
>  
> +	dict = PyDict_New();
> +	if (!dict)
> +		Py_FatalError("couldn't create Python dictionary");
> +
>  	snprintf(handler_name, sizeof(handler_name), "%s", "process_event");
>  
>  	handler = PyDict_GetItemString(main_dict, handler_name);
> @@ -353,14 +361,23 @@ static void python_process_general_event(union perf_event *pevent __unused,
>  		goto exit;
>  	}
>  
> -	/* Pass 4 parameters: event_attr, perf_sample, raw data, thread name */
> -	PyTuple_SetItem(t, n++, PyString_FromStringAndSize(
> +	PyDict_SetItemString(dict, "ev_name", PyString_FromString(event_name(evsel)));
> +	PyDict_SetItemString(dict, "attr", PyString_FromStringAndSize(
>  			(const char *)&evsel->attr, sizeof(evsel->attr)));
> -	PyTuple_SetItem(t, n++, PyString_FromStringAndSize(
> +	PyDict_SetItemString(dict, "sample", PyString_FromStringAndSize(
>  			(const char *)sample, sizeof(*sample)));
> -	PyTuple_SetItem(t, n++, PyString_FromStringAndSize(
> -			data, sample->raw_size));
> +	PyDict_SetItemString(dict, "raw_buf", PyString_FromStringAndSize(
> +			(const char *)sample->raw_data, sample->raw_size));
> +	PyDict_SetItemString(dict, "comm",
> +			PyString_FromString(thread->comm));
> +	PyDict_SetItemString(dict, "dso",
> +			PyString_FromString(al->map->dso->name));
> +	if (al->sym) {
> +		PyDict_SetItemString(dict, "symbol",
> +			PyString_FromString(al->sym->name));
> +	}
>  
> +	PyTuple_SetItem(t, n++, dict);

Now old scrips will break, as the tuple they expect:

 (attr, sample, raw_data)

will not be there.

To not break those scripts you must leave it there and add the new ones
in a python compatible way. I.e. have you tested the existing scripts
using the new perf tool with your patches applied? They must continue to
work.

I.e. leave the old tuple and add the dict at the end somehow.

The first two patches in this series don't apply anymore, please find
them attached fixed, please check if they work as expected.

- Arnaldo

[-- Attachment #2: 0001-perf-script-Add-general-python-handler-to-process-no.patch --]
[-- Type: text/plain, Size: 4546 bytes --]

>From 429bf5a8d5b27b444cbdfaea088d883d8921fb48 Mon Sep 17 00:00:00 2001
From: Feng Tang <feng.tang@intel.com>
Date: Wed, 27 Jun 2012 15:07:02 -0300
Subject: [PATCH 1/2] perf script: Add general python handler to process non-tracepoint events
Content-Type: text/plain; charset="UTF-8"

This patch just follows Robert Richter's idea and the commit 37a058ea0
	"perf script: Add generic perl handler to process events"
to similarly add a python handler for general events other than tracepoints.

For non-tracepoint events, this patch will try to find a function named
"process_event" in the python script, and pass the event attribute,
perf_sample, raw_data in format of raw string. And the python script can
use "struct" module's unpack function to disasemble the needed info and process.

Signed-off-by: Feng Tang <feng.tang@intel.com>
Cc: Andi Kleen <andi@firstfloor.org>
Cc: David Ahern <dsahern@gmail.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Robert Richter <robert.richter@amd.com>
Cc: Stephane Eranian <eranian@google.com>
http://lkml.kernel.org/r/1339999839-14007-2-git-send-email-feng.tang@intel.com
[ committer note: Fixed up wrt da37896, i.e. pevent parm in script event handlers ]
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 .../util/scripting-engines/trace-event-python.c    |   71 ++++++++++++++++++--
 1 files changed, 65 insertions(+), 6 deletions(-)

diff --git a/tools/perf/util/scripting-engines/trace-event-python.c b/tools/perf/util/scripting-engines/trace-event-python.c
index a8ca2f8..3fae022 100644
--- a/tools/perf/util/scripting-engines/trace-event-python.c
+++ b/tools/perf/util/scripting-engines/trace-event-python.c
@@ -31,6 +31,7 @@
 #include "../event.h"
 #include "../thread.h"
 #include "../trace-event.h"
+#include "../evsel.h"
 
 PyMODINIT_FUNC initperf_trace_context(void);
 
@@ -210,12 +211,12 @@ struct event_format *find_cache_event(struct pevent *pevent, int type)
 	return event;
 }
 
-static void python_process_event(union perf_event *perf_event __unused,
-				 struct pevent *pevent,
-				 struct perf_sample *sample,
-				 struct perf_evsel *evsel __unused,
-				 struct machine *machine __unused,
-				 struct thread *thread)
+static void python_process_tracepoint(union perf_event *perf_event __unused,
+				      struct pevent *pevent,
+				      struct perf_sample *sample,
+				      struct perf_evsel *evsel __unused,
+				      struct machine *machine __unused,
+				      struct thread *thread)
 {
 	PyObject *handler, *retval, *context, *t, *obj, *dict = NULL;
 	static char handler_name[256];
@@ -331,6 +332,64 @@ static void python_process_event(union perf_event *perf_event __unused,
 	Py_DECREF(t);
 }
 
+static void python_process_general_event(union perf_event *perf_event __unused,
+					 struct pevent *pevent,
+					 struct perf_sample *sample,
+					 struct perf_evsel *evsel,
+					 struct machine *machine __unused,
+					 struct thread *thread __unused)
+{
+	PyObject *handler, *retval, *t;
+	static char handler_name[64];
+	unsigned n = 0;
+	void *data = sample->raw_data;
+
+	t = PyTuple_New(MAX_FIELDS);
+	if (!t)
+		Py_FatalError("couldn't create Python tuple");
+
+	snprintf(handler_name, sizeof(handler_name), "%s", "process_event");
+
+	handler = PyDict_GetItemString(main_dict, handler_name);
+	if (handler && !PyCallable_Check(handler)) {
+		handler = NULL;
+		goto exit;
+	}
+
+	/* Pass 3 parameters: event_attr, perf_sample, raw data */
+	PyTuple_SetItem(t, n++, PyString_FromStringAndSize((void *)&evsel->attr, sizeof(evsel->attr)));
+	PyTuple_SetItem(t, n++, PyString_FromStringAndSize((void *)sample, sizeof(*sample)));
+	PyTuple_SetItem(t, n++, PyString_FromStringAndSize(data, sample->raw_size));
+
+	if (_PyTuple_Resize(&t, n) == -1)
+		Py_FatalError("error resizing Python tuple");
+
+	retval = PyObject_CallObject(handler, t);
+	if (retval == NULL)
+		handler_call_die(handler_name);
+exit:
+	Py_DECREF(t);
+}
+
+static void python_process_event(union perf_event *perf_event,
+				 struct pevent *pevent,
+				 struct perf_sample *sample,
+				 struct perf_evsel *evsel,
+				 struct machine *machine,
+				 struct thread *thread)
+{
+	switch (evsel->attr.type) {
+	case PERF_TYPE_TRACEPOINT:
+		python_process_tracepoint(perf_event, pevent, sample, evsel,
+					  machine, thread);
+		break;
+	/* Reserve for future process_hw/sw/raw APIs */
+	default:
+		python_process_general_event(perf_event, pevent, sample, evsel,
+					     machine, thread);
+	}
+}
+
 static int run_start_sub(void)
 {
 	PyObject *handler, *retval;
-- 
1.7.1


[-- Attachment #3: 0002-perf-script-Replace-struct-thread-with-struct-addr_l.patch --]
[-- Type: text/plain, Size: 8377 bytes --]

>From 16c5860d9f34236920ebb71b346417c638aa7198 Mon Sep 17 00:00:00 2001
From: Feng Tang <feng.tang@intel.com>
Date: Wed, 27 Jun 2012 15:24:16 -0300
Subject: [PATCH 2/2] perf script: Replace "struct thread" with "struct addr_location" as a parameter for "process_event()"
Content-Type: text/plain; charset="UTF-8"

Both perl and python script start processing events other than trace
points, and it's useful to pass the resolved symbol and the dso info to
the event handler in script for better analysis and statistics.

Struct thread is already a member of struct addr_location, using
addr_location will keep the thread info, while providing additional
symbol and dso info if exist, so that the script itself doesn't need to
bother to do the symbol resolving and dso searching work.

Signed-off-by: Feng Tang <feng.tang@intel.com>
Acked-by: David Ahern <dsahern@gmail.com>
Tested-by: David Ahern <dsahern@gmail.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Robert Richter <robert.richter@amd.com>
Cc: Andi Kleen <andi@firstfloor.org>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/r/1339999839-14007-3-git-send-email-feng.tang@intel.com>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/builtin-script.c                        |    5 +++--
 .../perf/util/scripting-engines/trace-event-perl.c |   13 +++++++------
 .../util/scripting-engines/trace-event-python.c    |   13 +++++++------
 tools/perf/util/trace-event-scripting.c            |    2 +-
 tools/perf/util/trace-event.h                      |    5 +++--
 5 files changed, 21 insertions(+), 17 deletions(-)

diff --git a/tools/perf/builtin-script.c b/tools/perf/builtin-script.c
index 1e60ab7..e588090 100644
--- a/tools/perf/builtin-script.c
+++ b/tools/perf/builtin-script.c
@@ -420,9 +420,10 @@ static void process_event(union perf_event *event __unused,
 			  struct perf_sample *sample,
 			  struct perf_evsel *evsel,
 			  struct machine *machine,
-			  struct thread *thread)
+			  struct addr_location *al)
 {
 	struct perf_event_attr *attr = &evsel->attr;
+	struct thread *thread = al->thread;
 
 	if (output[attr->type].fields == 0)
 		return;
@@ -538,7 +539,7 @@ static int process_sample_event(struct perf_tool *tool __used,
 		return 0;
 
 	scripting_ops->process_event(event, scr->session->pevent,
-				     sample, evsel, machine, thread);
+				     sample, evsel, machine, &al);
 
 	evsel->hists.stats.total_period += sample->period;
 	return 0;
diff --git a/tools/perf/util/scripting-engines/trace-event-perl.c b/tools/perf/util/scripting-engines/trace-event-perl.c
index b3620fe..3f64cd0 100644
--- a/tools/perf/util/scripting-engines/trace-event-perl.c
+++ b/tools/perf/util/scripting-engines/trace-event-perl.c
@@ -258,7 +258,7 @@ static void perl_process_tracepoint(union perf_event *perf_event __unused,
 				    struct perf_sample *sample,
 				    struct perf_evsel *evsel,
 				    struct machine *machine __unused,
-				    struct thread *thread)
+				    struct addr_location *al)
 {
 	struct format_field *field;
 	static char handler[256];
@@ -270,6 +270,7 @@ static void perl_process_tracepoint(union perf_event *perf_event __unused,
 	int cpu = sample->cpu;
 	void *data = sample->raw_data;
 	unsigned long long nsecs = sample->time;
+	struct thread *thread = al->thread;
 	char *comm = thread->comm;
 
 	dSP;
@@ -347,9 +348,9 @@ static void perl_process_tracepoint(union perf_event *perf_event __unused,
 
 static void perl_process_event_generic(union perf_event *pevent __unused,
 				       struct perf_sample *sample,
-				       struct perf_evsel *evsel __unused,
+				       struct perf_evsel *evsel,
 				       struct machine *machine __unused,
-				       struct thread *thread __unused)
+				       struct addr_location *al __unused)
 {
 	dSP;
 
@@ -376,10 +377,10 @@ static void perl_process_event(union perf_event *event,
 			       struct perf_sample *sample,
 			       struct perf_evsel *evsel,
 			       struct machine *machine,
-			       struct thread *thread)
+			       struct addr_location *al)
 {
-	perl_process_tracepoint(event, pevent, sample, evsel, machine, thread);
-	perl_process_event_generic(event, sample, evsel, machine, thread);
+	perl_process_tracepoint(event, pevent, sample, evsel, machine, al);
+	perl_process_event_generic(event, sample, evsel, machine, al);
 }
 
 static void run_start_sub(void)
diff --git a/tools/perf/util/scripting-engines/trace-event-python.c b/tools/perf/util/scripting-engines/trace-event-python.c
index 3fae022..0be4ec0 100644
--- a/tools/perf/util/scripting-engines/trace-event-python.c
+++ b/tools/perf/util/scripting-engines/trace-event-python.c
@@ -216,7 +216,7 @@ static void python_process_tracepoint(union perf_event *perf_event __unused,
 				      struct perf_sample *sample,
 				      struct perf_evsel *evsel __unused,
 				      struct machine *machine __unused,
-				      struct thread *thread)
+				      struct addr_location *al)
 {
 	PyObject *handler, *retval, *context, *t, *obj, *dict = NULL;
 	static char handler_name[256];
@@ -230,6 +230,7 @@ static void python_process_tracepoint(union perf_event *perf_event __unused,
 	int cpu = sample->cpu;
 	void *data = sample->raw_data;
 	unsigned long long nsecs = sample->time;
+	struct thread *thread = al->thread;
 	char *comm = thread->comm;
 
 	t = PyTuple_New(MAX_FIELDS);
@@ -337,7 +338,7 @@ static void python_process_general_event(union perf_event *perf_event __unused,
 					 struct perf_sample *sample,
 					 struct perf_evsel *evsel,
 					 struct machine *machine __unused,
-					 struct thread *thread __unused)
+					 struct addr_location *al __unused)
 {
 	PyObject *handler, *retval, *t;
 	static char handler_name[64];
@@ -356,7 +357,7 @@ static void python_process_general_event(union perf_event *perf_event __unused,
 		goto exit;
 	}
 
-	/* Pass 3 parameters: event_attr, perf_sample, raw data */
+	/* Pass 4 parameters: event_attr, perf_sample, raw data, thread name */
 	PyTuple_SetItem(t, n++, PyString_FromStringAndSize((void *)&evsel->attr, sizeof(evsel->attr)));
 	PyTuple_SetItem(t, n++, PyString_FromStringAndSize((void *)sample, sizeof(*sample)));
 	PyTuple_SetItem(t, n++, PyString_FromStringAndSize(data, sample->raw_size));
@@ -376,17 +377,17 @@ static void python_process_event(union perf_event *perf_event,
 				 struct perf_sample *sample,
 				 struct perf_evsel *evsel,
 				 struct machine *machine,
-				 struct thread *thread)
+				 struct addr_location *al)
 {
 	switch (evsel->attr.type) {
 	case PERF_TYPE_TRACEPOINT:
 		python_process_tracepoint(perf_event, pevent, sample, evsel,
-					  machine, thread);
+					  machine, al);
 		break;
 	/* Reserve for future process_hw/sw/raw APIs */
 	default:
 		python_process_general_event(perf_event, pevent, sample, evsel,
-					     machine, thread);
+					     machine, al);
 	}
 }
 
diff --git a/tools/perf/util/trace-event-scripting.c b/tools/perf/util/trace-event-scripting.c
index 474aa7a..c8dc56c 100644
--- a/tools/perf/util/trace-event-scripting.c
+++ b/tools/perf/util/trace-event-scripting.c
@@ -40,7 +40,7 @@ static void process_event_unsupported(union perf_event *event __unused,
 				      struct perf_sample *sample __unused,
 				      struct perf_evsel *evsel __unused,
 				      struct machine *machine __unused,
-				      struct thread *thread __unused)
+				      struct addr_location *al __unused)
 {
 }
 
diff --git a/tools/perf/util/trace-event.h b/tools/perf/util/trace-event.h
index 8fef1d6..8ec1de7 100644
--- a/tools/perf/util/trace-event.h
+++ b/tools/perf/util/trace-event.h
@@ -9,7 +9,6 @@ struct machine;
 struct perf_sample;
 union perf_event;
 struct perf_tool;
-struct thread;
 
 extern int header_page_size_size;
 extern int header_page_ts_size;
@@ -74,6 +73,8 @@ struct tracing_data *tracing_data_get(struct list_head *pattrs,
 void tracing_data_put(struct tracing_data *tdata);
 
 
+struct addr_location;
+
 struct scripting_ops {
 	const char *name;
 	int (*start_script) (const char *script, int argc, const char **argv);
@@ -83,7 +84,7 @@ struct scripting_ops {
 			       struct perf_sample *sample,
 			       struct perf_evsel *evsel,
 			       struct machine *machine,
-			       struct thread *thread);
+			       struct addr_location *al);
 	int (*generate_script) (struct pevent *pevent, const char *outfile);
 };
 
-- 
1.7.1

Re: [PATCH v3 3/3] perf script/python: Pass event/thread/dso name and symbol info to event handler in python

[Feng Tang]

Hi Arnaldo,

On Wed, 27 Jun 2012 15:58:17 -0300
Arnaldo Carvalho de Melo <acme@redhat.com> wrote:

> Em Mon, Jun 18, 2012 at 02:10:39PM +0800, Feng Tang escreveu:
> > Also as suggested by Arnaldo, pack all these parameters to a dictionary,
> > which is more expandable for adding new parameters while keep the
> > compatibility for old scripts.
> > 
> > Signed-off-by: Feng Tang <feng.tang@intel.com>
> > ---
> >  .../util/scripting-engines/trace-event-python.c    |   32
> > +++++++++++++++---- 1 files changed, 25 insertions(+), 7 deletions(-)
> > 
> > diff --git a/tools/perf/util/scripting-engines/trace-event-python.c
> > b/tools/perf/util/scripting-engines/trace-event-python.c index
> > 97f4fad..e8ac480 100644 ---
> > a/tools/perf/util/scripting-engines/trace-event-python.c +++
> > b/tools/perf/util/scripting-engines/trace-event-python.c @@ -336,15 +336,23
> > @@ static void python_process_general_event(union perf_event *pevent
> > __unused, struct machine *machine __unused, struct addr_location *al)
> >  {
> > -	PyObject *handler, *retval, *t;
> > +	PyObject *handler, *retval, *t, *dict;
> >  	static char handler_name[64];
> >  	unsigned n = 0;
> > -	void *data = sample->raw_data;
> > +	struct thread *thread = al->thread;
> >  
> > +	/*
> > +	 * Use the MAX_FIELDS to make the function expandable, though
> > +	 * currently there is only one itme for the tuple.
> > +	 */
> >  	t = PyTuple_New(MAX_FIELDS);
> >  	if (!t)
> >  		Py_FatalError("couldn't create Python tuple");
> >  
> > +	dict = PyDict_New();
> > +	if (!dict)
> > +		Py_FatalError("couldn't create Python dictionary");
> > +
> >  	snprintf(handler_name, sizeof(handler_name), "%s",
> > "process_event"); 
> >  	handler = PyDict_GetItemString(main_dict, handler_name);
> > @@ -353,14 +361,23 @@ static void python_process_general_event(union
> > perf_event *pevent __unused, goto exit;
> >  	}
> >  
> > -	/* Pass 4 parameters: event_attr, perf_sample, raw data, thread
> > name */
> > -	PyTuple_SetItem(t, n++, PyString_FromStringAndSize(
> > +	PyDict_SetItemString(dict, "ev_name",
> > PyString_FromString(event_name(evsel)));
> > +	PyDict_SetItemString(dict, "attr", PyString_FromStringAndSize(
> >  			(const char *)&evsel->attr, sizeof(evsel->attr)));
> > -	PyTuple_SetItem(t, n++, PyString_FromStringAndSize(
> > +	PyDict_SetItemString(dict, "sample", PyString_FromStringAndSize(
> >  			(const char *)sample, sizeof(*sample)));
> > -	PyTuple_SetItem(t, n++, PyString_FromStringAndSize(
> > -			data, sample->raw_size));
> > +	PyDict_SetItemString(dict, "raw_buf", PyString_FromStringAndSize(
> > +			(const char *)sample->raw_data, sample->raw_size));
> > +	PyDict_SetItemString(dict, "comm",
> > +			PyString_FromString(thread->comm));
> > +	PyDict_SetItemString(dict, "dso",
> > +			PyString_FromString(al->map->dso->name));
> > +	if (al->sym) {
> > +		PyDict_SetItemString(dict, "symbol",
> > +			PyString_FromString(al->sym->name));
> > +	}
> >  
> > +	PyTuple_SetItem(t, n++, dict);
> 
> Now old scrips will break, as the tuple they expect:
> 
>  (attr, sample, raw_data)
> 
> will not be there.

Actually, there are no such scripts out. Current perf python script only
supports the trace point type, while perl scriptt has the support for
general events. It is my first patch "perf script: Add general python 
handler to process non-tracepoint events" which bring this (attr, sample,
raw_data) in to be on the same page as perl. So this won't be a problem
when the 3 patches are merged together. 

> 
> The first two patches in this series don't apply anymore, please find
> them attached fixed, please check if they work as expected.
Thanks a lot for fixing them, I'll generate perf tool patch against the
"perf/core" branch of your git tree on kernel.org from now on.

Btw, I have 2 more python scripts based on these patches, will add them
to this serie for your review.

Thanks,
Feng

Re: [PATCH v3 3/3] perf script/python: Pass event/thread/dso name and symbol info to event handler in python

[Arnaldo Carvalho de Melo]

Em Thu, Jun 28, 2012 at 04:50:27PM +0800, Feng Tang escreveu:
> On Wed, 27 Jun 2012 15:58:17 -0300
> Arnaldo Carvalho de Melo <acme@redhat.com> wrote:
> > Now old scrips will break, as the tuple they expect:
> > 
> >  (attr, sample, raw_data)
> > 
> > will not be there.
> 
> Actually, there are no such scripts out. Current perf python script only
> supports the trace point type, while perl scriptt has the support for
> general events. It is my first patch "perf script: Add general python 
> handler to process non-tracepoint events" which bring this (attr, sample,
> raw_data) in to be on the same page as perl. So this won't be a problem
> when the 3 patches are merged together. 

Cool, I forgot about that, thanks for correcting me :-)

But even then, perf and python diverged in this... anyway, not a big
deal, I guess, the info available for perl scripts will be available in
the dict as well, right?
 
> > The first two patches in this series don't apply anymore, please find
> > them attached fixed, please check if they work as expected.

> Thanks a lot for fixing them, I'll generate perf tool patch against the
> "perf/core" branch of your git tree on kernel.org from now on.
> 
> Btw, I have 2 more python scripts based on these patches, will add them
> to this serie for your review.

Ok, will take a look at them.

- Arnaldo

Re: [PATCH v3 3/3] perf script/python: Pass event/thread/dso name and symbol info to event handler in python

[Feng Tang]

On Thu, 28 Jun 2012 14:29:31 -0300
Arnaldo Carvalho de Melo <acme@redhat.com> wrote:

> Em Thu, Jun 28, 2012 at 04:50:27PM +0800, Feng Tang escreveu:
> > On Wed, 27 Jun 2012 15:58:17 -0300
> > Arnaldo Carvalho de Melo <acme@redhat.com> wrote:
> > > Now old scrips will break, as the tuple they expect:
> > > 
> > >  (attr, sample, raw_data)
> > > 
> > > will not be there.
> > 
> > Actually, there are no such scripts out. Current perf python script only
> > supports the trace point type, while perl scriptt has the support for
> > general events. It is my first patch "perf script: Add general python 
> > handler to process non-tracepoint events" which bring this (attr, sample,
> > raw_data) in to be on the same page as perl. So this won't be a problem
> > when the 3 patches are merged together. 
> 
> Cool, I forgot about that, thanks for correcting me :-)
> 
> But even then, perf and python diverged in this... anyway, not a big
> deal, I guess, the info available for perl scripts will be available in
> the dict as well, right?

No, current perl code will pass 4 rather raw data structures to the
script's "process_event": pevent->header, evel->attr, sample, sample->raw_data

	XPUSHs(sv_2mortal(newSVpvn((const char *)pevent, pevent->header.size)));
	XPUSHs(sv_2mortal(newSVpvn((const char *)&evsel->attr, sizeof(evsel->attr))));
	XPUSHs(sv_2mortal(newSVpvn((const char *)sample, sizeof(*sample))));
	XPUSHs(sv_2mortal(newSVpvn((const char *)sample->raw_data, sample->raw_size)));

my patch only have the last 3 ones plus extra event_name/dso_name/symobl,
while missing the "pevent->header", do you want me to add it?

>  
> > > The first two patches in this series don't apply anymore, please find
> > > them attached fixed, please check if they work as expected.
> 
> > Thanks a lot for fixing them, I'll generate perf tool patch against the
> > "perf/core" branch of your git tree on kernel.org from now on.
> > 
> > Btw, I have 2 more python scripts based on these patches, will add them
> > to this serie for your review.
> 
> Ok, will take a look at them.

Thanks,

- Feng