* [PATCH 0/5] perf script: Add support for exporting to sqlite3
@ 2017-08-03 8:31 Adrian Hunter
2017-08-03 8:31 ` [PATCH 1/5] perf script: Fix missing call_path_id in export-to-postgresql script Adrian Hunter
` (5 more replies)
0 siblings, 6 replies; 13+ messages in thread
From: Adrian Hunter @ 2017-08-03 8:31 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo; +Cc: linux-kernel
Hi
Here is a script for exporting to SQLite 3 the same data as the PostgreSQL
export. The call-graph script is renamed and amended to work with both
PostgreSQL and SQLite.
Adrian Hunter (5):
perf script: Fix missing call_path_id in export-to-postgresql script
perf script: Fix query in call-graph-from-postgresql.py
perf script: Add support for exporting to sqlite3
perf script: Rename call-graph-from-postgresql.py to call-graph-from-sql.py
perf script: Add support for sqlite3 to call-graph-from-sql.py
tools/perf/Documentation/intel-pt.txt | 6 +-
.../scripts/python/bin/export-to-sqlite-record | 8 +
.../scripts/python/bin/export-to-sqlite-report | 29 ++
...h-from-postgresql.py => call-graph-from-sql.py} | 70 ++--
tools/perf/scripts/python/export-to-postgresql.py | 5 +-
tools/perf/scripts/python/export-to-sqlite.py | 451 +++++++++++++++++++++
6 files changed, 535 insertions(+), 34 deletions(-)
create mode 100644 tools/perf/scripts/python/bin/export-to-sqlite-record
create mode 100644 tools/perf/scripts/python/bin/export-to-sqlite-report
rename tools/perf/scripts/python/{call-graph-from-postgresql.py => call-graph-from-sql.py} (87%)
create mode 100644 tools/perf/scripts/python/export-to-sqlite.py
Regards
Adrian
^ permalink raw reply [flat|nested] 13+ messages in thread* [PATCH 1/5] perf script: Fix missing call_path_id in export-to-postgresql script 2017-08-03 8:31 [PATCH 0/5] perf script: Add support for exporting to sqlite3 Adrian Hunter @ 2017-08-03 8:31 ` Adrian Hunter 2017-08-17 7:50 ` [tip:perf/core] perf scripts python: " tip-bot for Adrian Hunter 2017-08-03 8:31 ` [PATCH 2/5] perf script: Fix query in call-graph-from-postgresql.py Adrian Hunter ` (4 subsequent siblings) 5 siblings, 1 reply; 13+ messages in thread From: Adrian Hunter @ 2017-08-03 8:31 UTC (permalink / raw) To: Arnaldo Carvalho de Melo; +Cc: linux-kernel The export does not work if only branches are exported because of a missing column in the samples table. Fix by adding the missing call_path_id. Signed-off-by: Adrian Hunter <adrian.hunter@intel.com> Fixes: 3521f3bc9dae ("perf script: Update export-to-postgresql to support callchain export") --- tools/perf/scripts/python/export-to-postgresql.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tools/perf/scripts/python/export-to-postgresql.py b/tools/perf/scripts/python/export-to-postgresql.py index 7656ff8aa066..f57811443beb 100644 --- a/tools/perf/scripts/python/export-to-postgresql.py +++ b/tools/perf/scripts/python/export-to-postgresql.py @@ -340,7 +340,8 @@ if branches: 'to_sym_offset bigint,' 'to_ip bigint,' 'branch_type integer,' - 'in_tx boolean)') + 'in_tx boolean,' + 'call_path_id bigint)') else: do_query(query, 'CREATE TABLE samples (' 'id bigint NOT NULL,' -- 1.9.1 ^ permalink raw reply related [flat|nested] 13+ messages in thread
* [tip:perf/core] perf scripts python: Fix missing call_path_id in export-to-postgresql script 2017-08-03 8:31 ` [PATCH 1/5] perf script: Fix missing call_path_id in export-to-postgresql script Adrian Hunter @ 2017-08-17 7:50 ` tip-bot for Adrian Hunter 0 siblings, 0 replies; 13+ messages in thread From: tip-bot for Adrian Hunter @ 2017-08-17 7:50 UTC (permalink / raw) To: linux-tip-commits; +Cc: acme, linux-kernel, adrian.hunter, mingo, hpa, tglx Commit-ID: c8a827285c33e7a19e81dfbff2740e1e67ca42df Gitweb: http://git.kernel.org/tip/c8a827285c33e7a19e81dfbff2740e1e67ca42df Author: Adrian Hunter <adrian.hunter@intel.com> AuthorDate: Thu, 3 Aug 2017 11:31:26 +0300 Committer: Arnaldo Carvalho de Melo <acme@redhat.com> CommitDate: Tue, 15 Aug 2017 16:05:36 -0300 perf scripts python: Fix missing call_path_id in export-to-postgresql script The export does not work if only branches are exported because of a missing column in the samples table. Fix by adding the missing call_path_id. Fixes: 3521f3bc9dae ("perf script: Update export-to-postgresql to support callchain export") Signed-off-by: Adrian Hunter <adrian.hunter@intel.com> Link: http://lkml.kernel.org/r/1501749090-20357-2-git-send-email-adrian.hunter@intel.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com> --- tools/perf/scripts/python/export-to-postgresql.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tools/perf/scripts/python/export-to-postgresql.py b/tools/perf/scripts/python/export-to-postgresql.py index 7656ff8..f578114 100644 --- a/tools/perf/scripts/python/export-to-postgresql.py +++ b/tools/perf/scripts/python/export-to-postgresql.py @@ -340,7 +340,8 @@ if branches: 'to_sym_offset bigint,' 'to_ip bigint,' 'branch_type integer,' - 'in_tx boolean)') + 'in_tx boolean,' + 'call_path_id bigint)') else: do_query(query, 'CREATE TABLE samples (' 'id bigint NOT NULL,' ^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH 2/5] perf script: Fix query in call-graph-from-postgresql.py 2017-08-03 8:31 [PATCH 0/5] perf script: Add support for exporting to sqlite3 Adrian Hunter 2017-08-03 8:31 ` [PATCH 1/5] perf script: Fix missing call_path_id in export-to-postgresql script Adrian Hunter @ 2017-08-03 8:31 ` Adrian Hunter 2017-08-17 7:50 ` [tip:perf/core] perf scripts python: " tip-bot for Adrian Hunter 2017-08-03 8:31 ` [PATCH 3/5] perf script: Add support for exporting to sqlite3 Adrian Hunter ` (3 subsequent siblings) 5 siblings, 1 reply; 13+ messages in thread From: Adrian Hunter @ 2017-08-03 8:31 UTC (permalink / raw) To: Arnaldo Carvalho de Melo; +Cc: linux-kernel Add a missing space which seemed not to affect PostgreSQL but upsets SQLite. Signed-off-by: Adrian Hunter <adrian.hunter@intel.com> --- tools/perf/scripts/python/call-graph-from-postgresql.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/perf/scripts/python/call-graph-from-postgresql.py b/tools/perf/scripts/python/call-graph-from-postgresql.py index e78fdc2a5a9d..ed9f7f3ccf22 100644 --- a/tools/perf/scripts/python/call-graph-from-postgresql.py +++ b/tools/perf/scripts/python/call-graph-from-postgresql.py @@ -160,7 +160,7 @@ class TreeItem(): '( SELECT short_name FROM dsos WHERE id = ( SELECT dso_id FROM symbols WHERE id = ( SELECT symbol_id FROM call_paths WHERE id = call_path_id ) ) ), ' '( SELECT ip FROM call_paths where id = call_path_id ) ' 'FROM calls WHERE parent_call_path_id = ' + str(self.call_path_id) + ' AND comm_id = ' + str(self.comm_id) + ' AND thread_id = ' + str(self.thread_id) + - 'ORDER BY call_path_id') + ' ORDER BY call_path_id') if not ret: raise Exception("Query failed: " + query.lastError().text()) last_call_path_id = 0 -- 1.9.1 ^ permalink raw reply related [flat|nested] 13+ messages in thread
* [tip:perf/core] perf scripts python: Fix query in call-graph-from-postgresql.py 2017-08-03 8:31 ` [PATCH 2/5] perf script: Fix query in call-graph-from-postgresql.py Adrian Hunter @ 2017-08-17 7:50 ` tip-bot for Adrian Hunter 0 siblings, 0 replies; 13+ messages in thread From: tip-bot for Adrian Hunter @ 2017-08-17 7:50 UTC (permalink / raw) To: linux-tip-commits; +Cc: linux-kernel, tglx, mingo, acme, hpa, adrian.hunter Commit-ID: 2295e9f850b0efbc57c81fccdd8bd8d26fe10029 Gitweb: http://git.kernel.org/tip/2295e9f850b0efbc57c81fccdd8bd8d26fe10029 Author: Adrian Hunter <adrian.hunter@intel.com> AuthorDate: Thu, 3 Aug 2017 11:31:27 +0300 Committer: Arnaldo Carvalho de Melo <acme@redhat.com> CommitDate: Tue, 15 Aug 2017 16:06:20 -0300 perf scripts python: Fix query in call-graph-from-postgresql.py Add a missing space which seemed not to affect PostgreSQL but upsets SQLite. Signed-off-by: Adrian Hunter <adrian.hunter@intel.com> Link: http://lkml.kernel.org/r/1501749090-20357-3-git-send-email-adrian.hunter@intel.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com> --- tools/perf/scripts/python/call-graph-from-postgresql.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/perf/scripts/python/call-graph-from-postgresql.py b/tools/perf/scripts/python/call-graph-from-postgresql.py index e78fdc2..ed9f7f3 100644 --- a/tools/perf/scripts/python/call-graph-from-postgresql.py +++ b/tools/perf/scripts/python/call-graph-from-postgresql.py @@ -160,7 +160,7 @@ class TreeItem(): '( SELECT short_name FROM dsos WHERE id = ( SELECT dso_id FROM symbols WHERE id = ( SELECT symbol_id FROM call_paths WHERE id = call_path_id ) ) ), ' '( SELECT ip FROM call_paths where id = call_path_id ) ' 'FROM calls WHERE parent_call_path_id = ' + str(self.call_path_id) + ' AND comm_id = ' + str(self.comm_id) + ' AND thread_id = ' + str(self.thread_id) + - 'ORDER BY call_path_id') + ' ORDER BY call_path_id') if not ret: raise Exception("Query failed: " + query.lastError().text()) last_call_path_id = 0 ^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH 3/5] perf script: Add support for exporting to sqlite3 2017-08-03 8:31 [PATCH 0/5] perf script: Add support for exporting to sqlite3 Adrian Hunter 2017-08-03 8:31 ` [PATCH 1/5] perf script: Fix missing call_path_id in export-to-postgresql script Adrian Hunter 2017-08-03 8:31 ` [PATCH 2/5] perf script: Fix query in call-graph-from-postgresql.py Adrian Hunter @ 2017-08-03 8:31 ` Adrian Hunter 2017-08-17 7:51 ` [tip:perf/core] perf script python: " tip-bot for Adrian Hunter 2017-08-03 8:31 ` [PATCH 4/5] perf script: Rename call-graph-from-postgresql.py to call-graph-from-sql.py Adrian Hunter ` (2 subsequent siblings) 5 siblings, 1 reply; 13+ messages in thread From: Adrian Hunter @ 2017-08-03 8:31 UTC (permalink / raw) To: Arnaldo Carvalho de Melo; +Cc: linux-kernel Add support for exporting to SQLite 3 the same data as the PostgreSQL export. Signed-off-by: Adrian Hunter <adrian.hunter@intel.com> --- tools/perf/Documentation/intel-pt.txt | 6 +- .../scripts/python/bin/export-to-sqlite-record | 8 + .../scripts/python/bin/export-to-sqlite-report | 29 ++ tools/perf/scripts/python/export-to-sqlite.py | 451 +++++++++++++++++++++ 4 files changed, 491 insertions(+), 3 deletions(-) create mode 100644 tools/perf/scripts/python/bin/export-to-sqlite-record create mode 100644 tools/perf/scripts/python/bin/export-to-sqlite-report create mode 100644 tools/perf/scripts/python/export-to-sqlite.py diff --git a/tools/perf/Documentation/intel-pt.txt b/tools/perf/Documentation/intel-pt.txt index 4b6cdbf8f935..8e8ae3ad4cbf 100644 --- a/tools/perf/Documentation/intel-pt.txt +++ b/tools/perf/Documentation/intel-pt.txt @@ -104,9 +104,9 @@ system, asynchronous, interrupt, transaction abort, trace begin, trace end, and in transaction, respectively. While it is possible to create scripts to analyze the data, an alternative -approach is available to export the data to a postgresql database. Refer to -script export-to-postgresql.py for more details, and to script -call-graph-from-postgresql.py for an example of using the database. +approach is available to export the data to a sqlite or postgresql database. +Refer to script export-to-sqlite.py or export-to-postgresql.py for more details, +and to script call-graph-from-postgresql.py for an example of using the database. There is also script intel-pt-events.py which provides an example of how to unpack the raw data for power events and PTWRITE. diff --git a/tools/perf/scripts/python/bin/export-to-sqlite-record b/tools/perf/scripts/python/bin/export-to-sqlite-record new file mode 100644 index 000000000000..070204fd6d00 --- /dev/null +++ b/tools/perf/scripts/python/bin/export-to-sqlite-record @@ -0,0 +1,8 @@ +#!/bin/bash + +# +# export perf data to a sqlite3 database. Can cover +# perf ip samples (excluding the tracepoints). No special +# record requirements, just record what you want to export. +# +perf record $@ diff --git a/tools/perf/scripts/python/bin/export-to-sqlite-report b/tools/perf/scripts/python/bin/export-to-sqlite-report new file mode 100644 index 000000000000..5ff6033e70ba --- /dev/null +++ b/tools/perf/scripts/python/bin/export-to-sqlite-report @@ -0,0 +1,29 @@ +#!/bin/bash +# description: export perf data to a sqlite3 database +# args: [database name] [columns] [calls] +n_args=0 +for i in "$@" +do + if expr match "$i" "-" > /dev/null ; then + break + fi + n_args=$(( $n_args + 1 )) +done +if [ "$n_args" -gt 3 ] ; then + echo "usage: export-to-sqlite-report [database name] [columns] [calls]" + exit +fi +if [ "$n_args" -gt 2 ] ; then + dbname=$1 + columns=$2 + calls=$3 + shift 3 +elif [ "$n_args" -gt 1 ] ; then + dbname=$1 + columns=$2 + shift 2 +elif [ "$n_args" -gt 0 ] ; then + dbname=$1 + shift +fi +perf script $@ -s "$PERF_EXEC_PATH"/scripts/python/export-to-sqlite.py $dbname $columns $calls diff --git a/tools/perf/scripts/python/export-to-sqlite.py b/tools/perf/scripts/python/export-to-sqlite.py new file mode 100644 index 000000000000..f827bf77e9d2 --- /dev/null +++ b/tools/perf/scripts/python/export-to-sqlite.py @@ -0,0 +1,451 @@ +# export-to-sqlite.py: export perf data to a sqlite3 database +# Copyright (c) 2017, Intel Corporation. +# +# This program is free software; you can redistribute it and/or modify it +# under the terms and conditions of the GNU General Public License, +# version 2, as published by the Free Software Foundation. +# +# This program is distributed in the hope it will be useful, but WITHOUT +# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or +# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for +# more details. + +import os +import sys +import struct +import datetime + +# To use this script you will need to have installed package python-pyside which +# provides LGPL-licensed Python bindings for Qt. You will also need the package +# libqt4-sql-sqlite for Qt sqlite3 support. +# +# An example of using this script with Intel PT: +# +# $ perf record -e intel_pt//u ls +# $ perf script -s ~/libexec/perf-core/scripts/python/export-to-sqlite.py pt_example branches calls +# 2017-07-31 14:26:07.326913 Creating database... +# 2017-07-31 14:26:07.538097 Writing records... +# 2017-07-31 14:26:09.889292 Adding indexes +# 2017-07-31 14:26:09.958746 Done +# +# To browse the database, sqlite3 can be used e.g. +# +# $ sqlite3 pt_example +# sqlite> .header on +# sqlite> select * from samples_view where id < 10; +# sqlite> .mode column +# sqlite> select * from samples_view where id < 10; +# sqlite> .tables +# sqlite> .schema samples_view +# sqlite> .quit +# +# An example of using the database is provided by the script +# call-graph-from-sql.py. Refer to that script for details. +# +# The database structure is practically the same as created by the script +# export-to-postgresql.py. Refer to that script for details. A notable +# difference is the 'transaction' column of the 'samples' table which is +# renamed 'transaction_' in sqlite because 'transaction' is a reserved word. + +from PySide.QtSql import * + +sys.path.append(os.environ['PERF_EXEC_PATH'] + \ + '/scripts/python/Perf-Trace-Util/lib/Perf/Trace') + +# These perf imports are not used at present +#from perf_trace_context import * +#from Core import * + +perf_db_export_mode = True +perf_db_export_calls = False +perf_db_export_callchains = False + +def usage(): + print >> sys.stderr, "Usage is: export-to-sqlite.py <database name> [<columns>] [<calls>] [<callchains>]" + print >> sys.stderr, "where: columns 'all' or 'branches'" + print >> sys.stderr, " calls 'calls' => create calls and call_paths table" + print >> sys.stderr, " callchains 'callchains' => create call_paths table" + raise Exception("Too few arguments") + +if (len(sys.argv) < 2): + usage() + +dbname = sys.argv[1] + +if (len(sys.argv) >= 3): + columns = sys.argv[2] +else: + columns = "all" + +if columns not in ("all", "branches"): + usage() + +branches = (columns == "branches") + +for i in range(3,len(sys.argv)): + if (sys.argv[i] == "calls"): + perf_db_export_calls = True + elif (sys.argv[i] == "callchains"): + perf_db_export_callchains = True + else: + usage() + +def do_query(q, s): + if (q.exec_(s)): + return + raise Exception("Query failed: " + q.lastError().text()) + +def do_query_(q): + if (q.exec_()): + return + raise Exception("Query failed: " + q.lastError().text()) + +print datetime.datetime.today(), "Creating database..." + +db_exists = False +try: + f = open(dbname) + f.close() + db_exists = True +except: + pass + +if db_exists: + raise Exception(dbname + " already exists") + +db = QSqlDatabase.addDatabase('QSQLITE') +db.setDatabaseName(dbname) +db.open() + +query = QSqlQuery(db) + +do_query(query, 'PRAGMA journal_mode = OFF') +do_query(query, 'BEGIN TRANSACTION') + +do_query(query, 'CREATE TABLE selected_events (' + 'id integer NOT NULL PRIMARY KEY,' + 'name varchar(80))') +do_query(query, 'CREATE TABLE machines (' + 'id integer NOT NULL PRIMARY KEY,' + 'pid integer,' + 'root_dir varchar(4096))') +do_query(query, 'CREATE TABLE threads (' + 'id integer NOT NULL PRIMARY KEY,' + 'machine_id bigint,' + 'process_id bigint,' + 'pid integer,' + 'tid integer)') +do_query(query, 'CREATE TABLE comms (' + 'id integer NOT NULL PRIMARY KEY,' + 'comm varchar(16))') +do_query(query, 'CREATE TABLE comm_threads (' + 'id integer NOT NULL PRIMARY KEY,' + 'comm_id bigint,' + 'thread_id bigint)') +do_query(query, 'CREATE TABLE dsos (' + 'id integer NOT NULL PRIMARY KEY,' + 'machine_id bigint,' + 'short_name varchar(256),' + 'long_name varchar(4096),' + 'build_id varchar(64))') +do_query(query, 'CREATE TABLE symbols (' + 'id integer NOT NULL PRIMARY KEY,' + 'dso_id bigint,' + 'sym_start bigint,' + 'sym_end bigint,' + 'binding integer,' + 'name varchar(2048))') +do_query(query, 'CREATE TABLE branch_types (' + 'id integer NOT NULL PRIMARY KEY,' + 'name varchar(80))') + +if branches: + do_query(query, 'CREATE TABLE samples (' + 'id integer NOT NULL PRIMARY KEY,' + 'evsel_id bigint,' + 'machine_id bigint,' + 'thread_id bigint,' + 'comm_id bigint,' + 'dso_id bigint,' + 'symbol_id bigint,' + 'sym_offset bigint,' + 'ip bigint,' + 'time bigint,' + 'cpu integer,' + 'to_dso_id bigint,' + 'to_symbol_id bigint,' + 'to_sym_offset bigint,' + 'to_ip bigint,' + 'branch_type integer,' + 'in_tx boolean,' + 'call_path_id bigint)') +else: + do_query(query, 'CREATE TABLE samples (' + 'id integer NOT NULL PRIMARY KEY,' + 'evsel_id bigint,' + 'machine_id bigint,' + 'thread_id bigint,' + 'comm_id bigint,' + 'dso_id bigint,' + 'symbol_id bigint,' + 'sym_offset bigint,' + 'ip bigint,' + 'time bigint,' + 'cpu integer,' + 'to_dso_id bigint,' + 'to_symbol_id bigint,' + 'to_sym_offset bigint,' + 'to_ip bigint,' + 'period bigint,' + 'weight bigint,' + 'transaction_ bigint,' + 'data_src bigint,' + 'branch_type integer,' + 'in_tx boolean,' + 'call_path_id bigint)') + +if perf_db_export_calls or perf_db_export_callchains: + do_query(query, 'CREATE TABLE call_paths (' + 'id integer NOT NULL PRIMARY KEY,' + 'parent_id bigint,' + 'symbol_id bigint,' + 'ip bigint)') +if perf_db_export_calls: + do_query(query, 'CREATE TABLE calls (' + 'id integer NOT NULL PRIMARY KEY,' + 'thread_id bigint,' + 'comm_id bigint,' + 'call_path_id bigint,' + 'call_time bigint,' + 'return_time bigint,' + 'branch_count bigint,' + 'call_id bigint,' + 'return_id bigint,' + 'parent_call_path_id bigint,' + 'flags integer)') + +# printf was added to sqlite in version 3.8.3 +sqlite_has_printf = False +try: + do_query(query, 'SELECT printf("") FROM machines') + sqlite_has_printf = True +except: + pass + +def emit_to_hex(x): + if sqlite_has_printf: + return 'printf("%x", ' + x + ')' + else: + return x + +do_query(query, 'CREATE VIEW machines_view AS ' + 'SELECT ' + 'id,' + 'pid,' + 'root_dir,' + 'CASE WHEN id=0 THEN \'unknown\' WHEN pid=-1 THEN \'host\' ELSE \'guest\' END AS host_or_guest' + ' FROM machines') + +do_query(query, 'CREATE VIEW dsos_view AS ' + 'SELECT ' + 'id,' + 'machine_id,' + '(SELECT host_or_guest FROM machines_view WHERE id = machine_id) AS host_or_guest,' + 'short_name,' + 'long_name,' + 'build_id' + ' FROM dsos') + +do_query(query, 'CREATE VIEW symbols_view AS ' + 'SELECT ' + 'id,' + 'name,' + '(SELECT short_name FROM dsos WHERE id=dso_id) AS dso,' + 'dso_id,' + 'sym_start,' + 'sym_end,' + 'CASE WHEN binding=0 THEN \'local\' WHEN binding=1 THEN \'global\' ELSE \'weak\' END AS binding' + ' FROM symbols') + +do_query(query, 'CREATE VIEW threads_view AS ' + 'SELECT ' + 'id,' + 'machine_id,' + '(SELECT host_or_guest FROM machines_view WHERE id = machine_id) AS host_or_guest,' + 'process_id,' + 'pid,' + 'tid' + ' FROM threads') + +do_query(query, 'CREATE VIEW comm_threads_view AS ' + 'SELECT ' + 'comm_id,' + '(SELECT comm FROM comms WHERE id = comm_id) AS command,' + 'thread_id,' + '(SELECT pid FROM threads WHERE id = thread_id) AS pid,' + '(SELECT tid FROM threads WHERE id = thread_id) AS tid' + ' FROM comm_threads') + +if perf_db_export_calls or perf_db_export_callchains: + do_query(query, 'CREATE VIEW call_paths_view AS ' + 'SELECT ' + 'c.id,' + + emit_to_hex('c.ip') + ' AS ip,' + 'c.symbol_id,' + '(SELECT name FROM symbols WHERE id = c.symbol_id) AS symbol,' + '(SELECT dso_id FROM symbols WHERE id = c.symbol_id) AS dso_id,' + '(SELECT dso FROM symbols_view WHERE id = c.symbol_id) AS dso_short_name,' + 'c.parent_id,' + + emit_to_hex('p.ip') + ' AS parent_ip,' + 'p.symbol_id AS parent_symbol_id,' + '(SELECT name FROM symbols WHERE id = p.symbol_id) AS parent_symbol,' + '(SELECT dso_id FROM symbols WHERE id = p.symbol_id) AS parent_dso_id,' + '(SELECT dso FROM symbols_view WHERE id = p.symbol_id) AS parent_dso_short_name' + ' FROM call_paths c INNER JOIN call_paths p ON p.id = c.parent_id') +if perf_db_export_calls: + do_query(query, 'CREATE VIEW calls_view AS ' + 'SELECT ' + 'calls.id,' + 'thread_id,' + '(SELECT pid FROM threads WHERE id = thread_id) AS pid,' + '(SELECT tid FROM threads WHERE id = thread_id) AS tid,' + '(SELECT comm FROM comms WHERE id = comm_id) AS command,' + 'call_path_id,' + + emit_to_hex('ip') + ' AS ip,' + 'symbol_id,' + '(SELECT name FROM symbols WHERE id = symbol_id) AS symbol,' + 'call_time,' + 'return_time,' + 'return_time - call_time AS elapsed_time,' + 'branch_count,' + 'call_id,' + 'return_id,' + 'CASE WHEN flags=1 THEN \'no call\' WHEN flags=2 THEN \'no return\' WHEN flags=3 THEN \'no call/return\' ELSE \'\' END AS flags,' + 'parent_call_path_id' + ' FROM calls INNER JOIN call_paths ON call_paths.id = call_path_id') + +do_query(query, 'CREATE VIEW samples_view AS ' + 'SELECT ' + 'id,' + 'time,' + 'cpu,' + '(SELECT pid FROM threads WHERE id = thread_id) AS pid,' + '(SELECT tid FROM threads WHERE id = thread_id) AS tid,' + '(SELECT comm FROM comms WHERE id = comm_id) AS command,' + '(SELECT name FROM selected_events WHERE id = evsel_id) AS event,' + + emit_to_hex('ip') + ' AS ip_hex,' + '(SELECT name FROM symbols WHERE id = symbol_id) AS symbol,' + 'sym_offset,' + '(SELECT short_name FROM dsos WHERE id = dso_id) AS dso_short_name,' + + emit_to_hex('to_ip') + ' AS to_ip_hex,' + '(SELECT name FROM symbols WHERE id = to_symbol_id) AS to_symbol,' + 'to_sym_offset,' + '(SELECT short_name FROM dsos WHERE id = to_dso_id) AS to_dso_short_name,' + '(SELECT name FROM branch_types WHERE id = branch_type) AS branch_type_name,' + 'in_tx' + ' FROM samples') + +do_query(query, 'END TRANSACTION') + +evsel_query = QSqlQuery(db) +evsel_query.prepare("INSERT INTO selected_events VALUES (?, ?)") +machine_query = QSqlQuery(db) +machine_query.prepare("INSERT INTO machines VALUES (?, ?, ?)") +thread_query = QSqlQuery(db) +thread_query.prepare("INSERT INTO threads VALUES (?, ?, ?, ?, ?)") +comm_query = QSqlQuery(db) +comm_query.prepare("INSERT INTO comms VALUES (?, ?)") +comm_thread_query = QSqlQuery(db) +comm_thread_query.prepare("INSERT INTO comm_threads VALUES (?, ?, ?)") +dso_query = QSqlQuery(db) +dso_query.prepare("INSERT INTO dsos VALUES (?, ?, ?, ?, ?)") +symbol_query = QSqlQuery(db) +symbol_query.prepare("INSERT INTO symbols VALUES (?, ?, ?, ?, ?, ?)") +branch_type_query = QSqlQuery(db) +branch_type_query.prepare("INSERT INTO branch_types VALUES (?, ?)") +sample_query = QSqlQuery(db) +if branches: + sample_query.prepare("INSERT INTO samples VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)") +else: + sample_query.prepare("INSERT INTO samples VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)") +if perf_db_export_calls or perf_db_export_callchains: + call_path_query = QSqlQuery(db) + call_path_query.prepare("INSERT INTO call_paths VALUES (?, ?, ?, ?)") +if perf_db_export_calls: + call_query = QSqlQuery(db) + call_query.prepare("INSERT INTO calls VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)") + +def trace_begin(): + print datetime.datetime.today(), "Writing records..." + do_query(query, 'BEGIN TRANSACTION') + # id == 0 means unknown. It is easier to create records for them than replace the zeroes with NULLs + evsel_table(0, "unknown") + machine_table(0, 0, "unknown") + thread_table(0, 0, 0, -1, -1) + comm_table(0, "unknown") + dso_table(0, 0, "unknown", "unknown", "") + symbol_table(0, 0, 0, 0, 0, "unknown") + sample_table(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0) + if perf_db_export_calls or perf_db_export_callchains: + call_path_table(0, 0, 0, 0) + +unhandled_count = 0 + +def trace_end(): + do_query(query, 'END TRANSACTION') + + print datetime.datetime.today(), "Adding indexes" + if perf_db_export_calls: + do_query(query, 'CREATE INDEX pcpid_idx ON calls (parent_call_path_id)') + + if (unhandled_count): + print datetime.datetime.today(), "Warning: ", unhandled_count, " unhandled events" + print datetime.datetime.today(), "Done" + +def trace_unhandled(event_name, context, event_fields_dict): + global unhandled_count + unhandled_count += 1 + +def sched__sched_switch(*x): + pass + +def bind_exec(q, n, x): + for xx in x[0:n]: + q.addBindValue(str(xx)) + do_query_(q) + +def evsel_table(*x): + bind_exec(evsel_query, 2, x) + +def machine_table(*x): + bind_exec(machine_query, 3, x) + +def thread_table(*x): + bind_exec(thread_query, 5, x) + +def comm_table(*x): + bind_exec(comm_query, 2, x) + +def comm_thread_table(*x): + bind_exec(comm_thread_query, 3, x) + +def dso_table(*x): + bind_exec(dso_query, 5, x) + +def symbol_table(*x): + bind_exec(symbol_query, 6, x) + +def branch_type_table(*x): + bind_exec(branch_type_query, 2, x) + +def sample_table(*x): + if branches: + bind_exec(sample_query, 18, x) + else: + bind_exec(sample_query, 22, x) + +def call_path_table(*x): + bind_exec(call_path_query, 4, x) + +def call_return_table(*x): + bind_exec(call_query, 11, x) -- 1.9.1 ^ permalink raw reply related [flat|nested] 13+ messages in thread
* [tip:perf/core] perf script python: Add support for exporting to sqlite3 2017-08-03 8:31 ` [PATCH 3/5] perf script: Add support for exporting to sqlite3 Adrian Hunter @ 2017-08-17 7:51 ` tip-bot for Adrian Hunter 0 siblings, 0 replies; 13+ messages in thread From: tip-bot for Adrian Hunter @ 2017-08-17 7:51 UTC (permalink / raw) To: linux-tip-commits; +Cc: mingo, acme, hpa, linux-kernel, adrian.hunter, tglx Commit-ID: 564b9527d1ccf5d581275391e39ac4b1f29f0d08 Gitweb: http://git.kernel.org/tip/564b9527d1ccf5d581275391e39ac4b1f29f0d08 Author: Adrian Hunter <adrian.hunter@intel.com> AuthorDate: Thu, 3 Aug 2017 11:31:28 +0300 Committer: Arnaldo Carvalho de Melo <acme@redhat.com> CommitDate: Tue, 15 Aug 2017 16:37:55 -0300 perf script python: Add support for exporting to sqlite3 Add support for exporting to SQLite 3 the same data as the PostgreSQL export. Committer note: Tested on RHEL 7.4 using the 1.2.2-4el python-pyside packages from EPEL. Signed-off-by: Adrian Hunter <adrian.hunter@intel.com> Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com> Link: http://lkml.kernel.org/r/1501749090-20357-4-git-send-email-adrian.hunter@intel.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com> --- tools/perf/Documentation/intel-pt.txt | 6 +- .../scripts/python/bin/export-to-sqlite-record | 8 + .../scripts/python/bin/export-to-sqlite-report | 29 ++ tools/perf/scripts/python/export-to-sqlite.py | 451 +++++++++++++++++++++ 4 files changed, 491 insertions(+), 3 deletions(-) diff --git a/tools/perf/Documentation/intel-pt.txt b/tools/perf/Documentation/intel-pt.txt index 4b6cdbf..8e8ae3a 100644 --- a/tools/perf/Documentation/intel-pt.txt +++ b/tools/perf/Documentation/intel-pt.txt @@ -104,9 +104,9 @@ system, asynchronous, interrupt, transaction abort, trace begin, trace end, and in transaction, respectively. While it is possible to create scripts to analyze the data, an alternative -approach is available to export the data to a postgresql database. Refer to -script export-to-postgresql.py for more details, and to script -call-graph-from-postgresql.py for an example of using the database. +approach is available to export the data to a sqlite or postgresql database. +Refer to script export-to-sqlite.py or export-to-postgresql.py for more details, +and to script call-graph-from-postgresql.py for an example of using the database. There is also script intel-pt-events.py which provides an example of how to unpack the raw data for power events and PTWRITE. diff --git a/tools/perf/scripts/python/bin/export-to-sqlite-record b/tools/perf/scripts/python/bin/export-to-sqlite-record new file mode 100644 index 0000000..070204f --- /dev/null +++ b/tools/perf/scripts/python/bin/export-to-sqlite-record @@ -0,0 +1,8 @@ +#!/bin/bash + +# +# export perf data to a sqlite3 database. Can cover +# perf ip samples (excluding the tracepoints). No special +# record requirements, just record what you want to export. +# +perf record $@ diff --git a/tools/perf/scripts/python/bin/export-to-sqlite-report b/tools/perf/scripts/python/bin/export-to-sqlite-report new file mode 100644 index 0000000..5ff6033 --- /dev/null +++ b/tools/perf/scripts/python/bin/export-to-sqlite-report @@ -0,0 +1,29 @@ +#!/bin/bash +# description: export perf data to a sqlite3 database +# args: [database name] [columns] [calls] +n_args=0 +for i in "$@" +do + if expr match "$i" "-" > /dev/null ; then + break + fi + n_args=$(( $n_args + 1 )) +done +if [ "$n_args" -gt 3 ] ; then + echo "usage: export-to-sqlite-report [database name] [columns] [calls]" + exit +fi +if [ "$n_args" -gt 2 ] ; then + dbname=$1 + columns=$2 + calls=$3 + shift 3 +elif [ "$n_args" -gt 1 ] ; then + dbname=$1 + columns=$2 + shift 2 +elif [ "$n_args" -gt 0 ] ; then + dbname=$1 + shift +fi +perf script $@ -s "$PERF_EXEC_PATH"/scripts/python/export-to-sqlite.py $dbname $columns $calls diff --git a/tools/perf/scripts/python/export-to-sqlite.py b/tools/perf/scripts/python/export-to-sqlite.py new file mode 100644 index 0000000..f827bf7 --- /dev/null +++ b/tools/perf/scripts/python/export-to-sqlite.py @@ -0,0 +1,451 @@ +# export-to-sqlite.py: export perf data to a sqlite3 database +# Copyright (c) 2017, Intel Corporation. +# +# This program is free software; you can redistribute it and/or modify it +# under the terms and conditions of the GNU General Public License, +# version 2, as published by the Free Software Foundation. +# +# This program is distributed in the hope it will be useful, but WITHOUT +# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or +# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for +# more details. + +import os +import sys +import struct +import datetime + +# To use this script you will need to have installed package python-pyside which +# provides LGPL-licensed Python bindings for Qt. You will also need the package +# libqt4-sql-sqlite for Qt sqlite3 support. +# +# An example of using this script with Intel PT: +# +# $ perf record -e intel_pt//u ls +# $ perf script -s ~/libexec/perf-core/scripts/python/export-to-sqlite.py pt_example branches calls +# 2017-07-31 14:26:07.326913 Creating database... +# 2017-07-31 14:26:07.538097 Writing records... +# 2017-07-31 14:26:09.889292 Adding indexes +# 2017-07-31 14:26:09.958746 Done +# +# To browse the database, sqlite3 can be used e.g. +# +# $ sqlite3 pt_example +# sqlite> .header on +# sqlite> select * from samples_view where id < 10; +# sqlite> .mode column +# sqlite> select * from samples_view where id < 10; +# sqlite> .tables +# sqlite> .schema samples_view +# sqlite> .quit +# +# An example of using the database is provided by the script +# call-graph-from-sql.py. Refer to that script for details. +# +# The database structure is practically the same as created by the script +# export-to-postgresql.py. Refer to that script for details. A notable +# difference is the 'transaction' column of the 'samples' table which is +# renamed 'transaction_' in sqlite because 'transaction' is a reserved word. + +from PySide.QtSql import * + +sys.path.append(os.environ['PERF_EXEC_PATH'] + \ + '/scripts/python/Perf-Trace-Util/lib/Perf/Trace') + +# These perf imports are not used at present +#from perf_trace_context import * +#from Core import * + +perf_db_export_mode = True +perf_db_export_calls = False +perf_db_export_callchains = False + +def usage(): + print >> sys.stderr, "Usage is: export-to-sqlite.py <database name> [<columns>] [<calls>] [<callchains>]" + print >> sys.stderr, "where: columns 'all' or 'branches'" + print >> sys.stderr, " calls 'calls' => create calls and call_paths table" + print >> sys.stderr, " callchains 'callchains' => create call_paths table" + raise Exception("Too few arguments") + +if (len(sys.argv) < 2): + usage() + +dbname = sys.argv[1] + +if (len(sys.argv) >= 3): + columns = sys.argv[2] +else: + columns = "all" + +if columns not in ("all", "branches"): + usage() + +branches = (columns == "branches") + +for i in range(3,len(sys.argv)): + if (sys.argv[i] == "calls"): + perf_db_export_calls = True + elif (sys.argv[i] == "callchains"): + perf_db_export_callchains = True + else: + usage() + +def do_query(q, s): + if (q.exec_(s)): + return + raise Exception("Query failed: " + q.lastError().text()) + +def do_query_(q): + if (q.exec_()): + return + raise Exception("Query failed: " + q.lastError().text()) + +print datetime.datetime.today(), "Creating database..." + +db_exists = False +try: + f = open(dbname) + f.close() + db_exists = True +except: + pass + +if db_exists: + raise Exception(dbname + " already exists") + +db = QSqlDatabase.addDatabase('QSQLITE') +db.setDatabaseName(dbname) +db.open() + +query = QSqlQuery(db) + +do_query(query, 'PRAGMA journal_mode = OFF') +do_query(query, 'BEGIN TRANSACTION') + +do_query(query, 'CREATE TABLE selected_events (' + 'id integer NOT NULL PRIMARY KEY,' + 'name varchar(80))') +do_query(query, 'CREATE TABLE machines (' + 'id integer NOT NULL PRIMARY KEY,' + 'pid integer,' + 'root_dir varchar(4096))') +do_query(query, 'CREATE TABLE threads (' + 'id integer NOT NULL PRIMARY KEY,' + 'machine_id bigint,' + 'process_id bigint,' + 'pid integer,' + 'tid integer)') +do_query(query, 'CREATE TABLE comms (' + 'id integer NOT NULL PRIMARY KEY,' + 'comm varchar(16))') +do_query(query, 'CREATE TABLE comm_threads (' + 'id integer NOT NULL PRIMARY KEY,' + 'comm_id bigint,' + 'thread_id bigint)') +do_query(query, 'CREATE TABLE dsos (' + 'id integer NOT NULL PRIMARY KEY,' + 'machine_id bigint,' + 'short_name varchar(256),' + 'long_name varchar(4096),' + 'build_id varchar(64))') +do_query(query, 'CREATE TABLE symbols (' + 'id integer NOT NULL PRIMARY KEY,' + 'dso_id bigint,' + 'sym_start bigint,' + 'sym_end bigint,' + 'binding integer,' + 'name varchar(2048))') +do_query(query, 'CREATE TABLE branch_types (' + 'id integer NOT NULL PRIMARY KEY,' + 'name varchar(80))') + +if branches: + do_query(query, 'CREATE TABLE samples (' + 'id integer NOT NULL PRIMARY KEY,' + 'evsel_id bigint,' + 'machine_id bigint,' + 'thread_id bigint,' + 'comm_id bigint,' + 'dso_id bigint,' + 'symbol_id bigint,' + 'sym_offset bigint,' + 'ip bigint,' + 'time bigint,' + 'cpu integer,' + 'to_dso_id bigint,' + 'to_symbol_id bigint,' + 'to_sym_offset bigint,' + 'to_ip bigint,' + 'branch_type integer,' + 'in_tx boolean,' + 'call_path_id bigint)') +else: + do_query(query, 'CREATE TABLE samples (' + 'id integer NOT NULL PRIMARY KEY,' + 'evsel_id bigint,' + 'machine_id bigint,' + 'thread_id bigint,' + 'comm_id bigint,' + 'dso_id bigint,' + 'symbol_id bigint,' + 'sym_offset bigint,' + 'ip bigint,' + 'time bigint,' + 'cpu integer,' + 'to_dso_id bigint,' + 'to_symbol_id bigint,' + 'to_sym_offset bigint,' + 'to_ip bigint,' + 'period bigint,' + 'weight bigint,' + 'transaction_ bigint,' + 'data_src bigint,' + 'branch_type integer,' + 'in_tx boolean,' + 'call_path_id bigint)') + +if perf_db_export_calls or perf_db_export_callchains: + do_query(query, 'CREATE TABLE call_paths (' + 'id integer NOT NULL PRIMARY KEY,' + 'parent_id bigint,' + 'symbol_id bigint,' + 'ip bigint)') +if perf_db_export_calls: + do_query(query, 'CREATE TABLE calls (' + 'id integer NOT NULL PRIMARY KEY,' + 'thread_id bigint,' + 'comm_id bigint,' + 'call_path_id bigint,' + 'call_time bigint,' + 'return_time bigint,' + 'branch_count bigint,' + 'call_id bigint,' + 'return_id bigint,' + 'parent_call_path_id bigint,' + 'flags integer)') + +# printf was added to sqlite in version 3.8.3 +sqlite_has_printf = False +try: + do_query(query, 'SELECT printf("") FROM machines') + sqlite_has_printf = True +except: + pass + +def emit_to_hex(x): + if sqlite_has_printf: + return 'printf("%x", ' + x + ')' + else: + return x + +do_query(query, 'CREATE VIEW machines_view AS ' + 'SELECT ' + 'id,' + 'pid,' + 'root_dir,' + 'CASE WHEN id=0 THEN \'unknown\' WHEN pid=-1 THEN \'host\' ELSE \'guest\' END AS host_or_guest' + ' FROM machines') + +do_query(query, 'CREATE VIEW dsos_view AS ' + 'SELECT ' + 'id,' + 'machine_id,' + '(SELECT host_or_guest FROM machines_view WHERE id = machine_id) AS host_or_guest,' + 'short_name,' + 'long_name,' + 'build_id' + ' FROM dsos') + +do_query(query, 'CREATE VIEW symbols_view AS ' + 'SELECT ' + 'id,' + 'name,' + '(SELECT short_name FROM dsos WHERE id=dso_id) AS dso,' + 'dso_id,' + 'sym_start,' + 'sym_end,' + 'CASE WHEN binding=0 THEN \'local\' WHEN binding=1 THEN \'global\' ELSE \'weak\' END AS binding' + ' FROM symbols') + +do_query(query, 'CREATE VIEW threads_view AS ' + 'SELECT ' + 'id,' + 'machine_id,' + '(SELECT host_or_guest FROM machines_view WHERE id = machine_id) AS host_or_guest,' + 'process_id,' + 'pid,' + 'tid' + ' FROM threads') + +do_query(query, 'CREATE VIEW comm_threads_view AS ' + 'SELECT ' + 'comm_id,' + '(SELECT comm FROM comms WHERE id = comm_id) AS command,' + 'thread_id,' + '(SELECT pid FROM threads WHERE id = thread_id) AS pid,' + '(SELECT tid FROM threads WHERE id = thread_id) AS tid' + ' FROM comm_threads') + +if perf_db_export_calls or perf_db_export_callchains: + do_query(query, 'CREATE VIEW call_paths_view AS ' + 'SELECT ' + 'c.id,' + + emit_to_hex('c.ip') + ' AS ip,' + 'c.symbol_id,' + '(SELECT name FROM symbols WHERE id = c.symbol_id) AS symbol,' + '(SELECT dso_id FROM symbols WHERE id = c.symbol_id) AS dso_id,' + '(SELECT dso FROM symbols_view WHERE id = c.symbol_id) AS dso_short_name,' + 'c.parent_id,' + + emit_to_hex('p.ip') + ' AS parent_ip,' + 'p.symbol_id AS parent_symbol_id,' + '(SELECT name FROM symbols WHERE id = p.symbol_id) AS parent_symbol,' + '(SELECT dso_id FROM symbols WHERE id = p.symbol_id) AS parent_dso_id,' + '(SELECT dso FROM symbols_view WHERE id = p.symbol_id) AS parent_dso_short_name' + ' FROM call_paths c INNER JOIN call_paths p ON p.id = c.parent_id') +if perf_db_export_calls: + do_query(query, 'CREATE VIEW calls_view AS ' + 'SELECT ' + 'calls.id,' + 'thread_id,' + '(SELECT pid FROM threads WHERE id = thread_id) AS pid,' + '(SELECT tid FROM threads WHERE id = thread_id) AS tid,' + '(SELECT comm FROM comms WHERE id = comm_id) AS command,' + 'call_path_id,' + + emit_to_hex('ip') + ' AS ip,' + 'symbol_id,' + '(SELECT name FROM symbols WHERE id = symbol_id) AS symbol,' + 'call_time,' + 'return_time,' + 'return_time - call_time AS elapsed_time,' + 'branch_count,' + 'call_id,' + 'return_id,' + 'CASE WHEN flags=1 THEN \'no call\' WHEN flags=2 THEN \'no return\' WHEN flags=3 THEN \'no call/return\' ELSE \'\' END AS flags,' + 'parent_call_path_id' + ' FROM calls INNER JOIN call_paths ON call_paths.id = call_path_id') + +do_query(query, 'CREATE VIEW samples_view AS ' + 'SELECT ' + 'id,' + 'time,' + 'cpu,' + '(SELECT pid FROM threads WHERE id = thread_id) AS pid,' + '(SELECT tid FROM threads WHERE id = thread_id) AS tid,' + '(SELECT comm FROM comms WHERE id = comm_id) AS command,' + '(SELECT name FROM selected_events WHERE id = evsel_id) AS event,' + + emit_to_hex('ip') + ' AS ip_hex,' + '(SELECT name FROM symbols WHERE id = symbol_id) AS symbol,' + 'sym_offset,' + '(SELECT short_name FROM dsos WHERE id = dso_id) AS dso_short_name,' + + emit_to_hex('to_ip') + ' AS to_ip_hex,' + '(SELECT name FROM symbols WHERE id = to_symbol_id) AS to_symbol,' + 'to_sym_offset,' + '(SELECT short_name FROM dsos WHERE id = to_dso_id) AS to_dso_short_name,' + '(SELECT name FROM branch_types WHERE id = branch_type) AS branch_type_name,' + 'in_tx' + ' FROM samples') + +do_query(query, 'END TRANSACTION') + +evsel_query = QSqlQuery(db) +evsel_query.prepare("INSERT INTO selected_events VALUES (?, ?)") +machine_query = QSqlQuery(db) +machine_query.prepare("INSERT INTO machines VALUES (?, ?, ?)") +thread_query = QSqlQuery(db) +thread_query.prepare("INSERT INTO threads VALUES (?, ?, ?, ?, ?)") +comm_query = QSqlQuery(db) +comm_query.prepare("INSERT INTO comms VALUES (?, ?)") +comm_thread_query = QSqlQuery(db) +comm_thread_query.prepare("INSERT INTO comm_threads VALUES (?, ?, ?)") +dso_query = QSqlQuery(db) +dso_query.prepare("INSERT INTO dsos VALUES (?, ?, ?, ?, ?)") +symbol_query = QSqlQuery(db) +symbol_query.prepare("INSERT INTO symbols VALUES (?, ?, ?, ?, ?, ?)") +branch_type_query = QSqlQuery(db) +branch_type_query.prepare("INSERT INTO branch_types VALUES (?, ?)") +sample_query = QSqlQuery(db) +if branches: + sample_query.prepare("INSERT INTO samples VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)") +else: + sample_query.prepare("INSERT INTO samples VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)") +if perf_db_export_calls or perf_db_export_callchains: + call_path_query = QSqlQuery(db) + call_path_query.prepare("INSERT INTO call_paths VALUES (?, ?, ?, ?)") +if perf_db_export_calls: + call_query = QSqlQuery(db) + call_query.prepare("INSERT INTO calls VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)") + +def trace_begin(): + print datetime.datetime.today(), "Writing records..." + do_query(query, 'BEGIN TRANSACTION') + # id == 0 means unknown. It is easier to create records for them than replace the zeroes with NULLs + evsel_table(0, "unknown") + machine_table(0, 0, "unknown") + thread_table(0, 0, 0, -1, -1) + comm_table(0, "unknown") + dso_table(0, 0, "unknown", "unknown", "") + symbol_table(0, 0, 0, 0, 0, "unknown") + sample_table(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0) + if perf_db_export_calls or perf_db_export_callchains: + call_path_table(0, 0, 0, 0) + +unhandled_count = 0 + +def trace_end(): + do_query(query, 'END TRANSACTION') + + print datetime.datetime.today(), "Adding indexes" + if perf_db_export_calls: + do_query(query, 'CREATE INDEX pcpid_idx ON calls (parent_call_path_id)') + + if (unhandled_count): + print datetime.datetime.today(), "Warning: ", unhandled_count, " unhandled events" + print datetime.datetime.today(), "Done" + +def trace_unhandled(event_name, context, event_fields_dict): + global unhandled_count + unhandled_count += 1 + +def sched__sched_switch(*x): + pass + +def bind_exec(q, n, x): + for xx in x[0:n]: + q.addBindValue(str(xx)) + do_query_(q) + +def evsel_table(*x): + bind_exec(evsel_query, 2, x) + +def machine_table(*x): + bind_exec(machine_query, 3, x) + +def thread_table(*x): + bind_exec(thread_query, 5, x) + +def comm_table(*x): + bind_exec(comm_query, 2, x) + +def comm_thread_table(*x): + bind_exec(comm_thread_query, 3, x) + +def dso_table(*x): + bind_exec(dso_query, 5, x) + +def symbol_table(*x): + bind_exec(symbol_query, 6, x) + +def branch_type_table(*x): + bind_exec(branch_type_query, 2, x) + +def sample_table(*x): + if branches: + bind_exec(sample_query, 18, x) + else: + bind_exec(sample_query, 22, x) + +def call_path_table(*x): + bind_exec(call_path_query, 4, x) + +def call_return_table(*x): + bind_exec(call_query, 11, x) ^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH 4/5] perf script: Rename call-graph-from-postgresql.py to call-graph-from-sql.py 2017-08-03 8:31 [PATCH 0/5] perf script: Add support for exporting to sqlite3 Adrian Hunter ` (2 preceding siblings ...) 2017-08-03 8:31 ` [PATCH 3/5] perf script: Add support for exporting to sqlite3 Adrian Hunter @ 2017-08-03 8:31 ` Adrian Hunter 2017-08-17 7:51 ` [tip:perf/core] perf script python: " tip-bot for Adrian Hunter 2017-08-03 8:31 ` [PATCH 5/5] perf script: Add support for sqlite3 " Adrian Hunter 2017-08-15 9:25 ` [PATCH 0/5] perf script: Add support for exporting to sqlite3 Adrian Hunter 5 siblings, 1 reply; 13+ messages in thread From: Adrian Hunter @ 2017-08-03 8:31 UTC (permalink / raw) To: Arnaldo Carvalho de Melo; +Cc: linux-kernel Rename call-graph-from-postgresql.py to call-graph-from-sql.py in preparation for adding support to it for SQLite 3. Signed-off-by: Adrian Hunter <adrian.hunter@intel.com> --- tools/perf/Documentation/intel-pt.txt | 2 +- .../scripts/python/call-graph-from-postgresql.py | 327 --------------------- tools/perf/scripts/python/call-graph-from-sql.py | 327 +++++++++++++++++++++ tools/perf/scripts/python/export-to-postgresql.py | 2 +- 4 files changed, 329 insertions(+), 329 deletions(-) delete mode 100644 tools/perf/scripts/python/call-graph-from-postgresql.py create mode 100644 tools/perf/scripts/python/call-graph-from-sql.py diff --git a/tools/perf/Documentation/intel-pt.txt b/tools/perf/Documentation/intel-pt.txt index 8e8ae3ad4cbf..ab1b0825130a 100644 --- a/tools/perf/Documentation/intel-pt.txt +++ b/tools/perf/Documentation/intel-pt.txt @@ -106,7 +106,7 @@ in transaction, respectively. While it is possible to create scripts to analyze the data, an alternative approach is available to export the data to a sqlite or postgresql database. Refer to script export-to-sqlite.py or export-to-postgresql.py for more details, -and to script call-graph-from-postgresql.py for an example of using the database. +and to script call-graph-from-sql.py for an example of using the database. There is also script intel-pt-events.py which provides an example of how to unpack the raw data for power events and PTWRITE. diff --git a/tools/perf/scripts/python/call-graph-from-postgresql.py b/tools/perf/scripts/python/call-graph-from-postgresql.py deleted file mode 100644 index ed9f7f3ccf22..000000000000 --- a/tools/perf/scripts/python/call-graph-from-postgresql.py +++ /dev/null @@ -1,327 +0,0 @@ -#!/usr/bin/python2 -# call-graph-from-postgresql.py: create call-graph from postgresql database -# Copyright (c) 2014, Intel Corporation. -# -# This program is free software; you can redistribute it and/or modify it -# under the terms and conditions of the GNU General Public License, -# version 2, as published by the Free Software Foundation. -# -# This program is distributed in the hope it will be useful, but WITHOUT -# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or -# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for -# more details. - -# To use this script you will need to have exported data using the -# export-to-postgresql.py script. Refer to that script for details. -# -# Following on from the example in the export-to-postgresql.py script, a -# call-graph can be displayed for the pt_example database like this: -# -# python tools/perf/scripts/python/call-graph-from-postgresql.py pt_example -# -# Note this script supports connecting to remote databases by setting hostname, -# port, username, password, and dbname e.g. -# -# python tools/perf/scripts/python/call-graph-from-postgresql.py "hostname=myhost username=myuser password=mypassword dbname=pt_example" -# -# The result is a GUI window with a tree representing a context-sensitive -# call-graph. Expanding a couple of levels of the tree and adjusting column -# widths to suit will display something like: -# -# Call Graph: pt_example -# Call Path Object Count Time(ns) Time(%) Branch Count Branch Count(%) -# v- ls -# v- 2638:2638 -# v- _start ld-2.19.so 1 10074071 100.0 211135 100.0 -# |- unknown unknown 1 13198 0.1 1 0.0 -# >- _dl_start ld-2.19.so 1 1400980 13.9 19637 9.3 -# >- _d_linit_internal ld-2.19.so 1 448152 4.4 11094 5.3 -# v-__libc_start_main@plt ls 1 8211741 81.5 180397 85.4 -# >- _dl_fixup ld-2.19.so 1 7607 0.1 108 0.1 -# >- __cxa_atexit libc-2.19.so 1 11737 0.1 10 0.0 -# >- __libc_csu_init ls 1 10354 0.1 10 0.0 -# |- _setjmp libc-2.19.so 1 0 0.0 4 0.0 -# v- main ls 1 8182043 99.6 180254 99.9 -# -# Points to note: -# The top level is a command name (comm) -# The next level is a thread (pid:tid) -# Subsequent levels are functions -# 'Count' is the number of calls -# 'Time' is the elapsed time until the function returns -# Percentages are relative to the level above -# 'Branch Count' is the total number of branches for that function and all -# functions that it calls - -import sys -from PySide.QtCore import * -from PySide.QtGui import * -from PySide.QtSql import * -from decimal import * - -class TreeItem(): - - def __init__(self, db, row, parent_item): - self.db = db - self.row = row - self.parent_item = parent_item - self.query_done = False; - self.child_count = 0 - self.child_items = [] - self.data = ["", "", "", "", "", "", ""] - self.comm_id = 0 - self.thread_id = 0 - self.call_path_id = 1 - self.branch_count = 0 - self.time = 0 - if not parent_item: - self.setUpRoot() - - def setUpRoot(self): - self.query_done = True - query = QSqlQuery(self.db) - ret = query.exec_('SELECT id, comm FROM comms') - if not ret: - raise Exception("Query failed: " + query.lastError().text()) - while query.next(): - if not query.value(0): - continue - child_item = TreeItem(self.db, self.child_count, self) - self.child_items.append(child_item) - self.child_count += 1 - child_item.setUpLevel1(query.value(0), query.value(1)) - - def setUpLevel1(self, comm_id, comm): - self.query_done = True; - self.comm_id = comm_id - self.data[0] = comm - self.child_items = [] - self.child_count = 0 - query = QSqlQuery(self.db) - ret = query.exec_('SELECT thread_id, ( SELECT pid FROM threads WHERE id = thread_id ), ( SELECT tid FROM threads WHERE id = thread_id ) FROM comm_threads WHERE comm_id = ' + str(comm_id)) - if not ret: - raise Exception("Query failed: " + query.lastError().text()) - while query.next(): - child_item = TreeItem(self.db, self.child_count, self) - self.child_items.append(child_item) - self.child_count += 1 - child_item.setUpLevel2(comm_id, query.value(0), query.value(1), query.value(2)) - - def setUpLevel2(self, comm_id, thread_id, pid, tid): - self.comm_id = comm_id - self.thread_id = thread_id - self.data[0] = str(pid) + ":" + str(tid) - - def getChildItem(self, row): - return self.child_items[row] - - def getParentItem(self): - return self.parent_item - - def getRow(self): - return self.row - - def timePercent(self, b): - if not self.time: - return "0.0" - x = (b * Decimal(100)) / self.time - return str(x.quantize(Decimal('.1'), rounding=ROUND_HALF_UP)) - - def branchPercent(self, b): - if not self.branch_count: - return "0.0" - x = (b * Decimal(100)) / self.branch_count - return str(x.quantize(Decimal('.1'), rounding=ROUND_HALF_UP)) - - def addChild(self, call_path_id, name, dso, count, time, branch_count): - child_item = TreeItem(self.db, self.child_count, self) - child_item.comm_id = self.comm_id - child_item.thread_id = self.thread_id - child_item.call_path_id = call_path_id - child_item.branch_count = branch_count - child_item.time = time - child_item.data[0] = name - if dso == "[kernel.kallsyms]": - dso = "[kernel]" - child_item.data[1] = dso - child_item.data[2] = str(count) - child_item.data[3] = str(time) - child_item.data[4] = self.timePercent(time) - child_item.data[5] = str(branch_count) - child_item.data[6] = self.branchPercent(branch_count) - self.child_items.append(child_item) - self.child_count += 1 - - def selectCalls(self): - self.query_done = True; - query = QSqlQuery(self.db) - ret = query.exec_('SELECT id, call_path_id, branch_count, call_time, return_time, ' - '( SELECT name FROM symbols WHERE id = ( SELECT symbol_id FROM call_paths WHERE id = call_path_id ) ), ' - '( SELECT short_name FROM dsos WHERE id = ( SELECT dso_id FROM symbols WHERE id = ( SELECT symbol_id FROM call_paths WHERE id = call_path_id ) ) ), ' - '( SELECT ip FROM call_paths where id = call_path_id ) ' - 'FROM calls WHERE parent_call_path_id = ' + str(self.call_path_id) + ' AND comm_id = ' + str(self.comm_id) + ' AND thread_id = ' + str(self.thread_id) + - ' ORDER BY call_path_id') - if not ret: - raise Exception("Query failed: " + query.lastError().text()) - last_call_path_id = 0 - name = "" - dso = "" - count = 0 - branch_count = 0 - total_branch_count = 0 - time = 0 - total_time = 0 - while query.next(): - if query.value(1) == last_call_path_id: - count += 1 - branch_count += query.value(2) - time += query.value(4) - query.value(3) - else: - if count: - self.addChild(last_call_path_id, name, dso, count, time, branch_count) - last_call_path_id = query.value(1) - name = query.value(5) - dso = query.value(6) - count = 1 - total_branch_count += branch_count - total_time += time - branch_count = query.value(2) - time = query.value(4) - query.value(3) - if count: - self.addChild(last_call_path_id, name, dso, count, time, branch_count) - total_branch_count += branch_count - total_time += time - # Top level does not have time or branch count, so fix that here - if total_branch_count > self.branch_count: - self.branch_count = total_branch_count - if self.branch_count: - for child_item in self.child_items: - child_item.data[6] = self.branchPercent(child_item.branch_count) - if total_time > self.time: - self.time = total_time - if self.time: - for child_item in self.child_items: - child_item.data[4] = self.timePercent(child_item.time) - - def childCount(self): - if not self.query_done: - self.selectCalls() - return self.child_count - - def columnCount(self): - return 7 - - def columnHeader(self, column): - headers = ["Call Path", "Object", "Count ", "Time (ns) ", "Time (%) ", "Branch Count ", "Branch Count (%) "] - return headers[column] - - def getData(self, column): - return self.data[column] - -class TreeModel(QAbstractItemModel): - - def __init__(self, db, parent=None): - super(TreeModel, self).__init__(parent) - self.db = db - self.root = TreeItem(db, 0, None) - - def columnCount(self, parent): - return self.root.columnCount() - - def rowCount(self, parent): - if parent.isValid(): - parent_item = parent.internalPointer() - else: - parent_item = self.root - return parent_item.childCount() - - def headerData(self, section, orientation, role): - if role == Qt.TextAlignmentRole: - if section > 1: - return Qt.AlignRight - if role != Qt.DisplayRole: - return None - if orientation != Qt.Horizontal: - return None - return self.root.columnHeader(section) - - def parent(self, child): - child_item = child.internalPointer() - if child_item is self.root: - return QModelIndex() - parent_item = child_item.getParentItem() - return self.createIndex(parent_item.getRow(), 0, parent_item) - - def index(self, row, column, parent): - if parent.isValid(): - parent_item = parent.internalPointer() - else: - parent_item = self.root - child_item = parent_item.getChildItem(row) - return self.createIndex(row, column, child_item) - - def data(self, index, role): - if role == Qt.TextAlignmentRole: - if index.column() > 1: - return Qt.AlignRight - if role != Qt.DisplayRole: - return None - index_item = index.internalPointer() - return index_item.getData(index.column()) - -class MainWindow(QMainWindow): - - def __init__(self, db, dbname, parent=None): - super(MainWindow, self).__init__(parent) - - self.setObjectName("MainWindow") - self.setWindowTitle("Call Graph: " + dbname) - self.move(100, 100) - self.resize(800, 600) - style = self.style() - icon = style.standardIcon(QStyle.SP_MessageBoxInformation) - self.setWindowIcon(icon); - - self.model = TreeModel(db) - - self.view = QTreeView() - self.view.setModel(self.model) - - self.setCentralWidget(self.view) - -if __name__ == '__main__': - if (len(sys.argv) < 2): - print >> sys.stderr, "Usage is: call-graph-from-postgresql.py <database name>" - raise Exception("Too few arguments") - - dbname = sys.argv[1] - - db = QSqlDatabase.addDatabase('QPSQL') - - opts = dbname.split() - for opt in opts: - if '=' in opt: - opt = opt.split('=') - if opt[0] == 'hostname': - db.setHostName(opt[1]) - elif opt[0] == 'port': - db.setPort(int(opt[1])) - elif opt[0] == 'username': - db.setUserName(opt[1]) - elif opt[0] == 'password': - db.setPassword(opt[1]) - elif opt[0] == 'dbname': - dbname = opt[1] - else: - dbname = opt - - db.setDatabaseName(dbname) - if not db.open(): - raise Exception("Failed to open database " + dbname + " error: " + db.lastError().text()) - - app = QApplication(sys.argv) - window = MainWindow(db, dbname) - window.show() - err = app.exec_() - db.close() - sys.exit(err) diff --git a/tools/perf/scripts/python/call-graph-from-sql.py b/tools/perf/scripts/python/call-graph-from-sql.py new file mode 100644 index 000000000000..f18406d3faf7 --- /dev/null +++ b/tools/perf/scripts/python/call-graph-from-sql.py @@ -0,0 +1,327 @@ +#!/usr/bin/python2 +# call-graph-from-sql.py: create call-graph from postgresql database +# Copyright (c) 2014-2017, Intel Corporation. +# +# This program is free software; you can redistribute it and/or modify it +# under the terms and conditions of the GNU General Public License, +# version 2, as published by the Free Software Foundation. +# +# This program is distributed in the hope it will be useful, but WITHOUT +# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or +# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for +# more details. + +# To use this script you will need to have exported data using the +# export-to-postgresql.py script. Refer to that script for details. +# +# Following on from the example in the export-to-postgresql.py script, a +# call-graph can be displayed for the pt_example database like this: +# +# python tools/perf/scripts/python/call-graph-from-sql.py pt_example +# +# Note this script supports connecting to remote databases by setting hostname, +# port, username, password, and dbname e.g. +# +# python tools/perf/scripts/python/call-graph-from-sql.py "hostname=myhost username=myuser password=mypassword dbname=pt_example" +# +# The result is a GUI window with a tree representing a context-sensitive +# call-graph. Expanding a couple of levels of the tree and adjusting column +# widths to suit will display something like: +# +# Call Graph: pt_example +# Call Path Object Count Time(ns) Time(%) Branch Count Branch Count(%) +# v- ls +# v- 2638:2638 +# v- _start ld-2.19.so 1 10074071 100.0 211135 100.0 +# |- unknown unknown 1 13198 0.1 1 0.0 +# >- _dl_start ld-2.19.so 1 1400980 13.9 19637 9.3 +# >- _d_linit_internal ld-2.19.so 1 448152 4.4 11094 5.3 +# v-__libc_start_main@plt ls 1 8211741 81.5 180397 85.4 +# >- _dl_fixup ld-2.19.so 1 7607 0.1 108 0.1 +# >- __cxa_atexit libc-2.19.so 1 11737 0.1 10 0.0 +# >- __libc_csu_init ls 1 10354 0.1 10 0.0 +# |- _setjmp libc-2.19.so 1 0 0.0 4 0.0 +# v- main ls 1 8182043 99.6 180254 99.9 +# +# Points to note: +# The top level is a command name (comm) +# The next level is a thread (pid:tid) +# Subsequent levels are functions +# 'Count' is the number of calls +# 'Time' is the elapsed time until the function returns +# Percentages are relative to the level above +# 'Branch Count' is the total number of branches for that function and all +# functions that it calls + +import sys +from PySide.QtCore import * +from PySide.QtGui import * +from PySide.QtSql import * +from decimal import * + +class TreeItem(): + + def __init__(self, db, row, parent_item): + self.db = db + self.row = row + self.parent_item = parent_item + self.query_done = False; + self.child_count = 0 + self.child_items = [] + self.data = ["", "", "", "", "", "", ""] + self.comm_id = 0 + self.thread_id = 0 + self.call_path_id = 1 + self.branch_count = 0 + self.time = 0 + if not parent_item: + self.setUpRoot() + + def setUpRoot(self): + self.query_done = True + query = QSqlQuery(self.db) + ret = query.exec_('SELECT id, comm FROM comms') + if not ret: + raise Exception("Query failed: " + query.lastError().text()) + while query.next(): + if not query.value(0): + continue + child_item = TreeItem(self.db, self.child_count, self) + self.child_items.append(child_item) + self.child_count += 1 + child_item.setUpLevel1(query.value(0), query.value(1)) + + def setUpLevel1(self, comm_id, comm): + self.query_done = True; + self.comm_id = comm_id + self.data[0] = comm + self.child_items = [] + self.child_count = 0 + query = QSqlQuery(self.db) + ret = query.exec_('SELECT thread_id, ( SELECT pid FROM threads WHERE id = thread_id ), ( SELECT tid FROM threads WHERE id = thread_id ) FROM comm_threads WHERE comm_id = ' + str(comm_id)) + if not ret: + raise Exception("Query failed: " + query.lastError().text()) + while query.next(): + child_item = TreeItem(self.db, self.child_count, self) + self.child_items.append(child_item) + self.child_count += 1 + child_item.setUpLevel2(comm_id, query.value(0), query.value(1), query.value(2)) + + def setUpLevel2(self, comm_id, thread_id, pid, tid): + self.comm_id = comm_id + self.thread_id = thread_id + self.data[0] = str(pid) + ":" + str(tid) + + def getChildItem(self, row): + return self.child_items[row] + + def getParentItem(self): + return self.parent_item + + def getRow(self): + return self.row + + def timePercent(self, b): + if not self.time: + return "0.0" + x = (b * Decimal(100)) / self.time + return str(x.quantize(Decimal('.1'), rounding=ROUND_HALF_UP)) + + def branchPercent(self, b): + if not self.branch_count: + return "0.0" + x = (b * Decimal(100)) / self.branch_count + return str(x.quantize(Decimal('.1'), rounding=ROUND_HALF_UP)) + + def addChild(self, call_path_id, name, dso, count, time, branch_count): + child_item = TreeItem(self.db, self.child_count, self) + child_item.comm_id = self.comm_id + child_item.thread_id = self.thread_id + child_item.call_path_id = call_path_id + child_item.branch_count = branch_count + child_item.time = time + child_item.data[0] = name + if dso == "[kernel.kallsyms]": + dso = "[kernel]" + child_item.data[1] = dso + child_item.data[2] = str(count) + child_item.data[3] = str(time) + child_item.data[4] = self.timePercent(time) + child_item.data[5] = str(branch_count) + child_item.data[6] = self.branchPercent(branch_count) + self.child_items.append(child_item) + self.child_count += 1 + + def selectCalls(self): + self.query_done = True; + query = QSqlQuery(self.db) + ret = query.exec_('SELECT id, call_path_id, branch_count, call_time, return_time, ' + '( SELECT name FROM symbols WHERE id = ( SELECT symbol_id FROM call_paths WHERE id = call_path_id ) ), ' + '( SELECT short_name FROM dsos WHERE id = ( SELECT dso_id FROM symbols WHERE id = ( SELECT symbol_id FROM call_paths WHERE id = call_path_id ) ) ), ' + '( SELECT ip FROM call_paths where id = call_path_id ) ' + 'FROM calls WHERE parent_call_path_id = ' + str(self.call_path_id) + ' AND comm_id = ' + str(self.comm_id) + ' AND thread_id = ' + str(self.thread_id) + + ' ORDER BY call_path_id') + if not ret: + raise Exception("Query failed: " + query.lastError().text()) + last_call_path_id = 0 + name = "" + dso = "" + count = 0 + branch_count = 0 + total_branch_count = 0 + time = 0 + total_time = 0 + while query.next(): + if query.value(1) == last_call_path_id: + count += 1 + branch_count += query.value(2) + time += query.value(4) - query.value(3) + else: + if count: + self.addChild(last_call_path_id, name, dso, count, time, branch_count) + last_call_path_id = query.value(1) + name = query.value(5) + dso = query.value(6) + count = 1 + total_branch_count += branch_count + total_time += time + branch_count = query.value(2) + time = query.value(4) - query.value(3) + if count: + self.addChild(last_call_path_id, name, dso, count, time, branch_count) + total_branch_count += branch_count + total_time += time + # Top level does not have time or branch count, so fix that here + if total_branch_count > self.branch_count: + self.branch_count = total_branch_count + if self.branch_count: + for child_item in self.child_items: + child_item.data[6] = self.branchPercent(child_item.branch_count) + if total_time > self.time: + self.time = total_time + if self.time: + for child_item in self.child_items: + child_item.data[4] = self.timePercent(child_item.time) + + def childCount(self): + if not self.query_done: + self.selectCalls() + return self.child_count + + def columnCount(self): + return 7 + + def columnHeader(self, column): + headers = ["Call Path", "Object", "Count ", "Time (ns) ", "Time (%) ", "Branch Count ", "Branch Count (%) "] + return headers[column] + + def getData(self, column): + return self.data[column] + +class TreeModel(QAbstractItemModel): + + def __init__(self, db, parent=None): + super(TreeModel, self).__init__(parent) + self.db = db + self.root = TreeItem(db, 0, None) + + def columnCount(self, parent): + return self.root.columnCount() + + def rowCount(self, parent): + if parent.isValid(): + parent_item = parent.internalPointer() + else: + parent_item = self.root + return parent_item.childCount() + + def headerData(self, section, orientation, role): + if role == Qt.TextAlignmentRole: + if section > 1: + return Qt.AlignRight + if role != Qt.DisplayRole: + return None + if orientation != Qt.Horizontal: + return None + return self.root.columnHeader(section) + + def parent(self, child): + child_item = child.internalPointer() + if child_item is self.root: + return QModelIndex() + parent_item = child_item.getParentItem() + return self.createIndex(parent_item.getRow(), 0, parent_item) + + def index(self, row, column, parent): + if parent.isValid(): + parent_item = parent.internalPointer() + else: + parent_item = self.root + child_item = parent_item.getChildItem(row) + return self.createIndex(row, column, child_item) + + def data(self, index, role): + if role == Qt.TextAlignmentRole: + if index.column() > 1: + return Qt.AlignRight + if role != Qt.DisplayRole: + return None + index_item = index.internalPointer() + return index_item.getData(index.column()) + +class MainWindow(QMainWindow): + + def __init__(self, db, dbname, parent=None): + super(MainWindow, self).__init__(parent) + + self.setObjectName("MainWindow") + self.setWindowTitle("Call Graph: " + dbname) + self.move(100, 100) + self.resize(800, 600) + style = self.style() + icon = style.standardIcon(QStyle.SP_MessageBoxInformation) + self.setWindowIcon(icon); + + self.model = TreeModel(db) + + self.view = QTreeView() + self.view.setModel(self.model) + + self.setCentralWidget(self.view) + +if __name__ == '__main__': + if (len(sys.argv) < 2): + print >> sys.stderr, "Usage is: call-graph-from-sql.py <database name>" + raise Exception("Too few arguments") + + dbname = sys.argv[1] + + db = QSqlDatabase.addDatabase('QPSQL') + + opts = dbname.split() + for opt in opts: + if '=' in opt: + opt = opt.split('=') + if opt[0] == 'hostname': + db.setHostName(opt[1]) + elif opt[0] == 'port': + db.setPort(int(opt[1])) + elif opt[0] == 'username': + db.setUserName(opt[1]) + elif opt[0] == 'password': + db.setPassword(opt[1]) + elif opt[0] == 'dbname': + dbname = opt[1] + else: + dbname = opt + + db.setDatabaseName(dbname) + if not db.open(): + raise Exception("Failed to open database " + dbname + " error: " + db.lastError().text()) + + app = QApplication(sys.argv) + window = MainWindow(db, dbname) + window.show() + err = app.exec_() + db.close() + sys.exit(err) diff --git a/tools/perf/scripts/python/export-to-postgresql.py b/tools/perf/scripts/python/export-to-postgresql.py index f57811443beb..efcaf6cac2eb 100644 --- a/tools/perf/scripts/python/export-to-postgresql.py +++ b/tools/perf/scripts/python/export-to-postgresql.py @@ -59,7 +59,7 @@ import datetime # pt_example=# \q # # An example of using the database is provided by the script -# call-graph-from-postgresql.py. Refer to that script for details. +# call-graph-from-sql.py. Refer to that script for details. # # Tables: # -- 1.9.1 ^ permalink raw reply related [flat|nested] 13+ messages in thread
* [tip:perf/core] perf script python: Rename call-graph-from-postgresql.py to call-graph-from-sql.py 2017-08-03 8:31 ` [PATCH 4/5] perf script: Rename call-graph-from-postgresql.py to call-graph-from-sql.py Adrian Hunter @ 2017-08-17 7:51 ` tip-bot for Adrian Hunter 0 siblings, 0 replies; 13+ messages in thread From: tip-bot for Adrian Hunter @ 2017-08-17 7:51 UTC (permalink / raw) To: linux-tip-commits; +Cc: linux-kernel, acme, adrian.hunter, mingo, hpa, tglx Commit-ID: 69e6e410f1a1e69cb656e8ebddaae0ba2138b235 Gitweb: http://git.kernel.org/tip/69e6e410f1a1e69cb656e8ebddaae0ba2138b235 Author: Adrian Hunter <adrian.hunter@intel.com> AuthorDate: Thu, 3 Aug 2017 11:31:29 +0300 Committer: Arnaldo Carvalho de Melo <acme@redhat.com> CommitDate: Tue, 15 Aug 2017 16:38:06 -0300 perf script python: Rename call-graph-from-postgresql.py to call-graph-from-sql.py Rename call-graph-from-postgresql.py to call-graph-from-sql.py in preparation for adding support to it for SQLite 3. Signed-off-by: Adrian Hunter <adrian.hunter@intel.com> Link: http://lkml.kernel.org/r/1501749090-20357-5-git-send-email-adrian.hunter@intel.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com> --- tools/perf/Documentation/intel-pt.txt | 2 +- .../{call-graph-from-postgresql.py => call-graph-from-sql.py} | 10 +++++----- tools/perf/scripts/python/export-to-postgresql.py | 2 +- 3 files changed, 7 insertions(+), 7 deletions(-) diff --git a/tools/perf/Documentation/intel-pt.txt b/tools/perf/Documentation/intel-pt.txt index 8e8ae3a..ab1b082 100644 --- a/tools/perf/Documentation/intel-pt.txt +++ b/tools/perf/Documentation/intel-pt.txt @@ -106,7 +106,7 @@ in transaction, respectively. While it is possible to create scripts to analyze the data, an alternative approach is available to export the data to a sqlite or postgresql database. Refer to script export-to-sqlite.py or export-to-postgresql.py for more details, -and to script call-graph-from-postgresql.py for an example of using the database. +and to script call-graph-from-sql.py for an example of using the database. There is also script intel-pt-events.py which provides an example of how to unpack the raw data for power events and PTWRITE. diff --git a/tools/perf/scripts/python/call-graph-from-postgresql.py b/tools/perf/scripts/python/call-graph-from-sql.py similarity index 96% rename from tools/perf/scripts/python/call-graph-from-postgresql.py rename to tools/perf/scripts/python/call-graph-from-sql.py index ed9f7f3..f18406d 100644 --- a/tools/perf/scripts/python/call-graph-from-postgresql.py +++ b/tools/perf/scripts/python/call-graph-from-sql.py @@ -1,6 +1,6 @@ #!/usr/bin/python2 -# call-graph-from-postgresql.py: create call-graph from postgresql database -# Copyright (c) 2014, Intel Corporation. +# call-graph-from-sql.py: create call-graph from postgresql database +# Copyright (c) 2014-2017, Intel Corporation. # # This program is free software; you can redistribute it and/or modify it # under the terms and conditions of the GNU General Public License, @@ -17,12 +17,12 @@ # Following on from the example in the export-to-postgresql.py script, a # call-graph can be displayed for the pt_example database like this: # -# python tools/perf/scripts/python/call-graph-from-postgresql.py pt_example +# python tools/perf/scripts/python/call-graph-from-sql.py pt_example # # Note this script supports connecting to remote databases by setting hostname, # port, username, password, and dbname e.g. # -# python tools/perf/scripts/python/call-graph-from-postgresql.py "hostname=myhost username=myuser password=mypassword dbname=pt_example" +# python tools/perf/scripts/python/call-graph-from-sql.py "hostname=myhost username=myuser password=mypassword dbname=pt_example" # # The result is a GUI window with a tree representing a context-sensitive # call-graph. Expanding a couple of levels of the tree and adjusting column @@ -291,7 +291,7 @@ class MainWindow(QMainWindow): if __name__ == '__main__': if (len(sys.argv) < 2): - print >> sys.stderr, "Usage is: call-graph-from-postgresql.py <database name>" + print >> sys.stderr, "Usage is: call-graph-from-sql.py <database name>" raise Exception("Too few arguments") dbname = sys.argv[1] diff --git a/tools/perf/scripts/python/export-to-postgresql.py b/tools/perf/scripts/python/export-to-postgresql.py index f578114..efcaf6c 100644 --- a/tools/perf/scripts/python/export-to-postgresql.py +++ b/tools/perf/scripts/python/export-to-postgresql.py @@ -59,7 +59,7 @@ import datetime # pt_example=# \q # # An example of using the database is provided by the script -# call-graph-from-postgresql.py. Refer to that script for details. +# call-graph-from-sql.py. Refer to that script for details. # # Tables: # ^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH 5/5] perf script: Add support for sqlite3 to call-graph-from-sql.py 2017-08-03 8:31 [PATCH 0/5] perf script: Add support for exporting to sqlite3 Adrian Hunter ` (3 preceding siblings ...) 2017-08-03 8:31 ` [PATCH 4/5] perf script: Rename call-graph-from-postgresql.py to call-graph-from-sql.py Adrian Hunter @ 2017-08-03 8:31 ` Adrian Hunter 2017-08-17 7:51 ` [tip:perf/core] perf script python: " tip-bot for Adrian Hunter 2017-08-15 9:25 ` [PATCH 0/5] perf script: Add support for exporting to sqlite3 Adrian Hunter 5 siblings, 1 reply; 13+ messages in thread From: Adrian Hunter @ 2017-08-03 8:31 UTC (permalink / raw) To: Arnaldo Carvalho de Melo; +Cc: linux-kernel Add support for SQLite 3 to the call-graph-from-sql.py script. The SQL statements work as is, so just detect the database type by checking if the SQLite 3 file exists. Signed-off-by: Adrian Hunter <adrian.hunter@intel.com> --- tools/perf/scripts/python/call-graph-from-sql.py | 60 ++++++++++++++---------- 1 file changed, 36 insertions(+), 24 deletions(-) diff --git a/tools/perf/scripts/python/call-graph-from-sql.py b/tools/perf/scripts/python/call-graph-from-sql.py index f18406d3faf7..b494a67a1c67 100644 --- a/tools/perf/scripts/python/call-graph-from-sql.py +++ b/tools/perf/scripts/python/call-graph-from-sql.py @@ -1,5 +1,5 @@ #!/usr/bin/python2 -# call-graph-from-sql.py: create call-graph from postgresql database +# call-graph-from-sql.py: create call-graph from sql database # Copyright (c) 2014-2017, Intel Corporation. # # This program is free software; you can redistribute it and/or modify it @@ -11,16 +11,17 @@ # FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for # more details. -# To use this script you will need to have exported data using the -# export-to-postgresql.py script. Refer to that script for details. +# To use this script you will need to have exported data using either the +# export-to-sqlite.py or the export-to-postgresql.py script. Refer to those +# scripts for details. # -# Following on from the example in the export-to-postgresql.py script, a +# Following on from the example in the export scripts, a # call-graph can be displayed for the pt_example database like this: # # python tools/perf/scripts/python/call-graph-from-sql.py pt_example # -# Note this script supports connecting to remote databases by setting hostname, -# port, username, password, and dbname e.g. +# Note that for PostgreSQL, this script supports connecting to remote databases +# by setting hostname, port, username, password, and dbname e.g. # # python tools/perf/scripts/python/call-graph-from-sql.py "hostname=myhost username=myuser password=mypassword dbname=pt_example" # @@ -296,24 +297,35 @@ if __name__ == '__main__': dbname = sys.argv[1] - db = QSqlDatabase.addDatabase('QPSQL') - - opts = dbname.split() - for opt in opts: - if '=' in opt: - opt = opt.split('=') - if opt[0] == 'hostname': - db.setHostName(opt[1]) - elif opt[0] == 'port': - db.setPort(int(opt[1])) - elif opt[0] == 'username': - db.setUserName(opt[1]) - elif opt[0] == 'password': - db.setPassword(opt[1]) - elif opt[0] == 'dbname': - dbname = opt[1] - else: - dbname = opt + is_sqlite3 = False + try: + f = open(dbname) + if f.read(15) == "SQLite format 3": + is_sqlite3 = True + f.close() + except: + pass + + if is_sqlite3: + db = QSqlDatabase.addDatabase('QSQLITE') + else: + db = QSqlDatabase.addDatabase('QPSQL') + opts = dbname.split() + for opt in opts: + if '=' in opt: + opt = opt.split('=') + if opt[0] == 'hostname': + db.setHostName(opt[1]) + elif opt[0] == 'port': + db.setPort(int(opt[1])) + elif opt[0] == 'username': + db.setUserName(opt[1]) + elif opt[0] == 'password': + db.setPassword(opt[1]) + elif opt[0] == 'dbname': + dbname = opt[1] + else: + dbname = opt db.setDatabaseName(dbname) if not db.open(): -- 1.9.1 ^ permalink raw reply related [flat|nested] 13+ messages in thread
* [tip:perf/core] perf script python: Add support for sqlite3 to call-graph-from-sql.py 2017-08-03 8:31 ` [PATCH 5/5] perf script: Add support for sqlite3 " Adrian Hunter @ 2017-08-17 7:51 ` tip-bot for Adrian Hunter 0 siblings, 0 replies; 13+ messages in thread From: tip-bot for Adrian Hunter @ 2017-08-17 7:51 UTC (permalink / raw) To: linux-tip-commits; +Cc: mingo, acme, linux-kernel, adrian.hunter, hpa, tglx Commit-ID: 1fe03b5f2d70b48c4a10785edf2678ff05506e59 Gitweb: http://git.kernel.org/tip/1fe03b5f2d70b48c4a10785edf2678ff05506e59 Author: Adrian Hunter <adrian.hunter@intel.com> AuthorDate: Thu, 3 Aug 2017 11:31:30 +0300 Committer: Arnaldo Carvalho de Melo <acme@redhat.com> CommitDate: Tue, 15 Aug 2017 17:03:38 -0300 perf script python: Add support for sqlite3 to call-graph-from-sql.py Add support for SQLite 3 to the call-graph-from-sql.py script. The SQL statements work as is, so just detect the database type by checking if the SQLite 3 file exists. Committer notes: Tested collecting the PT data on a RHEL7.4, generating the SQLite3 database there and then moving it to a Fedora 26 system where the call-graph-from-sql.py script was run, using python-pyside version 1.2.2-7fc26 to see the callgraphs using Qt4. Signed-off-by: Adrian Hunter <adrian.hunter@intel.com> Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com> Link: http://lkml.kernel.org/r/1501749090-20357-6-git-send-email-adrian.hunter@intel.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com> --- tools/perf/scripts/python/call-graph-from-sql.py | 60 ++++++++++++++---------- 1 file changed, 36 insertions(+), 24 deletions(-) diff --git a/tools/perf/scripts/python/call-graph-from-sql.py b/tools/perf/scripts/python/call-graph-from-sql.py index f18406d..b494a67 100644 --- a/tools/perf/scripts/python/call-graph-from-sql.py +++ b/tools/perf/scripts/python/call-graph-from-sql.py @@ -1,5 +1,5 @@ #!/usr/bin/python2 -# call-graph-from-sql.py: create call-graph from postgresql database +# call-graph-from-sql.py: create call-graph from sql database # Copyright (c) 2014-2017, Intel Corporation. # # This program is free software; you can redistribute it and/or modify it @@ -11,16 +11,17 @@ # FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for # more details. -# To use this script you will need to have exported data using the -# export-to-postgresql.py script. Refer to that script for details. +# To use this script you will need to have exported data using either the +# export-to-sqlite.py or the export-to-postgresql.py script. Refer to those +# scripts for details. # -# Following on from the example in the export-to-postgresql.py script, a +# Following on from the example in the export scripts, a # call-graph can be displayed for the pt_example database like this: # # python tools/perf/scripts/python/call-graph-from-sql.py pt_example # -# Note this script supports connecting to remote databases by setting hostname, -# port, username, password, and dbname e.g. +# Note that for PostgreSQL, this script supports connecting to remote databases +# by setting hostname, port, username, password, and dbname e.g. # # python tools/perf/scripts/python/call-graph-from-sql.py "hostname=myhost username=myuser password=mypassword dbname=pt_example" # @@ -296,24 +297,35 @@ if __name__ == '__main__': dbname = sys.argv[1] - db = QSqlDatabase.addDatabase('QPSQL') - - opts = dbname.split() - for opt in opts: - if '=' in opt: - opt = opt.split('=') - if opt[0] == 'hostname': - db.setHostName(opt[1]) - elif opt[0] == 'port': - db.setPort(int(opt[1])) - elif opt[0] == 'username': - db.setUserName(opt[1]) - elif opt[0] == 'password': - db.setPassword(opt[1]) - elif opt[0] == 'dbname': - dbname = opt[1] - else: - dbname = opt + is_sqlite3 = False + try: + f = open(dbname) + if f.read(15) == "SQLite format 3": + is_sqlite3 = True + f.close() + except: + pass + + if is_sqlite3: + db = QSqlDatabase.addDatabase('QSQLITE') + else: + db = QSqlDatabase.addDatabase('QPSQL') + opts = dbname.split() + for opt in opts: + if '=' in opt: + opt = opt.split('=') + if opt[0] == 'hostname': + db.setHostName(opt[1]) + elif opt[0] == 'port': + db.setPort(int(opt[1])) + elif opt[0] == 'username': + db.setUserName(opt[1]) + elif opt[0] == 'password': + db.setPassword(opt[1]) + elif opt[0] == 'dbname': + dbname = opt[1] + else: + dbname = opt db.setDatabaseName(dbname) if not db.open(): ^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: [PATCH 0/5] perf script: Add support for exporting to sqlite3 2017-08-03 8:31 [PATCH 0/5] perf script: Add support for exporting to sqlite3 Adrian Hunter ` (4 preceding siblings ...) 2017-08-03 8:31 ` [PATCH 5/5] perf script: Add support for sqlite3 " Adrian Hunter @ 2017-08-15 9:25 ` Adrian Hunter 2017-08-15 19:04 ` Arnaldo Carvalho de Melo 5 siblings, 1 reply; 13+ messages in thread From: Adrian Hunter @ 2017-08-15 9:25 UTC (permalink / raw) To: Arnaldo Carvalho de Melo; +Cc: linux-kernel On 03/08/17 11:31, Adrian Hunter wrote: > Hi > > Here is a script for exporting to SQLite 3 the same data as the PostgreSQL > export. The call-graph script is renamed and amended to work with both > PostgreSQL and SQLite. > > > Adrian Hunter (5): > perf script: Fix missing call_path_id in export-to-postgresql script > perf script: Fix query in call-graph-from-postgresql.py > perf script: Add support for exporting to sqlite3 > perf script: Rename call-graph-from-postgresql.py to call-graph-from-sql.py > perf script: Add support for sqlite3 to call-graph-from-sql.py > > tools/perf/Documentation/intel-pt.txt | 6 +- > .../scripts/python/bin/export-to-sqlite-record | 8 + > .../scripts/python/bin/export-to-sqlite-report | 29 ++ > ...h-from-postgresql.py => call-graph-from-sql.py} | 70 ++-- > tools/perf/scripts/python/export-to-postgresql.py | 5 +- > tools/perf/scripts/python/export-to-sqlite.py | 451 +++++++++++++++++++++ > 6 files changed, 535 insertions(+), 34 deletions(-) > create mode 100644 tools/perf/scripts/python/bin/export-to-sqlite-record > create mode 100644 tools/perf/scripts/python/bin/export-to-sqlite-report > rename tools/perf/scripts/python/{call-graph-from-postgresql.py => call-graph-from-sql.py} (87%) > create mode 100644 tools/perf/scripts/python/export-to-sqlite.py Any problem with these? ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH 0/5] perf script: Add support for exporting to sqlite3 2017-08-15 9:25 ` [PATCH 0/5] perf script: Add support for exporting to sqlite3 Adrian Hunter @ 2017-08-15 19:04 ` Arnaldo Carvalho de Melo 0 siblings, 0 replies; 13+ messages in thread From: Arnaldo Carvalho de Melo @ 2017-08-15 19:04 UTC (permalink / raw) To: Adrian Hunter; +Cc: linux-kernel Em Tue, Aug 15, 2017 at 12:25:24PM +0300, Adrian Hunter escreveu: > On 03/08/17 11:31, Adrian Hunter wrote: > > Hi > > > > Here is a script for exporting to SQLite 3 the same data as the PostgreSQL > > export. The call-graph script is renamed and amended to work with both > > PostgreSQL and SQLite. > > > > > > Adrian Hunter (5): > > perf script: Fix missing call_path_id in export-to-postgresql script > > perf script: Fix query in call-graph-from-postgresql.py > > perf script: Add support for exporting to sqlite3 > > perf script: Rename call-graph-from-postgresql.py to call-graph-from-sql.py > > perf script: Add support for sqlite3 to call-graph-from-sql.py > > > > tools/perf/Documentation/intel-pt.txt | 6 +- > > .../scripts/python/bin/export-to-sqlite-record | 8 + > > .../scripts/python/bin/export-to-sqlite-report | 29 ++ > > ...h-from-postgresql.py => call-graph-from-sql.py} | 70 ++-- > > tools/perf/scripts/python/export-to-postgresql.py | 5 +- > > tools/perf/scripts/python/export-to-sqlite.py | 451 +++++++++++++++++++++ > > 6 files changed, 535 insertions(+), 34 deletions(-) > > create mode 100644 tools/perf/scripts/python/bin/export-to-sqlite-record > > create mode 100644 tools/perf/scripts/python/bin/export-to-sqlite-report > > rename tools/perf/scripts/python/{call-graph-from-postgresql.py => call-graph-from-sql.py} (87%) > > create mode 100644 tools/perf/scripts/python/export-to-sqlite.py > > Any problem with these? Nope, will try and process them soon, sorry for the delay... - Arnaldo ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2017-08-17 7:53 UTC | newest] Thread overview: 13+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2017-08-03 8:31 [PATCH 0/5] perf script: Add support for exporting to sqlite3 Adrian Hunter 2017-08-03 8:31 ` [PATCH 1/5] perf script: Fix missing call_path_id in export-to-postgresql script Adrian Hunter 2017-08-17 7:50 ` [tip:perf/core] perf scripts python: " tip-bot for Adrian Hunter 2017-08-03 8:31 ` [PATCH 2/5] perf script: Fix query in call-graph-from-postgresql.py Adrian Hunter 2017-08-17 7:50 ` [tip:perf/core] perf scripts python: " tip-bot for Adrian Hunter 2017-08-03 8:31 ` [PATCH 3/5] perf script: Add support for exporting to sqlite3 Adrian Hunter 2017-08-17 7:51 ` [tip:perf/core] perf script python: " tip-bot for Adrian Hunter 2017-08-03 8:31 ` [PATCH 4/5] perf script: Rename call-graph-from-postgresql.py to call-graph-from-sql.py Adrian Hunter 2017-08-17 7:51 ` [tip:perf/core] perf script python: " tip-bot for Adrian Hunter 2017-08-03 8:31 ` [PATCH 5/5] perf script: Add support for sqlite3 " Adrian Hunter 2017-08-17 7:51 ` [tip:perf/core] perf script python: " tip-bot for Adrian Hunter 2017-08-15 9:25 ` [PATCH 0/5] perf script: Add support for exporting to sqlite3 Adrian Hunter 2017-08-15 19:04 ` Arnaldo Carvalho de Melo
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox