[PATCH 1/2] perf tools: Do not use __perf_session__process_events() directly

[PATCH 1/2] perf tools: Do not use __perf_session__process_events() directly

[Namhyung Kim]

It's only used for perf record to process build-id because its file
size it's not fixed at this time due to remaining header features.
However data offset and size is available so that we can use the
perf_session__process_events() once we set the file size as the
current offset like for now.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/builtin-record.c | 7 +++----
 tools/perf/util/session.c   | 6 +++---
 tools/perf/util/session.h   | 3 ---
 3 files changed, 6 insertions(+), 10 deletions(-)

diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index 8648c6d3003d..1134de22979e 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -194,12 +194,13 @@ static int process_buildids(struct record *rec)
 {
 	struct perf_data_file *file  = &rec->file;
 	struct perf_session *session = rec->session;
-	u64 start = session->header.data_offset;
 
 	u64 size = lseek(file->fd, 0, SEEK_CUR);
 	if (size == 0)
 		return 0;
 
+	file->size = size;
+
 	/*
 	 * During this process, it'll load kernel map and replace the
 	 * dso->long_name to a real pathname it found.  In this case
@@ -211,9 +212,7 @@ static int process_buildids(struct record *rec)
 	 */
 	symbol_conf.ignore_vmlinux_buildid = true;
 
-	return __perf_session__process_events(session, start,
-					      size - start,
-					      size, &build_id__mark_dso_hit_ops);
+	return perf_session__process_events(session, &build_id__mark_dso_hit_ops);
 }
 
 static void perf_event__synthesize_guest_os(struct machine *machine, void *data)
diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index 6ac62ae6b8fa..88aa2f09df93 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -1252,9 +1252,9 @@ fetch_mmaped_event(struct perf_session *session,
 #define NUM_MMAPS 128
 #endif
 
-int __perf_session__process_events(struct perf_session *session,
-				   u64 data_offset, u64 data_size,
-				   u64 file_size, struct perf_tool *tool)
+static int __perf_session__process_events(struct perf_session *session,
+					  u64 data_offset, u64 data_size,
+					  u64 file_size, struct perf_tool *tool)
 {
 	int fd = perf_data_file__fd(session->file);
 	u64 head, page_offset, file_offset, file_pos, size;
diff --git a/tools/perf/util/session.h b/tools/perf/util/session.h
index dc26ebf60fe4..6d663dc76404 100644
--- a/tools/perf/util/session.h
+++ b/tools/perf/util/session.h
@@ -49,9 +49,6 @@ int perf_session__peek_event(struct perf_session *session, off_t file_offset,
 			     union perf_event **event_ptr,
 			     struct perf_sample *sample);
 
-int __perf_session__process_events(struct perf_session *session,
-				   u64 data_offset, u64 data_size, u64 size,
-				   struct perf_tool *tool);
 int perf_session__process_events(struct perf_session *session,
 				 struct perf_tool *tool);
 
-- 
2.1.2

[PATCH 2/2] perf record: Show precise number of samples

[Namhyung Kim]

After perf record finishes, it prints file size and number of samples
in the file but this info is wrong since it assumes typical sample
size of 24 bytes and divides file size by the value.

However as we post-process recorded samples for build-id, it can show
correct number like below.  If build-id post-processing is not requested
just omit the wrong number of samples.

  $ perf record noploop 1
  [ perf record: Woken up 1 times to write data ]
  [ perf record: Captured and wrote 0.159 MB perf.data (3989 samples) ]

  $ perf report --stdio -n
  # To display the perf.data header info, please use --header/--header-only options.
  #
  # Samples: 3K of event 'cycles'
  # Event count (approx.): 3771330663
  #
  # Overhead       Samples  Command  Shared Object     Symbol
  # ........  ............  .......  ................  ..........................
  #
      99.90%          3982  noploop  noploop           [.] main
       0.09%             1  noploop  ld-2.17.so        [.] _dl_check_map_versions
       0.01%             1  noploop  [kernel.vmlinux]  [k] setup_arg_pages
       0.00%             5  noploop  [kernel.vmlinux]  [k] intel_pmu_enable_all

Reported-by: Milian Wolff <mail@milianw.de>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/builtin-record.c | 50 +++++++++++++++++++++++++++++++++------------
 1 file changed, 37 insertions(+), 13 deletions(-)

diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index 1134de22979e..e4e2b5b583b3 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -190,6 +190,19 @@ static int record__open(struct record *rec)
 	return rc;
 }
 
+static int process_sample_event(struct perf_tool *tool,
+				union perf_event *event,
+				struct perf_sample *sample,
+				struct perf_evsel *evsel,
+				struct machine *machine)
+{
+	struct record *rec = container_of(tool, struct record, tool);
+
+	rec->samples++;
+
+	return build_id__mark_dso_hit(tool, event, sample, evsel, machine);
+}
+
 static int process_buildids(struct record *rec)
 {
 	struct perf_data_file *file  = &rec->file;
@@ -200,6 +213,7 @@ static int process_buildids(struct record *rec)
 		return 0;
 
 	file->size = size;
+	rec->samples = 0;
 
 	/*
 	 * During this process, it'll load kernel map and replace the
@@ -212,7 +226,7 @@ static int process_buildids(struct record *rec)
 	 */
 	symbol_conf.ignore_vmlinux_buildid = true;
 
-	return perf_session__process_events(session, &build_id__mark_dso_hit_ops);
+	return perf_session__process_events(session, &rec->tool);
 }
 
 static void perf_event__synthesize_guest_os(struct machine *machine, void *data)
@@ -503,19 +517,9 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 		goto out_child;
 	}
 
-	if (!quiet) {
+	if (!quiet)
 		fprintf(stderr, "[ perf record: Woken up %ld times to write data ]\n", waking);
 
-		/*
-		 * Approximate RIP event size: 24 bytes.
-		 */
-		fprintf(stderr,
-			"[ perf record: Captured and wrote %.3f MB %s (~%" PRIu64 " samples) ]\n",
-			(double)rec->bytes_written / 1024.0 / 1024.0,
-			file->path,
-			rec->bytes_written / 24);
-	}
-
 out_child:
 	if (forks) {
 		int exit_status;
@@ -537,8 +541,21 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 	if (!err && !file->is_pipe) {
 		rec->session->header.data_size += rec->bytes_written;
 
-		if (!rec->no_buildid)
+		if (!rec->no_buildid) {
 			process_buildids(rec);
+
+			if (!quiet) {
+				fprintf(stderr,	"[ perf record: Captured and wrote %.3f MB %s (%"PRIu64" samples) ]\n",
+					(double)rec->bytes_written / 1024.0 / 1024.0,
+					file->path, rec->samples);
+			}
+		}
+		if (!quiet) {
+			fprintf(stderr,	"[ perf record: Captured and wrote %.3f MB %s ]\n",
+				(double)rec->bytes_written / 1024.0 / 1024.0,
+				file->path);
+		}
+
 		perf_session__write_header(rec->session, rec->evlist,
 					   file->fd, true);
 	}
@@ -719,6 +736,13 @@ static struct record record = {
 			.default_per_cpu = true,
 		},
 	},
+	.tool = {
+		.sample		= process_sample_event,
+		.fork		= perf_event__process_fork,
+		.comm		= perf_event__process_comm,
+		.mmap		= perf_event__process_mmap,
+		.mmap2		= perf_event__process_mmap2,
+	},
 };
 
 #define CALLCHAIN_HELP "setup and enables call-graph (stack chain/backtrace) recording: "
-- 
2.1.2

[PATCH v2 2/2] perf record: Show precise number of samples

[Namhyung Kim]

After perf record finishes, it prints file size and number of samples
in the file but this info is wrong since it assumes typical sample
size of 24 bytes and divides file size by the value.

However as we post-process recorded samples for build-id, it can show
correct number like below.  If build-id post-processing is not requested
just omit the wrong number of samples.

  $ perf record noploop 1
  [ perf record: Woken up 1 times to write data ]
  [ perf record: Captured and wrote 0.159 MB perf.data (3989 samples) ]

  $ perf report --stdio -n
  # To display the perf.data header info, please use --header/--header-only options.
  #
  # Samples: 3K of event 'cycles'
  # Event count (approx.): 3771330663
  #
  # Overhead       Samples  Command  Shared Object     Symbol
  # ........  ............  .......  ................  ..........................
  #
      99.90%          3982  noploop  noploop           [.] main
       0.09%             1  noploop  ld-2.17.so        [.] _dl_check_map_versions
       0.01%             1  noploop  [kernel.vmlinux]  [k] setup_arg_pages
       0.00%             5  noploop  [kernel.vmlinux]  [k] intel_pmu_enable_all

Reported-by: Milian Wolff <mail@milianw.de>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/builtin-record.c | 53 ++++++++++++++++++++++++++++++++++-----------
 1 file changed, 40 insertions(+), 13 deletions(-)

diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index 1134de22979e..f88f5ccb96dc 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -190,6 +190,19 @@ static int record__open(struct record *rec)
 	return rc;
 }
 
+static int process_sample_event(struct perf_tool *tool,
+				union perf_event *event,
+				struct perf_sample *sample,
+				struct perf_evsel *evsel,
+				struct machine *machine)
+{
+	struct record *rec = container_of(tool, struct record, tool);
+
+	rec->samples++;
+
+	return build_id__mark_dso_hit(tool, event, sample, evsel, machine);
+}
+
 static int process_buildids(struct record *rec)
 {
 	struct perf_data_file *file  = &rec->file;
@@ -200,6 +213,7 @@ static int process_buildids(struct record *rec)
 		return 0;
 
 	file->size = size;
+	rec->samples = 0;
 
 	/*
 	 * During this process, it'll load kernel map and replace the
@@ -212,7 +226,7 @@ static int process_buildids(struct record *rec)
 	 */
 	symbol_conf.ignore_vmlinux_buildid = true;
 
-	return perf_session__process_events(session, &build_id__mark_dso_hit_ops);
+	return perf_session__process_events(session, &rec->tool);
 }
 
 static void perf_event__synthesize_guest_os(struct machine *machine, void *data)
@@ -503,19 +517,9 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 		goto out_child;
 	}
 
-	if (!quiet) {
+	if (!quiet)
 		fprintf(stderr, "[ perf record: Woken up %ld times to write data ]\n", waking);
 
-		/*
-		 * Approximate RIP event size: 24 bytes.
-		 */
-		fprintf(stderr,
-			"[ perf record: Captured and wrote %.3f MB %s (~%" PRIu64 " samples) ]\n",
-			(double)rec->bytes_written / 1024.0 / 1024.0,
-			file->path,
-			rec->bytes_written / 24);
-	}
-
 out_child:
 	if (forks) {
 		int exit_status;
@@ -535,10 +539,26 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 		status = err;
 
 	if (!err && !file->is_pipe) {
+		bool print_nr_samples = false;
+
 		rec->session->header.data_size += rec->bytes_written;
 
-		if (!rec->no_buildid)
+		if (!rec->no_buildid) {
 			process_buildids(rec);
+			print_nr_samples = true;
+		}
+
+		if (!quiet) {
+			fprintf(stderr,	"[ perf record: Captured and wrote %.3f MB %s",
+				(double)rec->bytes_written / 1024.0 / 1024.0,
+				file->path);
+
+			if (print_nr_samples)
+				fprintf(stderr, " (%"PRIu64" samples)", rec->samples);
+
+			fputs(" ]\n", stderr);
+		}
+
 		perf_session__write_header(rec->session, rec->evlist,
 					   file->fd, true);
 	}
@@ -719,6 +739,13 @@ static struct record record = {
 			.default_per_cpu = true,
 		},
 	},
+	.tool = {
+		.sample		= process_sample_event,
+		.fork		= perf_event__process_fork,
+		.comm		= perf_event__process_comm,
+		.mmap		= perf_event__process_mmap,
+		.mmap2		= perf_event__process_mmap2,
+	},
 };
 
 #define CALLCHAIN_HELP "setup and enables call-graph (stack chain/backtrace) recording: "
-- 
2.1.2

Re: [PATCH v2 2/2] perf record: Show precise number of samples

[Jiri Olsa]

On Fri, Nov 28, 2014 at 04:14:19PM +0900, Namhyung Kim wrote:
> After perf record finishes, it prints file size and number of samples
> in the file but this info is wrong since it assumes typical sample
> size of 24 bytes and divides file size by the value.
> 
> However as we post-process recorded samples for build-id, it can show
> correct number like below.  If build-id post-processing is not requested
> just omit the wrong number of samples.
> 
>   $ perf record noploop 1
>   [ perf record: Woken up 1 times to write data ]
>   [ perf record: Captured and wrote 0.159 MB perf.data (3989 samples) ]
> 
>   $ perf report --stdio -n
>   # To display the perf.data header info, please use --header/--header-only options.
>   #
>   # Samples: 3K of event 'cycles'
>   # Event count (approx.): 3771330663
>   #
>   # Overhead       Samples  Command  Shared Object     Symbol
>   # ........  ............  .......  ................  ..........................
>   #
>       99.90%          3982  noploop  noploop           [.] main
>        0.09%             1  noploop  ld-2.17.so        [.] _dl_check_map_versions
>        0.01%             1  noploop  [kernel.vmlinux]  [k] setup_arg_pages
>        0.00%             5  noploop  [kernel.vmlinux]  [k] intel_pmu_enable_all
> 
> Reported-by: Milian Wolff <mail@milianw.de>
> Signed-off-by: Namhyung Kim <namhyung@kernel.org>

eeeey great, I mean.. finally ;-) 

Reviewed-by: Jiri Olsa <jolsa@kernel.org>

thanks,
jirka

Re: [PATCH v2 2/2] perf record: Show precise number of samples

[Andi Kleen]

> > Reported-by: Milian Wolff <mail@milianw.de>
> > Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> 
> eeeey great, I mean.. finally ;-) 

Yes I like it too.

BTW it would be nice if we could show down a break down on how many
bytes the different options passed to perf record added, by accounting
the different header fields. That would guide people who suffer
from too large perf.data files.

-Andi

Re: [PATCH v2 2/2] perf record: Show precise number of samples

[Arnaldo Carvalho de Melo]

Em Fri, Nov 28, 2014 at 06:56:13PM +0100, Andi Kleen escreveu:
> > > Reported-by: Milian Wolff <mail@milianw.de>
> > > Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> > 
> > eeeey great, I mean.. finally ;-) 
> 
> Yes I like it too.
> 
> BTW it would be nice if we could show down a break down on how many
> bytes the different options passed to perf record added, by accounting
> the different header fields. That would guide people who suffer
> from too large perf.data files.

That is a good idea, after perf_evsel__parse_sample() we add up the
things that are variable: callchains, branch_stack, etc.

Will implement if nobody does it first :-)

- Arnaldo