* [Buildroot] [PATCH] graph-build-time: generate graphs based on timing data
@ 2013-12-27 18:16 Yann E. MORIN
2013-12-27 18:21 ` Yann E. MORIN
2013-12-27 18:49 ` Thomas Petazzoni
0 siblings, 2 replies; 4+ messages in thread
From: Yann E. MORIN @ 2013-12-27 18:16 UTC (permalink / raw)
To: buildroot
From: "Yann E. MORIN" <yann.morin.1998@free.fr>
This script generates graphs of packages build time, from the timing
data generated by Buildroot in the $(O)/build-time.log file.
Example usage:
cat $(O)/build-time.data | \
./support/scripts/graph-build-time \
--type=histogram --output=foobar.pdf
Three graph types are available :
* histogram, which creates an histogram of the build time for each
package, decomposed by each step (extract, patch, configure,
etc.). The order in which the packages are shown is
configurable: by package name, by build order, or by duration
order. See the --order option.
* pie-packages, which creates a pie chart of the build time of
each package (without decomposition in steps). Packages that
contributed to less than 1% of the overall build time are all
grouped together in an "Other" entry.
* pie-steps, which creates a pie chart of the time spent globally
on each step (extract, patch, configure, etc...)
The default is to generate an histogram ordered by package name.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
[yann.morin.1998 at free.fr: adapt to the format of the step-hooks build-time.log,
add sort order by name]
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
---
So, this is a respin of Thomas' scripts from more than two years ago, that
I adapted to the new build-time.log format we now have with step-hooks.
I have put a few graphs on-line:
http://ymorin.is-a-geek.org/download/tmp/buildroot/graphs/matplotlib/
For comparisson, some I made with gnuplot:
http://ymorin.is-a-geek.org/download/tmp/buildroot/graphs/gnuplot/
To be honnest, I prefer the graphs made with gnuplot, but it is mostly
because of the colors (hey, Thomas, are you colorblind? ;-p)
But since this script is cleaner (much, much cleaner) than what I was able
to get with gnuplot, I won't even try to submit my gnuplot scripts... :-/
---
support/scripts/graph-build-time | 273 +++++++++++++++++++++++++++++++++++++++
1 file changed, 273 insertions(+)
create mode 100755 support/scripts/graph-build-time
diff --git a/support/scripts/graph-build-time b/support/scripts/graph-build-time
new file mode 100755
index 0000000..13509ef
--- /dev/null
+++ b/support/scripts/graph-build-time
@@ -0,0 +1,273 @@
+#!/usr/bin/env python
+
+# Copyright (C) 2011 by Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
+# Copyright (C) 2013 by Yann E. MORIN <yann.morin.1998@free.fr>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+# This script generates graphs of packages build time, from the timing
+# data generated by Buildroot in the $(O)/build-time.log file.
+#
+# Example usage:
+#
+# cat $(O)/build-time.data | ./support/scripts/graph-build-time --type=histogram --output=foobar.pdf
+#
+# Three graph types are available :
+#
+# * histogram, which creates an histogram of the build time for each
+# package, decomposed by each step (extract, patch, configure,
+# etc.). The order in which the packages are shown is
+# configurable: by package name, by build order, or by duration
+# order. See the --order option.
+#
+# * pie-packages, which creates a pie chart of the build time of
+# each package (without decomposition in steps). Packages that
+# contributed to less than 1% of the overall build time are all
+# grouped together in an "Other" entry.
+#
+# * pie-steps, which creates a pie chart of the time spent globally
+# on each step (extract, patch, configure, etc...)
+#
+# The default is to generate an histogram ordered by package name.
+#
+# Requirements:
+#
+# * matplotlib (python-matplotlib on Debian/Ubuntu systems)
+# * numpy (python-numpy on Debian/Ubuntu systems)
+# * argparse (by default in Python 2.7, requires python-argparse if
+# Python 2.6 is used)
+
+import matplotlib
+import numpy
+matplotlib.use('PDF')
+
+import matplotlib.pyplot as plt
+import matplotlib.font_manager as fm
+import csv
+import argparse
+import sys
+
+steps = [ 'extract', 'patch', 'configure', 'build',
+ 'install-target', 'install-host', 'install-images',
+ 'install-staging' ]
+
+histogram_colors = ['#e60004', '#009836', '#2e1d86', '#ffed00',
+ '#0068b5', '#f28e00', '#940084', '#97c000',
+ '#00469b', '#f9c000' ]
+
+class Package:
+ def __init__(self, name):
+ self.name = name
+ self.steps_duration = {}
+ self.steps_start = {}
+ self.steps_end = {}
+
+ def add_step(self, step, state, time):
+ if state == "start":
+ self.steps_start[step] = time
+ else:
+ self.steps_end[step] = time
+ if self.steps_start.has_key(step) and self.steps_end.has_key(step):
+ self.steps_duration[step] = self.steps_end[step] - self.steps_start[step]
+
+ def get_duration(self, step=None):
+ if step == None:
+ duration = 0
+ for step in self.steps_duration.keys():
+ duration += self.steps_duration[step]
+ return duration
+ if self.steps_duration.has_key(step):
+ return self.steps_duration[step]
+ return 0
+
+# Generate an histogram of the time spent in each step of each
+# package.
+def pkg_histogram(data, output, order="build"):
+ n_pkgs = len(data)
+ ind = numpy.arange(n_pkgs)
+
+ if order == "duration":
+ data = sorted(data, key=lambda p: p.get_duration(), reverse=True)
+ elif order == "name":
+ data = sorted(data, key=lambda p: p.name, reverse=False)
+
+ # Prepare the vals array, containing one entry for each step
+ vals = []
+ for step in steps:
+ val = []
+ for p in data:
+ val.append(p.get_duration(step))
+ vals.append(val)
+
+ bottom = [0] * n_pkgs
+ legenditems = []
+
+ plt.figure()
+
+ # Draw the bars, step by step
+ for i in range(0, len(vals)):
+ b = plt.bar(ind+0.1, vals[i], width=0.8, color=histogram_colors[i], bottom=bottom, linewidth=0.25)
+ legenditems.append(b[0])
+ bottom = [ bottom[j] + vals[i][j] for j in range(0, len(vals[i])) ]
+
+ # Draw the package names
+ plt.xticks(ind + .6, [ p.name for p in data ], rotation=-60, rotation_mode="anchor", fontsize=8, ha='left')
+
+ # Adjust size of graph (double the width)
+ sz = plt.gcf().get_size_inches()
+ plt.gcf().set_size_inches(sz[0] * 2, sz[1])
+
+ # Add more space for the package names at the bottom
+ plt.gcf().subplots_adjust(bottom=0.2)
+
+ # Remove ticks in the graph for each package
+ axes = plt.gcf().gca()
+ for line in axes.get_xticklines():
+ line.set_markersize(0)
+
+ axes.set_ylabel('Time (seconds)')
+
+ # Reduce size of legend text
+ leg_prop = fm.FontProperties(size=6)
+
+ # Draw legend
+ plt.legend(legenditems, steps, prop=leg_prop)
+
+ if order == "name":
+ plt.title('Build time of packages')
+ elif order == "build":
+ plt.title('Build time of packages, by build order')
+ elif order == "duration":
+ plt.title('Build time of packages, by duration order')
+
+ # Save graph
+ plt.savefig(output)
+
+# Generate a pie chart with the time spent building each package.
+def pkg_pie_time_per_package(data, output):
+ # Compute total build duration
+ total = 0
+ for p in data:
+ total += p.get_duration()
+
+ # Build the list of labels and values, and filter the packages
+ # that account for less than 1% of the build time.
+ labels = []
+ values = []
+ other_value = 0
+ for p in data:
+ if p.get_duration() < (total * 0.01):
+ other_value += p.get_duration()
+ else:
+ labels.append(p.name)
+ values.append(p.get_duration())
+
+ labels.append('Other')
+ values.append(other_value)
+
+ plt.figure()
+
+ # Draw pie graph
+ patches, texts, autotexts = plt.pie(values, labels=labels, autopct='%1.1f%%', shadow=True)
+
+ # Reduce text size
+ proptease = fm.FontProperties()
+ proptease.set_size('xx-small')
+ plt.setp(autotexts, fontproperties=proptease)
+ plt.setp(texts, fontproperties=proptease)
+
+ plt.title('Build time per package')
+ plt.savefig(output)
+
+# Generate a pie chart with a portion for the overall time spent in
+# each step for all packages.
+def pkg_pie_time_per_step(data, output):
+ steps_values = []
+ for step in steps:
+ val = 0
+ for p in data:
+ val += p.get_duration(step)
+ steps_values.append(val)
+
+ plt.figure()
+
+ # Draw pie graph
+ patches, texts, autotexts = plt.pie(steps_values, labels=steps,
+ autopct='%1.1f%%', shadow=True)
+
+ # Reduce text size
+ proptease = fm.FontProperties()
+ proptease.set_size('xx-small')
+ plt.setp(autotexts, fontproperties=proptease)
+ plt.setp(texts, fontproperties=proptease)
+
+ plt.title('Build time per step')
+ plt.savefig(output)
+
+# Parses the csv file passed on standard input and returns a list of
+# Package objects, filed with the duration of each step and the total
+# duration of the package.
+def read_data():
+ reader = csv.reader(sys.stdin, delimiter=':')
+ pkgs = []
+
+ # Auxilliary function to find a package by name in the list.
+ def getpkg(name):
+ for p in pkgs:
+ if p.name == name:
+ return p
+ return None
+
+ for row in reader:
+ time = int(row[0].strip())
+ state = row[1].strip()
+ step = row[2].strip()
+ pkg = row[3].strip()
+
+ p = getpkg(pkg)
+ if p is None:
+ p = Package(pkg)
+ pkgs.append(p)
+
+ p.add_step(step, state, time)
+
+ return pkgs
+
+parser = argparse.ArgumentParser(description='Draw build time graphs')
+parser.add_argument("--type", metavar="GRAPH_TYPE", required=True,
+ help="Type of graph (histogram, pie-packages, pie-steps)")
+parser.add_argument("--order", metavar="GRAPH_ORDER",
+ help="Ordering of packages: build or duration (for histogram only)")
+parser.add_argument("--output", metavar="OUTPUT", required=True,
+ help="Output file (PDF extension)")
+args = parser.parse_args()
+
+d = read_data()
+
+if args.type == "histogram" or args.type == None:
+ if args.order == "build" or args.order == "duration" or args.order == "name":
+ pkg_histogram(d, args.output, args.order)
+ elif args.order == None:
+ pkg_histogram(d, args.output, "name")
+ else:
+ sys.stderr.write("Unknown ordering: %s\n" % args.order)
+ exit(1)
+elif args.type == "pie-packages":
+ pkg_pie_time_per_package(d, args.output)
+elif args.type == "pie-steps":
+ pkg_pie_time_per_step(d, args.output)
+else:
+ sys.stderr.write("Unknown type: %s\n" % args.type)
+ exit(1)
--
1.8.1.2
^ permalink raw reply related [flat|nested] 4+ messages in thread
* [Buildroot] [PATCH] graph-build-time: generate graphs based on timing data
2013-12-27 18:16 [Buildroot] [PATCH] graph-build-time: generate graphs based on timing data Yann E. MORIN
@ 2013-12-27 18:21 ` Yann E. MORIN
2013-12-27 18:49 ` Thomas Petazzoni
1 sibling, 0 replies; 4+ messages in thread
From: Yann E. MORIN @ 2013-12-27 18:21 UTC (permalink / raw)
To: buildroot
On 2013-12-27 19:16 +0100, Yann E. MORIN spake thusly:
> From: "Yann E. MORIN" <yann.morin.1998@free.fr>
Doh... This hould have been:
From: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
I fsckup when I applied the patch, sorry... :-/
Regards,
Yann E. MORIN.
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 4+ messages in thread
* [Buildroot] [PATCH] graph-build-time: generate graphs based on timing data
2013-12-27 18:16 [Buildroot] [PATCH] graph-build-time: generate graphs based on timing data Yann E. MORIN
2013-12-27 18:21 ` Yann E. MORIN
@ 2013-12-27 18:49 ` Thomas Petazzoni
2013-12-27 19:34 ` Yann E. MORIN
1 sibling, 1 reply; 4+ messages in thread
From: Thomas Petazzoni @ 2013-12-27 18:49 UTC (permalink / raw)
To: buildroot
Dear Yann E. MORIN,
On Fri, 27 Dec 2013 19:16:29 +0100, Yann E. MORIN wrote:
> From: "Yann E. MORIN" <yann.morin.1998@free.fr>
>
> This script generates graphs of packages build time, from the timing
> data generated by Buildroot in the $(O)/build-time.log file.
>
> Example usage:
>
> cat $(O)/build-time.data | \
> ./support/scripts/graph-build-time \
> --type=histogram --output=foobar.pdf
Just like the dependency graph tool, I'm wondering if we shouldn't try
to integrate this as a make target, to give it more visibility. This
can of course be done at a later stage.
> Three graph types are available :
>
> * histogram, which creates an histogram of the build time for each
> package, decomposed by each step (extract, patch, configure,
> etc.). The order in which the packages are shown is
> configurable: by package name, by build order, or by duration
> order. See the --order option.
>
> * pie-packages, which creates a pie chart of the build time of
> each package (without decomposition in steps). Packages that
> contributed to less than 1% of the overall build time are all
> grouped together in an "Other" entry.
>
> * pie-steps, which creates a pie chart of the time spent globally
> on each step (extract, patch, configure, etc...)
>
> The default is to generate an histogram ordered by package name.
>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> [yann.morin.1998 at free.fr: adapt to the format of the step-hooks build-time.log,
> add sort order by name]
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> ---
>
> So, this is a respin of Thomas' scripts from more than two years ago, that
> I adapted to the new build-time.log format we now have with step-hooks.
>
> I have put a few graphs on-line:
> http://ymorin.is-a-geek.org/download/tmp/buildroot/graphs/matplotlib/
>
> For comparisson, some I made with gnuplot:
> http://ymorin.is-a-geek.org/download/tmp/buildroot/graphs/gnuplot/
>
> To be honnest, I prefer the graphs made with gnuplot, but it is mostly
> because of the colors (hey, Thomas, are you colorblind? ;-p)
I'm not colorblind. To me, the histogram colors look nice (they are
defined in the script). The colors for the pie charts are horrible, but
I believe they are automatically chosen by matplotlib, no?
Feel free to improve the colors :)
> But since this script is cleaner (much, much cleaner) than what I was able
> to get with gnuplot, I won't even try to submit my gnuplot scripts... :-/
>
> ---
> support/scripts/graph-build-time | 273 +++++++++++++++++++++++++++++++++++++++
> 1 file changed, 273 insertions(+)
> create mode 100755 support/scripts/graph-build-time
Thanks a lot for submitting this!
Thomas
--
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
^ permalink raw reply [flat|nested] 4+ messages in thread
* [Buildroot] [PATCH] graph-build-time: generate graphs based on timing data
2013-12-27 18:49 ` Thomas Petazzoni
@ 2013-12-27 19:34 ` Yann E. MORIN
0 siblings, 0 replies; 4+ messages in thread
From: Yann E. MORIN @ 2013-12-27 19:34 UTC (permalink / raw)
To: buildroot
Thomas, All,
On 2013-12-27 19:49 +0100, Thomas Petazzoni spake thusly:
> On Fri, 27 Dec 2013 19:16:29 +0100, Yann E. MORIN wrote:
> > From: "Yann E. MORIN" <yann.morin.1998@free.fr>
> >
> > This script generates graphs of packages build time, from the timing
> > data generated by Buildroot in the $(O)/build-time.log file.
> >
> > Example usage:
> >
> > cat $(O)/build-time.data | \
> > ./support/scripts/graph-build-time \
> > --type=histogram --output=foobar.pdf
>
> Just like the dependency graph tool, I'm wondering if we shouldn't try
> to integrate this as a make target, to give it more visibility. This
> can of course be done at a later stage.
That's my opinion, too. I'll tackle this as following patches.
> > To be honnest, I prefer the graphs made with gnuplot, but it is mostly
> > because of the colors (hey, Thomas, are you colorblind? ;-p)
>
> I'm not colorblind.
I was just kidding! ;-)
> To me, the histogram colors look nice (they are
> defined in the script).
Yes, I've seen that, but I did not change them, as it's a minor nit.
> The colors for the pie charts are horrible, but
> I believe they are automatically chosen by matplotlib, no?
Maybe we can also force the collors we want. I'll see if we can do that
with matplotlib...
I'll do some changes and will re-submit later.
Regards,
Yann E. MORIN.
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2013-12-27 19:34 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-12-27 18:16 [Buildroot] [PATCH] graph-build-time: generate graphs based on timing data Yann E. MORIN
2013-12-27 18:21 ` Yann E. MORIN
2013-12-27 18:49 ` Thomas Petazzoni
2013-12-27 19:34 ` Yann E. MORIN
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox