[Buildroot] [pull request] Pull request for branch yem/graphs

[Buildroot] [pull request] Pull request for branch yem/graphs

[Yann E. MORIN]

From: "Yann E. MORIN" <yann.morin.1998@free.fr>

Hello All!

Here is a small series that adds some graphing abilities to Buildroot:
  - graph build-times, originally by Thomas, revamped now that our step-hooks
    generate $(O)/build/build-time.log for all builds
  - expose this and graph-depends as make targets
  - allow generating either PDF or PNG graphs

Regards,
Yann E. MORIN.


The following changes since commit 269da64af5ad72e1d522b54a6203970c3eb344d0:

  package: deprecate some more development tools (2013-12-27 19:54:05 +0100)

are available in the git repository at:

  git://ymorin.is-a-geek.org/buildroot yem/graphs

for you to fetch changes up to 22105197db33dde1f1cee82511ce9c6d00d86853:

  graphs: support generating png graphs (2013-12-28 00:00:24 +0100)

----------------------------------------------------------------
Thomas Petazzoni (1):
      graph-build-time: generate graphs based on timing data

Yann E. MORIN (3):
      Makefile: expose target 'graph-build' to generate the build-time graphs
      Makefile: expose 'graph-depends' to generate a graph of the dependency tree
      graphs: support generating png graphs

 Makefile                         |  31 +++++
 package/pkg-generic.mk           |   3 +
 support/scripts/graph-build-time | 284 +++++++++++++++++++++++++++++++++++++++
 3 files changed, 318 insertions(+)
 create mode 100755 support/scripts/graph-build-time

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 1/4] graph-build-time: generate graphs based on timing data

[Yann E. MORIN]

From: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>

This script generates graphs of packages build time, from the timing
data generated by Buildroot in the $(O)/build-time.log file.

Example usage:

  cat $(O)/build-time.log | \
      ./support/scripts/graph-build-time \
      --type=histogram --output=foobar.pdf

Three graph types are available :

  * histogram, which creates an histogram of the build time for each
    package, decomposed by each step (extract, patch, configure,
    etc.). The order in which the packages are shown is
    configurable: by package name, by build order, or by duration
    order. See the --order option.

  * pie-packages, which creates a pie chart of the build time of
    each package (without decomposition in steps). Packages that
    contributed to less than 1% of the overall build time are all
    grouped together in an "Other" entry.

  * pie-steps, which creates a pie chart of the time spent globally
    on each step (extract, patch, configure, etc...)

The default is to generate an histogram ordered by package name.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
[yann.morin.1998 at free.fr: adapt to the format of the step-hooks build-time.log,
    add sort order by name, default to name-ordered histogram, use our colours
    for pie-charts, add alternate color-scheme]
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
---
 support/scripts/graph-build-time | 285 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 285 insertions(+)
 create mode 100755 support/scripts/graph-build-time

diff --git a/support/scripts/graph-build-time b/support/scripts/graph-build-time
new file mode 100755
index 0000000..a1a82eb
--- /dev/null
+++ b/support/scripts/graph-build-time
@@ -0,0 +1,285 @@
+#!/usr/bin/env python
+
+# Copyright (C) 2011 by Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
+# Copyright (C) 2013 by Yann E. MORIN <yann.morin.1998@free.fr>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+# This script generates graphs of packages build time, from the timing
+# data generated by Buildroot in the $(O)/build-time.log file.
+#
+# Example usage:
+#
+#   cat $(O)/build-time.log | ./support/scripts/graph-build-time --type=histogram --output=foobar.pdf
+#
+# Three graph types are available :
+#
+#   * histogram, which creates an histogram of the build time for each
+#     package, decomposed by each step (extract, patch, configure,
+#     etc.). The order in which the packages are shown is
+#     configurable: by package name, by build order, or by duration
+#     order. See the --order option.
+#
+#   * pie-packages, which creates a pie chart of the build time of
+#     each package (without decomposition in steps). Packages that
+#     contributed to less than 1% of the overall build time are all
+#     grouped together in an "Other" entry.
+#
+#   * pie-steps, which creates a pie chart of the time spent globally
+#     on each step (extract, patch, configure, etc...)
+#
+# The default is to generate an histogram ordered by package name.
+#
+# Requirements:
+#
+#   * matplotlib (python-matplotlib on Debian/Ubuntu systems)
+#   * numpy (python-numpy on Debian/Ubuntu systems)
+#   * argparse (by default in Python 2.7, requires python-argparse if
+#     Python 2.6 is used)
+
+import matplotlib
+import numpy
+matplotlib.use('PDF')
+
+import matplotlib.pyplot as plt
+import matplotlib.font_manager as fm
+import csv
+import argparse
+import sys
+
+steps = [ 'extract', 'patch', 'configure', 'build',
+          'install-target', 'install-staging', 'install-images',
+          'install-host']
+
+default_colors = ['#e60004', '#009836', '#2e1d86', '#ffed00',
+                  '#0068b5', '#f28e00', '#940084', '#97c000']
+
+alternate_colors = ['#00e0e0', '#3f7f7f', '#ff0000', '#00c000',
+                    '#0080ff', '#c000ff', '#00eeee', '#e0e000']
+
+class Package:
+    def __init__(self, name):
+        self.name = name
+        self.steps_duration = {}
+        self.steps_start = {}
+        self.steps_end = {}
+
+    def add_step(self, step, state, time):
+        if state == "start":
+            self.steps_start[step] = time
+        else:
+            self.steps_end[step] = time
+        if self.steps_start.has_key(step) and self.steps_end.has_key(step):
+            self.steps_duration[step] = self.steps_end[step] - self.steps_start[step]
+
+    def get_duration(self, step=None):
+        if step == None:
+            duration = 0
+            for step in self.steps_duration.keys():
+                duration += self.steps_duration[step]
+            return duration
+        if self.steps_duration.has_key(step):
+            return self.steps_duration[step]
+        return 0
+
+# Generate an histogram of the time spent in each step of each
+# package.
+def pkg_histogram(data, output, order="build"):
+    n_pkgs = len(data)
+    ind = numpy.arange(n_pkgs)
+
+    if order == "duration":
+        data = sorted(data, key=lambda p: p.get_duration(), reverse=True)
+    elif order == "name":
+        data = sorted(data, key=lambda p: p.name, reverse=False)
+
+    # Prepare the vals array, containing one entry for each step
+    vals = []
+    for step in steps:
+        val = []
+        for p in data:
+            val.append(p.get_duration(step))
+        vals.append(val)
+
+    bottom = [0] * n_pkgs
+    legenditems = []
+
+    plt.figure()
+
+    # Draw the bars, step by step
+    for i in range(0, len(vals)):
+        b = plt.bar(ind+0.1, vals[i], width=0.8, color=colors[i], bottom=bottom, linewidth=0.25)
+        legenditems.append(b[0])
+        bottom = [ bottom[j] + vals[i][j] for j in range(0, len(vals[i])) ]
+
+    # Draw the package names
+    plt.xticks(ind + .6, [ p.name for p in data ], rotation=-60, rotation_mode="anchor", fontsize=8, ha='left')
+
+    # Adjust size of graph (double the width)
+    sz = plt.gcf().get_size_inches()
+    plt.gcf().set_size_inches(sz[0] * 2, sz[1])
+
+    # Add more space for the package names at the bottom
+    plt.gcf().subplots_adjust(bottom=0.2)
+
+    # Remove ticks in the graph for each package
+    axes = plt.gcf().gca()
+    for line in axes.get_xticklines():
+        line.set_markersize(0)
+
+    axes.set_ylabel('Time (seconds)')
+
+    # Reduce size of legend text
+    leg_prop = fm.FontProperties(size=6)
+
+    # Draw legend
+    plt.legend(legenditems, steps, prop=leg_prop)
+
+    if order == "name":
+        plt.title('Build time of packages\n')
+    elif order == "build":
+        plt.title('Build time of packages, by build order\n')
+    elif order == "duration":
+        plt.title('Build time of packages, by duration order\n')
+
+    # Save graph
+    plt.savefig(output)
+
+# Generate a pie chart with the time spent building each package.
+def pkg_pie_time_per_package(data, output):
+    # Compute total build duration
+    total = 0
+    for p in data:
+        total += p.get_duration()
+
+    # Build the list of labels and values, and filter the packages
+    # that account for less than 1% of the build time.
+    labels = []
+    values = []
+    other_value = 0
+    for p in data:
+        if p.get_duration() < (total * 0.01):
+            other_value += p.get_duration()
+        else:
+            labels.append(p.name)
+            values.append(p.get_duration())
+
+    labels.append('Other')
+    values.append(other_value)
+
+    plt.figure()
+
+    # Draw pie graph
+    patches, texts, autotexts = plt.pie(values, labels=labels,
+                                        autopct='%1.1f%%', shadow=True,
+                                        colors=colors)
+
+    # Reduce text size
+    proptease = fm.FontProperties()
+    proptease.set_size('xx-small')
+    plt.setp(autotexts, fontproperties=proptease)
+    plt.setp(texts, fontproperties=proptease)
+
+    plt.title('Build time per package')
+    plt.savefig(output)
+
+# Generate a pie chart with a portion for the overall time spent in
+# each step for all packages.
+def pkg_pie_time_per_step(data, output):
+    steps_values = []
+    for step in steps:
+        val = 0
+        for p in data:
+            val += p.get_duration(step)
+        steps_values.append(val)
+
+    plt.figure()
+
+    # Draw pie graph
+    patches, texts, autotexts = plt.pie(steps_values, labels=steps,
+                                        autopct='%1.1f%%', shadow=True,
+                                        colors=colors)
+
+    # Reduce text size
+    proptease = fm.FontProperties()
+    proptease.set_size('xx-small')
+    plt.setp(autotexts, fontproperties=proptease)
+    plt.setp(texts, fontproperties=proptease)
+
+    plt.title('Build time per step')
+    plt.savefig(output)
+
+# Parses the csv file passed on standard input and returns a list of
+# Package objects, filed with the duration of each step and the total
+# duration of the package.
+def read_data():
+    reader = csv.reader(sys.stdin, delimiter=':')
+    pkgs = []
+
+    # Auxilliary function to find a package by name in the list.
+    def getpkg(name):
+        for p in pkgs:
+            if p.name == name:
+                return p
+        return None
+
+    for row in reader:
+        time = int(row[0].strip())
+        state = row[1].strip()
+        step = row[2].strip()
+        pkg = row[3].strip()
+
+        p = getpkg(pkg)
+        if p is None:
+            p = Package(pkg)
+            pkgs.append(p)
+
+        p.add_step(step, state, time)
+
+    return pkgs
+
+parser = argparse.ArgumentParser(description='Draw build time graphs')
+parser.add_argument("--type", metavar="GRAPH_TYPE",
+                    help="Type of graph (histogram, pie-packages, pie-steps)")
+parser.add_argument("--order", metavar="GRAPH_ORDER",
+                    help="Ordering of packages: build or duration (for histogram only)")
+parser.add_argument("--alternate-colors", action="store_true",
+                    help="Use alternate colour-scheme")
+parser.add_argument("--output", metavar="OUTPUT", required=True,
+                    help="Output file (PDF extension)")
+args = parser.parse_args()
+
+d = read_data()
+
+if args.alternate_colors:
+    colors = alternate_colors
+else:
+    colors = default_colors
+
+if args.type == "histogram" or args.type == None:
+    if args.order == "build" or args.order == "duration" or args.order == "name":
+        pkg_histogram(d, args.output, args.order)
+    elif args.order == None:
+        pkg_histogram(d, args.output, "name")
+    else:
+        sys.stderr.write("Unknown ordering: %s\n" % args.order)
+        exit(1)
+elif args.type == "pie-packages":
+    pkg_pie_time_per_package(d, args.output)
+elif args.type == "pie-steps":
+    pkg_pie_time_per_step(d, args.output)
+else:
+    sys.stderr.write("Unknown type: %s\n" % args.type)
+    exit(1)
-- 
1.8.1.2

[Buildroot] [PATCH 2/4] Makefile: expose target 'graph-build' to generate the build-time graphs

[Yann E. MORIN]

From: "Yann E. MORIN" <yann.morin.1998@free.fr>

Generate the build-time graphs by calling:
    make graph-build

This generates the graphs in $(O)/graphs/

It is possible to use the alternate color-scheme by setting the variable
GRAPH_ALT=1 on the command line:
    make GRAPH_ALT=1 graph-build

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 Makefile | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/Makefile b/Makefile
index 3190541..9df72e9 100644
--- a/Makefile
+++ b/Makefile
@@ -631,6 +631,19 @@ legal-info: dirs legal-info-clean legal-info-prepare $(TARGETS_LEGAL_INFO) \
 show-targets:
 	@echo $(TARGETS)
 
+graph-build: $(O)/build/build-time.log
+	@install -d $(O)/graphs
+	$(foreach o,name build duration,./support/scripts/graph-build-time \
+					--type=histogram --order=$(o) \
+					--output=$(O)/graphs/build.hist-$(o).pdf \
+					$(if $(GRAPH_ALT),--alternate-colors) \
+					<$(<)$(sep))
+	$(foreach t,packages steps,./support/scripts/graph-build-time \
+				   --type=pie-$(t) \
+				   --output=$(O)/graphs/build.pie-$(t).pdf \
+				   $(if $(GRAPH_ALT),--alternate-colors) \
+				   <$(<)$(sep))
+
 else # ifeq ($(BR2_HAVE_DOT_CONFIG),y)
 
 all: menuconfig
@@ -843,6 +856,7 @@ endif
 	@echo '  manual-pdf             - build manual in PDF'
 	@echo '  manual-text            - build manual in text'
 	@echo '  manual-epub            - build manual in ePub'
+	@echo '  graph-build            - generate graphs of the build times'
 	@echo
 	@echo 'Miscellaneous:'
 	@echo '  source                 - download all sources needed for offline-build'
@@ -852,6 +866,7 @@ endif
 	@echo
 	@echo '  make V=0|1             - 0 => quiet build (default), 1 => verbose build'
 	@echo '  make O=dir             - Locate all output files in "dir", including .config'
+	@echo '  make GRAPH_ALT=1       - use alternate color-scheme for build-time graphs'
 	@echo
 	@echo 'Built-in configs:'
 	@$(foreach b, $(sort $(notdir $(wildcard $(TOPDIR)/configs/*_defconfig))), \
-- 
1.8.1.2

[Buildroot] [PATCH 3/4] Makefile: expose 'graph-depends' to generate a graph of the dependency tree

[Yann E. MORIN]

From: "Yann E. MORIN" <yann.morin.1998@free.fr>

Generate the graph of the complete dependency tree by calling:
    make graph-depends

It's also possible to generate the graph-depends for a single package:
    make PKG-graph-depends

The graphs are generated in $(O)/graphs/

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 Makefile               | 13 +++++++++++++
 package/pkg-generic.mk |  3 +++
 2 files changed, 16 insertions(+)

diff --git a/Makefile b/Makefile
index 9df72e9..f09f744 100644
--- a/Makefile
+++ b/Makefile
@@ -644,6 +644,18 @@ graph-build: $(O)/build/build-time.log
 				   $(if $(GRAPH_ALT),--alternate-colors) \
 				   <$(<)$(sep))
 
+graph-depends:
+	@install -d $(O)/graphs
+	@./support/scripts/graph-depends \
+	|dot -Tpdf -o $(O)/graphs/graph-depends.pdf
+	-o $(O)/graphs/graph-depends.pdf
+
+%-graph-depends:
+	@install -d $(O)/graphs
+	@./support/scripts/graph-depends $(@:-graph-depends=) \
+	|dot -Tpdf \
+	-o $(O)/graphs/$(@).pdf
+
 else # ifeq ($(BR2_HAVE_DOT_CONFIG),y)
 
 all: menuconfig
@@ -857,6 +869,7 @@ endif
 	@echo '  manual-text            - build manual in text'
 	@echo '  manual-epub            - build manual in ePub'
 	@echo '  graph-build            - generate graphs of the build times'
+	@echo '  graph-depends          - generate graph of the dependency tree'
 	@echo
 	@echo 'Miscellaneous:'
 	@echo '  source                 - download all sources needed for offline-build'
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 66034ba..aa7e4ef 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -468,6 +468,9 @@ endif
 $(1)-show-depends:
 			@echo $$($(2)_DEPENDENCIES)
 
+# For auto-completion, handled in top-level Makefile
+$(1)-graph-depends:
+
 $(1)-dirclean:		$$($(2)_TARGET_DIRCLEAN)
 
 $(1)-clean-for-rebuild:
-- 
1.8.1.2

[Buildroot] [PATCH 4/4] graphs: support generating png graphs

[Yann E. MORIN]

From: "Yann E. MORIN" <yann.morin.1998@free.fr>

PDF files can not be easily embedded in other documents (eg. ODT, or HTML).

Add support for generating PNG graphs, by setting the GRAPH_OUT=pdf|png on
the command line:
    make GRAPH_OUT=png graph-build graph-depends

The default is still to generate PDF graphs.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 Makefile                         | 15 +++++++++------
 support/scripts/graph-build-time |  3 +--
 2 files changed, 10 insertions(+), 8 deletions(-)

diff --git a/Makefile b/Makefile
index f09f744..e85eec3 100644
--- a/Makefile
+++ b/Makefile
@@ -631,30 +631,32 @@ legal-info: dirs legal-info-clean legal-info-prepare $(TARGETS_LEGAL_INFO) \
 show-targets:
 	@echo $(TARGETS)
 
+BR2_GRAPH_OUT := $(or $(GRAPH_OUT),pdf)
+
 graph-build: $(O)/build/build-time.log
 	@install -d $(O)/graphs
 	$(foreach o,name build duration,./support/scripts/graph-build-time \
 					--type=histogram --order=$(o) \
-					--output=$(O)/graphs/build.hist-$(o).pdf \
+					--output=$(O)/graphs/build.hist-$(o).$(BR2_GRAPH_OUT) \
 					$(if $(GRAPH_ALT),--alternate-colors) \
 					<$(<)$(sep))
 	$(foreach t,packages steps,./support/scripts/graph-build-time \
 				   --type=pie-$(t) \
-				   --output=$(O)/graphs/build.pie-$(t).pdf \
+				   --output=$(O)/graphs/build.pie-$(t).$(BR2_GRAPH_OUT) \
 				   $(if $(GRAPH_ALT),--alternate-colors) \
 				   <$(<)$(sep))
 
 graph-depends:
 	@install -d $(O)/graphs
 	@./support/scripts/graph-depends \
-	|dot -Tpdf -o $(O)/graphs/graph-depends.pdf
-	-o $(O)/graphs/graph-depends.pdf
+	|dot -T$(BR2_GRAPH_OUT) \
+	-o $(O)/graphs/$(@).$(BR2_GRAPH_OUT)
 
 %-graph-depends:
 	@install -d $(O)/graphs
 	@./support/scripts/graph-depends $(@:-graph-depends=) \
-	|dot -Tpdf \
-	-o $(O)/graphs/$(@).pdf
+	|dot -T$(BR2_GRAPH_OUT) \
+	-o $(O)/graphs/$(@).$(BR2_GRAPH_OUT)
 
 else # ifeq ($(BR2_HAVE_DOT_CONFIG),y)
 
@@ -880,6 +882,7 @@ endif
 	@echo '  make V=0|1             - 0 => quiet build (default), 1 => verbose build'
 	@echo '  make O=dir             - Locate all output files in "dir", including .config'
 	@echo '  make GRAPH_ALT=1       - use alternate color-scheme for build-time graphs'
+	@echo '  make GRAPH_OUT=pdf|png - generate either pdf (default) or png graphs'
 	@echo
 	@echo 'Built-in configs:'
 	@$(foreach b, $(sort $(notdir $(wildcard $(TOPDIR)/configs/*_defconfig))), \
diff --git a/support/scripts/graph-build-time b/support/scripts/graph-build-time
index a1a82eb..2611c2c 100755
--- a/support/scripts/graph-build-time
+++ b/support/scripts/graph-build-time
@@ -51,7 +51,6 @@
 
 import matplotlib
 import numpy
-matplotlib.use('PDF')
 
 import matplotlib.pyplot as plt
 import matplotlib.font_manager as fm
@@ -258,7 +257,7 @@ parser.add_argument("--order", metavar="GRAPH_ORDER",
 parser.add_argument("--alternate-colors", action="store_true",
                     help="Use alternate colour-scheme")
 parser.add_argument("--output", metavar="OUTPUT", required=True,
-                    help="Output file (PDF extension)")
+                    help="Output file (.pdf or .png extension)")
 args = parser.parse_args()
 
 d = read_data()
-- 
1.8.1.2

[Buildroot] [PATCH 3/4] Makefile: expose 'graph-depends' to generate a graph of the dependency tree

[Thomas Petazzoni]

Dear Yann E. MORIN,

On Sat, 28 Dec 2013 00:12:51 +0100, Yann E. MORIN wrote:

> +%-graph-depends:
> +	@install -d $(O)/graphs
> +	@./support/scripts/graph-depends $(@:-graph-depends=) \
> +	|dot -Tpdf \
> +	-o $(O)/graphs/$(@).pdf

Why isn't that one done in the package infrastructure, i.e in
pkg-generic.mk ?

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 3/4] Makefile: expose 'graph-depends' to generate a graph of the dependency tree

[Yann E. MORIN]

Thomas, All,

On 2013-12-28 12:46 +0100, Thomas Petazzoni spake thusly:
> Dear Yann E. MORIN,
> 
> On Sat, 28 Dec 2013 00:12:51 +0100, Yann E. MORIN wrote:
> 
> > +%-graph-depends:
> > +	@install -d $(O)/graphs
> > +	@./support/scripts/graph-depends $(@:-graph-depends=) \
> > +	|dot -Tpdf \
> > +	-o $(O)/graphs/$(@).pdf
> 
> Why isn't that one done in the package infrastructure, i.e in
> pkg-generic.mk ?

For two reasons:
  - to have both graph-depends and %-graph-depends side-by-side, so it
    is easier to update the rules,
  - to limit the number of rules in the Makefile, which is already a bit
    long to parse.

I initially added that in package/pkg-generic.mk:
    $(1)-graph-depends:
        @install -d $(O)/graphs
        @./support/scripts/graph-depends $(1) \
        |dot -Tpdf \
        -o $(O)/graphs/$$(@).pdf

But this adds yet another rule per-package to the Makefile. So for the
reasons above, I decided to move it with graph-depends.

But I don't really care, I can change it back to a per-package rule.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 3/4] Makefile: expose 'graph-depends' to generate a graph of the dependency tree

[Thomas Petazzoni]

Dear Yann E. MORIN,

On Sat, 28 Dec 2013 12:56:34 +0100, Yann E. MORIN wrote:

> > Why isn't that one done in the package infrastructure, i.e in
> > pkg-generic.mk ?
> 
> For two reasons:
>   - to have both graph-depends and %-graph-depends side-by-side, so it
>     is easier to update the rules,

Right, that's a pretty good reason.

>   - to limit the number of rules in the Makefile, which is already a bit
>     long to parse.

Well, you still have the rule in pkg-generic.mk, which you added to
make the completion work. So practically speaking, the number of rules
is the same.

> I initially added that in package/pkg-generic.mk:
>     $(1)-graph-depends:
>         @install -d $(O)/graphs

Should be $(INSTALL) maybe?

>         @./support/scripts/graph-depends $(1) \
>         |dot -Tpdf \
>         -o $(O)/graphs/$$(@).pdf
> 
> But this adds yet another rule per-package to the Makefile. So for the
> reasons above, I decided to move it with graph-depends.
> 
> But I don't really care, I can change it back to a per-package rule.

To me, it makes more sense to have all the per-package rules in the
package infrastructure, but that's not a very strong opinion, since the
argument of having both the per-package and the global rules together
also makes sense.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 3/4] Makefile: expose 'graph-depends' to generate a graph of the dependency tree

[Yann E. MORIN]

On 2013-12-28 13:06 +0100, Thomas Petazzoni spake thusly:
> Dear Yann E. MORIN,
> 
> On Sat, 28 Dec 2013 12:56:34 +0100, Yann E. MORIN wrote:
> 
> > > Why isn't that one done in the package infrastructure, i.e in
> > > pkg-generic.mk ?
> > 
> > For two reasons:
> >   - to have both graph-depends and %-graph-depends side-by-side, so it
> >     is easier to update the rules,
> 
> Right, that's a pretty good reason.
> 
> >   - to limit the number of rules in the Makefile, which is already a bit
> >     long to parse.
> 
> Well, you still have the rule in pkg-generic.mk, which you added to
> make the completion work. So practically speaking, the number of rules
> is the same.

Doh, you're right...

> > I initially added that in package/pkg-generic.mk:
> >     $(1)-graph-depends:
> >         @install -d $(O)/graphs
> 
> Should be $(INSTALL) maybe?

Yes, I'll change that.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 1/4] graph-build-time: generate graphs based on timing data

[Thomas Petazzoni]

Dear Yann E. MORIN,

On Sat, 28 Dec 2013 00:12:49 +0100, Yann E. MORIN wrote:

> Example usage:
> 
>   cat $(O)/build-time.log | \
>       ./support/scripts/graph-build-time \
>       --type=histogram --output=foobar.pdf

I'm wondering: since the script has a --output/-o option, would it make
more sense to have a --input/-i option instead of passing the
build-time.log contents on the standard input?

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 3/4] Makefile: expose 'graph-depends' to generate a graph of the dependency tree

[Thomas Petazzoni]

Dear Yann E. MORIN,

On Sat, 28 Dec 2013 00:12:51 +0100, Yann E. MORIN wrote:

>  all: menuconfig
> @@ -857,6 +869,7 @@ endif
>  	@echo '  manual-text            - build manual in text'
>  	@echo '  manual-epub            - build manual in ePub'
>  	@echo '  graph-build            - generate graphs of the build times'
> +	@echo '  graph-depends          - generate graph of the dependency tree'

You could also describe:

	@echo '  <package>-graph-depends'

near the help text for <package>-rebuild and <package>-reconfigure

Or at least mention it somewhere in the manual.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 4/4] graphs: support generating png graphs

[Thomas Petazzoni]

Dear Yann E. MORIN,

On Sat, 28 Dec 2013 00:12:52 +0100, Yann E. MORIN wrote:

>  	@echo '  make V=0|1             - 0 => quiet build (default), 1 => verbose build'
>  	@echo '  make O=dir             - Locate all output files in "dir", including .config'
>  	@echo '  make GRAPH_ALT=1       - use alternate color-scheme for build-time graphs'
> +	@echo '  make GRAPH_OUT=pdf|png - generate either pdf (default) or png graphs'

I'm a little bit concerned about these variables. They are specific to
the graph-build-time thing, but still they clutter the 'make help'
output, just for some very special functionality.

Since it's quite specific, shouldn't we just document them in the
manual instead, and keep only the graph-build target in make help?

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 1/4] graph-build-time: generate graphs based on timing data

[Yann E. MORIN]

Thomas, All,

On 2013-12-28 16:26 +0100, Thomas Petazzoni spake thusly:
> On Sat, 28 Dec 2013 00:12:49 +0100, Yann E. MORIN wrote:
> 
> > Example usage:
> > 
> >   cat $(O)/build-time.log | \
> >       ./support/scripts/graph-build-time \
> >       --type=histogram --output=foobar.pdf
> 
> I'm wondering: since the script has a --output/-o option, would it make
> more sense to have a --input/-i option instead of passing the
> build-time.log contents on the standard input?

Agreed. If no --input is specified, then read stdin, otherwise, read
file specified with --input.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 1/4] graph-build-time: generate graphs based on timing data

[Yann E. MORIN]

From: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>

This script generates graphs of packages build time, from the timing
data generated by Buildroot in the $(O)/build-time.log file.

Example usage:

  cat $(O)/build-time.log | \
      ./support/scripts/graph-build-time \
      --type=histogram --output=foobar.pdf

Three graph types are available :

  * histogram, which creates an histogram of the build time for each
    package, decomposed by each step (extract, patch, configure,
    etc.). The order in which the packages are shown is
    configurable: by package name, by build order, or by duration
    order. See the --order option.

  * pie-packages, which creates a pie chart of the build time of
    each package (without decomposition in steps). Packages that
    contributed to less than 1% of the overall build time are all
    grouped together in an "Other" entry.

  * pie-steps, which creates a pie chart of the time spent globally
    on each step (extract, patch, configure, etc...)

The default is to generate an histogram ordered by package name.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
[yann.morin.1998 at free.fr: adapt to the format of the step-hooks build-time.log,
    add sort order by name, default to name-ordered histogram, use our colours
    for pie-charts, add alternate color-scheme, add short-options, add
    --input/-i]
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
---
 support/scripts/graph-build-time | 291 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 291 insertions(+)
 create mode 100755 support/scripts/graph-build-time

diff --git a/support/scripts/graph-build-time b/support/scripts/graph-build-time
new file mode 100755
index 0000000..2216db2
--- /dev/null
+++ b/support/scripts/graph-build-time
@@ -0,0 +1,291 @@
+#!/usr/bin/env python
+
+# Copyright (C) 2011 by Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
+# Copyright (C) 2013 by Yann E. MORIN <yann.morin.1998@free.fr>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+# This script generates graphs of packages build time, from the timing
+# data generated by Buildroot in the $(O)/build-time.log file.
+#
+# Example usage:
+#
+#   cat $(O)/build-time.log | ./support/scripts/graph-build-time --type=histogram --output=foobar.pdf
+#
+# Three graph types are available :
+#
+#   * histogram, which creates an histogram of the build time for each
+#     package, decomposed by each step (extract, patch, configure,
+#     etc.). The order in which the packages are shown is
+#     configurable: by package name, by build order, or by duration
+#     order. See the --order option.
+#
+#   * pie-packages, which creates a pie chart of the build time of
+#     each package (without decomposition in steps). Packages that
+#     contributed to less than 1% of the overall build time are all
+#     grouped together in an "Other" entry.
+#
+#   * pie-steps, which creates a pie chart of the time spent globally
+#     on each step (extract, patch, configure, etc...)
+#
+# The default is to generate an histogram ordered by package name.
+#
+# Requirements:
+#
+#   * matplotlib (python-matplotlib on Debian/Ubuntu systems)
+#   * numpy (python-numpy on Debian/Ubuntu systems)
+#   * argparse (by default in Python 2.7, requires python-argparse if
+#     Python 2.6 is used)
+
+import matplotlib
+import numpy
+matplotlib.use('PDF')
+
+import matplotlib.pyplot as plt
+import matplotlib.font_manager as fm
+import csv
+import argparse
+import sys
+
+steps = [ 'extract', 'patch', 'configure', 'build',
+          'install-target', 'install-staging', 'install-images',
+          'install-host']
+
+default_colors = ['#e60004', '#009836', '#2e1d86', '#ffed00',
+                  '#0068b5', '#f28e00', '#940084', '#97c000']
+
+alternate_colors = ['#00e0e0', '#3f7f7f', '#ff0000', '#00c000',
+                    '#0080ff', '#c000ff', '#00eeee', '#e0e000']
+
+class Package:
+    def __init__(self, name):
+        self.name = name
+        self.steps_duration = {}
+        self.steps_start = {}
+        self.steps_end = {}
+
+    def add_step(self, step, state, time):
+        if state == "start":
+            self.steps_start[step] = time
+        else:
+            self.steps_end[step] = time
+        if self.steps_start.has_key(step) and self.steps_end.has_key(step):
+            self.steps_duration[step] = self.steps_end[step] - self.steps_start[step]
+
+    def get_duration(self, step=None):
+        if step == None:
+            duration = 0
+            for step in self.steps_duration.keys():
+                duration += self.steps_duration[step]
+            return duration
+        if self.steps_duration.has_key(step):
+            return self.steps_duration[step]
+        return 0
+
+# Generate an histogram of the time spent in each step of each
+# package.
+def pkg_histogram(data, output, order="build"):
+    n_pkgs = len(data)
+    ind = numpy.arange(n_pkgs)
+
+    if order == "duration":
+        data = sorted(data, key=lambda p: p.get_duration(), reverse=True)
+    elif order == "name":
+        data = sorted(data, key=lambda p: p.name, reverse=False)
+
+    # Prepare the vals array, containing one entry for each step
+    vals = []
+    for step in steps:
+        val = []
+        for p in data:
+            val.append(p.get_duration(step))
+        vals.append(val)
+
+    bottom = [0] * n_pkgs
+    legenditems = []
+
+    plt.figure()
+
+    # Draw the bars, step by step
+    for i in range(0, len(vals)):
+        b = plt.bar(ind+0.1, vals[i], width=0.8, color=colors[i], bottom=bottom, linewidth=0.25)
+        legenditems.append(b[0])
+        bottom = [ bottom[j] + vals[i][j] for j in range(0, len(vals[i])) ]
+
+    # Draw the package names
+    plt.xticks(ind + .6, [ p.name for p in data ], rotation=-60, rotation_mode="anchor", fontsize=8, ha='left')
+
+    # Adjust size of graph (double the width)
+    sz = plt.gcf().get_size_inches()
+    plt.gcf().set_size_inches(sz[0] * 2, sz[1])
+
+    # Add more space for the package names at the bottom
+    plt.gcf().subplots_adjust(bottom=0.2)
+
+    # Remove ticks in the graph for each package
+    axes = plt.gcf().gca()
+    for line in axes.get_xticklines():
+        line.set_markersize(0)
+
+    axes.set_ylabel('Time (seconds)')
+
+    # Reduce size of legend text
+    leg_prop = fm.FontProperties(size=6)
+
+    # Draw legend
+    plt.legend(legenditems, steps, prop=leg_prop)
+
+    if order == "name":
+        plt.title('Build time of packages\n')
+    elif order == "build":
+        plt.title('Build time of packages, by build order\n')
+    elif order == "duration":
+        plt.title('Build time of packages, by duration order\n')
+
+    # Save graph
+    plt.savefig(output)
+
+# Generate a pie chart with the time spent building each package.
+def pkg_pie_time_per_package(data, output):
+    # Compute total build duration
+    total = 0
+    for p in data:
+        total += p.get_duration()
+
+    # Build the list of labels and values, and filter the packages
+    # that account for less than 1% of the build time.
+    labels = []
+    values = []
+    other_value = 0
+    for p in data:
+        if p.get_duration() < (total * 0.01):
+            other_value += p.get_duration()
+        else:
+            labels.append(p.name)
+            values.append(p.get_duration())
+
+    labels.append('Other')
+    values.append(other_value)
+
+    plt.figure()
+
+    # Draw pie graph
+    patches, texts, autotexts = plt.pie(values, labels=labels,
+                                        autopct='%1.1f%%', shadow=True,
+                                        colors=colors)
+
+    # Reduce text size
+    proptease = fm.FontProperties()
+    proptease.set_size('xx-small')
+    plt.setp(autotexts, fontproperties=proptease)
+    plt.setp(texts, fontproperties=proptease)
+
+    plt.title('Build time per package')
+    plt.savefig(output)
+
+# Generate a pie chart with a portion for the overall time spent in
+# each step for all packages.
+def pkg_pie_time_per_step(data, output):
+    steps_values = []
+    for step in steps:
+        val = 0
+        for p in data:
+            val += p.get_duration(step)
+        steps_values.append(val)
+
+    plt.figure()
+
+    # Draw pie graph
+    patches, texts, autotexts = plt.pie(steps_values, labels=steps,
+                                        autopct='%1.1f%%', shadow=True,
+                                        colors=colors)
+
+    # Reduce text size
+    proptease = fm.FontProperties()
+    proptease.set_size('xx-small')
+    plt.setp(autotexts, fontproperties=proptease)
+    plt.setp(texts, fontproperties=proptease)
+
+    plt.title('Build time per step')
+    plt.savefig(output)
+
+# Parses the csv file passed on standard input and returns a list of
+# Package objects, filed with the duration of each step and the total
+# duration of the package.
+def read_data(input_file):
+    if input_file is None:
+        input_file = sys.stdin
+    else:
+        input_file = open(input_file)
+    reader = csv.reader(input_file, delimiter=':')
+    pkgs = []
+
+    # Auxilliary function to find a package by name in the list.
+    def getpkg(name):
+        for p in pkgs:
+            if p.name == name:
+                return p
+        return None
+
+    for row in reader:
+        time = int(row[0].strip())
+        state = row[1].strip()
+        step = row[2].strip()
+        pkg = row[3].strip()
+
+        p = getpkg(pkg)
+        if p is None:
+            p = Package(pkg)
+            pkgs.append(p)
+
+        p.add_step(step, state, time)
+
+    return pkgs
+
+parser = argparse.ArgumentParser(description='Draw build time graphs')
+parser.add_argument("--type", '-t', metavar="GRAPH_TYPE",
+                    help="Type of graph (histogram, pie-packages, pie-steps)")
+parser.add_argument("--order", '-O', metavar="GRAPH_ORDER",
+                    help="Ordering of packages: build or duration (for histogram only)")
+parser.add_argument("--alternate-colors", '-c', action="store_true",
+                    help="Use alternate colour-scheme")
+parser.add_argument("--input", '-i', metavar="OUTPUT",
+                    help="Input file (usually $(O)/build/build-time.log)")
+parser.add_argument("--output", '-o', metavar="OUTPUT", required=True,
+                    help="Output file (PDF extension)")
+args = parser.parse_args()
+
+d = read_data(args.input)
+
+if args.alternate_colors:
+    colors = alternate_colors
+else:
+    colors = default_colors
+
+if args.type == "histogram" or args.type == None:
+    if args.order == "build" or args.order == "duration" or args.order == "name":
+        pkg_histogram(d, args.output, args.order)
+    elif args.order == None:
+        pkg_histogram(d, args.output, "name")
+    else:
+        sys.stderr.write("Unknown ordering: %s\n" % args.order)
+        exit(1)
+elif args.type == "pie-packages":
+    pkg_pie_time_per_package(d, args.output)
+elif args.type == "pie-steps":
+    pkg_pie_time_per_step(d, args.output)
+else:
+    sys.stderr.write("Unknown type: %s\n" % args.type)
+    exit(1)
-- 
1.8.1.2

[Buildroot] [PATCH 4/4] graphs: support generating png graphs

[Peter Korsgaard]

>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@free-electrons.com> writes:

 > Dear Yann E. MORIN,
 > On Sat, 28 Dec 2013 00:12:52 +0100, Yann E. MORIN wrote:

 >> @echo '  make V=0|1             - 0 => quiet build (default), 1 => verbose build'
 >> @echo '  make O=dir             - Locate all output files in "dir", including .config'
 >> @echo '  make GRAPH_ALT=1       - use alternate color-scheme for build-time graphs'
 >> +	@echo '  make GRAPH_OUT=pdf|png - generate either pdf (default) or png graphs'

 > I'm a little bit concerned about these variables. They are specific to
 > the graph-build-time thing, but still they clutter the 'make help'
 > output, just for some very special functionality.

 > Since it's quite specific, shouldn't we just document them in the
 > manual instead, and keep only the graph-build target in make help?

Yes, I also think the manual is a better place for this.

-- 
Bye, Peter Korsgaard