[PATCH 0/1] bb-matrix performance metrics gathering and visualization scripts

[PATCH 0/1] bb-matrix performance metrics gathering and visualization scripts

[Darren Hart]

Add bb-matrix.sh and bb-matrix-plot.sh. Example output of these scripts is
viewable here:

https://wiki.yoctoproject.org/wiki/Build_Performance#bb-matrix

The following changes since commit 9f4eaeef33da5595748253d59d95c7ca548e28fa:

  libx11: enable xcb support (2011-07-08 23:02:08 +0100)

are available in the git repository at:
  git://git.pokylinux.org/poky-contrib dvhart/bb-perf
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=dvhart/bb-perf

Darren Hart (1):
  bb-matrix: initial scripts to record TIME(1) metrics for BB and PM
    combinations

 scripts/contrib/bb-perf/bb-matrix-plot.sh |  137 +++++++++++++++++++++++++++++
 scripts/contrib/bb-perf/bb-matrix.sh      |   78 ++++++++++++++++
 2 files changed, 215 insertions(+), 0 deletions(-)
 create mode 100755 scripts/contrib/bb-perf/bb-matrix-plot.sh
 create mode 100755 scripts/contrib/bb-perf/bb-matrix.sh

[PATCH 1/1] bb-matrix: initial scripts to record TIME(1) metrics for BB and PM combinations

[Darren Hart]

The bb-matrix.sh script will run a bitbake command, building core-image-minimal
by default, for various combinations of BB_NUMBER_THREADS and PARALLEL_MAKE. It
records all relevant metrics of the TIME(1) command for each combination in a
data file.

The bb-matrix-plot.sh script can be used to visualize each of these metrics via
a 3d surface plot, either solid surface or wireframe with a value-map
projection on the XY plane.

Signed-off-by: Darren Hart <dvhart@linux.intel.com>
---
 scripts/contrib/bb-perf/bb-matrix-plot.sh |  137 +++++++++++++++++++++++++++++
 scripts/contrib/bb-perf/bb-matrix.sh      |   78 ++++++++++++++++
 2 files changed, 215 insertions(+), 0 deletions(-)
 create mode 100755 scripts/contrib/bb-perf/bb-matrix-plot.sh
 create mode 100755 scripts/contrib/bb-perf/bb-matrix.sh

diff --git a/scripts/contrib/bb-perf/bb-matrix-plot.sh b/scripts/contrib/bb-perf/bb-matrix-plot.sh
new file mode 100755
index 0000000..62aa66d
--- /dev/null
+++ b/scripts/contrib/bb-perf/bb-matrix-plot.sh
@@ -0,0 +1,137 @@
+#!/bin/bash
+#
+# Copyright (c) 2011, Intel Corporation.
+# All rights reserved.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
+#
+# DESCRIPTION
+# This script operates on the .dat file generated by bb-matrix.sh. It tolerates
+# the header by skipping the first line, but error messages and bad data records
+# need to be removed first. It will generate three views of the plot, and leave
+# an interactive view open for further analysis.
+#
+# AUTHORS
+# Darren Hart <dvhart@linux.intel.com>
+#
+
+# Setup the defaults
+DATFILE="bb-matrix.dat"
+XLABEL="BB_NUMBER_THREADS"
+YLABEL="PARALLEL_MAKE"
+FIELD=3
+DEF_TITLE="Elapsed Time (seconds)"
+PM3D_FRAGMENT="unset surface; set pm3d at s hidden3d 100"
+SIZE="640,480"
+
+function usage {
+CMD=$(basename $0)
+cat <<EOM
+Usage: $CMD [-d datfile] [-f field] [-h] [-t title] [-w]
+  -d datfile    The data file generated by bb-matrix.sh (default: $DATFILE)
+  -f field      The field index to plot as the Z axis from the data file
+                (default: $FIELD, "$DEF_TITLE")
+  -h            Display this help message
+  -s W,H        PNG and window size in pixels (default: $SIZE)
+  -t title      The title to display, should describe the field (-f) and units
+                (default: "$DEF_TITLE")
+  -w            Render the plot as wireframe with a 2D colormap projected on the
+                XY plane rather than as the texture for the surface
+EOM
+}
+
+# Parse and validate arguments
+while getopts "d:f:hs:t:w" OPT; do
+	case $OPT in
+	d)
+		DATFILE="$OPTARG"
+		;;
+	f)
+		FIELD="$OPTARG"
+		;;
+	h)
+		usage
+		exit 0
+		;;
+	s)
+		SIZE="$OPTARG"
+		;;
+	t)
+		TITLE="$OPTARG"
+		;;
+	w)
+		PM3D_FRAGMENT="set pm3d at b"
+		W="-w"
+		;;
+	*)
+		usage
+		exit 1
+		;;
+	esac
+done
+
+# Ensure the data file exists
+if [ ! -f "$DATFILE" ]; then
+	echo "ERROR: $DATFILE does not exist"
+	usage
+	exit 1
+fi
+PLOT_BASENAME=${DATFILE%.*}-f$FIELD$W
+
+# Set a sane title
+# TODO: parse the header and define titles for each format parameter for TIME(1)
+if [ -z "$TITLE" ]; then
+	if [ ! "$FIELD" == "3" ]; then
+		TITLE="Field $FIELD"
+	else
+		TITLE="$DEF_TITLE"
+	fi
+fi
+
+# Determine the dgrid3d mesh dimensions size
+MIN=$(tail -n +2 "$DATFILE" | cut -d ' ' -f 1 | sort | uniq | head -n1)
+MAX=$(tail -n +2 "$DATFILE" | cut -d ' ' -f 1 | sort | uniq | tail -n1)
+BB_CNT=$[${MAX#*0} - $MIN + 1]
+MIN=$(tail -n +2 "$DATFILE" | cut -d ' ' -f 2 | sort | uniq | head -n1)
+MAX=$(tail -n +2 "$DATFILE" | cut -d ' ' -f 2 | sort | uniq | tail -n1)
+PM_CNT=$[${MAX#*0} - $MIN + 1]
+
+
+(cat <<EOF
+set title "$TITLE"
+set xlabel "$XLABEL"
+set ylabel "$YLABEL"
+set style line 100 lt 5 lw 1.5
+$PM3D_FRAGMENT
+set dgrid3d $PM_CNT,$BB_CNT
+set ticslevel 0.2
+
+set term png size $SIZE
+set output "$PLOT_BASENAME.png"
+splot "$DATFILE" every ::1 using 1:2:$FIELD with lines ls 100
+
+set view 90,0
+set output "$PLOT_BASENAME-bb.png"
+replot
+
+set view 90,90
+set output "$PLOT_BASENAME-pm.png"
+replot
+
+set view 60,30
+set term wxt size $SIZE
+replot
+EOF
+) | gnuplot --persist
diff --git a/scripts/contrib/bb-perf/bb-matrix.sh b/scripts/contrib/bb-perf/bb-matrix.sh
new file mode 100755
index 0000000..64d5513
--- /dev/null
+++ b/scripts/contrib/bb-perf/bb-matrix.sh
@@ -0,0 +1,78 @@
+#!/bin/bash
+#
+# Copyright (c) 2011, Intel Corporation.
+# All rights reserved.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
+#
+# DESCRIPTION
+# This script runs BB_CMD (typically building core-image-sato) for all
+# combincations of BB_RANGE and PM_RANGE values. It saves off all the console
+# logs, the buildstats directories, and creates a bb-pm-runtime.dat file which
+# can be used to postprocess the results with a plotting tool, spreadsheet, etc.
+# Before running this script, it is recommended that you pre-download all the
+# necessary sources by performing the BB_CMD once manually. It is also a good
+# idea to disable cron to avoid runtime variations caused by things like the
+# locate process. Be sure to sanitize the dat file prior to post-processing as
+# it may contain error messages or bad runs that should be removed.
+#
+# AUTHORS
+# Darren Hart <dvhart@linux.intel.com>
+#
+
+# The following ranges are appropriate for a 4 core system with 8 logical units
+BB_RANGE="04 05 06 07 08 09 10 11 12 13 14 15 16"
+PM_RANGE="04 05 06 07 08 09 10 11 12 13 14 15 16"
+
+DATADIR="bb-matrix-$$"
+BB_CMD="bitbake core-image-minimal"
+RUNTIME_LOG="$DATADIR/bb-matrix.dat"
+
+# See TIME(1) for a description of the time format parameters
+# The following all report 0: W K r s t w
+TIME_STR="%e %S %U %P %c %w %R %F %M %x"
+
+# Prepare the DATADIR
+mkdir $DATADIR
+if [ $? -ne 0 ]; then
+	echo "Failed to create $DATADIR."
+	exit 1
+fi
+
+# Add a simple header
+echo "BB PM $TIME_STR" > $RUNTIME_LOG
+for BB in $BB_RANGE; do
+	for PM in $PM_RANGE; do
+		RUNDIR="$DATADIR/$BB-$PM-build"
+		mkdir $RUNDIR
+		BB_LOG=$RUNDIR/$BB-$PM-bitbake.log
+		date
+		echo "BB=$BB PM=$PM Logging to $BB_LOG"
+
+		# Export the variables under test and run the bitbake command
+		export BB_NUMBER_THREADS="${BB##*0}"
+		export PARALLEL_MAKE="-j ${PM##*0}"
+		/usr/bin/time -f "$BB $PM $TIME_STR" -a -o $RUNTIME_LOG $BB_CMD &> $BB_LOG
+		
+		echo "  $(tail -n1 $RUNTIME_LOG)"
+		echo -n "  Cleaning up..."
+		mv tmp/buildstats $RUNDIR/$BB-$PM-buildstats
+		rm -f pseudodone &> /dev/null
+		rm -rf tmp &> /dev/null
+		rm -rf sstate-cache &> /dev/null
+		rm -rf tmp-eglibc &> /dev/null
+		echo "done"
+	done
+done
-- 
1.7.1

Re: [PATCH 0/1] bb-matrix performance metrics gathering and visualization scripts

[Otavio Salvador]

On Fri, Jul 8, 2011 at 23:02, Darren Hart <dvhart@linux.intel.com> wrote:
>  bb-matrix: initial scripts to record TIME(1) metrics for BB and PM
>    combinations

please drop initial scripts; it doesn't matter et all if it is a
script and if it is merged it is because it is usable and worth having
it so the initial status makes no useful contribution to the short
description. Long description can make clear and more detailed
explanation.

-- 
Otavio Salvador                             O.S. Systems
E-mail: otavio@ossystems.com.br  http://www.ossystems.com.br
Mobile: +55 53 9981-7854              http://projetos.ossystems.com.br

Re: [PATCH 1/1] bb-matrix: initial scripts to record TIME(1) metrics for BB and PM combinations

[Richard Purdie]

On Fri, 2011-07-08 at 19:02 -0700, Darren Hart wrote:
> The bb-matrix.sh script will run a bitbake command, building core-image-minimal
> by default, for various combinations of BB_NUMBER_THREADS and PARALLEL_MAKE. It
> records all relevant metrics of the TIME(1) command for each combination in a
> data file.
> 
> The bb-matrix-plot.sh script can be used to visualize each of these metrics via
> a 3d surface plot, either solid surface or wireframe with a value-map
> projection on the XY plane.

Merged to master, thanks.

Richard