[PATCH 0/2] Toaster-related improvements

[PATCH 0/2] Toaster-related improvements

[Paul Eggleton]

Adds a class that allows Toaster (the new web-based UI for analysing
build output) to receive information about packages and image files,
and updates documentation.conf (which is used by Toaster to display
descriptions in the variable listing).


The following changes since commit 0636093711547957a8f5b25322bd3e0da367cfc4:

  sgml-common: make postinst run successfully at rootfs time (2013-11-01 11:08:31 +0000)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib paule/toaster-oe
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/toaster-oe

Alexandru DAMIAN (1):
  toaster: add class to dump toaster-tracked data

Belen Barros Pena (1):
  documentation.conf: update contents

 meta/classes/toaster.bbclass | 110 +++++++++++++++++++++++++++++++++++++++++++
 meta/conf/documentation.conf |  23 +++++----
 2 files changed, 124 insertions(+), 9 deletions(-)
 create mode 100644 meta/classes/toaster.bbclass

-- 
1.8.1.2

[PATCH 1/2] documentation.conf: update contents

[Paul Eggleton]

From: Belen Barros Pena <belen.barros.pena@intel.com>

This patch updates descriptions and brings more
info for the variable configurations as documented
in the OE-Core manual.

This file is used by Toaster to display help for
the configuration variables.

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/conf/documentation.conf | 23 ++++++++++++++---------
 1 file changed, 14 insertions(+), 9 deletions(-)

diff --git a/meta/conf/documentation.conf b/meta/conf/documentation.conf
index ecc7c8f..6666648 100644
--- a/meta/conf/documentation.conf
+++ b/meta/conf/documentation.conf
@@ -42,6 +42,7 @@ CFLAGS[doc] = "Flags passed to the C compiler for the target system. This variab
 COMBINED_FEATURES[doc] = "A set of features common between MACHINE_FEATURES and DISTRO_FEATURES."
 COMPATIBLE_HOST[doc] = "A regular expression that resolves to one or more hosts (when the recipe is native) or one or more targets (when the recipe is non-native) with which a recipe is compatible."
 COMPATIBLE_MACHINE[doc] = "A regular expression that resolves to one or more target machines with which a recipe is compatible."
+CONF_VERSION[doc] = "Increased each time build/conf/ changes incompatibly and used to track the version of local.conf"
 CONFFILES[doc] = "Identifies editable or configurable files that are part of a package."
 CONFIG_SITE[doc] = "A list of files that contains autoconf test results relevant to the current build. This variable is used by the Autotools utilities when running configure."
 CORE_IMAGE_EXTRA_INSTALL[doc] = "Specifies the list of packages to be added to the image. You should only set this variable in the conf/local.conf file in the Build Directory."
@@ -117,6 +118,7 @@ IMAGE_LINGUAS[doc] = "Specifies the list of locales to install into the image du
 IMAGE_OVERHEAD_FACTOR[doc] = "Defines a multiplier that the build system applies to the initial image size for cases when the multiplier times the returned disk usage value for the image is greater than the sum of IMAGE_ROOTFS_SIZE and IMAGE_ROOTFS_EXTRA_SPACE."
 IMAGE_ROOTFS_EXTRA_SPACE[doc] = "Defines additional free disk space created in the image in Kbytes. By default, this variable is set to '0'."
 IMAGE_ROOTFS_SIZE[doc] = "Defines the size in Kbytes for the generated image."
+IMAGETEST[doc] = "Enable test booting of virtual machine images under the qemu emulator after any root filesystems are created and run tests against those images."
 INC_PR[doc] = "Helps define the recipe revision for recipes that share a common include file."
 INHIBIT_PACKAGE_STRIP[doc] = "If set to "1", causes the build to not strip binaries in resulting packages."
 INHERIT[doc] = "Causes the named class to be inherited at this point during parsing. The variable is only valid in configuration files."
@@ -170,18 +172,10 @@ MLPREFIX[doc] = "Specifies a prefix has been added to PN to create a special ver
 MODULE_TARBALL_DEPLOY[doc] = "Controls creation of the modules-*.tgz file. Set this variable to "0" to disable creation of this file, which contains all of the kernel modules resulting from a kernel build."
 MULTIMACH_TARGET_SYS[doc] = "Separates files for different machines such that you can build for multiple target machines using the same output directories."
 
-BAD_RECOMMENDATIONS[doc] = "List of packages to not install if recommended.  These will still be installed if required."
-NO_RECOMMENDATIONS[doc] = "When set to 1, no recommended packages will be installed.  Note: some recommended packages may be required for certain system functionality, such as kernel-modules.  It is up to the user to add packages to IMAGE_INSTALL as needed."
-
-IMAGE_INSTALL[doc] = "Used by an image recipe to list the packages to be installed.  See PACKAGE_INSTALL."
-
-PACKAGE_EXCLUDE[doc] = "Packages to exclude from the installation, if required an error will be generated."
-PACKAGE_INSTALL[doc] = "Generally not user defined.  List of the packages to be installed into the image, uses IMAGE_INSTALL as part of the list."
-PACKAGE_INSTALL_ATTEMPTONLY[doc] = "Generally not user defined.  List of packages that will be attempted to be installed, but no error will generate if any of them fail to install."
-
 #N
 
 NATIVELSBSTRING[doc] = "A string identifying the host distribution."
+NO_RECOMMENDATIONS[doc] = "When set to 1, no recommended packages will be installed. Note: some recommended packages may be required for certain system functionality, such as kernel-modules. It is up to the user to add packages to IMAGE_INSTALL as needed."
 
 #O
 
@@ -194,12 +188,16 @@ PACKAGE_ARCH[doc] = "The architecture of the resulting package or packages."
 PACKAGE_ARCHS[doc] = "A list of architectures compatible with the given target in order of priority."
 PACKAGE_BEFORE_PN[doc] = "Enables easily adding packages to PACKAGES before ${PN} so that the packages can pick up files that would normally be included in the default package."
 PACKAGE_CLASSES[doc] = "This variable specifies the package manager to use when packaging data. It is set in the conf/local.conf file in the Build Directory."
+PACKAGE_EXCLUDE[doc] = "Packages to exclude from the installation, if required an error will be generated."
 PACKAGE_EXTRA_ARCHS[doc] = "Specifies the list of architectures compatible with the device CPU. This variable is useful when you build for several different devices that use miscellaneous processors."
+PACKAGE_INSTALL[doc] = "Generally not user defined. List of the packages to be installed into the image, uses IMAGE_INSTALL as part of the list."
+PACKAGE_INSTALL_ATTEMPTONLY[doc] = "Generally not user defined. List of packages that will be attempted to be installed, but no error will generate if any of them fail to install."
 PACKAGECONFIG[doc] = "This variable provides a means of enabling or disabling features of a recipe on a per-recipe basis."
 PACKAGES[doc] = "The list of packages to be created from the recipe."
 PACKAGES_DYNAMIC[doc] = "A promise that your recipe satisfies runtime dependencies for optional modules that are found in other recipes."
 PALMTOP_USE_MULTITHREADED_QT[doc] = "Set to yes, if you want to build qt apps with CONFIG+=thread"
 PARALLEL_MAKE[doc] = "Specifies extra options that are passed to the make command during the compile tasks. This variable is usually in the form -j 4, where the number represents the maximum number of parallel threads make can run."
+PATCHRESOLVE[doc] = "Enable / disable interactive patch resolution."
 PE[doc] = "The epoch of the recipe. The default value is '0'. The field is used to make upgrades possible when the versioning scheme changes in some backwards incompatible way."
 PF[doc] = "Specifies the recipe or package name and includes all version and revision numbers. This variable is comprised of ${PN}-${EXTENDPE}${PV}-${PR}"
 PN[doc] = "PN refers to a recipe name in the context of a file used by the OpenEmbedded build system as input to create a package. It refers to a package name in the context of a file created or produced by the OpenEmbedded build system."
@@ -227,6 +225,7 @@ RSUGGESTS[doc] = "A list of additional packages that you can suggest for install
 S[doc] = "The location in the Build Directory where unpacked package source code resides."
 SANITY_TESTED_DISTROS[doc] = "A list of the host distribution identifiers that the build system has been tested against."
 SDKIMAGE_FEATURES[doc] = "Equivalent to IMAGE_FEATURES. However, this variable applies to the SDK generated from an image using the command '$ bitbake -c populate_sdk imagename'."
+SDKMACHINE[doc] = "Specifies the architecture (i686 or x86_64) to build SDK/ADT tiems for."
 SECTION[doc] = "The section in which packages should be categorized. Package management utilities can make use of this variable."
 SELECTED_OPTIMIZATION[doc] = "The variable takes the value of FULL_OPTIMIZATION unless DEBUG_BUILD = "1". In this case the value of DEBUG_OPTIMIZATION is used."
 SERIAL_CONSOLE[doc] = "The speed and device for the serial port used to attach the serial console. This variable is given to the kernel as the 'console' parameter and after booting occurs getty is started on that port so remote login is possible."
@@ -263,6 +262,8 @@ TARGET_PREFIX[doc] = "The prefix for the cross compile toolchain. E.g arm-linux-
 TARGET_SYS[doc] = "The target system is composed out of TARGET_ARCH,TARGET_VENDOR and TARGET_OS."
 TCLIBC[doc] = "Specifies which variant of the GNU standard C library (libc) to use during the build process. You can select eglibc or uclibc."
 TCMODE[doc] = "The toolchain selector. It selects the external toolchain built using the OpenEmbedded build system or a few supported combinations of the upstream GCC or CodeSourcery Labs toolchain."
+TEST_SCEN[doc] = "Controls which tests are run against virtual images if testing is enabled with IMAGETEST."
+TEST_SERIALIZE[doc] = "Controls the time taken by the sanity tests."
 TIME[doc] = "The time the build was started HMS"
 TMPDIR[doc] = "This variable is the temporary directory the OpenEmbedded build system uses when it does its work building images. By default, the TMPDIR variable is named tmp within the Build Directory."
 TOPDIR[doc] = "This variable is the Build Directory. BitBake automatically sets this variable. The OpenEmbedded build system uses the Build Directory when building images."
@@ -272,6 +273,10 @@ TUNEABI_WHITELIST[doc] = "A whitelist of permissible TUNEABI values; if unset, a
 TUNECONFLICTS[doc] = "List of conflicting features for a given feature."
 TUNEVALID[doc] = "Descriptions of valid tuning features, stored as flags."
 
+#T
+
+USER_CLASSES[doc] = "List of additional classes to use when building images which enable extra features."
+
 #W
 
 WORKDIR[doc] = "The pathname of the working directory in which the OpenEmbedded build system builds a recipe. This directory is located within the TMPDIR directory structure and changes as different packages are built."
-- 
1.8.1.2

[PATCH 2/2] toaster: add class to dump toaster-tracked data

[Paul Eggleton]

From: Alexandru DAMIAN <alexandru.damian@intel.com>

Adding a new bbclass that will collect and send relevant
data from the task context to the Toaster UI.

This bbclass consists of postfuncs that get executed
right after the main task func, and in the same context.
This allows data gathering in a synchronous manner during
the build, guaranteeing data integrity. This approach also
preserves the task signatures.

The data is moved to the UI through the event system.

There is no performance impact if the class is disabled.

License is MIT.

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/toaster.bbclass | 110 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 110 insertions(+)
 create mode 100644 meta/classes/toaster.bbclass

diff --git a/meta/classes/toaster.bbclass b/meta/classes/toaster.bbclass
new file mode 100644
index 0000000..7dbb384
--- /dev/null
+++ b/meta/classes/toaster.bbclass
@@ -0,0 +1,110 @@
+#
+# Toaster helper class
+#
+# Copyright (C) 2013 Intel Corporation
+#
+# Released under the MIT license (see COPYING.MIT)
+#
+# This bbclass is designed to extract data used by OE-Core during the build process,
+# for recording in the Toaster system.
+# The data access is synchronous, preserving the build data integrity across
+# different builds.
+#
+# The data is transferred through the event system, using the MetadataEvent objects.
+#
+# The model is to enable the datadump functions as postfuncs, and have the dump
+# executed after the real taskfunc has been executed. This prevents task signature changing
+# is toaster is enabled or not. Build performance is not affected if Toaster is not enabled.
+#
+# To enable, use INHERIT in local.conf:
+#
+#       INHERIT += "toaster"
+#
+#
+#
+#
+
+# 1. Dump package file info data
+
+python toaster_package_dumpdata() {
+    """
+    Dumps the data created by emit_pkgdata
+    """
+    # replicate variables from the package.bbclass
+
+    packages = d.getVar('PACKAGES', True)
+    pkgdest = d.getVar('PKGDEST', True)
+
+    pkgdatadir = d.getVar('PKGDESTWORK', True)
+
+
+    # scan and send data for each package
+    import ast
+    import fnmatch
+
+    lpkgdata = {}
+    for pkg in packages.split():
+
+        subdata_file = pkgdatadir + "/runtime/%s" % pkg
+        lpkgdata = {}
+
+        sf = open(subdata_file, "r")
+        line = sf.readline()
+        while line:
+            (n, v) = line.rstrip().split(":", 1)
+            if pkg in n:
+                n = n.replace("_" + pkg, "")
+            lpkgdata[n] = v.strip()
+            line = sf.readline()
+            pkgsplitname = os.path.join(pkgdest, pkg)
+            # replace FILES_INFO data with a dictionary of file name - file size
+            if n == 'FILES_INFO':
+                filesizedata = {}
+                val = v.strip().replace('\\\'', '\'')
+                dictval = ast.literal_eval(val)
+                for parent, dirlist in dictval.items():
+                    idx = parent.find(pkgsplitname)
+                    if idx > -1:
+                        parent = parent[idx+len(pkgsplitname):]
+                    else:
+                        bb.error("Invalid path while looking for file ", parent)
+                    for basename in dirlist:
+                        fullpath = os.path.join(parent, basename)
+                        try:
+                            filesizedata[fullpath] = os.stat(pkgsplitname + fullpath).st_size
+                        except OSError:
+                            # we may hit a symlink that is not pointing correctly over package-split
+                            filesizedata[fullpath] = 0
+                lpkgdata[n] = filesizedata
+
+        # Fire an event containing the pkg data
+        bb.event.fire(bb.event.MetadataEvent("SinglePackageInfo", lpkgdata), d)
+}
+
+# 2. Dump output image files information
+
+python toaster_image_dumpdata() {
+    """
+    Image filename for output images is not standardized.
+    image_types.bbclass will spell out IMAGE_CMD_xxx variables that actually
+    have hardcoded ways to create image file names in them.
+    So we look for files starting with the set name.
+    """
+
+    deploy_dir_image = d.getVar('DEPLOY_DIR_IMAGE', True);
+    image_name = d.getVar('IMAGE_NAME', True);
+
+    image_info_data = {}
+
+    for dirpath, dirnames, filenames in os.walk(deploy_dir_image):
+        for fn in filenames:
+            if fn.startswith(image_name):
+                image_info_data[dirpath + fn] = os.stat(os.path.join(dirpath, fn)).st_size
+
+    bb.event.fire(bb.event.MetadataEvent("ImageFileSize",image_info_data), d)
+}
+
+
+do_package[postfuncs] += "toaster_package_dumpdata "
+
+do_rootfs[postfuncs] += "toaster_image_dumpdata "
-- 
1.8.1.2