[PATCH 00/18] Provide list of deployment artifacts

[PATCH 00/18] Provide list of deployment artifacts

[Ed Bartosh]

Hi,

This is a fix for Bug #9869 - Provide a per-target manifest of files which were, or would have been, produced

The list of artifacts produced by deployment tasks (do_deploy, do_image_complete and do_populate_sdk[_ext] is
obtained from sstate manifests and fired as a TaskArtifacts metadata event. This should allow Toaster to
handle artifacts in simple way and remove a lot of current Toaster code doing guess work.

To generate manifests for do_image_complete and do_populate_sdk they have been put under sstate control.

To avoid storing big files(images and sdk installer) in sstate new variable SSTATE_SKIP_CREATION has been
set in image.bbclass and populate_sdk_base.bbclass and sstate code was modified to avoid adding files
to sstate if SSTATE_SKIP_CREATION is set.

The following changes since commit 087c580b286816265f487e02746bfa6e26081554:

  init-install: Fixes the install script failing when not finding any mmcblk devices (2016-08-30 07:57:50 +0100)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib ed/oe-core/artifacts-9869
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/oe-core/artifacts-9869

Ed Bartosh (18):
  image-live.bbclass: deploy images to DEPLOYDIR
  image-vm.bbclass: deploy images to DEPLOYDIR
  image.bbclass: deploy images to DEPLOYDIR
  image.bbclass: deploy images to DEPLOYDIR
  image_types_uboot.bbclass: deploy images to DEPLOYDIR
  syslinux.bbclass: deploy bootloader to DEPLOYDIR
  build-appliance-image: process images in DEPLOYDIR
  populate_sdk_base.bbclass: deploy sdk artifacts to DEPLOYDIR
  rootfs-postcommands.bbclass: generate manifest in DEPLOYDIR
  selftest: renamed variable
  rootfs.py: use DEPLOYDIR instead of DEPLOY_DIR_IMAGE
  image.bbclass: put image_complete under sstate control
  image.bbclass: cleanup DEPLOYDIR
  populate_sdk_base: put populate_sdk under sstate control
  sstate.bbclass: skip packaging if SSTATE_SKIP_CREATION is set
  image: populate_sdk_base: skip sstate creation
  image: populate_sdk_base: set stamp-extra-info flag
  toaster: fire TaskArtifacts event

 meta/classes/image-live.bbclass                    | 12 +++---
 meta/classes/image-vm.bbclass                      | 22 +++++------
 meta/classes/image.bbclass                         | 14 +++++--
 meta/classes/image_types.bbclass                   | 44 +++++++++++-----------
 meta/classes/image_types_uboot.bbclass             |  2 +-
 meta/classes/populate_sdk_base.bbclass             | 28 +++++++++-----
 meta/classes/rootfs-postcommands.bbclass           |  4 +-
 meta/classes/sstate.bbclass                        |  2 +
 meta/classes/syslinux.bbclass                      |  2 +-
 meta/classes/toaster.bbclass                       | 17 +++++++++
 meta/lib/oe/rootfs.py                              |  6 +--
 meta/lib/oeqa/selftest/imagefeatures.py            |  4 +-
 .../images/build-appliance-image_15.0.0.bb         |  8 ++--
 13 files changed, 100 insertions(+), 65 deletions(-)

--
Regards,
Ed

[PATCH 01/18] image-live.bbclass: deploy images to DEPLOYDIR

[Ed Bartosh]

Changed deployment directory from DEPLOY_DIR_IMAGE to DEPLOYDIR
to make sstate machinery to do final deployment and generate
manifest.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/classes/image-live.bbclass | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/meta/classes/image-live.bbclass b/meta/classes/image-live.bbclass
index f0e6647..1294b11 100644
--- a/meta/classes/image-live.bbclass
+++ b/meta/classes/image-live.bbclass
@@ -43,7 +43,7 @@ ROOT_LIVE ?= "root=/dev/ram0"
 INITRD_IMAGE_LIVE ?= "core-image-minimal-initramfs"
 INITRD_LIVE ?= "${DEPLOY_DIR_IMAGE}/${INITRD_IMAGE_LIVE}-${MACHINE}.cpio.gz"
 
-ROOTFS ?= "${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.ext4"
+ROOTFS ?= "${DEPLOYDIR}/${IMAGE_LINK_NAME}.ext4"
 
 IMAGE_TYPEDEP_live = "ext4"
 IMAGE_TYPEDEP_iso = "ext4"
@@ -144,14 +144,14 @@ build_iso() {
 	if [ "${PCBIOS}" = "1" ] && [ "${EFI}" != "1" ] ; then
 		# PCBIOS only media
 		mkisofs -V ${BOOTIMG_VOLUME_ID} \
-		        -o ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.iso \
+		        -o ${DEPLOYDIR}/${IMAGE_NAME}.iso \
 			-b ${ISO_BOOTIMG} -c ${ISO_BOOTCAT} \
 			$mkisofs_compress_opts \
 			${MKISOFS_OPTIONS} $mkisofs_iso_level ${ISODIR}
 	else
 		# EFI only OR EFI+PCBIOS
 		mkisofs -A ${BOOTIMG_VOLUME_ID} -V ${BOOTIMG_VOLUME_ID} \
-		        -o ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.iso \
+		        -o ${DEPLOYDIR}/${IMAGE_NAME}.iso \
 			-b ${ISO_BOOTIMG} -c ${ISO_BOOTCAT} \
 			$mkisofs_compress_opts ${MKISOFS_OPTIONS} $mkisofs_iso_level \
 			-eltorito-alt-boot -eltorito-platform efi \
@@ -160,7 +160,7 @@ build_iso() {
 		isohybrid_args="-u"
 	fi
 
-	isohybrid $isohybrid_args ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.iso
+	isohybrid $isohybrid_args ${DEPLOYDIR}/${IMAGE_NAME}.iso
 }
 
 build_fat_img() {
@@ -252,13 +252,13 @@ build_hddimg() {
 			fi
 		fi
 
-		build_fat_img ${HDDDIR} ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.hddimg
+		build_fat_img ${HDDDIR} ${DEPLOYDIR}/${IMAGE_NAME}.hddimg
 
 		if [ "${PCBIOS}" = "1" ]; then
 			syslinux_hddimg_install
 		fi
 
-		chmod 644 ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.hddimg
+		chmod 644 ${DEPLOYDIR}/${IMAGE_NAME}.hddimg
 	fi
 }
 
-- 
2.1.4

[PATCH 02/18] image-vm.bbclass: deploy images to DEPLOYDIR

[Ed Bartosh]

Changed deployment directory from DEPLOY_DIR_IMAGE to DEPLOYDIR
to make sstate machinery to do final deployment and generate
manifest.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/classes/image-vm.bbclass | 22 +++++++++++-----------
 1 file changed, 11 insertions(+), 11 deletions(-)

diff --git a/meta/classes/image-vm.bbclass b/meta/classes/image-vm.bbclass
index bf57e2c..2a86b40 100644
--- a/meta/classes/image-vm.bbclass
+++ b/meta/classes/image-vm.bbclass
@@ -33,14 +33,14 @@ IMAGE_TYPEDEP_hdddirect = "${VM_ROOTFS_TYPE}"
 IMAGE_TYPES_MASKED += "vmdk vdi qcow2 hdddirect"
 
 VM_ROOTFS_TYPE ?= "ext4"
-ROOTFS ?= "${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.${VM_ROOTFS_TYPE}"
+ROOTFS ?= "${DEPLOYDIR}/${IMAGE_LINK_NAME}.${VM_ROOTFS_TYPE}"
 
 # Used by bootloader
 LABELS_VM ?= "boot"
 ROOT_VM ?= "root=/dev/sda2"
 # Using an initramfs is optional. Enable it by setting INITRD_IMAGE_VM.
 INITRD_IMAGE_VM ?= ""
-INITRD_VM ?= "${@'${DEPLOY_DIR_IMAGE}/${INITRD_IMAGE_VM}-${MACHINE}.cpio.gz' if '${INITRD_IMAGE_VM}' else ''}"
+INITRD_VM ?= "${@'${DEPLOYDIR}/${INITRD_IMAGE_VM}-${MACHINE}.cpio.gz' if '${INITRD_IMAGE_VM}' else ''}"
 do_bootdirectdisk[depends] += "${@'${INITRD_IMAGE_VM}:do_image_complete' if '${INITRD_IMAGE_VM}' else ''}"
 
 BOOTDD_VOLUME_ID   ?= "boot"
@@ -52,7 +52,7 @@ DISK_SIGNATURE[vardepsexclude] = "DISK_SIGNATURE_GENERATED"
 build_boot_dd() {
 	HDDDIR="${S}/hdd/boot"
 	HDDIMG="${S}/hdd.image"
-	IMAGE=${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.hdddirect
+	IMAGE=${DEPLOYDIR}/${IMAGE_NAME}.hdddirect
 
 	populate_kernel $HDDDIR
 
@@ -104,13 +104,13 @@ build_boot_dd() {
 	dd if=$HDDIMG of=$IMAGE conv=notrunc seek=1 bs=512
 	dd if=${ROOTFS} of=$IMAGE conv=notrunc seek=$OFFSET bs=512
 
-	cd ${DEPLOY_DIR_IMAGE}
+	cd ${DEPLOYDIR}
 
-	if [ "${RM_OLD_IMAGE}" = "1" ] && [ -L ${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.hdddirect ]; then
-		rm -f $(readlink -f ${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.hdddirect)
+	if [ "${RM_OLD_IMAGE}" = "1" ] && [ -L ${DEPLOYDIR}/${IMAGE_LINK_NAME}.hdddirect ]; then
+		rm -f $(readlink -f ${DEPLOYDIR}/${IMAGE_LINK_NAME}.hdddirect)
 	fi
 
-	ln -sf ${IMAGE_NAME}.hdddirect ${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.hdddirect
+	ln -sf ${IMAGE_NAME}.hdddirect ${DEPLOYDIR}/${IMAGE_LINK_NAME}.hdddirect
 } 
 
 python do_bootdirectdisk() {
@@ -145,13 +145,13 @@ DISK_SIGNATURE_GENERATED := "${@generate_disk_signature()}"
 
 run_qemu_img (){
     type="$1"
-    qemu-img convert -O $type ${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.hdddirect ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.$type
+    qemu-img convert -O $type ${DEPLOYDIR}/${IMAGE_LINK_NAME}.hdddirect ${DEPLOYDIR}/${IMAGE_NAME}.$type
 
-    if [ "${RM_OLD_IMAGE}" = "1" ] && [ -L ${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.$type ]; then
-        rm -f $(readlink -f ${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.$type)
+    if [ "${RM_OLD_IMAGE}" = "1" ] && [ -L ${DEPLOYDIR}/${IMAGE_LINK_NAME}.$type ]; then
+        rm -f $(readlink -f ${DEPLOYDIR}/${IMAGE_LINK_NAME}.$type)
     fi
 
-    ln -sf ${IMAGE_NAME}.$type ${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.$type
+    ln -sf ${IMAGE_NAME}.$type ${DEPLOYDIR}/${IMAGE_LINK_NAME}.$type
 }
 create_vmdk_image () {
     run_qemu_img vmdk
-- 
2.1.4

[PATCH 03/18] image.bbclass: deploy images to DEPLOYDIR

[Ed Bartosh]

Changed deployment directory from DEPLOY_DIR_IMAGE to DEPLOYDIR
to make sstate machinery to do final deployment and generate
manifest.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/classes/image.bbclass | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index c06dee2..82a07d5 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -440,7 +440,7 @@ python () {
             cmds.append("\t" + image_cmd)
         else:
             bb.fatal("No IMAGE_CMD defined for IMAGE_FSTYPES entry '%s' - possibly invalid type name or missing support class" % t)
-        cmds.append(localdata.expand("\tcd ${DEPLOY_DIR_IMAGE}"))
+        cmds.append(localdata.expand("\tcd ${DEPLOYDIR}"))
 
         # Since a copy of IMAGE_CMD_xxx will be inlined within do_image_xxx,
         # prevent a redundant copy of IMAGE_CMD_xxx being emitted as a function.
@@ -558,7 +558,7 @@ python set_image_size () {
 #
 python create_symlinks() {
 
-    deploy_dir = d.getVar('DEPLOY_DIR_IMAGE', True)
+    deploy_dir = d.getVar('DEPLOYDIR', True)
     img_name = d.getVar('IMAGE_NAME', True)
     link_name = d.getVar('IMAGE_LINK_NAME', True)
     manifest_name = d.getVar('IMAGE_MANIFEST', True)
-- 
2.1.4

[PATCH 04/18] image.bbclass: deploy images to DEPLOYDIR

[Ed Bartosh]

Changed deployment directory from DEPLOY_DIR_IMAGE to DEPLOYDIR
to make sstate machinery to do final deployment and generate
manifest.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/classes/image.bbclass       |  2 +-
 meta/classes/image_types.bbclass | 44 ++++++++++++++++++++--------------------
 2 files changed, 23 insertions(+), 23 deletions(-)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 82a07d5..ea3792a7 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -118,7 +118,7 @@ def rootfs_variables(d):
                  'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','RM_OLD_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS',
                  'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS',
                  'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS',
-                 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED']
+                 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'DEPLOYDIR']
     variables.extend(rootfs_command_variables(d))
     variables.extend(variable_depends(d))
     return " ".join(variables)
diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index 2e852af..6e6fbd7 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -41,9 +41,9 @@ XZ_THREADS ?= "-T 0"
 ZIP_COMPRESSION_LEVEL ?= "-9"
 
 JFFS2_SUM_EXTRA_ARGS ?= ""
-IMAGE_CMD_jffs2 = "mkfs.jffs2 --root=${IMAGE_ROOTFS} --faketime --output=${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.jffs2 ${EXTRA_IMAGECMD}"
+IMAGE_CMD_jffs2 = "mkfs.jffs2 --root=${IMAGE_ROOTFS} --faketime --output=${DEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.jffs2 ${EXTRA_IMAGECMD}"
 
-IMAGE_CMD_cramfs = "mkfs.cramfs ${IMAGE_ROOTFS} ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cramfs ${EXTRA_IMAGECMD}"
+IMAGE_CMD_cramfs = "mkfs.cramfs ${IMAGE_ROOTFS} ${DEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cramfs ${EXTRA_IMAGECMD}"
 
 oe_mkext234fs () {
 	fstype=$1
@@ -63,8 +63,8 @@ oe_mkext234fs () {
 		eval COUNT=\"$MIN_COUNT\"
 	fi
 	# Create a sparse image block
-	dd if=/dev/zero of=${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.$fstype seek=$ROOTFS_SIZE count=$COUNT bs=1024
-	mkfs.$fstype -F $extra_imagecmd ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.$fstype -d ${IMAGE_ROOTFS}
+	dd if=/dev/zero of=${DEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.$fstype seek=$ROOTFS_SIZE count=$COUNT bs=1024
+	mkfs.$fstype -F $extra_imagecmd ${DEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.$fstype -d ${IMAGE_ROOTFS}
 }
 
 IMAGE_CMD_ext2 = "oe_mkext234fs ext2 ${EXTRA_IMAGECMD}"
@@ -74,16 +74,16 @@ IMAGE_CMD_ext4 = "oe_mkext234fs ext4 ${EXTRA_IMAGECMD}"
 MIN_BTRFS_SIZE ?= "16384"
 IMAGE_CMD_btrfs () {
 	if [ ${ROOTFS_SIZE} -gt ${MIN_BTRFS_SIZE} ]; then
-		dd if=/dev/zero of=${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.btrfs count=${ROOTFS_SIZE} bs=1024
-		mkfs.btrfs ${EXTRA_IMAGECMD} -r ${IMAGE_ROOTFS} ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.btrfs
+		dd if=/dev/zero of=${DEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.btrfs count=${ROOTFS_SIZE} bs=1024
+		mkfs.btrfs ${EXTRA_IMAGECMD} -r ${IMAGE_ROOTFS} ${DEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.btrfs
 	else
 		bbfatal "Rootfs is too small for BTRFS (Rootfs Actual Size: ${ROOTFS_SIZE}, BTRFS Minimum Size: ${MIN_BTRFS_SIZE})"
 	fi
 }
 
-IMAGE_CMD_squashfs = "mksquashfs ${IMAGE_ROOTFS} ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs ${EXTRA_IMAGECMD} -noappend"
-IMAGE_CMD_squashfs-xz = "mksquashfs ${IMAGE_ROOTFS} ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-xz ${EXTRA_IMAGECMD} -noappend -comp xz"
-IMAGE_CMD_squashfs-lzo = "mksquashfs ${IMAGE_ROOTFS} ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-lzo ${EXTRA_IMAGECMD} -noappend -comp lzo"
+IMAGE_CMD_squashfs = "mksquashfs ${IMAGE_ROOTFS} ${DEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs ${EXTRA_IMAGECMD} -noappend"
+IMAGE_CMD_squashfs-xz = "mksquashfs ${IMAGE_ROOTFS} ${DEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-xz ${EXTRA_IMAGECMD} -noappend -comp xz"
+IMAGE_CMD_squashfs-lzo = "mksquashfs ${IMAGE_ROOTFS} ${DEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-lzo ${EXTRA_IMAGECMD} -noappend -comp lzo"
 
 # By default, tar from the host is used, which can be quite old. If
 # you need special parameters (like --xattrs) which are only supported
@@ -96,11 +96,11 @@ IMAGE_CMD_squashfs-lzo = "mksquashfs ${IMAGE_ROOTFS} ${DEPLOY_DIR_IMAGE}/${IMAGE
 # In practice, it turned out to be not needed when creating archives and
 # required when extracting, but it seems prudent to use it in both cases.
 IMAGE_CMD_TAR ?= "tar"
-IMAGE_CMD_tar = "${IMAGE_CMD_TAR} -cvf ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.tar -C ${IMAGE_ROOTFS} ."
+IMAGE_CMD_tar = "${IMAGE_CMD_TAR} -cvf ${DEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.tar -C ${IMAGE_ROOTFS} ."
 
 do_image_cpio[cleandirs] += "${WORKDIR}/cpio_append"
 IMAGE_CMD_cpio () {
-	(cd ${IMAGE_ROOTFS} && find . | cpio -o -H newc >${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cpio)
+	(cd ${IMAGE_ROOTFS} && find . | cpio -o -H newc >${DEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cpio)
 	# We only need the /init symlink if we're building the real
 	# image. The -dbg image doesn't need it! By being clever
 	# about this we also avoid 'touch' below failing, as it
@@ -113,7 +113,7 @@ IMAGE_CMD_cpio () {
 			else
 				touch ${WORKDIR}/cpio_append/init
 			fi
-			(cd  ${WORKDIR}/cpio_append && echo ./init | cpio -oA -H newc -F ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cpio)
+			(cd  ${WORKDIR}/cpio_append && echo ./init | cpio -oA -H newc -F ${DEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cpio)
 		fi
 	fi
 }
@@ -122,8 +122,8 @@ ELF_KERNEL ?= "${DEPLOY_DIR_IMAGE}/${KERNEL_IMAGETYPE}"
 ELF_APPEND ?= "ramdisk_size=32768 root=/dev/ram0 rw console="
 
 IMAGE_CMD_elf () {
-	test -f ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.elf && rm -f ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.elf
-	mkelfImage --kernel=${ELF_KERNEL} --initrd=${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.cpio.gz --output=${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.elf --append='${ELF_APPEND}' ${EXTRA_IMAGECMD}
+	test -f ${DEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.elf && rm -f ${DEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.elf
+	mkelfImage --kernel=${ELF_KERNEL} --initrd=${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.cpio.gz --output=${DEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.elf --append='${ELF_APPEND}' ${EXTRA_IMAGECMD}
 }
 
 IMAGE_TYPEDEP_elf = "cpio.gz"
@@ -141,20 +141,20 @@ multiubi_mkfs() {
 
 	echo \[ubifs\] > ubinize${vname}-${IMAGE_NAME}.cfg
 	echo mode=ubi >> ubinize${vname}-${IMAGE_NAME}.cfg
-	echo image=${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${vname}${IMAGE_NAME_SUFFIX}.ubifs >> ubinize${vname}-${IMAGE_NAME}.cfg
+	echo image=${DEPLOYDIR}/${IMAGE_NAME}${vname}${IMAGE_NAME_SUFFIX}.ubifs >> ubinize${vname}-${IMAGE_NAME}.cfg
 	echo vol_id=0 >> ubinize${vname}-${IMAGE_NAME}.cfg
 	echo vol_type=dynamic >> ubinize${vname}-${IMAGE_NAME}.cfg
 	echo vol_name=${UBI_VOLNAME} >> ubinize${vname}-${IMAGE_NAME}.cfg
 	echo vol_flags=autoresize >> ubinize${vname}-${IMAGE_NAME}.cfg
-	mkfs.ubifs -r ${IMAGE_ROOTFS} -o ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${vname}${IMAGE_NAME_SUFFIX}.ubifs ${mkubifs_args}
-	ubinize -o ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${vname}${IMAGE_NAME_SUFFIX}.ubi ${ubinize_args} ubinize${vname}-${IMAGE_NAME}.cfg
+	mkfs.ubifs -r ${IMAGE_ROOTFS} -o ${DEPLOYDIR}/${IMAGE_NAME}${vname}${IMAGE_NAME_SUFFIX}.ubifs ${mkubifs_args}
+	ubinize -o ${DEPLOYDIR}/${IMAGE_NAME}${vname}${IMAGE_NAME_SUFFIX}.ubi ${ubinize_args} ubinize${vname}-${IMAGE_NAME}.cfg
 
 	# Cleanup cfg file
-	mv ubinize${vname}-${IMAGE_NAME}.cfg ${DEPLOY_DIR_IMAGE}/
+	mv ubinize${vname}-${IMAGE_NAME}.cfg ${DEPLOYDIR}/
 
 	# Create own symlinks for 'named' volumes
 	if [ -n "$vname" ]; then
-		cd ${DEPLOY_DIR_IMAGE}
+		cd ${DEPLOYDIR}
 		if [ -e ${IMAGE_NAME}${vname}${IMAGE_NAME_SUFFIX}.ubifs ]; then
 			ln -sf ${IMAGE_NAME}${vname}${IMAGE_NAME_SUFFIX}.ubifs \
 			${IMAGE_LINK_NAME}${vname}.ubifs
@@ -181,7 +181,7 @@ IMAGE_CMD_ubi () {
 	multiubi_mkfs "${MKUBIFS_ARGS}" "${UBINIZE_ARGS}"
 }
 
-IMAGE_CMD_ubifs = "mkfs.ubifs -r ${IMAGE_ROOTFS} -o ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.ubifs ${MKUBIFS_ARGS}"
+IMAGE_CMD_ubifs = "mkfs.ubifs -r ${IMAGE_ROOTFS} -o ${DEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.ubifs ${MKUBIFS_ARGS}"
 
 WKS_FILE ?= "${IMAGE_BASENAME}.${MACHINE}.wks"
 WKS_FILES ?= "${WKS_FILE} ${IMAGE_BASENAME}.wks"
@@ -201,7 +201,7 @@ def wks_search(files, search_path):
 WIC_CREATE_EXTRA_ARGS ?= ""
 
 IMAGE_CMD_wic () {
-	out="${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}"
+	out="${DEPLOYDIR}/${IMAGE_NAME}"
 	wks="${WKS_FULL_PATH}"
 	if [ -z "$wks" ]; then
 		bbfatal "No kickstart files from WKS_FILES were found: ${WKS_FILES}. Please set WKS_FILE or WKS_FILES appropriately."
@@ -360,4 +360,4 @@ IMAGE_TYPES_MASKED ?= ""
 
 # The WICVARS variable is used to define list of bitbake variables used in wic code
 # variables from this list is written to <image>.env file
-WICVARS ?= "BBLAYERS DEPLOY_DIR_IMAGE HDDDIR IMAGE_BASENAME IMAGE_BOOT_FILES IMAGE_LINK_NAME IMAGE_ROOTFS INITRAMFS_FSTYPES INITRD ISODIR MACHINE_ARCH ROOTFS_SIZE STAGING_DATADIR STAGING_DIR_NATIVE STAGING_LIBDIR TARGET_SYS"
+WICVARS ?= "BBLAYERS DEPLOYDIR DEPLOY_DIR_IMAGE HDDDIR IMAGE_BASENAME IMAGE_BOOT_FILES IMAGE_LINK_NAME IMAGE_ROOTFS INITRAMFS_FSTYPES INITRD ISODIR MACHINE_ARCH ROOTFS_SIZE STAGING_DATADIR STAGING_DIR_NATIVE STAGING_LIBDIR TARGET_SYS"
-- 
2.1.4

[PATCH 05/18] image_types_uboot.bbclass: deploy images to DEPLOYDIR

[Ed Bartosh]

Changed deployment directory from DEPLOY_DIR_IMAGE to DEPLOYDIR
to make sstate machinery to do final deployment and generate
manifest.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/classes/image_types_uboot.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/image_types_uboot.bbclass b/meta/classes/image_types_uboot.bbclass
index f72d9b2..b8a281e 100644
--- a/meta/classes/image_types_uboot.bbclass
+++ b/meta/classes/image_types_uboot.bbclass
@@ -2,7 +2,7 @@ inherit image_types kernel-arch
 
 oe_mkimage () {
     mkimage -A ${UBOOT_ARCH} -O linux -T ramdisk -C $2 -n ${IMAGE_NAME} \
-        -d ${DEPLOY_DIR_IMAGE}/$1 ${DEPLOY_DIR_IMAGE}/$1.u-boot
+        -d ${DEPLOYDIR}/$1 ${DEPLOYDIR}/$1.u-boot
     if [ x$3 = x"clean" ]; then
         rm $1
     fi
-- 
2.1.4

[PATCH 06/18] syslinux.bbclass: deploy bootloader to DEPLOYDIR

[Ed Bartosh]

Changed deployment directory from DEPLOY_DIR_IMAGE to DEPLOYDIR
to make sstate machinery to do final deployment and generate
manifest.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/classes/syslinux.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/syslinux.bbclass b/meta/classes/syslinux.bbclass
index 35d9eed..fd4c32e 100644
--- a/meta/classes/syslinux.bbclass
+++ b/meta/classes/syslinux.bbclass
@@ -72,7 +72,7 @@ syslinux_hddimg_populate() {
 }
 
 syslinux_hddimg_install() {
-	syslinux ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.hddimg
+	syslinux ${DEPLOYDIR}/${IMAGE_NAME}.hddimg
 }
 
 syslinux_hdddirect_install() {
-- 
2.1.4

[PATCH 07/18] build-appliance-image: process images in DEPLOYDIR

[Ed Bartosh]

Updated path to images as deployment directory is changed
from DEPLOY_DIR_IMAGE to DEPLOYDIR.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/recipes-core/images/build-appliance-image_15.0.0.bb | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/meta/recipes-core/images/build-appliance-image_15.0.0.bb b/meta/recipes-core/images/build-appliance-image_15.0.0.bb
index dc621d6..f7d57a8 100644
--- a/meta/recipes-core/images/build-appliance-image_15.0.0.bb
+++ b/meta/recipes-core/images/build-appliance-image_15.0.0.bb
@@ -32,7 +32,7 @@ BA_INCLUDE_SOURCES ??= "0"
 
 IMAGE_CMD_ext4_append () {
 	# We don't need to reserve much space for root, 0.5% is more than enough
-	tune2fs -m 0.5 ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.ext4
+	tune2fs -m 0.5 ${DEPLOYDIR}/${IMAGE_NAME}.rootfs.ext4
 }
 
 fakeroot do_populate_poky_src () {
@@ -101,9 +101,9 @@ create_bundle_files () {
 	cd ${WORKDIR}
 	mkdir -p Yocto_Build_Appliance
 	cp *.vmx* Yocto_Build_Appliance
-	ln -sf ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.vmdk Yocto_Build_Appliance/Yocto_Build_Appliance.vmdk
-	zip -r ${DEPLOY_DIR_IMAGE}/Yocto_Build_Appliance-${DATETIME}.zip Yocto_Build_Appliance
-	ln -sf Yocto_Build_Appliance-${DATETIME}.zip ${DEPLOY_DIR_IMAGE}/Yocto_Build_Appliance.zip 
+	ln -sf ${DEPLOYDIR}/${IMAGE_NAME}.vmdk Yocto_Build_Appliance/Yocto_Build_Appliance.vmdk
+	zip -r ${DEPLOYDIR}/Yocto_Build_Appliance-${DATETIME}.zip Yocto_Build_Appliance
+	ln -sf Yocto_Build_Appliance-${DATETIME}.zip ${DEPLOYDIR}/Yocto_Build_Appliance.zip
 }
 create_bundle_files[vardepsexclude] = "DATETIME"
 
-- 
2.1.4

[PATCH 08/18] populate_sdk_base.bbclass: deploy sdk artifacts to DEPLOYDIR

[Ed Bartosh]

Changed deployment directory from SDK_DEPLOY to DEPLOYDIR
to make sstate machinery to do final deployment and generate
manifest.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/classes/populate_sdk_base.bbclass | 22 ++++++++++++----------
 1 file changed, 12 insertions(+), 10 deletions(-)

diff --git a/meta/classes/populate_sdk_base.bbclass b/meta/classes/populate_sdk_base.bbclass
index 645a7f4..4d2bc9a 100644
--- a/meta/classes/populate_sdk_base.bbclass
+++ b/meta/classes/populate_sdk_base.bbclass
@@ -26,6 +26,8 @@ SDK_DIR = "${WORKDIR}/sdk"
 SDK_OUTPUT = "${SDK_DIR}/image"
 SDK_DEPLOY = "${DEPLOY_DIR}/sdk"
 
+DEPLOYDIR = "${WORKDIR}/deploy-${PN}"
+
 B_task-populate-sdk = "${SDK_DIR}"
 
 SDKTARGETSYSROOT = "${SDKPATH}/sysroots/${REAL_MULTIMACH_TARGET_SYS}"
@@ -58,8 +60,8 @@ SDK_RELOCATE_AFTER_INSTALL ?= "1"
 SDKEXTPATH ?= "~/${@d.getVar('DISTRO', True)}_sdk"
 SDK_TITLE ?= "${@d.getVar('DISTRO_NAME', True) or d.getVar('DISTRO', True)} SDK"
 
-SDK_TARGET_MANIFEST = "${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.target.manifest"
-SDK_HOST_MANIFEST = "${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.host.manifest"
+SDK_TARGET_MANIFEST = "${DEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.target.manifest"
+SDK_HOST_MANIFEST = "${DEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.host.manifest"
 python write_target_sdk_manifest () {
     from oe.sdk import sdk_list_installed_packages
     from oe.utils import format_pkg_list
@@ -180,14 +182,14 @@ SDKTAROPTS = "--owner=root --group=root"
 
 fakeroot tar_sdk() {
 	# Package it up
-	mkdir -p ${SDK_DEPLOY}
+	mkdir -p ${DEPLOYDIR}
 	cd ${SDK_OUTPUT}/${SDKPATH}
-	tar ${SDKTAROPTS} -cf - . | pixz > ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.tar.xz
+	tar ${SDKTAROPTS} -cf - . | pixz > ${DEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz
 }
 
 fakeroot create_shar() {
 	# copy in the template shar extractor script
-	cp ${COREBASE}/meta/files/toolchain-shar-extract.sh ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh
+	cp ${COREBASE}/meta/files/toolchain-shar-extract.sh ${DEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.sh
 
 	rm -f ${T}/pre_install_command ${T}/post_install_command
 
@@ -203,7 +205,7 @@ ${SDK_POST_INSTALL_COMMAND}
 EOF
 	sed -i -e '/@SDK_PRE_INSTALL_COMMAND@/r ${T}/pre_install_command' \
 		-e '/@SDK_POST_INSTALL_COMMAND@/r ${T}/post_install_command' \
-		${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh
+		${DEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.sh
 
 	# substitute variables
 	sed -i -e 's#@SDK_ARCH@#${SDK_ARCH}#g' \
@@ -215,16 +217,16 @@ EOF
 		-e 's#@SDK_VERSION@#${SDK_VERSION}#g' \
 		-e '/@SDK_PRE_INSTALL_COMMAND@/d' \
 		-e '/@SDK_POST_INSTALL_COMMAND@/d' \
-		${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh
+		${DEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.sh
 
 	# add execution permission
-	chmod +x ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh
+	chmod +x ${DEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.sh
 
 	# append the SDK tarball
-	cat ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.tar.xz >> ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh
+	cat ${DEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz >> ${DEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.sh
 
 	# delete the old tarball, we don't need it anymore
-	rm ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.tar.xz
+	rm ${DEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz
 }
 
 populate_sdk_log_check() {
-- 
2.1.4

[PATCH 09/18] rootfs-postcommands.bbclass: generate manifest in DEPLOYDIR

[Ed Bartosh]

Updated manifest path as deployment directory is changed
from DEPLOY_DIR_IMAGE to DEPLOYDIR.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/classes/rootfs-postcommands.bbclass | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/classes/rootfs-postcommands.bbclass b/meta/classes/rootfs-postcommands.bbclass
index b38b6a5..f8a63b0 100644
--- a/meta/classes/rootfs-postcommands.bbclass
+++ b/meta/classes/rootfs-postcommands.bbclass
@@ -15,7 +15,7 @@ ROOTFS_POSTPROCESS_COMMAND += "rootfs_update_timestamp ; "
 ROOTFS_POSTPROCESS_COMMAND += '${@bb.utils.contains("IMAGE_FEATURES", "read-only-rootfs", "read_only_rootfs_hook; ", "",d)}'
 
 # Write manifest
-IMAGE_MANIFEST = "${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.manifest"
+IMAGE_MANIFEST = "${DEPLOYDIR}/${IMAGE_NAME}.rootfs.manifest"
 ROOTFS_POSTUNINSTALL_COMMAND =+ "write_image_manifest ; "
 # Set default postinst log file
 POSTINST_LOGFILE ?= "${localstatedir}/log/postinstall.log"
@@ -217,7 +217,7 @@ python write_image_manifest () {
     from oe.rootfs import image_list_installed_packages
     from oe.utils import format_pkg_list
 
-    deploy_dir = d.getVar('DEPLOY_DIR_IMAGE', True)
+    deploy_dir = d.getVar('DEPLOYDIR', True)
     link_name = d.getVar('IMAGE_LINK_NAME', True)
     manifest_name = d.getVar('IMAGE_MANIFEST', True)
 
-- 
2.1.4

[PATCH 10/18] selftest: renamed variable

[Ed Bartosh]

Renamed variable deploy_dir to deploy_dir_image to avoid
confusion with DEPLOYDIR variable.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/lib/oeqa/selftest/imagefeatures.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oeqa/selftest/imagefeatures.py b/meta/lib/oeqa/selftest/imagefeatures.py
index 08e382f..d015c49 100644
--- a/meta/lib/oeqa/selftest/imagefeatures.py
+++ b/meta/lib/oeqa/selftest/imagefeatures.py
@@ -113,9 +113,9 @@ class ImageFeatures(oeSelfTest):
         image_name = 'core-image-minimal'
         bitbake(image_name)
 
-        deploy_dir = get_bb_var('DEPLOY_DIR_IMAGE')
+        deploy_dir_image = get_bb_var('DEPLOY_DIR_IMAGE')
         link_name = get_bb_var('IMAGE_LINK_NAME', image_name)
-        image_path = os.path.join(deploy_dir, "%s.ext4" % link_name)
+        image_path = os.path.join(deploy_dir_image, "%s.ext4" % link_name)
         bmap_path = "%s.bmap" % image_path
 
         # check if result image and bmap file are in deploy directory
-- 
2.1.4

[PATCH 11/18] rootfs.py: use DEPLOYDIR instead of DEPLOY_DIR_IMAGE

[Ed Bartosh]

Updated the code of Rootfs class to use DEPLOYDIR variable
as it's now used as a new deployment destination.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/lib/oe/rootfs.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/meta/lib/oe/rootfs.py b/meta/lib/oe/rootfs.py
index 7c620e9..076bc33 100644
--- a/meta/lib/oe/rootfs.py
+++ b/meta/lib/oe/rootfs.py
@@ -19,7 +19,7 @@ class Rootfs(object, metaclass=ABCMeta):
         self.d = d
         self.pm = None
         self.image_rootfs = self.d.getVar('IMAGE_ROOTFS', True)
-        self.deploy_dir_image = self.d.getVar('DEPLOY_DIR_IMAGE', True)
+        self.deploydir = self.d.getVar('DEPLOYDIR', True)
         self.progress_reporter = progress_reporter
 
         self.install_order = Manifest.INSTALL_ORDER
@@ -182,12 +182,12 @@ class Rootfs(object, metaclass=ABCMeta):
 
         bb.utils.mkdirhier(self.image_rootfs)
 
-        bb.utils.mkdirhier(self.deploy_dir_image)
+        bb.utils.mkdirhier(self.deploydir)
 
         shutil.copytree(postinst_intercepts_dir, intercepts_dir)
 
         shutil.copy(self.d.expand("${COREBASE}/meta/files/deploydir_readme.txt"),
-                    self.deploy_dir_image +
+                    self.deploydir +
                     "/README_-_DO_NOT_DELETE_FILES_IN_THIS_DIRECTORY.txt")
 
         execute_pre_post_process(self.d, pre_process_cmds)
-- 
2.1.4

[PATCH 12/18] image.bbclass: put image_complete under sstate control

[Ed Bartosh]

Adding image_complete task should make sstate machinery
to generate manifest for deployed images and do final
deployment to DEPLOY_DIR_IMAGE.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/classes/image.bbclass | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index ea3792a7..3bfc328 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -264,6 +264,9 @@ do_image[dirs] = "${TOPDIR}"
 do_image[umask] = "022"
 addtask do_image after do_rootfs before do_build
 
+DEPLOYDIR = "${WORKDIR}/deploy-${PN}"
+SSTATETASKS += "do_image_complete"
+
 fakeroot python do_image_complete () {
     from oe.utils import execute_pre_post_process
 
@@ -273,6 +276,8 @@ fakeroot python do_image_complete () {
 }
 do_image_complete[dirs] = "${TOPDIR}"
 do_image_complete[umask] = "022"
+do_image_complete[sstate-inputdirs] = "${DEPLOYDIR}"
+do_image_complete[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
 addtask do_image_complete after do_image before do_build
 
 # Add image-level QA/sanity checks to IMAGE_QA_COMMANDS
-- 
2.1.4

[PATCH 13/18] image.bbclass: cleanup DEPLOYDIR

[Ed Bartosh]

Make sure DEPLOYDIR doesn't contain images from past deployments
to prevent them to be included into sstate manifests.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/classes/image.bbclass | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 3bfc328..bbdfd8a 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -262,6 +262,7 @@ fakeroot python do_image () {
 }
 do_image[dirs] = "${TOPDIR}"
 do_image[umask] = "022"
+do_image[cleandirs] = "${DEPLOYDIR}"
 addtask do_image after do_rootfs before do_build
 
 DEPLOYDIR = "${WORKDIR}/deploy-${PN}"
-- 
2.1.4

[PATCH 14/18] populate_sdk_base: put populate_sdk under sstate control

[Ed Bartosh]

Adding populate_sdk task to SSTATE_TASKS should make sstate machinery
to generate manifest for deployed sdk artifacts and do final deployment
to SDK_DEPLOY.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/classes/populate_sdk_base.bbclass | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/meta/classes/populate_sdk_base.bbclass b/meta/classes/populate_sdk_base.bbclass
index 4d2bc9a..c1dac39 100644
--- a/meta/classes/populate_sdk_base.bbclass
+++ b/meta/classes/populate_sdk_base.bbclass
@@ -27,6 +27,7 @@ SDK_OUTPUT = "${SDK_DIR}/image"
 SDK_DEPLOY = "${DEPLOY_DIR}/sdk"
 
 DEPLOYDIR = "${WORKDIR}/deploy-${PN}"
+SSTATETASKS += "do_populate_sdk"
 
 B_task-populate-sdk = "${SDK_DIR}"
 
@@ -118,6 +119,9 @@ fakeroot python do_populate_sdk() {
     populate_sdk(d)
 }
 
+do_populate_sdk[sstate-inputdirs] = "${DEPLOYDIR}"
+do_populate_sdk[sstate-outputdirs] = "${SDK_DEPLOY}"
+
 fakeroot create_sdk_files() {
 	cp ${COREBASE}/scripts/relocate_sdk.py ${SDK_OUTPUT}/${SDKPATH}/
 
-- 
2.1.4

[PATCH 15/18] sstate.bbclass: skip packaging if SSTATE_SKIP_CREATION is set

[Ed Bartosh]

SSTATE_SKIP_CREATION variable will be used to skip creation of
sstate .tgz files. It makes sense for image creation tasks as
tarring images and keeping them in sstate would consume a lot of
disk space.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/classes/sstate.bbclass | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index 2496928..0f0baeb 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -566,6 +566,8 @@ def sstate_package(ss, d):
     for state in ss['dirs']:
         if not os.path.exists(state[1]):
             continue
+        if d.getVar('SSTATE_SKIP_CREATION', True) == '1':
+            continue
         srcbase = state[0].rstrip("/").rsplit('/', 1)[0]
         for walkroot, dirs, files in os.walk(state[1]):
             for file in files:
-- 
2.1.4

[PATCH 16/18] image: populate_sdk_base: skip sstate creation

[Ed Bartosh]

Set SSTATE_SKIP_CREATION variable to '1' for image_complete and
populate_sdk tasks as we don't want to put images and sdk installer
into sstate due to their big size. We added these tasks under sstate
control only for two reasons:
 - to generate sstate manifest
 - to make final deployments

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/classes/image.bbclass             | 1 +
 meta/classes/populate_sdk_base.bbclass | 1 +
 2 files changed, 2 insertions(+)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index bbdfd8a..8e84b1b 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -267,6 +267,7 @@ addtask do_image after do_rootfs before do_build
 
 DEPLOYDIR = "${WORKDIR}/deploy-${PN}"
 SSTATETASKS += "do_image_complete"
+SSTATE_SKIP_CREATION_task-image-complete = '1'
 
 fakeroot python do_image_complete () {
     from oe.utils import execute_pre_post_process
diff --git a/meta/classes/populate_sdk_base.bbclass b/meta/classes/populate_sdk_base.bbclass
index c1dac39..1914a3b 100644
--- a/meta/classes/populate_sdk_base.bbclass
+++ b/meta/classes/populate_sdk_base.bbclass
@@ -28,6 +28,7 @@ SDK_DEPLOY = "${DEPLOY_DIR}/sdk"
 
 DEPLOYDIR = "${WORKDIR}/deploy-${PN}"
 SSTATETASKS += "do_populate_sdk"
+SSTATE_SKIP_CREATION_task-populate-sdk = '1'
 
 B_task-populate-sdk = "${SDK_DIR}"
 
-- 
2.1.4

[PATCH 17/18] image: populate_sdk_base: set stamp-extra-info flag

[Ed Bartosh]

This flag is used in the name of sstate manifest.

Setting it to predetermined value for image_complete and populate_sdk
tasks should help to get correct manifest filenames when processing
runQueueTask events.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/classes/image.bbclass             | 1 +
 meta/classes/populate_sdk_base.bbclass | 1 +
 2 files changed, 2 insertions(+)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 8e84b1b..49fe500 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -280,6 +280,7 @@ do_image_complete[dirs] = "${TOPDIR}"
 do_image_complete[umask] = "022"
 do_image_complete[sstate-inputdirs] = "${DEPLOYDIR}"
 do_image_complete[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
+do_image_complete[stamp-extra-info] = "${MACHINE}"
 addtask do_image_complete after do_image before do_build
 
 # Add image-level QA/sanity checks to IMAGE_QA_COMMANDS
diff --git a/meta/classes/populate_sdk_base.bbclass b/meta/classes/populate_sdk_base.bbclass
index 1914a3b..ede3a7b 100644
--- a/meta/classes/populate_sdk_base.bbclass
+++ b/meta/classes/populate_sdk_base.bbclass
@@ -122,6 +122,7 @@ fakeroot python do_populate_sdk() {
 
 do_populate_sdk[sstate-inputdirs] = "${DEPLOYDIR}"
 do_populate_sdk[sstate-outputdirs] = "${SDK_DEPLOY}"
+do_populate_sdk[stamp-extra-info] = "${MACHINE}"
 
 fakeroot create_sdk_files() {
 	cp ${COREBASE}/scripts/relocate_sdk.py ${SDK_OUTPUT}/${SDKPATH}/
-- 
2.1.4

[PATCH 18/18] toaster: fire TaskArtifacts event

[Ed Bartosh]

Fire TaskArtifact MetaData event for deployment tasks when task
either completed or skipped. Event contains full task id
(recipe+task) and list of deployment artifacts from sstate
manifest.

This should allow Toaster to always get notified about deployment
artifacts produced by the build.

[YOCTO #9869]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/classes/toaster.bbclass | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/meta/classes/toaster.bbclass b/meta/classes/toaster.bbclass
index 972efb9..4bddf34 100644
--- a/meta/classes/toaster.bbclass
+++ b/meta/classes/toaster.bbclass
@@ -324,6 +324,20 @@ python toaster_buildhistory_dump() {
 
 }
 
+# get list of artifacts from sstate manifest
+python toaster_artifacts() {
+    if e.taskname in ["do_deploy", "do_image_complete", "do_populate_sdk", "do_populate_sdk_ext"]:
+        d2 = d.createCopy()
+        d2.setVar('FILE', e.taskfile)
+        d2.setVar('SSTATE_MANMACH', d2.expand("${MACHINE}"))
+        manifest = oe.sstatesig.sstate_get_manifest_filename(e.taskname[3:], d2)[0]
+        if os.access(manifest, os.R_OK):
+            with open(manifest) as fmanifest:
+                artifacts = [fname.strip() for fname in fmanifest]
+                data = {"task": e.taskid, "artifacts": artifacts}
+                bb.event.fire(bb.event.MetadataEvent("TaskArtifacts", data), d2)
+}
+
 # set event handlers
 addhandler toaster_layerinfo_dumpdata
 toaster_layerinfo_dumpdata[eventmask] = "bb.event.TreeDataPreparationCompleted"
@@ -334,6 +348,9 @@ toaster_collect_task_stats[eventmask] = "bb.event.BuildCompleted bb.build.TaskSu
 addhandler toaster_buildhistory_dump
 toaster_buildhistory_dump[eventmask] = "bb.event.BuildCompleted"
 
+addhandler toaster_artifacts
+toaster_artifacts[eventmask] = "bb.runqueue.runQueueTaskSkipped bb.runqueue.runQueueTaskCompleted"
+
 do_packagedata_setscene[postfuncs] += "toaster_package_dumpdata "
 do_packagedata_setscene[vardepsexclude] += "toaster_package_dumpdata "
 
-- 
2.1.4

Re: [PATCH 00/18] Provide list of deployment artifacts

[Richard Purdie]

Hi Ed,

On Tue, 2016-08-30 at 12:29 +0300, Ed Bartosh wrote:
> This is a fix for Bug #9869 - Provide a per-target manifest of files
> which were, or would have been, produced
> 
> The list of artifacts produced by deployment tasks (do_deploy,
> do_image_complete and do_populate_sdk[_ext] is
> obtained from sstate manifests and fired as a TaskArtifacts metadata
> event. This should allow Toaster to
> handle artifacts in simple way and remove a lot of current Toaster
> code doing guess work.
> 
> To generate manifests for do_image_complete and do_populate_sdk they
> have been put under sstate control.
> 
> To avoid storing big files(images and sdk installer) in sstate new
> variable SSTATE_SKIP_CREATION has been
> set in image.bbclass and populate_sdk_base.bbclass and sstate code
> was modified to avoid adding files
> to sstate if SSTATE_SKIP_CREATION is set.

I'd like to see this series reordered to:

a) Set DEPLOYDIR in image.bbclass/populate_sdk_base.bbclass to point at
DEPLOY_DIR_IMAGE
b) Add SSTATE_SKIP_CREATION support
c) Make all the DEPLOY_DIR_IMAGE -> DEPLOYDIR changes in one patch
d) Enable sstate for images (change DEPLOYDIR definition, add the flags
including cleandirs, stamp-extra, set SSTATE_SKIP_CREATION)
e) As per d but for populate_sdk_base

That way it applies incrementally and like changes are all grouped
together. It will mean some commit messages are longer and squashed
together but I think that is fine.

Cheers,

Richard

Re: [PATCH 00/18] Provide list of deployment artifacts

[Otavio Salvador]

On Tue, Aug 30, 2016 at 6:29 AM, Ed Bartosh <ed.bartosh@linux.intel.com> wrote:
> Hi,
>
> This is a fix for Bug #9869 - Provide a per-target manifest of files which were, or would have been, produced
>
> The list of artifacts produced by deployment tasks (do_deploy, do_image_complete and do_populate_sdk[_ext] is
> obtained from sstate manifests and fired as a TaskArtifacts metadata event. This should allow Toaster to
> handle artifacts in simple way and remove a lot of current Toaster code doing guess work.
>
> To generate manifests for do_image_complete and do_populate_sdk they have been put under sstate control.
>
> To avoid storing big files(images and sdk installer) in sstate new variable SSTATE_SKIP_CREATION has been
> set in image.bbclass and populate_sdk_base.bbclass and sstate code was modified to avoid adding files
> to sstate if SSTATE_SKIP_CREATION is set.

SKIP creation of what? Variable name seems a little vague for me. Even
needing the manifest, does it makes sense to store the image and SDK
on sstate?

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Re: [PATCH 00/18] Provide list of deployment artifacts

[Ed Bartosh]

On Tue, Aug 30, 2016 at 07:21:20AM -0300, Otavio Salvador wrote:
> On Tue, Aug 30, 2016 at 6:29 AM, Ed Bartosh <ed.bartosh@linux.intel.com> wrote:
> > Hi,
> >
> > This is a fix for Bug #9869 - Provide a per-target manifest of files which were, or would have been, produced
> >
> > The list of artifacts produced by deployment tasks (do_deploy, do_image_complete and do_populate_sdk[_ext] is
> > obtained from sstate manifests and fired as a TaskArtifacts metadata event. This should allow Toaster to
> > handle artifacts in simple way and remove a lot of current Toaster code doing guess work.
> >
> > To generate manifests for do_image_complete and do_populate_sdk they have been put under sstate control.
> >
> > To avoid storing big files(images and sdk installer) in sstate new variable SSTATE_SKIP_CREATION has been
> > set in image.bbclass and populate_sdk_base.bbclass and sstate code was modified to avoid adding files
> > to sstate if SSTATE_SKIP_CREATION is set.
> 
> SKIP creation of what?
SKIP creation of SSTATE entity.

> Variable name seems a little vague for me. Even
> needing the manifest, does it makes sense to store the image and SDK
> on sstate?
It doesn't and skipping this is a purpose of introducing SSTATE_SKIP_CREATION variable.

--
Regards,
Ed

Re: [PATCH 00/18] Provide list of deployment artifacts

[Otavio Salvador]

On Tue, Aug 30, 2016 at 7:35 AM, Ed Bartosh <ed.bartosh@linux.intel.com> wrote:
> On Tue, Aug 30, 2016 at 07:21:20AM -0300, Otavio Salvador wrote:
>> On Tue, Aug 30, 2016 at 6:29 AM, Ed Bartosh <ed.bartosh@linux.intel.com> wrote:
>> > Hi,
>> >
>> > This is a fix for Bug #9869 - Provide a per-target manifest of files which were, or would have been, produced
>> >
>> > The list of artifacts produced by deployment tasks (do_deploy, do_image_complete and do_populate_sdk[_ext] is
>> > obtained from sstate manifests and fired as a TaskArtifacts metadata event. This should allow Toaster to
>> > handle artifacts in simple way and remove a lot of current Toaster code doing guess work.
>> >
>> > To generate manifests for do_image_complete and do_populate_sdk they have been put under sstate control.
>> >
>> > To avoid storing big files(images and sdk installer) in sstate new variable SSTATE_SKIP_CREATION has been
>> > set in image.bbclass and populate_sdk_base.bbclass and sstate code was modified to avoid adding files
>> > to sstate if SSTATE_SKIP_CREATION is set.
>>
>> SKIP creation of what?
> SKIP creation of SSTATE entity.

I know but of WHAT?

SSTATE_SKIP_IMAGE_CREATION would be a clearer name for me.

>> Variable name seems a little vague for me. Even
>> needing the manifest, does it makes sense to store the image and SDK
>> on sstate?
> It doesn't and skipping this is a purpose of introducing SSTATE_SKIP_CREATION variable.

So it is always going to be skipped?

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Re: [PATCH 00/18] Provide list of deployment artifacts

[Ed Bartosh]

On Tue, Aug 30, 2016 at 07:58:31AM -0300, Otavio Salvador wrote:
> On Tue, Aug 30, 2016 at 7:35 AM, Ed Bartosh <ed.bartosh@linux.intel.com> wrote:
> > On Tue, Aug 30, 2016 at 07:21:20AM -0300, Otavio Salvador wrote:
> >> On Tue, Aug 30, 2016 at 6:29 AM, Ed Bartosh <ed.bartosh@linux.intel.com> wrote:
> >> > Hi,
> >> >
> >> > This is a fix for Bug #9869 - Provide a per-target manifest of files which were, or would have been, produced
> >> >
> >> > The list of artifacts produced by deployment tasks (do_deploy, do_image_complete and do_populate_sdk[_ext] is
> >> > obtained from sstate manifests and fired as a TaskArtifacts metadata event. This should allow Toaster to
> >> > handle artifacts in simple way and remove a lot of current Toaster code doing guess work.
> >> >
> >> > To generate manifests for do_image_complete and do_populate_sdk they have been put under sstate control.
> >> >
> >> > To avoid storing big files(images and sdk installer) in sstate new variable SSTATE_SKIP_CREATION has been
> >> > set in image.bbclass and populate_sdk_base.bbclass and sstate code was modified to avoid adding files
> >> > to sstate if SSTATE_SKIP_CREATION is set.
> >>
> >> SKIP creation of what?
> > SKIP creation of SSTATE entity.
> 
> I know but of WHAT?
Of anything we don't want to put to sstate. In this patchset it's set
for do_image_complete and do_populate_sdk, so it's for images and sdk
installer.

> SSTATE_SKIP_IMAGE_CREATION would be a clearer name for me.
I'm not sure about it. I'd need one more variable
SSTATE_SKIP_SDK_INSTALLER if I go this way.

> >> Variable name seems a little vague for me. Even
> >> needing the manifest, does it makes sense to store the image and SDK
> >> on sstate?
> > It doesn't and skipping this is a purpose of introducing SSTATE_SKIP_CREATION variable.
>
> So it is always going to be skipped?
Yes it is for do_image_complete and do_populate_sdk tasks for now.

--
Regards,
Ed

Re: [PATCH 00/18] Provide list of deployment artifacts

[Otavio Salvador]

On Tue, Aug 30, 2016 at 8:21 AM, Ed Bartosh <ed.bartosh@linux.intel.com> wrote:
>> So it is always going to be skipped?
> Yes it is for do_image_complete and do_populate_sdk tasks for now.

Oh ok.

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Re: [PATCH 00/18] Provide list of deployment artifacts

[Richard Purdie]

On Tue, 2016-08-30 at 10:36 -0300, Otavio Salvador wrote:
> On Tue, Aug 30, 2016 at 8:21 AM, Ed Bartosh <
> ed.bartosh@linux.intel.com> wrote:
> > > So it is always going to be skipped?
> > Yes it is for do_image_complete and do_populate_sdk tasks for now.
> 
> Oh ok.

Its a performance issue. I can see the attraction of putting these into
sstate however it would add significant time to the builds which many
people would find unacceptable.

Having manifests of deployed files is a huge win for a whole variety of
reasons and it standardises our output tasks. This compromise gives us
that without the performance hit of compressing everything into an
sstate artefact.

Cheers,

Richard