[PATCH 0/6 V3] refactor the archiver*.bbclass

[PATCH 0/6 V3] refactor the archiver*.bbclass

[Robert Yang]

=== V3:
* Do the unpack, patch and configure in another WORKDIR as RP and Ross
  suggested, so that it would not affact the default build dependencies,
  thanks for Ross' prototype.
* Don't archive the logs (temp/log.do*), mainly because of:
  - We can easily get the log via reset the T rather than use the
    archiver.
  - If we need archive the logs, we must let the task to run as the last
    one, but it's hard to do this.

=== V2:
* Fix the warning between different machines which is reported by Martin
* Fix the error when the archiver.bbclass is not inherited but
  ARCHIVER_MODE[type] = "srpm" is set, reported by Ross.
* Fix the archiving for gcc staff which uses the shared source according
  to the recently changes of prefuncs and postfuncs in bitbake, so it only
  works well with bitbake's up to date master branch, if we want to backport it,
  we also need backport the pre/postfuncs related patches.

=== V1:
* The archive*.bbclass didn't work, and there were a few problems, for
  example:
  1) There were a few duplicated code
  2) There was no src_dir.org (or orig), but the diff command still use
     it, and it is not easy to fix this issue if we don't change a lot
     of the code.
  3) It didn't archive the source for the native or gcc
  4) The work flow is not very well
  5) The "subprocess.call('fakeroot cp xxxx'" should be removed
  6) And others ...

* So that we have to refactor it, the benefits are:
  1) Fix the problems and make it work well.
  2) Reduce more than 400 lines in total.
  3) Make it easy to use.

// Robert


The following changes since commit 23f7decb84922a3467c5b10b013e66596926b5cf:

  .templateconf: New file for customized template defaults (2014-03-11 08:11:07 -0700)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib rbt/archiver_v3
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=rbt/archiver_v3

Robert Yang (6):
  classes/archive*.bbclass: remove archive-*-source.bbclass
  archiver.bbclass: refactor it
  package_rpm.bbclass: archive the source to srpm package
  archiver.bbclass: move a few code to copyleft_compliance.bbclass
  local.conf.sample.extended: update for the archiver
  patch.bbclass: do_patch: unset TMPDIR from environment

 meta/classes/archive-configured-source.bbclass |  65 --
 meta/classes/archive-original-source.bbclass   |  65 --
 meta/classes/archive-patched-source.bbclass    |  65 --
 meta/classes/archiver.bbclass                  | 791 +++++++++----------------
 meta/classes/copyleft_compliance.bbclass       |  55 +-
 meta/classes/package_rpm.bbclass               |  33 +-
 meta/classes/patch.bbclass                     |   1 +
 meta/conf/local.conf.sample.extended           |  63 +-
 8 files changed, 367 insertions(+), 771 deletions(-)
 delete mode 100644 meta/classes/archive-configured-source.bbclass
 delete mode 100644 meta/classes/archive-original-source.bbclass
 delete mode 100644 meta/classes/archive-patched-source.bbclass

-- 
1.8.3.1

[PATCH 1/6] classes/archive*.bbclass: remove archive-*-source.bbclass

[Robert Yang]

Nearly all of the codes in the following 3 files are the same, we can
move the code to archiver.bbclass and remove them:

  archive-configured-source.bbclass
  archive-original-source.bbclass
  archive-patched-source.bbclass

[YOCTO #5113]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta/classes/archive-configured-source.bbclass | 65 --------------------------
 meta/classes/archive-original-source.bbclass   | 65 --------------------------
 meta/classes/archive-patched-source.bbclass    | 65 --------------------------
 meta/classes/archiver.bbclass                  | 63 ++++++++++++++++++++++++-
 4 files changed, 62 insertions(+), 196 deletions(-)
 delete mode 100644 meta/classes/archive-configured-source.bbclass
 delete mode 100644 meta/classes/archive-original-source.bbclass
 delete mode 100644 meta/classes/archive-patched-source.bbclass

diff --git a/meta/classes/archive-configured-source.bbclass b/meta/classes/archive-configured-source.bbclass
deleted file mode 100644
index 961a532..0000000
--- a/meta/classes/archive-configured-source.bbclass
+++ /dev/null
@@ -1,65 +0,0 @@
-# This file is for getting archiving packages with configured
-# sources(archive ${S} after configure stage), logs(archive 'temp' after
-# package_write_rpm), dump data and creating diff file(get all
-# environment variables and functions in building and mapping all
-# content in ${S} including patches to xxx.diff.gz. All archived
-# packages will be deployed in ${DEPLOY_DIR}/sources
-
-inherit archiver
-
-# Get archiving package with configured sources including patches
-addtask do_archive_configured_sources after do_configure
-
-# Get archiving package with temp(logs) and scripts(.bb and inc files)
-addtask do_archive_scripts_logs
-
-# Get dump date and create diff file 
-addtask do_dumpdata_create_diff_gz
-
-python () {
-    pn = d.getVar('PN', True)
-    packaging = d.getVar('IMAGE_PKGTYPE', True)
-
-    if tar_filter(d):
-        return
-
-    d.appendVarFlag('do_dumpdata_create_diff_gz', 'depends', ' %s:do_package_write_' %pn + packaging)
-    build_deps = ' %s:do_dumpdata_create_diff_gz' %pn
-
-    if d.getVar('SOURCE_ARCHIVE_LOG_WITH_SCRIPTS', True) == 'logs_with_scripts':
-        d.appendVarFlag('do_archive_scripts_logs', 'depends', ' %s:do_package_write_' %pn + packaging)
-        build_deps += ' %s:do_archive_scripts_logs' %pn
-
-    if not not_tarball(d):
-        d.appendVarFlag('do_compile', 'depends', ' %s:do_archive_configured_sources' %pn)
-        build_deps += ' %s:do_archive_configured_sources' %pn
-
-    if bb.data.inherits_class('image', d):
-        d.appendVarFlag('do_rootfs', 'depends', build_deps)
-    else:
-        d.appendVarFlag('do_build', 'depends', build_deps)
-}
-
-ARCHIVE_SSTATE_OUTDIR = "${DEPLOY_DIR}/sources/"
-ARCHIVE_SSTATE_SCRIPTS_LOGS_INDIR = "${WORKDIR}/script-logs/"
-ARCHIVE_SSTATE_DIFFGZ_ENVDATA_INDIR = "${WORKDIR}/diffgz-envdata/"
-
-SSTATETASKS += "do_archive_scripts_logs"
-do_archive_scripts_logs[sstate-inputdirs] = "${ARCHIVE_SSTATE_SCRIPTS_LOGS_INDIR}"
-do_archive_scripts_logs[sstate-outputdirs] = "${ARCHIVE_SSTATE_OUTDIR}"
-
-python do_archive_scripts_logs_setscene () {
-    sstate_setscene(d)
-}
-
-addtask do_archive_scripts_logs_setscene
-
-SSTATETASKS += "do_dumpdata_create_diff_gz"
-do_dumpdata_create_diff_gz[sstate-inputdirs] = "${ARCHIVE_SSTATE_DIFFGZ_ENVDATA_INDIR}"
-do_dumpdata_create_diff_gz[sstate-outputdirs] = "${ARCHIVE_SSTATE_OUTDIR}"
-
-python do_dumpdata_create_diff_gz_setscene () {
-    sstate_setscene(d)
-}
-
-addtask do_dumpdata_create_diff_gz_setscene
diff --git a/meta/classes/archive-original-source.bbclass b/meta/classes/archive-original-source.bbclass
deleted file mode 100644
index e271a08..0000000
--- a/meta/classes/archive-original-source.bbclass
+++ /dev/null
@@ -1,65 +0,0 @@
-# This file is for getting archiving packages with original
-# sources(archive ${S} after unpack stage), patches, logs(archive 'temp'
-# after package_write_rpm), dump data and creating diff file(get all
-# environment variables and functions in building and mapping all
-# content in ${S} including patches to xxx.diff.gz. All archived packages
-# will be deployed in ${DEPLOY_DIR}/sources
-
-inherit archiver
-
-# Get original sources archiving package with patches
-addtask do_archive_original_sources_patches after do_unpack
-
-# Get archiving package with temp(logs) and scripts(.bb and inc files)
-addtask do_archive_scripts_logs
-
-# Get dump date and create diff file 
-addtask do_dumpdata_create_diff_gz
-
-python () {
-    pn = d.getVar('PN', True)
-    packaging = d.getVar('IMAGE_PKGTYPE', True)
-
-    if tar_filter(d):
-        return
-
-    d.appendVarFlag('do_dumpdata_create_diff_gz', 'depends', ' %s:do_package_write_' %pn + packaging)
-    build_deps = ' %s:do_dumpdata_create_diff_gz' %pn
-
-    if d.getVar('SOURCE_ARCHIVE_LOG_WITH_SCRIPTS', True) == 'logs_with_scripts':
-        d.appendVarFlag('do_archive_scripts_logs', 'depends', ' %s:do_package_write_' %pn + packaging)
-        build_deps += ' %s:do_archive_scripts_logs' %pn
-
-    if not not_tarball(d):
-        d.appendVarFlag('do_patch', 'depends', ' %s:do_archive_original_sources_patches' %pn)
-        build_deps += ' %s:do_archive_original_sources_patches' %pn
-
-    if bb.data.inherits_class('image', d):
-        d.appendVarFlag('do_rootfs', 'depends', build_deps)
-    else:
-        d.appendVarFlag('do_build', 'depends', build_deps)
-}
-
-ARCHIVE_SSTATE_OUTDIR = "${DEPLOY_DIR}/sources/"
-ARCHIVE_SSTATE_SCRIPTS_LOGS_INDIR = "${WORKDIR}/script-logs/"
-ARCHIVE_SSTATE_DIFFGZ_ENVDATA_INDIR = "${WORKDIR}/diffgz-envdata/"
-
-SSTATETASKS += "do_archive_scripts_logs"
-do_archive_scripts_logs[sstate-inputdirs] = "${ARCHIVE_SSTATE_SCRIPTS_LOGS_INDIR}"
-do_archive_scripts_logs[sstate-outputdirs] = "${ARCHIVE_SSTATE_OUTDIR}"
-
-python do_archive_scripts_logs_setscene () {
-    sstate_setscene(d)
-}
-
-addtask do_archive_scripts_logs_setscene
-
-SSTATETASKS += "do_dumpdata_create_diff_gz"
-do_dumpdata_create_diff_gz[sstate-inputdirs] = "${ARCHIVE_SSTATE_DIFFGZ_ENVDATA_INDIR}"
-do_dumpdata_create_diff_gz[sstate-outputdirs] = "${ARCHIVE_SSTATE_OUTDIR}"
-
-python do_dumpdata_create_diff_gz_setscene () {
-    sstate_setscene(d)
-}
-
-addtask do_dumpdata_create_diff_gz_setscene
diff --git a/meta/classes/archive-patched-source.bbclass b/meta/classes/archive-patched-source.bbclass
deleted file mode 100644
index c5f5cca..0000000
--- a/meta/classes/archive-patched-source.bbclass
+++ /dev/null
@@ -1,65 +0,0 @@
-# This file is for getting archiving packages with patched
-# sources(archive ${S} before do_patch stage), logs(archive 'temp' after
-# package_write_rpm), dump data and creating diff file(get all
-# environment variables and functions in building and mapping all
-# content in ${S} including patches to xxx.diff.gz. All archived
-# packages will be deployed in ${DEPLOY_DIR}/sources
-
-inherit archiver
-
-# Get archiving package with patched sources including patches
-addtask do_archive_patched_sources after do_patch
-
-# Get archiving package with logs(temp) and scripts(.bb and .inc files)
-addtask do_archive_scripts_logs
-
-# Get dump date and create diff file 
-addtask do_dumpdata_create_diff_gz
-
-python () {
-    pn = d.getVar('PN', True)
-    packaging = d.getVar('IMAGE_PKGTYPE', True)
-
-    if tar_filter(d):
-        return
-
-    d.appendVarFlag('do_dumpdata_create_diff_gz', 'depends', ' %s:do_package_write_' %pn + packaging)
-    build_deps = ' %s:do_dumpdata_create_diff_gz' %pn
-
-    if d.getVar('SOURCE_ARCHIVE_LOG_WITH_SCRIPTS', True) == 'logs_with_scripts':
-        d.appendVarFlag('do_archive_scripts_logs', 'depends', ' %s:do_package_write_' %pn + packaging)
-        build_deps += ' %s:do_archive_scripts_logs' %pn
-
-    if not not_tarball(d):
-        d.appendVarFlag('do_configure', 'depends', ' %s:do_archive_patched_sources' %pn)
-        build_deps += ' %s:do_archive_patched_sources' %pn
-
-    if bb.data.inherits_class('image', d):
-        d.appendVarFlag('do_rootfs', 'depends', build_deps)
-    else:
-        d.appendVarFlag('do_build', 'depends', build_deps)
-}
-
-ARCHIVE_SSTATE_OUTDIR = "${DEPLOY_DIR}/sources/"
-ARCHIVE_SSTATE_SCRIPTS_LOGS_INDIR = "${WORKDIR}/script-logs/"
-ARCHIVE_SSTATE_DIFFGZ_ENVDATA_INDIR = "${WORKDIR}/diffgz-envdata/"
-
-SSTATETASKS += "do_archive_scripts_logs"
-do_archive_scripts_logs[sstate-inputdirs] = "${ARCHIVE_SSTATE_SCRIPTS_LOGS_INDIR}"
-do_archive_scripts_logs[sstate-outputdirs] = "${ARCHIVE_SSTATE_OUTDIR}"
-
-python do_archive_scripts_logs_setscene () {
-    sstate_setscene(d)
-}
-
-addtask do_archive_scripts_logs_setscene
-
-SSTATETASKS += "do_dumpdata_create_diff_gz"
-do_dumpdata_create_diff_gz[sstate-inputdirs] = "${ARCHIVE_SSTATE_DIFFGZ_ENVDATA_INDIR}"
-do_dumpdata_create_diff_gz[sstate-outputdirs] = "${ARCHIVE_SSTATE_OUTDIR}"
-
-python do_dumpdata_create_diff_gz_setscene () {
-    sstate_setscene(d)
-}
-
-addtask do_dumpdata_create_diff_gz_setscene
diff --git a/meta/classes/archiver.bbclass b/meta/classes/archiver.bbclass
index 66efe7d..43373ae 100644
--- a/meta/classes/archiver.bbclass
+++ b/meta/classes/archiver.bbclass
@@ -36,6 +36,33 @@ COPYLEFT_AVAILABLE_RECIPE_TYPES = 'target native nativesdk cross crosssdk cross-
 COPYLEFT_AVAILABLE_RECIPE_TYPES[type] = 'list'
 COPYLEFT_AVAILABLE_RECIPE_TYPES[doc] = 'Space separated list of available recipe types'
 
+python () {
+    pn = d.getVar('PN', True)
+    packaging = d.getVar('IMAGE_PKGTYPE', True)
+
+    if tar_filter(d):
+        return
+
+    if d.getVar('PACKAGES', True) != '':
+        d.appendVarFlag('do_dumpdata_create_diff_gz', 'depends', ' %s:do_package_write_%s' % (pn, packaging))
+
+    build_deps = ' %s:do_dumpdata_create_diff_gz' % pn
+
+    if d.getVar('SOURCE_ARCHIVE_LOG_WITH_SCRIPTS', True) == 'logs_with_scripts':
+        d.appendVarFlag('do_archive_scripts_logs', 'depends', ' %s:do_package_write_%s' % (pn, packaging))
+        build_deps += ' %s:do_archive_scripts_logs' % pn
+
+    if not not_tarball(d):
+        archiver_mode = d.getVar('ARCHIVER_MODE')
+        d.appendVarFlag('do_compile', 'depends', ' %s:do_archive_%s_sources' % (pn, archiver_mode))
+        build_deps += ' %s:do_archive_%s_sources' % (pn, archiver_mode)
+
+    if bb.data.inherits_class('image', d):
+        d.appendVarFlag('do_rootfs', 'depends', build_deps)
+    else:
+        d.appendVarFlag('do_build', 'depends', build_deps)
+}
+
 def copyleft_recipe_type(d):
     for recipe_type in oe.data.typed_value('COPYLEFT_AVAILABLE_RECIPE_TYPES', d):
         if oe.utils.inherits(d, recipe_type):
@@ -516,7 +543,7 @@ def create_diff_gz(d):
 
 # This function will run when user want to get tarball for sources and
 # patches after do_unpack
-python do_archive_original_sources_patches(){
+python do_archive_original_sources(){
     archive_sources_patches(d, 'prepatch')
 }
 
@@ -567,3 +594,37 @@ python do_delete_tarlist(){
 }
 do_delete_tarlist[deptask] = "do_archive_scripts_logs"
 do_package_write_rpm[postfuncs] += "do_delete_tarlist "
+
+# Get archiving package with temp(logs) and scripts(.bb and .inc files)
+addtask do_archive_scripts_logs
+
+# Get dump date and create diff file
+addtask do_dumpdata_create_diff_gz
+
+ARCHIVE_SSTATE_OUTDIR = "${DEPLOY_DIR}/sources/"
+ARCHIVE_SSTATE_SCRIPTS_LOGS_INDIR = "${WORKDIR}/script-logs/"
+ARCHIVE_SSTATE_DIFFGZ_ENVDATA_INDIR = "${WORKDIR}/diffgz-envdata/"
+
+SSTATETASKS += "do_archive_scripts_logs"
+do_archive_scripts_logs[sstate-inputdirs] = "${ARCHIVE_SSTATE_SCRIPTS_LOGS_INDIR}"
+do_archive_scripts_logs[sstate-outputdirs] = "${ARCHIVE_SSTATE_OUTDIR}"
+
+python do_archive_scripts_logs_setscene () {
+    sstate_setscene(d)
+}
+
+addtask do_archive_scripts_logs_setscene
+
+SSTATETASKS += "do_dumpdata_create_diff_gz"
+do_dumpdata_create_diff_gz[sstate-inputdirs] = "${ARCHIVE_SSTATE_DIFFGZ_ENVDATA_INDIR}"
+do_dumpdata_create_diff_gz[sstate-outputdirs] = "${ARCHIVE_SSTATE_OUTDIR}"
+
+python do_dumpdata_create_diff_gz_setscene () {
+    sstate_setscene(d)
+}
+
+addtask do_dumpdata_create_diff_gz_setscene
+
+addtask do_archive_original_sources after do_unpack
+addtask do_archive_patched_sources after do_patch
+addtask do_archive_configured_sources after do_configure
-- 
1.8.3.1

[PATCH 2/6] archiver.bbclass: refactor it

[Robert Yang]

The archiver didn't work, and there were a few problems, for example:
1) There was no src_dir.org (or orig), but the diff command still use it
2) There were a few duplicated code
3) It didn't archive the source for the native or gcc
4) The work flow is not very well
5) The "subprocess.call('fakeroot cp xxxx'" should be removed
6) And others ...

So that we have to refactor it, the benefits are:
1) Fix the problems and make it work well.
2) Reduce more than 300 lines
3) Make it easy to use.

Hre are the explanation about the bbclass:
This bbclass is used for creating archive for:
1) original (or unpacked) source: ARCHIVER_MODE[src] = "original"
2) patched source: ARCHIVER_MODE[src] = "patched" (default)
3) configured source: ARCHIVER_MODE[src] = "configured"
4) The patches between do_unpack and do_patch:
   ARCHIVER_MODE[diff] = "1"
   And you can set the one that you'd like to exclude from the diff:
   ARCHIVER_MODE[diff-exclude] ?= ".pc autom4te.cache patches"
5) The environment data, similar to 'bitbake -e recipe':
   ARCHIVER_MODE[dumpdata] = "1"
6) The recipe (.bb and .inc): ARCHIVER_MODE[recipe] = "1"

All of the above can be packed into a .src.rpm package: (when PACKAGES
!= "")
ARCHIVER_MODE[srpm] = "1"

[YOCTO #4986]
[YOCTO #5113]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta/classes/archiver.bbclass | 819 ++++++++++++++----------------------------
 1 file changed, 265 insertions(+), 554 deletions(-)

diff --git a/meta/classes/archiver.bbclass b/meta/classes/archiver.bbclass
index 43373ae..b1109dc 100644
--- a/meta/classes/archiver.bbclass
+++ b/meta/classes/archiver.bbclass
@@ -1,20 +1,38 @@
 # ex:ts=4:sw=4:sts=4:et
 # -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
 #
-# This file is used for archiving sources, patches, and logs to a
-# tarball.  It also output building environment to xxx.dump.data and
-# create xxx.diff.gz to record all content in ${S} to a diff file.
+# This bbclass is used for creating archive for:
+# 1) original (or unpacked) source: ARCHIVER_MODE[src] = "original"
+# 2) patched source: ARCHIVER_MODE[src] = "patched" (default)
+# 3) configured source: ARCHIVER_MODE[src] = "configured"
+# 4) The patches between do_unpack and do_patch:
+#    ARCHIVER_MODE[diff] = "1"
+#    And you can set the one that you'd like to exclude from the diff:
+#    ARCHIVER_MODE[diff-exclude] ?= ".pc autom4te.cache patches"
+# 5) The environment data, similar to 'bitbake -e recipe':
+#    ARCHIVER_MODE[dumpdata] = "1"
+# 6) The recipe (.bb and .inc): ARCHIVER_MODE[recipe] = "1"
 #
+# All of the above can be packed into a .src.rpm package: (when PACKAGES != "")
+# ARCHIVER_MODE[srpm] = "1"
 
-ARCHIVE_EXCLUDE_FROM ?= ".pc autom4te.cache"
-ARCHIVE_TYPE ?= "tar srpm"
-PATCHES_ARCHIVE_WITH_SERIES = 'yes'
-SOURCE_ARCHIVE_LOG_WITH_SCRIPTS ?= '${@d.getVarFlag('ARCHIVER_MODE', 'log_type') \
-    if d.getVarFlag('ARCHIVER_MODE', 'log_type') != 'none' else 'logs_with_scripts'}'
-SOURCE_ARCHIVE_PACKAGE_TYPE ?= '${@d.getVarFlag('ARCHIVER_MODE', 'type') \
-    if d.getVarFlag('ARCHIVER_MODE', 'log_type') != 'none' else 'tar'}'
-FILTER ?= '${@d.getVarFlag('ARCHIVER_MODE', 'filter') \
-    if d.getVarFlag('ARCHIVER_MODE', 'filter')!= 'none' else 'no'}'
+ARCHIVER_MODE[srpm] ?= "0"
+ARCHIVER_MODE[src] ?= "patched"
+ARCHIVER_MODE[diff] ?= "0"
+ARCHIVER_MODE[diff-exclude] ?= ".pc autom4te.cache patches"
+ARCHIVER_MODE[dumpdata] ?= "0"
+ARCHIVER_MODE[recipe] ?= "0"
+
+DEPLOY_DIR_SRC ?= "${DEPLOY_DIR}/sources"
+AR_TOPDIR = "${WORKDIR}/deploy-sources"
+ARCHIVER_OUTDIR = "${AR_TOPDIR}/${TARGET_SYS}/${PF}/"
+ARCHIVER_WORKDIR = "${WORKDIR}/archiver-work/"
+
+do_dumpdata[dirs] = "${ARCHIVER_OUTDIR}"
+do_ar_recipe[dirs] = "${ARCHIVER_OUTDIR}"
+do_ar_original[dirs] = "${ARCHIVER_OUTDIR} ${ARCHIVER_WORKDIR}"
+
+# This is a convenience for the shell script to use it
 
 
 COPYLEFT_LICENSE_INCLUDE ?= 'GPL* LGPL*'
@@ -23,7 +41,7 @@ COPYLEFT_LICENSE_INCLUDE[doc] = 'Space separated list of globs which include lic
 
 COPYLEFT_LICENSE_EXCLUDE ?= 'CLOSED Proprietary'
 COPYLEFT_LICENSE_EXCLUDE[type] = 'list'
-COPYLEFT_LICENSE_INCLUDE[doc] = 'Space separated list of globs which exclude licenses'
+COPYLEFT_LICENSE_EXCLUDE[doc] = 'Space separated list of globs which exclude licenses'
 
 COPYLEFT_RECIPE_TYPE ?= '${@copyleft_recipe_type(d)}'
 COPYLEFT_RECIPE_TYPE[doc] = 'The "type" of the current recipe (e.g. target, native, cross)'
@@ -38,593 +56,286 @@ COPYLEFT_AVAILABLE_RECIPE_TYPES[doc] = 'Space separated list of available recipe
 
 python () {
     pn = d.getVar('PN', True)
-    packaging = d.getVar('IMAGE_PKGTYPE', True)
 
-    if tar_filter(d):
-        return
-
-    if d.getVar('PACKAGES', True) != '':
-        d.appendVarFlag('do_dumpdata_create_diff_gz', 'depends', ' %s:do_package_write_%s' % (pn, packaging))
+    ar_src = d.getVarFlag('ARCHIVER_MODE', 'src', True)
+    ar_dumpdata = d.getVarFlag('ARCHIVER_MODE', 'dumpdata', True)
+    ar_recipe = d.getVarFlag('ARCHIVER_MODE', 'recipe', True)
+
+    if ar_src == "original":
+        d.appendVarFlag('do_deploy_archives', 'depends', ' %s:do_ar_original' % pn)
+    elif ar_src == "patched":
+        d.appendVarFlag('do_deploy_archives', 'depends', ' %s:do_ar_patched' % pn)
+    elif ar_src == "configured":
+        # We can't use "addtask do_ar_configured after do_configure" since it
+        # will cause the deptask of do_populate_sysroot to run not matter what
+        # archives we need, so we add the depends here.
+        d.appendVarFlag('do_ar_configured', 'depends', ' %s:do_configure' % pn)
+        d.appendVarFlag('do_deploy_archives', 'depends', ' %s:do_ar_configured' % pn)
+    elif ar_src:
+        bb.fatal("Invalid ARCHIVER_MODE[src]: %s" % ar_src)
+
+    if ar_dumpdata == "1":
+        d.appendVarFlag('do_deploy_archives', 'depends', ' %s:do_dumpdata' % pn)
+
+    if ar_recipe == "1":
+        d.appendVarFlag('do_deploy_archives', 'depends', ' %s:do_ar_recipe' % pn)
+
+    # Output the srpm package
+    ar_srpm = d.getVarFlag('ARCHIVER_MODE', 'srpm', True)
+    if ar_srpm == "1":
+        if d.getVar('PACKAGES', True) != '' and d.getVar('IMAGE_PKGTYPE', True) == 'rpm':
+            d.appendVarFlag('do_deploy_archives', 'depends', ' %s:do_package_write_rpm' % pn)
+            if ar_dumpdata == "1":
+                d.appendVarFlag('do_package_write_rpm', 'depends', ' %s:do_dumpdata' % pn)
+            if ar_recipe == "1":
+                d.appendVarFlag('do_package_write_rpm', 'depends', ' %s:do_ar_recipe' % pn)
+            if ar_src == "original":
+                d.appendVarFlag('do_package_write_rpm', 'depends', ' %s:do_ar_original' % pn)
+            elif ar_src == "patched":
+                d.appendVarFlag('do_package_write_rpm', 'depends', ' %s:do_ar_patched' % pn)
+            elif ar_src == "configured":
+                d.appendVarFlag('do_package_write_rpm', 'depends', ' %s:do_ar_configured' % pn)
+
+    # The gcc staff uses shared source
+    flag = d.getVarFlag("do_unpack", "stamp-base", True)
+    if flag:
+        if ar_src in [ 'original', 'patched' ]:
+            ar_outdir = os.path.join(d.getVar('AR_TOPDIR', True), 'work-shared')
+            d.setVar('ARCHIVER_OUTDIR', ar_outdir)
+        d.setVarFlag('do_ar_original', 'stamp-base', flag)
+        d.setVarFlag('do_ar_patched', 'stamp-base', flag)
+        d.setVarFlag('do_unpack_and_patch', 'stamp-base', flag)
+        d.setVarFlag('do_ar_original', 'vardepsexclude', 'PN PF ARCHIVER_OUTDIR WORKDIR')
+        d.setVarFlag('do_unpack_and_patch', 'vardepsexclude', 'PN PF ARCHIVER_OUTDIR WORKDIR')
+        d.setVarFlag('do_ar_patched', 'vardepsexclude', 'PN PF ARCHIVER_OUTDIR WORKDIR')
+        d.setVarFlag('create_diff_gz', 'vardepsexclude', 'PF')
+        d.setVarFlag('create_tarball', 'vardepsexclude', 'PF')
+
+        flag_clean = d.getVarFlag('do_unpack', 'stamp-base-clean', True)
+        if flag_clean:
+            d.setVarFlag('do_ar_original', 'stamp-base-clean', flag_clean)
+            d.setVarFlag('do_ar_patched', 'stamp-base-clean', flag_clean)
+            d.setVarFlag('do_unpack_and_patch', 'stamp-base-clean', flag_clean)
+}
 
-    build_deps = ' %s:do_dumpdata_create_diff_gz' % pn
+# Take all the sources for a recipe and puts them in WORKDIR/archiver-work/.
+# Files in SRC_URI are copied directly, anything that's a directory
+# (e.g. git repositories) is "unpacked" and then put into a tarball.
+python do_ar_original() {
 
-    if d.getVar('SOURCE_ARCHIVE_LOG_WITH_SCRIPTS', True) == 'logs_with_scripts':
-        d.appendVarFlag('do_archive_scripts_logs', 'depends', ' %s:do_package_write_%s' % (pn, packaging))
-        build_deps += ' %s:do_archive_scripts_logs' % pn
+    import shutil, tarfile, tempfile
 
-    if not not_tarball(d):
-        archiver_mode = d.getVar('ARCHIVER_MODE')
-        d.appendVarFlag('do_compile', 'depends', ' %s:do_archive_%s_sources' % (pn, archiver_mode))
-        build_deps += ' %s:do_archive_%s_sources' % (pn, archiver_mode)
+    if d.getVarFlag('ARCHIVER_MODE', 'src', True) != "original":
+        return
 
-    if bb.data.inherits_class('image', d):
-        d.appendVarFlag('do_rootfs', 'depends', build_deps)
-    else:
-        d.appendVarFlag('do_build', 'depends', build_deps)
+    ar_outdir = d.getVar('ARCHIVER_OUTDIR', True)
+    bb.note('Archiving the original source...')
+    fetch = bb.fetch2.Fetch([], d)
+    for url in fetch.urls:
+        local = fetch.localpath(url)
+        if os.path.isfile(local):
+            shutil.copy(local, ar_outdir)
+        elif os.path.isdir(local):
+            basename = os.path.basename(local)
+
+            tmpdir = tempfile.mkdtemp(dir=d.getVar('ARCHIVER_WORKDIR', True))
+            fetch.unpack(tmpdir, (url,))
+
+            os.chdir(tmpdir)
+            tarname = os.path.join(ar_outdir, basename + '.tar.gz')
+            tar = tarfile.open(tarname, 'w:gz')
+            tar.add('.')
+            tar.close()
 }
 
-def copyleft_recipe_type(d):
-    for recipe_type in oe.data.typed_value('COPYLEFT_AVAILABLE_RECIPE_TYPES', d):
-        if oe.utils.inherits(d, recipe_type):
-            return recipe_type
-    return 'target'
-
-def copyleft_should_include(d):
-    """
-    Determine if this recipe's sources should be deployed for compliance
-    """
-    import ast
-    import oe.license
-    from fnmatch import fnmatchcase as fnmatch
+python do_ar_patched() {
 
-    recipe_type = d.getVar('COPYLEFT_RECIPE_TYPE', True)
-    if recipe_type not in oe.data.typed_value('COPYLEFT_RECIPE_TYPES', d):
-        return False, 'recipe type "%s" is excluded' % recipe_type
-
-    include = oe.data.typed_value('COPYLEFT_LICENSE_INCLUDE', d)
-    exclude = oe.data.typed_value('COPYLEFT_LICENSE_EXCLUDE', d)
+    if d.getVarFlag('ARCHIVER_MODE', 'src', True) != 'patched':
+        return
 
-    try:
-        is_included, reason = oe.license.is_included(d.getVar('LICENSE', True), include, exclude)
-    except oe.license.LicenseError as exc:
-        bb.fatal('%s: %s' % (d.getVar('PF', True), exc))
-    else:
-        if is_included:
-            if reason:
-                return True, 'recipe has included licenses: %s' % ', '.join(reason)
-            else:
-                return False, 'recipe does not include a copyleft license'
-        else:
-            return False, 'recipe has excluded licenses: %s' % ', '.join(reason)
-
-def tar_filter(d):
-    """
-    Only archive the package belongs to COPYLEFT_LICENSE_INCLUDE
-    and ignore the one in COPYLEFT_LICENSE_EXCLUDE. Don't exclude any
-    packages when \"FILTER\" is \"no\"
-    """
-    if d.getVar('FILTER', True) == "yes":
-        included, reason = copyleft_should_include(d)
-        return not included
+    # Get the ARCHIVER_OUTDIR before we reset the WORKDIR
+    ar_outdir = d.getVar('ARCHIVER_OUTDIR', True)
+    bb.note('Archiving the patched source...')
+    d.setVar('WORKDIR', d.getVar('ARCHIVER_WORKDIR', True))
+    # The gcc staff uses shared source
+    flag = d.getVarFlag('do_unpack', 'stamp-base', True)
+    if flag:
+        create_tarball(d, d.getVar('S', True), 'patched', ar_outdir, 'gcc')
     else:
-        return False
-
-def get_bb_inc(d):
-    """
-    create a directory "script-logs" including .bb and .inc file in ${WORKDIR}
-    """
-    import re
-    import shutil
-
-    bbinc = []
-    pat=re.compile('require\s*([^\s]*\.*)(.*)')
-    work_dir = d.getVar('WORKDIR', True)
-    bbfile = d.getVar('FILE', True)
-    bbdir = os.path.dirname(bbfile)
-    target_sys = d.getVar('TARGET_SYS', True)
-    pf = d.getVar('PF', True)
-    licenses = get_licenses(d)
-    script_logs = os.path.join(work_dir, 'script-logs/'+ target_sys + '/' + licenses + '/' + pf + '/script-logs')
-    bb_inc = os.path.join(script_logs, 'bb_inc')
-    bb.utils.mkdirhier(bb_inc)
-
-    def find_file(dir, file):
-        for root, dirs, files in os.walk(dir):
-            if file in files:
-                return os.path.join(root, file)
-
-    def get_inc (file):
-        f = open(file, 'r')
-        for line in f.readlines():
-            if 'require' not  in line:
-                bbinc.append(file)
-            else:
-                try:
-                    incfile = pat.match(line).group(1)
-                    incfile = bb.data.expand(os.path.basename(incfile), d)
-                    abs_incfile = find_file(bbdir, incfile)
-                    if abs_incfile:
-                        bbinc.append(abs_incfile)
-                        get_inc(abs_incfile)
-                except AttributeError:
-                    pass
-    get_inc(bbfile)
-    bbinc = list(set(bbinc))
-    for bbincfile in bbinc:
-        shutil.copy(bbincfile, bb_inc)
-
-    return script_logs
-
-def get_logs(d):
-    """
-    create a directory "script-logs" in ${WORKDIR}
-    """
-    work_dir = d.getVar('WORKDIR', True)
-    target_sys = d.getVar('TARGET_SYS', True)
-    pf = d.getVar('PF', True)
-    licenses = get_licenses(d)
-    script_logs = os.path.join(work_dir, 'script-logs/'+ target_sys + '/' + licenses + '/' + pf + '/script-logs')
-
-    try:
-        bb.utils.mkdirhier(os.path.join(script_logs, 'temp'))
-        oe.path.copytree(os.path.join(work_dir, 'temp'), os.path.join(script_logs, 'temp'))
-    except (IOError, AttributeError):
-        pass
-    return script_logs
-
-def get_series(d):
-    """
-    copy patches and series file to a pointed directory which will be
-    archived to tarball in ${WORKDIR}
-    """
-    import shutil
+        create_tarball(d, d.getVar('S', True), 'patched', ar_outdir)
+}
 
-    src_patches=[]
-    pf = d.getVar('PF', True)
-    work_dir = d.getVar('WORKDIR', True)
-    s = d.getVar('S', True)
-    dest = os.path.join(work_dir, pf + '-series')
-    shutil.rmtree(dest, ignore_errors=True)
-    bb.utils.mkdirhier(dest)
-
-    src_uri = d.getVar('SRC_URI', True).split()
-    fetch = bb.fetch2.Fetch(src_uri, d)
-    locals = (fetch.localpath(url) for url in fetch.urls)
-    for local in locals:
-        src_patches.append(local)
-    if not cmp(work_dir, s):
-        tmp_list = src_patches
-    else:
-        tmp_list = src_patches[1:]
-
-    for patch in tmp_list:
-        try:
-            shutil.copy(patch, dest)
-        except IOError:
-            if os.path.isdir(patch):
-                bb.utils.mkdirhier(os.path.join(dest, patch))
-                oe.path.copytree(patch, os.path.join(dest, patch))
-    return dest
-
-def get_applying_patches(d):
-    """
-    only copy applying patches to a pointed directory which will be
-    archived to tarball
-    """
+python do_ar_configured() {
     import shutil
 
-    pf = d.getVar('PF', True)
-    work_dir = d.getVar('WORKDIR', True)
-    dest = os.path.join(work_dir, pf + '-patches')
-    shutil.rmtree(dest, ignore_errors=True)
-    bb.utils.mkdirhier(dest)
-
-    patches = src_patches(d)
-    for patch in patches:
-        _, _, local, _, _, parm = bb.fetch.decodeurl(patch)
-        if local:
-             shutil.copy(local, dest)
-    return dest
-
-def not_tarball(d):
-    """
-    packages including key words 'work-shared', 'native', 'packagegroup-' will be passed
-    """
-    workdir = d.getVar('WORKDIR', True)
-    s = d.getVar('S', True)
-    if 'work-shared' in s or 'packagegroup-' in workdir or 'native' in workdir:
-        return True
-    else:
-        return False
+    ar_outdir = d.getVar('ARCHIVER_OUTDIR', True)
+    if d.getVarFlag('ARCHIVER_MODE', 'src', True) == 'configured':
+        bb.note('Archiving the configured source...')
+        # The libtool-native's do_configure will remove the
+        # ${STAGING_DATADIR}/aclocal/libtool.m4, so we can't re-run the
+        # do_configure, we archive the already configured ${S} to
+        # instead of.
+        if d.getVar('PN', True) != 'libtool-native':
+            # Change the WORKDIR to make do_configure run in another dir.
+            d.setVar('WORKDIR', d.getVar('ARCHIVER_WORKDIR', True))
+        if bb.data.inherits_class('kernel-yocto', d):
+            bb.build.exec_func('do_kernel_configme', d)
+            bb.build.exec_func('do_configure', d)
+        srcdir = d.getVar('S', True)
+        builddir = d.getVar('B', True)
+        if srcdir != builddir:
+            if os.path.exists(builddir):
+                oe.path.copytree(builddir, os.path.join(srcdir, \
+                    'build.%s.ar_configured' % d.getVar('PF', True)))
+        create_tarball(d, srcdir, 'configured', ar_outdir)
+}
 
-def get_source_from_downloads(d, stage_name):
+def create_tarball(d, srcdir, suffix, ar_outdir, pf=None):
     """
-    copy tarball of $P to $WORKDIR when this tarball exists in $DL_DIR
-    """
-    if stage_name in 'patched' 'configured':
-        return
-    pf = d.getVar('PF', True)
-    dl_dir = d.getVar('DL_DIR', True)
-    try:
-        source = os.path.join(dl_dir, os.path.basename(d.getVar('SRC_URI', True).split()[0]))
-        if os.path.exists(source) and not os.path.isdir(source):
-            return source
-    except (IndexError, OSError):
-        pass
-    return ''
-
-def do_tarball(workdir, srcdir, tarname):
-    """
-    tar "srcdir" under "workdir" to "tarname"
+    create the tarball from srcdir
     """
     import tarfile
 
-    sav_dir = os.getcwd()
-    os.chdir(workdir)
-    if (len(os.listdir(srcdir))) != 0:
-        tar = tarfile.open(tarname, "w:gz")
-        tar.add(srcdir)
-        tar.close()
-    else:
-        tarname = ''
-    os.chdir(sav_dir)
-    return tarname
-
-def archive_sources_from_directory(d, stage_name):
-    """
-    archive sources codes tree to tarball when tarball of $P doesn't
-    exist in $DL_DIR
-    """
-
-    s = d.getVar('S', True)
-    work_dir=d.getVar('WORKDIR', True)
-    PF = d.getVar('PF', True)
-    tarname = PF + '-' + stage_name + ".tar.gz"
-
-    if os.path.exists(s) and work_dir in s:
-        try:
-            source_dir = os.path.join(work_dir, [ i for i in s.replace(work_dir, '').split('/') if i][0])
-        except IndexError:
-            if not cmp(s, work_dir):
-                return ''
-    else:
-        return ''
-    source = os.path.basename(source_dir)
-    return do_tarball(work_dir, source, tarname)
-
-def archive_sources(d, stage_name):
-    """
-    copy tarball from $DL_DIR to $WORKDIR if have tarball, archive
-    source codes tree in $WORKDIR if $P is directory instead of tarball
-    """
-    import shutil
-
-    work_dir = d.getVar('WORKDIR', True)
-    file = get_source_from_downloads(d, stage_name)
-    if file:
-        shutil.copy(file, work_dir)
-        file = os.path.basename(file)
+    bb.utils.mkdirhier(ar_outdir)
+    if pf:
+        tarname = os.path.join(ar_outdir, '%s-%s.tar.gz' % (pf, suffix))
     else:
-        file = archive_sources_from_directory(d, stage_name)
-    return file
+        tarname = os.path.join(ar_outdir, '%s-%s.tar.gz' % \
+            (d.getVar('PF', True), suffix))
 
-def archive_patches(d, patchdir, series):
-    """
-    archive patches to tarball and also include series files if 'series' is True
-    """
-    import shutil
+    srcdir = srcdir.rstrip('/')
+    dirname = os.path.dirname(srcdir)
+    basename = os.path.basename(srcdir)
+    os.chdir(dirname)
+    bb.note('Creating %s' % tarname)
+    tar = tarfile.open(tarname, 'w:gz')
+    tar.add(basename)
+    tar.close()
 
-    s = d.getVar('S', True)
-    work_dir = d.getVar('WORKDIR', True)
-    patch_dir = os.path.basename(patchdir)
-    tarname = patch_dir + ".tar.gz"
-    if series  == 'all' and os.path.exists(os.path.join(s, 'patches/series')):
-        shutil.copy(os.path.join(s, 'patches/series'), patchdir)
-    tarname = do_tarball(work_dir, patch_dir, tarname)
-    shutil.rmtree(patchdir, ignore_errors=True)
-    return tarname
-
-def select_archive_patches(d, option):
-    """
-    select to archive all patches including non-applying and series or
-    applying patches
-    """
-    if option == "all":
-        patchdir = get_series(d)
-    elif option == "applying":
-        patchdir = get_applying_patches(d)
-    try:
-        os.rmdir(patchdir)
-    except OSError:
-            tarpatch = archive_patches(d, patchdir, option)
-            return tarpatch
-    return
-
-def archive_logs(d, logdir, bbinc=False):
-    """
-    archive logs in temp to tarball and .bb and .inc files if bbinc is True
-    """
-    import shutil
+# creating .diff.gz between source.orig and source
+def create_diff_gz(d, src_orig, src, ar_outdir):
 
-    pf = d.getVar('PF', True)
-    work_dir = d.getVar('WORKDIR', True)
-    log_dir =  os.path.basename(logdir)
-    tarname = pf + '-' + log_dir + ".tar.gz"
-    archive_dir = os.path.join( logdir, '..' )
-    tarname = do_tarball(archive_dir, log_dir, tarname)
-    if bbinc:
-        shutil.rmtree(logdir, ignore_errors=True)
-    return tarname
-
-def get_licenses(d):
-    """get licenses for running .bb file"""
-    import oe.license
-
-    licenses_type = d.getVar('LICENSE', True) or ""
-    lics = oe.license.is_included(licenses_type)[1:][0]
-    lice = ''
-    for lic in lics:
-        licens = d.getVarFlag('SPDXLICENSEMAP', lic)
-        if licens != None:
-            lice += licens
-        else:
-            lice += lic
-    return lice
-
-
-def move_tarball_deploy(d, tarball_list):
-    """move tarball in location to ${DEPLOY_DIR}/sources"""
-    import shutil
+    import subprocess
 
-    if tarball_list is []:
+    if not os.path.isdir(src) or not os.path.isdir(src_orig):
         return
-    target_sys = d.getVar('TARGET_SYS', True)
-    pf = d.getVar('PF', True)
-    licenses = get_licenses(d)
-    work_dir = d.getVar('WORKDIR', True)
-    tar_sources = d.getVar('DEPLOY_DIR', True) + '/sources/' + target_sys + '/' + licenses + '/' + pf
-    if not os.path.exists(tar_sources):
-        bb.utils.mkdirhier(tar_sources)
-    for source in tarball_list:
-        if source:
-            if os.path.exists(os.path.join(tar_sources, source)):
-                os.remove(os.path.join(tar_sources, source))
-            shutil.move(os.path.join(work_dir, source), tar_sources)
-
-def check_archiving_type(d):
-    """check the type for archiving package('tar' or 'srpm')"""
-    if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True) not in d.getVar('ARCHIVE_TYPE', True).split():
-        bb.fatal("\"SOURCE_ARCHIVE_PACKAGE_TYPE\" is \'tar\' or \'srpm\', no other types")
-
-def store_package(d, package_name):
-    """
-    store tarbablls name to file "tar-package"
-    """
-    f = open(os.path.join(d.getVar('WORKDIR', True), 'tar-package'), 'a')
-    f.write(package_name + ' ')
-    f.close()
-
-def get_package(d):
-    """
-    get tarballs name from "tar-package"
-    """
-    work_dir = (d.getVar('WORKDIR', True))
-    tarlist = os.path.join(work_dir, 'tar-package')
-    if os.path.exists(tarlist):
-        f = open(tarlist, 'r')
-        line = f.readline().rstrip('\n').split()
-        f.close()
-        return line
-    return []
 
+    # The diff --exclude can't exclude the file with path, so we copy
+    # the patched source, and remove the files that we'd like to
+    # exclude.
+    src_patched = src + '.patched'
+    oe.path.copyhardlinktree(src, src_patched)
+    for i in d.getVarFlag('ARCHIVER_MODE', 'diff-exclude', True).split():
+        bb.utils.remove(os.path.join(src_orig, i), recurse=True)
+        bb.utils.remove(os.path.join(src_patched, i), recurse=True)
+
+    dirname = os.path.dirname(src)
+    basename = os.path.basename(src)
+    os.chdir(dirname)
+    out_file = os.path.join(ar_outdir, '%s-diff.gz' % d.getVar('PF', True))
+    diff_cmd = 'diff -Naur %s.orig %s.patched | gzip -c > %s' % (basename, basename, out_file)
+    subprocess.call(diff_cmd, shell=True)
+    bb.utils.remove(src_patched, recurse=True)
+
+# Run do_unpack and do_patch
+python do_unpack_and_patch() {
+    if d.getVarFlag('ARCHIVER_MODE', 'src', True) not in \
+            [ 'patched', 'configured'] and \
+            d.getVarFlag('ARCHIVER_MODE', 'diff', True) != '1':
+        return
 
-def archive_sources_patches(d, stage_name):
-    """
-    archive sources and patches to tarball. stage_name will append
-    strings ${stage_name} to ${PR} as middle name. for example,
-    zlib-1.4.6-prepatch(stage_name).tar.gz
-    """
-    import shutil
+    ar_outdir = d.getVar('ARCHIVER_OUTDIR', True)
 
-    check_archiving_type(d)
+    # Change the WORKDIR to make do_unpack do_patch run in another dir.
+    d.setVar('WORKDIR', d.getVar('ARCHIVER_WORKDIR', True))
 
-    source_tar_name = archive_sources(d, stage_name)
-    if stage_name == "prepatch":
-        if d.getVar('PATCHES_ARCHIVE_WITH_SERIES', True) == 'yes':
-            patch_tar_name = select_archive_patches(d, "all")
-        elif d.getVar('PATCHES_ARCHIVE_WITH_SERIES', True) == 'no':
-            patch_tar_name = select_archive_patches(d, "applying")
-        else:
-            bb.fatal("Please define 'PATCHES_ARCHIVE_WITH_SERIES' to 'yes' or 'no' ")
+    # The kernel source is ready after do_validate_branches
+    if bb.data.inherits_class('kernel-yocto', d):
+        bb.build.exec_func('do_unpack', d)
+        bb.build.exec_func('do_kernel_checkout', d)
+        bb.build.exec_func('do_validate_branches', d)
     else:
-        patch_tar_name = ''
+        bb.build.exec_func('do_unpack', d)
+
+    # Save the original source for creating the patches
+    if d.getVarFlag('ARCHIVER_MODE', 'diff', True) == '1':
+        src = d.getVar('S', True).rstrip('/')
+        src_orig = '%s.orig' % src
+        oe.path.copytree(src, src_orig)
+    bb.build.exec_func('do_patch', d)
+    # Create the patches
+    if d.getVarFlag('ARCHIVER_MODE', 'diff', True) == '1':
+        bb.note('Creating diff gz...')
+        create_diff_gz(d, src_orig, src, ar_outdir)
+        bb.utils.remove(src_orig, recurse=True)
+}
 
-    if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True) != 'srpm':
-        move_tarball_deploy(d, [source_tar_name, patch_tar_name])
-    else:
-        tarlist = os.path.join(d.getVar('WORKDIR', True), 'tar-package')
-        if os.path.exists(tarlist):
-            os.remove(tarlist)
-        for package in os.path.basename(source_tar_name), patch_tar_name:
-            if package:
-                store_package(d, str(package) + ' ')
-
-def archive_scripts_logs(d):
+python do_ar_recipe () {
     """
-    archive scripts and logs. scripts include .bb and .inc files and
-    logs include stuff in "temp".
+    archive the recipe, including .bb and .inc.
     """
+    import re
     import shutil
 
-    work_dir = d.getVar('WORKDIR', True)
-    temp_dir = os.path.join(work_dir, 'temp')
-    source_archive_log_with_scripts = d.getVar('SOURCE_ARCHIVE_LOG_WITH_SCRIPTS', True)
-    if source_archive_log_with_scripts == 'logs_with_scripts':
-        logdir = get_logs(d)
-        logdir = get_bb_inc(d)
-    elif source_archive_log_with_scripts == 'logs':
-        logdir = get_logs(d)
-    else:
-        return
-
-    tarlog = archive_logs(d, logdir, True)
-
-    if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True) == 'srpm':
-        store_package(d, tarlog)
+    require_re = re.compile( r"require\s+(.+)" )
+    include_re = re.compile( r"include\s+(.+)" )
+    bbfile = d.getVar('FILE', True)
+    outdir = os.path.join(d.getVar('WORKDIR', True), \
+            '%s-recipe' % d.getVar('PF', True))
+    bb.utils.mkdirhier(outdir)
+    shutil.copy(bbfile, outdir)
+
+    dirname = os.path.dirname(bbfile)
+    bbpath = '%s:%s' % (dirname, d.getVar('BBPATH', True))
+    f = open(bbfile, 'r')
+    for line in f.readlines():
+        incfile = None
+        if require_re.match(line):
+            incfile = require_re.match(line).group(1)
+        elif include_re.match(line):
+            incfile = include_re.match(line).group(1)
+        if incfile:
+            incfile = bb.data.expand(incfile, d)
+            incfile = bb.utils.which(bbpath, incfile)
+            if incfile:
+                shutil.copy(incfile, outdir)
+
+    create_tarball(d, outdir, 'recipe', d.getVar('ARCHIVER_OUTDIR', True))
+    bb.utils.remove(outdir, recurse=True)
+}
 
-def dumpdata(d):
+python do_dumpdata () {
     """
-    dump environment to "${P}-${PR}.showdata.dump" including all
-    kinds of variables and functions when running a task
+    dump environment data to ${PF}-showdata.dump
     """
 
-    workdir = bb.data.getVar('WORKDIR', d, 1)
-    distro = bb.data.getVar('DISTRO', d, 1)
-    s = d.getVar('S', True)
-    pf = d.getVar('PF', True)
-    target_sys = d.getVar('TARGET_SYS', True)
-    licenses = get_licenses(d)
-    dumpdir = os.path.join(workdir, 'diffgz-envdata/'+ target_sys + '/' + licenses + '/' + pf )
-    if not os.path.exists(dumpdir):
-        bb.utils.mkdirhier(dumpdir)
-
-    dumpfile = os.path.join(dumpdir, bb.data.expand("${P}-${PR}.showdata.dump", d))
-
-    bb.note("Dumping metadata into '%s'" % dumpfile)
-    f = open(dumpfile, "w")
+    dumpfile = os.path.join(d.getVar('ARCHIVER_OUTDIR', True), \
+        '%s-showdata.dump' % d.getVar('PF', True))
+    bb.note('Dumping metadata into %s' % dumpfile)
+    f = open(dumpfile, 'w')
     # emit variables and shell functions
     bb.data.emit_env(f, d, True)
     # emit the metadata which isn't valid shell
     for e in d.keys():
         if bb.data.getVarFlag(e, 'python', d):
-            f.write("\npython %s () {\n%s}\n" % (e, bb.data.getVar(e, d, 1)))
-    f.close()
-
-def create_diff_gz(d):
-    """
-    creating .diff.gz in ${DEPLOY_DIR_SRC}/${P}-${PR}.diff.g gz for
-    mapping all content in 's' including patches to  xxx.diff.gz
-    """
-    import shutil
-    import subprocess
-
-    work_dir = d.getVar('WORKDIR', True)
-    exclude_from = d.getVar('ARCHIVE_EXCLUDE_FROM', True).split()
-    pf = d.getVar('PF', True)
-    licenses = get_licenses(d)
-    target_sys = d.getVar('TARGET_SYS', True)
-    diff_dir = os.path.join(work_dir, 'diffgz-envdata/'+ target_sys + '/' + licenses + '/' + pf )
-    diff_file = os.path.join(diff_dir, bb.data.expand("${P}-${PR}.diff.gz",d))
-
-    f = open(os.path.join(work_dir,'temp/exclude-from-file'), 'a')
-    for i in exclude_from:
-        f.write(i)
-        f.write("\n")
+            f.write("\npython %s () {\n%s}\n" % (e, bb.data.getVar(e, d, True)))
     f.close()
-
-    s=d.getVar('S', True)
-    distro = d.getVar('DISTRO',True) or ""
-    dest = s + '/' + distro + '/files'
-    if not os.path.exists(dest):
-        bb.utils.mkdirhier(dest)
-    for i in os.listdir(os.getcwd()):
-        if os.path.isfile(i):
-            try:
-                shutil.copy(i, dest)
-            except IOError:
-                subprocess.call('fakeroot cp -rf ' + i + " " + dest, shell=True)
-
-    bb.note("Creating .diff.gz in ${DEPLOY_DIR_SRC}/${P}-${PR}.diff.gz")
-    cmd = "LC_ALL=C TZ=UTC0 diff --exclude-from=" + work_dir + "/temp/exclude-from-file -Naur " + s + '.org' + ' ' +  s + " | gzip -c > " + diff_file
-    d.setVar('DIFF', cmd + "\n")
-    d.setVarFlag('DIFF', 'func', '1')
-    bb.build.exec_func('DIFF', d)
-    shutil.rmtree(s + '.org', ignore_errors=True)
-
-# This function will run when user want to get tarball for sources and
-# patches after do_unpack
-python do_archive_original_sources(){
-    archive_sources_patches(d, 'prepatch')
-}
-
-# This function will run when user want to get tarball for patched
-# sources after do_patch
-python do_archive_patched_sources(){
-    archive_sources_patches(d, 'patched')
-}
-
-# This function will run when user want to get tarball for configured
-# sources after do_configure
-python do_archive_configured_sources(){
-    archive_sources_patches(d, 'configured')
-}
-
-# This function will run when user want to get tarball for logs or both
-# logs and scripts(.bb and .inc files)
-python do_archive_scripts_logs(){
-    archive_scripts_logs(d)
-}
-
-# This function will run when user want to know what variable and
-# functions in a running task are and also can get a diff file including
-# all content a package should include.
-python do_dumpdata_create_diff_gz(){
-    dumpdata(d)
-    create_diff_gz(d)
-}
-
-# This functions prepare for archiving "linux-yocto" because this
-# package create directory 's' before do_patch instead of after
-# do_unpack.  This is special control for archiving linux-yocto only.
-python do_archive_linux_yocto(){
-    s = d.getVar('S', True)
-    if 'linux-yocto' in s:
-        source_tar_name = archive_sources(d, '')
-    if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True) != 'srpm':
-        move_tarball_deploy(d, [source_tar_name, ''])
-}
-do_kernel_checkout[postfuncs] += "do_archive_linux_yocto "
-
-# remove tarball for sources, patches and logs after creating srpm.
-python do_delete_tarlist(){
-    work_dir = d.getVar('WORKDIR', True)
-    tarlist = os.path.join(work_dir, 'tar-package')
-    if os.path.exists(tarlist):
-        os.remove(tarlist)
 }
-do_delete_tarlist[deptask] = "do_archive_scripts_logs"
-do_package_write_rpm[postfuncs] += "do_delete_tarlist "
-
-# Get archiving package with temp(logs) and scripts(.bb and .inc files)
-addtask do_archive_scripts_logs
-
-# Get dump date and create diff file
-addtask do_dumpdata_create_diff_gz
-
-ARCHIVE_SSTATE_OUTDIR = "${DEPLOY_DIR}/sources/"
-ARCHIVE_SSTATE_SCRIPTS_LOGS_INDIR = "${WORKDIR}/script-logs/"
-ARCHIVE_SSTATE_DIFFGZ_ENVDATA_INDIR = "${WORKDIR}/diffgz-envdata/"
 
-SSTATETASKS += "do_archive_scripts_logs"
-do_archive_scripts_logs[sstate-inputdirs] = "${ARCHIVE_SSTATE_SCRIPTS_LOGS_INDIR}"
-do_archive_scripts_logs[sstate-outputdirs] = "${ARCHIVE_SSTATE_OUTDIR}"
-
-python do_archive_scripts_logs_setscene () {
-    sstate_setscene(d)
+SSTATETASKS += "do_deploy_archives"
+do_deploy_archives () {
+    echo "Deploying source archive files ..."
 }
-
-addtask do_archive_scripts_logs_setscene
-
-SSTATETASKS += "do_dumpdata_create_diff_gz"
-do_dumpdata_create_diff_gz[sstate-inputdirs] = "${ARCHIVE_SSTATE_DIFFGZ_ENVDATA_INDIR}"
-do_dumpdata_create_diff_gz[sstate-outputdirs] = "${ARCHIVE_SSTATE_OUTDIR}"
-
-python do_dumpdata_create_diff_gz_setscene () {
+python do_deploy_archives_setscene () {
     sstate_setscene(d)
 }
-
-addtask do_dumpdata_create_diff_gz_setscene
-
-addtask do_archive_original_sources after do_unpack
-addtask do_archive_patched_sources after do_patch
-addtask do_archive_configured_sources after do_configure
+do_deploy_archives[sstate-inputdirs] = "${AR_TOPDIR}"
+do_deploy_archives[sstate-outputdirs] = "${DEPLOY_DIR_SRC}"
+
+addtask do_ar_original after do_unpack
+addtask do_unpack_and_patch after do_patch
+addtask do_ar_patched after do_unpack_and_patch
+addtask do_ar_configured after do_unpack_and_patch
+addtask do_dumpdata
+addtask do_ar_recipe
+addtask do_deploy_archives before do_build
-- 
1.8.3.1

[PATCH 3/6] package_rpm.bbclass: archive the source to srpm package

[Robert Yang]

The archiver.bbclass will put the sources to ARCHIVER_OUTDIR according
to configuration, then the rpmbuild -bs will create the srpm.

[YOCTO #4986]
[YOCTO #5113]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta/classes/package_rpm.bbclass | 33 ++++++++++++---------------------
 1 file changed, 12 insertions(+), 21 deletions(-)

diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass
index bce5648..9e63878 100644
--- a/meta/classes/package_rpm.bbclass
+++ b/meta/classes/package_rpm.bbclass
@@ -6,7 +6,6 @@ RPM="rpm"
 RPMBUILD="rpmbuild"
 
 PKGWRITEDIRRPM = "${WORKDIR}/deploy-rpms"
-PKGWRITEDIRSRPM = "${DEPLOY_DIR}/sources/deploy-srpm"
 
 # Maintaining the perfile dependencies has singificant overhead when writing the 
 # packages. When set, this value merges them for efficiency.
@@ -87,23 +86,26 @@ python write_specfile () {
 
     # append information for logs and patches to %prep
     def add_prep(d,spec_files_bottom):
-        if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True) == 'srpm':
+        if d.getVarFlag('ARCHIVER_MODE', 'srpm', True) == '1' and bb.data.inherits_class('archiver', d):
             spec_files_bottom.append('%%prep -n %s' % d.getVar('PN', True) )
             spec_files_bottom.append('%s' % "echo \"include logs and patches, Please check them in SOURCES\"")
             spec_files_bottom.append('')
 
     # append the name of tarball to key word 'SOURCE' in xxx.spec.
     def tail_source(d):
-        if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True) == 'srpm':
-            source_list = get_package(d)
+        if d.getVarFlag('ARCHIVER_MODE', 'srpm', True) == '1' and bb.data.inherits_class('archiver', d):
+            ar_outdir = d.getVar('ARCHIVER_OUTDIR', True)
+            if not os.path.exists(ar_outdir):
+                return
+            source_list = os.listdir(ar_outdir)
             source_number = 0
-            workdir = d.getVar('WORKDIR', True)
             for source in source_list:
                 # The rpmbuild doesn't need the root permission, but it needs
                 # to know the file's user and group name, the only user and
                 # group in fakeroot is "root" when working in fakeroot.
-                os.chown("%s/%s" % (workdir, source), 0, 0)
-                spec_preamble_top.append('Source' + str(source_number) + ': %s' % source)
+                f = os.path.join(ar_outdir, source)
+                os.chown(f, 0, 0)
+                spec_preamble_top.append('Source%s: %s' % (source_number, source))
                 source_number += 1
     # We need a simple way to remove the MLPREFIX from the package name,
     # and dependency information...
@@ -611,15 +613,6 @@ python write_specfile () {
 }
 
 python do_package_rpm () {
-    def creat_srpm_dir(d):
-        if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True) == 'srpm':
-            clean_licenses = get_licenses(d)
-            pkgwritesrpmdir = bb.data.expand('${PKGWRITEDIRSRPM}/${PACKAGE_ARCH_EXTEND}', d)
-            pkgwritesrpmdir = pkgwritesrpmdir + '/' + clean_licenses
-            bb.utils.mkdirhier(pkgwritesrpmdir)
-            os.chmod(pkgwritesrpmdir, 0755)
-            return pkgwritesrpmdir
-            
     # We need a simple way to remove the MLPREFIX from the package name,
     # and dependency information...
     def strip_multilib(name, d):
@@ -687,16 +680,14 @@ python do_package_rpm () {
     cmd = cmd + " --define 'debug_package %{nil}'"
     cmd = cmd + " --define '_rpmfc_magic_path " + magicfile + "'"
     cmd = cmd + " --define '_tmppath " + workdir + "'"
-    if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True) == 'srpm':
-        cmd = cmd + " --define '_sourcedir " + workdir + "'"
-        cmdsrpm = cmd + " --define '_srcrpmdir " + creat_srpm_dir(d) + "'"
+    if d.getVarFlag('ARCHIVER_MODE', 'srpm', True) == '1' and bb.data.inherits_class('archiver', d):
+        cmd = cmd + " --define '_sourcedir " + d.getVar('ARCHIVER_OUTDIR', True) + "'"
+        cmdsrpm = cmd + " --define '_srcrpmdir " + d.getVar('ARCHIVER_OUTDIR', True) + "'"
         cmdsrpm = cmdsrpm + " -bs " + outspecfile
         # Build the .src.rpm
         d.setVar('SBUILDSPEC', cmdsrpm + "\n")
         d.setVarFlag('SBUILDSPEC', 'func', '1')
         bb.build.exec_func('SBUILDSPEC', d)
-        # Remove the source (SOURCE0, SOURCE1 ...)
-        cmd = cmd + " --rmsource "
     cmd = cmd + " -bb " + outspecfile
 
     # Build the rpm package!
-- 
1.8.3.1

[PATCH 4/6] archiver.bbclass: move a few code to copyleft_compliance.bbclass

[Robert Yang]

Move the code which is only used by copyleft_compliance.bbclass from
archiver.bbclassc, and remove the "inherit archiver" from
copyleft_compliance.bbclass.

The archiver.bbclass is used for archiving various types of sources, but
the copyleft_compliance.bbclass is used for analysing the license, they
don't have much relationships.

[YOCTO #4986]
[YOCTO #5113]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta/classes/archiver.bbclass            | 19 -----------
 meta/classes/copyleft_compliance.bbclass | 55 ++++++++++++++++++++++++++++++--
 2 files changed, 53 insertions(+), 21 deletions(-)

diff --git a/meta/classes/archiver.bbclass b/meta/classes/archiver.bbclass
index b1109dc..54b498d 100644
--- a/meta/classes/archiver.bbclass
+++ b/meta/classes/archiver.bbclass
@@ -35,25 +35,6 @@ do_ar_original[dirs] = "${ARCHIVER_OUTDIR} ${ARCHIVER_WORKDIR}"
 # This is a convenience for the shell script to use it
 
 
-COPYLEFT_LICENSE_INCLUDE ?= 'GPL* LGPL*'
-COPYLEFT_LICENSE_INCLUDE[type] = 'list'
-COPYLEFT_LICENSE_INCLUDE[doc] = 'Space separated list of globs which include licenses'
-
-COPYLEFT_LICENSE_EXCLUDE ?= 'CLOSED Proprietary'
-COPYLEFT_LICENSE_EXCLUDE[type] = 'list'
-COPYLEFT_LICENSE_EXCLUDE[doc] = 'Space separated list of globs which exclude licenses'
-
-COPYLEFT_RECIPE_TYPE ?= '${@copyleft_recipe_type(d)}'
-COPYLEFT_RECIPE_TYPE[doc] = 'The "type" of the current recipe (e.g. target, native, cross)'
-
-COPYLEFT_RECIPE_TYPES ?= 'target'
-COPYLEFT_RECIPE_TYPES[type] = 'list'
-COPYLEFT_RECIPE_TYPES[doc] = 'Space separated list of recipe types to include'
-
-COPYLEFT_AVAILABLE_RECIPE_TYPES = 'target native nativesdk cross crosssdk cross-canadian'
-COPYLEFT_AVAILABLE_RECIPE_TYPES[type] = 'list'
-COPYLEFT_AVAILABLE_RECIPE_TYPES[doc] = 'Space separated list of available recipe types'
-
 python () {
     pn = d.getVar('PN', True)
 
diff --git a/meta/classes/copyleft_compliance.bbclass b/meta/classes/copyleft_compliance.bbclass
index 32aa757..b47c2c9 100644
--- a/meta/classes/copyleft_compliance.bbclass
+++ b/meta/classes/copyleft_compliance.bbclass
@@ -6,11 +6,62 @@
 #
 # vi:sts=4:sw=4:et
 
-# Need the copyleft_should_include
-inherit archiver
+COPYLEFT_LICENSE_INCLUDE ?= 'GPL* LGPL*'
+COPYLEFT_LICENSE_INCLUDE[type] = 'list'
+COPYLEFT_LICENSE_INCLUDE[doc] = 'Space separated list of globs which include licenses'
+
+COPYLEFT_LICENSE_EXCLUDE ?= 'CLOSED Proprietary'
+COPYLEFT_LICENSE_EXCLUDE[type] = 'list'
+COPYLEFT_LICENSE_EXCLUDE[doc] = 'Space separated list of globs which exclude licenses'
+
+COPYLEFT_RECIPE_TYPE ?= '${@copyleft_recipe_type(d)}'
+COPYLEFT_RECIPE_TYPE[doc] = 'The "type" of the current recipe (e.g. target, native, cross)'
+
+COPYLEFT_RECIPE_TYPES ?= 'target'
+COPYLEFT_RECIPE_TYPES[type] = 'list'
+COPYLEFT_RECIPE_TYPES[doc] = 'Space separated list of recipe types to include'
+
+COPYLEFT_AVAILABLE_RECIPE_TYPES = 'target native nativesdk cross crosssdk cross-canadian'
+COPYLEFT_AVAILABLE_RECIPE_TYPES[type] = 'list'
+COPYLEFT_AVAILABLE_RECIPE_TYPES[doc] = 'Space separated list of available recipe types'
 
 COPYLEFT_SOURCES_DIR ?= '${DEPLOY_DIR}/copyleft_sources'
 
+def copyleft_recipe_type(d):
+    for recipe_type in oe.data.typed_value('COPYLEFT_AVAILABLE_RECIPE_TYPES', d):
+        if oe.utils.inherits(d, recipe_type):
+            return recipe_type
+    return 'target'
+
+def copyleft_should_include(d):
+    """
+    Determine if this recipe's sources should be deployed for compliance
+    """
+    import ast
+    import oe.license
+    from fnmatch import fnmatchcase as fnmatch
+
+    recipe_type = d.getVar('COPYLEFT_RECIPE_TYPE', True)
+    if recipe_type not in oe.data.typed_value('COPYLEFT_RECIPE_TYPES', d):
+        return False, 'recipe type "%s" is excluded' % recipe_type
+
+    include = oe.data.typed_value('COPYLEFT_LICENSE_INCLUDE', d)
+    exclude = oe.data.typed_value('COPYLEFT_LICENSE_EXCLUDE', d)
+
+    try:
+        is_included, reason = oe.license.is_included(d.getVar('LICENSE', True), include, exclude)
+    except oe.license.LicenseError as exc:
+        bb.fatal('%s: %s' % (d.getVar('PF', True), exc))
+    else:
+        if is_included:
+            if reason:
+                return True, 'recipe has included licenses: %s' % ', '.join(reason)
+            else:
+                return False, 'recipe does not include a copyleft license'
+        else:
+            return False, 'recipe has excluded licenses: %s' % ', '.join(reason)
+
+
 python do_prepare_copyleft_sources () {
     """Populate a tree of the recipe sources and emit patch series files"""
     import os.path
-- 
1.8.3.1

[PATCH 5/6] local.conf.sample.extended: update for the archiver

[Robert Yang]

Updated it since we have refactored the archiver.bbclass.

[YOCTO #5113]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta/conf/local.conf.sample.extended | 63 +++++++++++++++++-------------------
 1 file changed, 29 insertions(+), 34 deletions(-)

diff --git a/meta/conf/local.conf.sample.extended b/meta/conf/local.conf.sample.extended
index 5f10886..aa01917 100644
--- a/meta/conf/local.conf.sample.extended
+++ b/meta/conf/local.conf.sample.extended
@@ -188,40 +188,35 @@
 # when the disk space reduces 50M (or the amount of inode reduces 5k).
 #BB_DISKMON_WARNINTERVAL = "50M,5K"
 
-# Archiving source code, configure what kind of sources will be archived
-# and the output format. The output files will be put in
-# ${DEPLOY_DIR}/sources/.
-#
-# You can add the following 3 lines to the conf file to get a quick
-# usage:
-#ARCHIVER_MODE ?= "original"
-#ARCHIVER_CLASS = "${@'archive-${ARCHIVER_MODE}-source' if ARCHIVER_MODE != 'none' else ''}"
-#INHERIT += "${ARCHIVER_CLASS}"
-#
-# Detailed configuration:
-# What kind of sources will be archived, the ARCHIVER_MODE could be:
-# original: the ${S} after do_unpack
-# patched : the ${S} after do_patch
-# configured: the ${S} after do_configure
-#ARCHIVER_MODE ?= "original"
-#
-# The output format type, tar or srpm, the default is "tar".
-#ARCHIVER_MODE[type] ?= "tar"
-#
-# Whether include the log file under ${T} and the recipe (.bb and .inc),
-# the default is "logs_with_scripts".
-#ARCHIVER_MODE[log_type] ?= "logs_with_scripts"
-#
-# license filter:
-# yes: Only the COPYLEFT_LICENSE_INCLUDE recipe will be archived
-# no: All kinds of license will be archived
-# The default is "no"
-#ARCHIVER_MODE[filter] ?= "no"
-#
-# The following lines should be added to your conf file verbatim
-# (uncommented though of course):
-#ARCHIVER_CLASS = "${@'archive-${ARCHIVER_MODE}-source' if ARCHIVER_MODE != 'none' else ''}"
-#INHERIT += "${ARCHIVER_CLASS}"
+# Archive the source and put them to ${DEPLOY_DIR}/sources/.
+#
+#INHERIT += "archiver"
+#
+# The tarball for the patched source will be created by default, and you
+# can configure the archiver as follow:
+#
+# Create archive for:
+# 1) original (or unpacked) source:
+#ARCHIVER_MODE[src] = "original"
+# 2) patched source: (default)
+#ARCHIVER_MODE[src] = "patched"
+# 3) configured source:
+#ARCHIVER_MODE[src] = "configured"
+#
+# 4) the patches between do_unpack and do_patch:
+#ARCHIVER_MODE[diff] = "1"
+# set the files that you'd like to exclude from the diff:
+#ARCHIVER_MODE[diff-exclude] ?= ".pc autom4te.cache patches"
+#
+# 5) the environment data, similar to 'bitbake -e recipe':
+#ARCHIVER_MODE[dumpdata] = "1"
+#
+# 6) the recipe (.bb and .inc):
+#ARCHIVER_MODE[recipe] = "1"
+#
+# Whether output the .src.rpm package:
+#ARCHIVER_MODE[srpm] = "1"
+#
 
 # Remove the old image before the new one generated to save disk space
 #RM_OLD_IMAGE = "1"
-- 
1.8.3.1

[PATCH 6/6] patch.bbclass: do_patch: unset TMPDIR from environment

[Robert Yang]

The do_patch set TMPDIR via:
os.environ['TMPDIR'] = process_tmpdir

We need unset it after we use it, otherwise it would affect the env
after we run "bb.build.exec_func('do_patch', d)", and will cause
unexpected errors.

[YOCTO #5113]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta/classes/patch.bbclass | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/classes/patch.bbclass b/meta/classes/patch.bbclass
index 86c65b3..2c92463 100644
--- a/meta/classes/patch.bbclass
+++ b/meta/classes/patch.bbclass
@@ -177,6 +177,7 @@ python patch_do_patch() {
             bb.fatal(str(e))
 
     bb.utils.remove(process_tmpdir, True)
+    os.unsetenv('TMPDIR')
 }
 patch_do_patch[vardepsexclude] = "PATCHRESOLVE"
 
-- 
1.8.3.1

Re: [PATCH 6/6] patch.bbclass: do_patch: unset TMPDIR from environment

[Burton, Ross]

On 11 March 2014 17:08, Robert Yang <liezhi.yang@windriver.com> wrote:
> We need unset it after we use it, otherwise it would affect the env
> after we run "bb.build.exec_func('do_patch', d)", and will cause
> unexpected errors.

This assignment has been there since October 2012, so what are the
problems that this causes?

Ross

Re: [PATCH 6/6] patch.bbclass: do_patch: unset TMPDIR from environment

[Burton, Ross]

On 13 March 2014 16:46, Burton, Ross <ross.burton@intel.com> wrote:
> On 11 March 2014 17:08, Robert Yang <liezhi.yang@windriver.com> wrote:
>> We need unset it after we use it, otherwise it would affect the env
>> after we run "bb.build.exec_func('do_patch', d)", and will cause
>> unexpected errors.
>
> This assignment has been there since October 2012, so what are the
> problems that this causes?

So what bothers me about this (and the original patch) is that TMPDIR
has special meaning to bitbake and the conflict with patch's temporary
directory is unfortunate.  Maybe a better solution is to change
meta/lib/oe/patch.py to set TMPDIR when invoking patch so the changed
TMPDIR is constrained to the patch process and nothing else.

Ross

Re: [PATCH 3/6] package_rpm.bbclass: archive the source to srpm package

[Burton, Ross]

On 11 March 2014 17:08, Robert Yang <liezhi.yang@windriver.com> wrote:
> -        # Remove the source (SOURCE0, SOURCE1 ...)
> -        cmd = cmd + " --rmsource "

You probably want to keep this as otherwise
deploy/sources/MACHINE/RECIPE/ contains both the .src.rpm and the
separate sources.

Ross

Re: [PATCH 0/6 V3] refactor the archiver*.bbclass

[Burton, Ross]

On 11 March 2014 17:08, Robert Yang <liezhi.yang@windriver.com> wrote:
> Robert Yang (6):
>   classes/archive*.bbclass: remove archive-*-source.bbclass
>   archiver.bbclass: refactor it
>   package_rpm.bbclass: archive the source to srpm package
>   archiver.bbclass: move a few code to copyleft_compliance.bbclass
>   local.conf.sample.extended: update for the archiver
>   patch.bbclass: do_patch: unset TMPDIR from environment

This series appears to work for me and doesn't cause rebuilds (after
removing the patch change, as that obviously does).  Can anyone else
who actively uses the archiver verify that it still works for them?

Cheers,
Ross

Re: [PATCH 6/6] patch.bbclass: do_patch: unset TMPDIR from environment

[Chris Larson]

[-- Attachment #1: Type: text/plain, Size: 1067 bytes --]

On Thu, Mar 13, 2014 at 9:54 AM, Burton, Ross <ross.burton@intel.com> wrote:

> On 13 March 2014 16:46, Burton, Ross <ross.burton@intel.com> wrote:
> > On 11 March 2014 17:08, Robert Yang <liezhi.yang@windriver.com> wrote:
> >> We need unset it after we use it, otherwise it would affect the env
> >> after we run "bb.build.exec_func('do_patch', d)", and will cause
> >> unexpected errors.
> >
> > This assignment has been there since October 2012, so what are the
> > problems that this causes?
>
> So what bothers me about this (and the original patch) is that TMPDIR
> has special meaning to bitbake and the conflict with patch's temporary
> directory is unfortunate.  Maybe a better solution is to change
> meta/lib/oe/patch.py to set TMPDIR when invoking patch so the changed
> TMPDIR is constrained to the patch process and nothing else.


That sounds like a clean approach to me.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 1644 bytes --]

Re: [PATCH 3/6] package_rpm.bbclass: archive the source to srpm package

[Robert Yang]

On 03/14/2014 01:39 AM, Burton, Ross wrote:
> On 11 March 2014 17:08, Robert Yang <liezhi.yang@windriver.com> wrote:
>> -        # Remove the source (SOURCE0, SOURCE1 ...)
>> -        cmd = cmd + " --rmsource "
>
> You probably want to keep this as otherwise
> deploy/sources/MACHINE/RECIPE/ contains both the .src.rpm and the
> separate sources.
>

I'd like to remove the sources in the first thought, but that would
cause a problem: if we only re-run the do_package_write_rpm, for example,
when the FILES is changed, then the sources are gone, and we would get
nothing in the .src.rpm. I thing that keep both the .src.rpm and.tar.gz
is not a big problem, so I changed the usage to:

We will always get .tar.gz and other sources, and the .src.rpm is optional:

ARCHIVER_MODE[srpm] ?= "1"


I think that we can move the rpmbuild related code to archiver.*bbclass to
fix  the problem, but that would change a lot of the code, I'd like to fix
it in  the next release:-)

// Robert

> Ross
>
>

Re: [PATCH 6/6] patch.bbclass: do_patch: unset TMPDIR from environment

[Robert Yang]

On 03/14/2014 12:54 AM, Burton, Ross wrote:
> On 13 March 2014 16:46, Burton, Ross <ross.burton@intel.com> wrote:
>> On 11 March 2014 17:08, Robert Yang <liezhi.yang@windriver.com> wrote:
>>> We need unset it after we use it, otherwise it would affect the env
>>> after we run "bb.build.exec_func('do_patch', d)", and will cause
>>> unexpected errors.
>>
>> This assignment has been there since October 2012, so what are the
>> problems that this causes?
>
> So what bothers me about this (and the original patch) is that TMPDIR
> has special meaning to bitbake and the conflict with patch's temporary
> directory is unfortunate.  Maybe a better solution is to change
> meta/lib/oe/patch.py to set TMPDIR when invoking patch so the changed
> TMPDIR is constrained to the patch process and nothing else.
>

Hi Ross,

The problem is that:
1) do_patch set "TMPDIR" in env, it would remove the directory but didn't
    unset the "TMPDIR".

2) autoconf will use the "TMPDIR" from the env:
autoconf/Autom4te/General.pm:  my $TMPDIR = $ENV{'TMPDIR'} || '/tmp';

And if the dir doesn't exist, there would be errors.

I had met this problem because I used the do_patch and do_configure
in one task:

bb.build.exec_func('do_patch', d)
bb.build.exec_func('do_configure', d)

I had split them into 2 tasks, so the "TMPDIR" doesn't affact the
archiver.bbclass any more.

I've tried to set the TMPDIR in meta/lib/oe/patch.py, but met more
problems, the PATCHTOOL = "patch" doesn't work (Seems only on
OpenSuse 12.2), I've filed a bug for it:

https://bugzilla.yoctoproject.org/show_bug.cgi?id=5968

I think that we can fix it in another thread since it doesn't affect
the archiver.bbclass, as we have talked, we may drop the
PATCHTOOL = "patch"

// Robert

> Ross
>
>

Re: [PATCH 3/6] package_rpm.bbclass: archive the source to srpm package

[Burton, Ross]

On 14 March 2014 02:14, Robert Yang <liezhi.yang@windriver.com> wrote:
> I'd like to remove the sources in the first thought, but that would
> cause a problem: if we only re-run the do_package_write_rpm, for example,
> when the FILES is changed, then the sources are gone, and we would get
> nothing in the .src.rpm. I thing that keep both the .src.rpm and.tar.gz
> is not a big problem, so I changed the usage to:

Good argument, fair enough.

Ross

Re: [PATCH 0/6 V3] refactor the archiver*.bbclass

[Robert Yang]

On 03/14/2014 01:45 AM, Burton, Ross wrote:
> On 11 March 2014 17:08, Robert Yang <liezhi.yang@windriver.com> wrote:
>> Robert Yang (6):
>>    classes/archive*.bbclass: remove archive-*-source.bbclass
>>    archiver.bbclass: refactor it
>>    package_rpm.bbclass: archive the source to srpm package
>>    archiver.bbclass: move a few code to copyleft_compliance.bbclass
>>    local.conf.sample.extended: update for the archiver
>>    patch.bbclass: do_patch: unset TMPDIR from environment
>
> This series appears to work for me and doesn't cause rebuilds (after
> removing the patch change, as that obviously does).  Can anyone else
> who actively uses the archiver verify that it still works for them?
>

Thank you very much for the testing and suggestions, I've updated it
a little:

* Remove the [PATCH 6/6] which is used for unset the TMPDIR, we can fix
   it in another thread as bug 5968 describes since it doesn't affect the
   archiver.

* Modified the do_ar_configured a little and tested the "bitbake
   core-image-sato world meta-toolchain", it worked well.

Here is the repo: (the same repo as before)

   git://git.openembedded.org/openembedded-core-contrib rbt/archiver_v3
 
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=rbt/archiver_v3

Robert Yang (5):
   classes/archive*.bbclass: remove archive-*-source.bbclass
   archiver.bbclass: refactor it
   package_rpm.bbclass: archive the source to srpm package
   archiver.bbclass: move a few code to copyleft_compliance.bbclass
   local.conf.sample.extended: update for the archiver

// Robert

> Cheers,
> Ross
>
>

Re: [PATCH 4/6] archiver.bbclass: move a few code to copyleft_compliance.bbclass

[Paul Eggleton]

On Wednesday 12 March 2014 01:08:22 Robert Yang wrote:
> Move the code which is only used by copyleft_compliance.bbclass from
> archiver.bbclassc, and remove the "inherit archiver" from
> copyleft_compliance.bbclass.
> 
> The archiver.bbclass is used for archiving various types of sources, but
> the copyleft_compliance.bbclass is used for analysing the license, they
> don't have much relationships.
> 
> [YOCTO #4986]
> [YOCTO #5113]

I can see the benefit of tidying this up, however I think it's still desirable 
for archiver.bbclass to have the ability to filter the recipes that it archives 
by LICENSE value, so that you can easily exclude archiving proprietary sources 
if you wish. At the moment it doesn't seem like there's an easy way even to 
disable archiving on an individual recipe basis.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 2/6] archiver.bbclass: refactor it

[Paul Eggleton]

On Wednesday 12 March 2014 01:08:20 Robert Yang wrote:
> +AR_TOPDIR = "${WORKDIR}/deploy-sources"

Could we please be consistent here and use ARCHIVER_TOPDIR, and also set it 
with ?= so the user can override it if desired?

Thanks,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 0/6 V3] refactor the archiver*.bbclass

[Paul Eggleton]

Hi Robert,

On Wednesday 12 March 2014 01:08:18 Robert Yang wrote:
> === V3:
> * Do the unpack, patch and configure in another WORKDIR as RP and Ross
>   suggested, so that it would not affact the default build dependencies,
>   thanks for Ross' prototype.
> * Don't archive the logs (temp/log.do*), mainly because of:
>   - We can easily get the log via reset the T rather than use the
>     archiver.
>   - If we need archive the logs, we must let the task to run as the last
>     one, but it's hard to do this.
> 
> === V2:
> * Fix the warning between different machines which is reported by Martin
> * Fix the error when the archiver.bbclass is not inherited but
>   ARCHIVER_MODE[type] = "srpm" is set, reported by Ross.
> * Fix the archiving for gcc staff which uses the shared source according
>   to the recently changes of prefuncs and postfuncs in bitbake, so it only
>   works well with bitbake's up to date master branch, if we want to backport
> it, we also need backport the pre/postfuncs related patches.
> 
> === V1:
> * The archive*.bbclass didn't work, and there were a few problems, for
>   example:
>   1) There were a few duplicated code
>   2) There was no src_dir.org (or orig), but the diff command still use
>      it, and it is not easy to fix this issue if we don't change a lot
>      of the code.
>   3) It didn't archive the source for the native or gcc
>   4) The work flow is not very well
>   5) The "subprocess.call('fakeroot cp xxxx'" should be removed
>   6) And others ...
> 
> * So that we have to refactor it, the benefits are:
>   1) Fix the problems and make it work well.
>   2) Reduce more than 400 lines in total.
>   3) Make it easy to use.
> 
> // Robert
> 
> 
> The following changes since commit 23f7decb84922a3467c5b10b013e66596926b5cf:
> 
>   .templateconf: New file for customized template defaults (2014-03-11
> 08:11:07 -0700)
> 
> are available in the git repository at:
> 
>   git://git.openembedded.org/openembedded-core-contrib rbt/archiver_v3
>  
> http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=rbt/
> archiver_v3
> 
> Robert Yang (6):
>   classes/archive*.bbclass: remove archive-*-source.bbclass
>   archiver.bbclass: refactor it
>   package_rpm.bbclass: archive the source to srpm package
>   archiver.bbclass: move a few code to copyleft_compliance.bbclass
>   local.conf.sample.extended: update for the archiver

I've had a look at this series - great work on the refactoring and cleanup! 
There were a couple of things I noticed, I've replied to the individual 
patches.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 4/6] archiver.bbclass: move a few code to copyleft_compliance.bbclass

[Robert Yang]

On 03/19/2014 08:09 PM, Paul Eggleton wrote:
> On Wednesday 12 March 2014 01:08:22 Robert Yang wrote:
>> Move the code which is only used by copyleft_compliance.bbclass from
>> archiver.bbclassc, and remove the "inherit archiver" from
>> copyleft_compliance.bbclass.
>>
>> The archiver.bbclass is used for archiving various types of sources, but
>> the copyleft_compliance.bbclass is used for analysing the license, they
>> don't have much relationships.
>>
>> [YOCTO #4986]
>> [YOCTO #5113]
>
> I can see the benefit of tidying this up, however I think it's still desirable
> for archiver.bbclass to have the ability to filter the recipes that it archives
> by LICENSE value, so that you can easily exclude archiving proprietary sources
> if you wish. At the moment it doesn't seem like there's an easy way even to
> disable archiving on an individual recipe basis.
>

Thanks, sounds reasonable, we can treat this as an enhancement, I notice
that it is already on master-next, how about we let them get merged atm?
And I will send another patch to fix it.

// Robert

> Cheers,
> Paul
>

Re: [PATCH 2/6] archiver.bbclass: refactor it

[Robert Yang]

On 03/19/2014 08:12 PM, Paul Eggleton wrote:
> On Wednesday 12 March 2014 01:08:20 Robert Yang wrote:
>> +AR_TOPDIR = "${WORKDIR}/deploy-sources"
>
> Could we please be consistent here and use ARCHIVER_TOPDIR, and also set it
> with ?= so the user can override it if desired?
>

Thanks, it is already on master-next now, how about we let them
get merged atm, and I will send another patch to fix it.

// Robert

> Thanks,
> Paul
>

Re: [PATCH 2/6] archiver.bbclass: refactor it

[Robert Yang]

On 03/19/2014 08:12 PM, Paul Eggleton wrote:
> On Wednesday 12 March 2014 01:08:20 Robert Yang wrote:
>> +AR_TOPDIR = "${WORKDIR}/deploy-sources"
>
> Could we please be consistent here and use ARCHIVER_TOPDIR, and also set it
> with ?= so the user can override it if desired?
>

Hi Paul,

I've updated it in the repo since it is easy to fix:

   git://git.openembedded.org/openembedded-core-contrib rbt/archiver_v3
 
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=rbt/archiver_v3

// Robert

> Thanks,
> Paul
>

Re: [PATCH 0/6 V3] refactor the archiver*.bbclass

[Richard Purdie]

On Fri, 2014-03-14 at 23:03 +0800, Robert Yang wrote:
> 
> On 03/14/2014 01:45 AM, Burton, Ross wrote:
> > On 11 March 2014 17:08, Robert Yang <liezhi.yang@windriver.com> wrote:
> >> Robert Yang (6):
> >>    classes/archive*.bbclass: remove archive-*-source.bbclass
> >>    archiver.bbclass: refactor it
> >>    package_rpm.bbclass: archive the source to srpm package
> >>    archiver.bbclass: move a few code to copyleft_compliance.bbclass
> >>    local.conf.sample.extended: update for the archiver
> >>    patch.bbclass: do_patch: unset TMPDIR from environment
> >
> > This series appears to work for me and doesn't cause rebuilds (after
> > removing the patch change, as that obviously does).  Can anyone else
> > who actively uses the archiver verify that it still works for them?
> >
> 
> Thank you very much for the testing and suggestions, I've updated it
> a little:
> 
> * Remove the [PATCH 6/6] which is used for unset the TMPDIR, we can fix
>    it in another thread as bug 5968 describes since it doesn't affect the
>    archiver.
> 
> * Modified the do_ar_configured a little and tested the "bitbake
>    core-image-sato world meta-toolchain", it worked well.
> 
> Here is the repo: (the same repo as before)
> 
>    git://git.openembedded.org/openembedded-core-contrib rbt/archiver_v3
>  
> http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=rbt/archiver_v3
> 
> Robert Yang (5):
>    classes/archive*.bbclass: remove archive-*-source.bbclass
>    archiver.bbclass: refactor it
>    package_rpm.bbclass: archive the source to srpm package
>    archiver.bbclass: move a few code to copyleft_compliance.bbclass
>    local.conf.sample.extended: update for the archiver

I'm going to merge this into master (its queued in master-next for final
testing) but this is on the condition that we work on follow up patches
to the issues raised during review.

Cheers,

Richard