[PATCH 0/2] fix do_ar_original (multiple git sources, absolute path)

[PATCH 0/2] fix do_ar_original (multiple git sources, absolute path)

[Patrick Ohly]

meta-oic's iotivity_1.1.1.bb recently started using destsuffix=${S}
for different git source URLs. This breaks randomly (depending on task
scheduling) when archive.bbclass and ARCHIVER_MODE[src] = "original"
are active, because the archiver then unpacks again into ${S} in
parallel to the running build.

While fixing that it was noticed that the archiver also did not
archive all upstream sources because it used the same .tar.gz filename
for all git repos.

The following changes since commit 1e13d1fd22186af5544e7248dc12635cd2f2e08b:

  meta-environment: ensure corret TOOLCHAIN_CONFIGSITE_NOCACHE value (2016-09-26 10:05:53 +0100)

are available in the git repository at:

  git://github.com/pohly/openembedded-core do_ar_original
  https://github.com/pohly/openembedded-core/tree/do_ar_original

Patrick Ohly (2):
  archiver.bbclass: ignore unpack sub-directories in do_ar_original
  archive.bbclass: fix do_ar_original archiving of multiple source repos

 meta/classes/archiver.bbclass | 37 +++++++++++++++++++++++++++++++++++--
 1 file changed, 35 insertions(+), 2 deletions(-)

-- 
2.1.4

[PATCH 1/2] archiver.bbclass: ignore unpack sub-directories in do_ar_original

[Patrick Ohly]

Support for absolute paths in the "subdir" parameter was recently
added (bitbake rev: c3873346c6fa). The git fetcher has supported
absolute paths in "destsuffix" already before.

When the path is absolute as in destsuffix=${S}/foobar, the tmpdir
used by do_ar_original gets ignored, which breaks:
- source code archiving (tmpdir is empty)
- compilation due to race conditions (for example, ${S} getting
  modified by do_ar_original while do_compile runs)

To solve this, these parameters get removed from URLs before
instantiating the fetcher for them.

This is done unconditionally also for relative paths, because these
paths are not useful when archiving the original source (upstream
source does not have them, they only get used by the recipe during
compilation).

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
---
 meta/classes/archiver.bbclass | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)

diff --git a/meta/classes/archiver.bbclass b/meta/classes/archiver.bbclass
index 2f3b278..5f9c91d 100644
--- a/meta/classes/archiver.bbclass
+++ b/meta/classes/archiver.bbclass
@@ -132,7 +132,25 @@ python do_ar_original() {
 
     ar_outdir = d.getVar('ARCHIVER_OUTDIR', True)
     bb.note('Archiving the original source...')
-    fetch = bb.fetch2.Fetch([], d)
+    urls = d.getVar("SRC_URI", True).split()
+    # destsuffix (git fetcher) and subdir (everything else) are allowed to be
+    # absolute paths (for example, destsuffix=${S}/foobar).
+    # That messes with unpacking inside our tmpdir below, because the fetchers
+    # will then unpack in that directory and completely ignore the tmpdir.
+    # That breaks parallel tasks relying on ${S}, like do_compile.
+    #
+    # To solve this, we remove these parameters from all URLs.
+    # We do this even for relative paths because it makes the content of the
+    # archives more useful (no extra paths that are only used during
+    # compilation).
+    for i, url in enumerate(urls):
+        decoded = bb.fetch2.decodeurl(url)
+        for param in ('destsuffix', 'subdir'):
+            if param in decoded[5]:
+                del decoded[5][param]
+        encoded = bb.fetch2.encodeurl(decoded)
+        urls[i] = encoded
+    fetch = bb.fetch2.Fetch(urls, d)
     for url in fetch.urls:
         local = fetch.localpath(url).rstrip("/");
         if os.path.isfile(local):
-- 
2.1.4

[PATCH 2/2] archive.bbclass: fix do_ar_original archiving of multiple source repos

[Patrick Ohly]

When a recipe uses more than one source which isn't a plain file (for
example, multiple git repos), then do_ar_original created the source
archives using the same filename and thus only archived one source.

The "name" parameter is used as file suffix to create unique names for
each source, leading to archives following this pattern:
deploy/${TARGET_SYS}/${PF}/${PF}[-<name>].tar.gz.

The ${PF} part is a bit redundant, which may or may not be
desirable. The patch is more localized this way (no need to modify
create_tarball()).

For example, meta-oic's iotivity_1.1.1.bb uses:

   url_iotivity = "git://github.com/iotivity/iotivity.git"
   branch_iotivity = "1.1-rel"
   SRC_URI = "${url_iotivity};destsuffix=${S};branch=${branch_iotivity};protocol=http;"

   url_tinycbor = "git://github.com/01org/tinycbor.git"
   SRC_URI += "${url_tinycbor};name=tinycbor;destsuffix=${S}/extlibs/tinycbor/tinycbor;protocol=http"

   url_hippomocks = "git://github.com/dascandy/hippomocks.git"
   SRC_URI += "${url_hippomocks};name=hippomocks;destsuffix=${S}/extlibs/hippomocks-master;protocol=http"
   SRC_URI += "file://hippomocks_mips_patch"

   url_gtest = "http://pkgs.fedoraproject.org/repo/pkgs/gtest/gtest-1.7.0.zip/2d6ec8ccdf5c46b05ba54a9fd1d130d7/gtest-1.7.0.zip"
   SRC_URI += "${url_gtest};name=gtest;subdir=${BP}/extlibs/gtest"

   url_sqlite = "http://www.sqlite.org/2015/sqlite-amalgamation-3081101.zip"
   SRC_URI += "${url_sqlite};name=sqlite3;subdir=${BP}/extlibs/sqlite3;unpack=false"

These now get archived in deploy/sources/*/iotivity-1.1.1-r2/ as:

gtest-1.7.0.zip                      iotivity-1.1.1-r2-recipe.tar.gz    sqlite-amalgamation-3081101.zip
hippomocks_mips_patch                iotivity-1.1.1-r2.tar.gz
iotivity-1.1.1-r2-hippomocks.tar.gz  iotivity-1.1.1-r2-tinycbor.tar.gz

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
---
 meta/classes/archiver.bbclass | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)

diff --git a/meta/classes/archiver.bbclass b/meta/classes/archiver.bbclass
index 5f9c91d..1d8e863 100644
--- a/meta/classes/archiver.bbclass
+++ b/meta/classes/archiver.bbclass
@@ -151,6 +151,7 @@ python do_ar_original() {
         encoded = bb.fetch2.encodeurl(decoded)
         urls[i] = encoded
     fetch = bb.fetch2.Fetch(urls, d)
+    tarball_suffix = {}
     for url in fetch.urls:
         local = fetch.localpath(url).rstrip("/");
         if os.path.isfile(local):
@@ -158,7 +159,21 @@ python do_ar_original() {
         elif os.path.isdir(local):
             tmpdir = tempfile.mkdtemp(dir=d.getVar('ARCHIVER_WORKDIR', True))
             fetch.unpack(tmpdir, (url,))
-            create_tarball(d, tmpdir + '/.', '', ar_outdir)
+            # To handle recipes with more than one source, we add the "name"
+            # URL parameter as suffix. We treat it as an error when
+            # there's more than one URL without a name, or a name gets reused.
+            # This is an additional safety net, in practice the name has
+            # to be set when using the git fetcher, otherwise SRCREV cannot
+            # be set separately for each URL.
+            params = bb.fetch2.decodeurl(url)[5]
+            name = params.get('name', '')
+            if name in tarball_suffix:
+                if not name:
+                    bb.fatal("Cannot determine archive names for original source because 'name' URL parameter is unset in more than one URL. Add it to at least one of these: %s %s" % (tarball_suffix[name], url))
+                else:
+                    bb.fatal("Cannot determine archive names for original source because 'name=' URL parameter '%s' is used twice. Make it unique in: %s %s" % (tarball_suffix[name], url))
+            tarball_suffix[name] = url
+            create_tarball(d, tmpdir + '/.', name, ar_outdir)
 
     # Emit patch series files for 'original'
     bb.note('Writing patch series files...')
-- 
2.1.4