[PATCH 0/3] Add oe.fetch, rework fetch/unpack deps from SRC_URI

[PATCH 0/3] Add oe.fetch, rework fetch/unpack deps from SRC_URI

[Chris Larson]

From: Chris Larson <chris_larson@mentor.com>

This was intended to clean up and consolidate the code in base.bbclass which
sets up dependencies conditionally based upon the contents of SRC_URI.  To get
there, I decided to incorporate and rewrite some Uri handling code I had
written previously.  This is much more convenient to use than urldata and
SRC_URI, and the caching can be easily re-implemented if we move it into
bitbake at some point (e.g. create an lru cache decorator, inject its
construction/sync into the standard caching spots where cache/signature code
caching is done, if necessary).  Finally, this makes .zip files pull in
unzip-native.

Pull URL: https://github.com/kergoth/oe-core
  Branch: unpack-fetch-deps
  Browse: https://github.com/kergoth/oe-core/commits/unpack-fetch-deps

Thanks,
    Chris Larson <chris_larson@mentor.com>
---


Chris Larson (3):
  oe.fetch: add module with uri convenience code
  Clean up how we handle unpack/fetch deps from SRC_URI
  base: when using zip files, depend on unzip-native

 meta/classes/base.bbclass |   58 ++++++++++++++++--------------
 meta/conf/bitbake.conf    |    1 +
 meta/lib/oe/fetch.py      |   87 +++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 119 insertions(+), 27 deletions(-)
 create mode 100644 meta/lib/oe/fetch.py

-- 
1.7.4.1

[PATCH 1/3] oe.fetch: add module with uri convenience code

[Chris Larson]

From: Chris Larson <chris_larson@mentor.com>

Signed-off-by: Chris Larson <chris_larson@mentor.com>
---
 meta/conf/bitbake.conf |    1 +
 meta/lib/oe/fetch.py   |   87 ++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 88 insertions(+), 0 deletions(-)
 create mode 100644 meta/lib/oe/fetch.py

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 27b8a6b..27b4f74 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -527,6 +527,7 @@ AUTOREV = "${@bb.fetch2.get_autorev(d)}"
 SRCPV = "${@bb.fetch2.get_srcrev(d)}"
 
 SRC_URI = "file://${FILE}"
+SRC_URI[type] = "list"
 
 # We can choose which provider of fake root privileges to use
 # default is fakeroot but in Poky we use pseudo
diff --git a/meta/lib/oe/fetch.py b/meta/lib/oe/fetch.py
new file mode 100644
index 0000000..c99adcf
--- /dev/null
+++ b/meta/lib/oe/fetch.py
@@ -0,0 +1,87 @@
+import oe.data
+import urlparse
+from collections import namedtuple
+
+class MissingScheme(Exception):
+    def __str__(self):
+        return "Missing scheme in uri '%s'" % self.args[0]
+
+def src_uri(d):
+    return [uri(url) for url in oe.data.typed_value('SRC_URI', d)]
+
+def uri(url):
+    import urlparse
+
+    scheme, netloc, path, params, query, fragment = urlparse.urlparse(url)
+
+    if scheme == 'file':
+        if netloc:
+            path = netloc + path
+            netloc = ''
+    elif not scheme:
+        raise MissingScheme(url)
+
+    if ';' in path:
+        path, params = urlparse._splitparams(path)
+
+    params = parse_params(params)
+    query = urlparse.parse_qs(query)
+
+    return Uri(scheme, netloc, path, params, query, fragment)
+
+def parse_params(params):
+    values = {}
+    if params:
+        for param in params.split(';'):
+            try:
+                key, value = param.split('=', 1)
+            except ValueError:
+                key, value = param, True
+            values[key] = value
+    return values
+
+class Uri(namedtuple('Uri', 'scheme netloc path params query fragment'),
+          urlparse.ResultMixin):
+    """Representation of a Uniform Resource Identifier"""
+
+    __slots__ = ()
+
+    @property
+    def querystring(self):
+        """Reassembled query string"""
+        if self.query:
+            query = ';'.join('%s=%s' % (key, v)
+                             for key, value in self.query.iteritems()
+                             for v in value)
+            return query
+        else:
+            return self.query
+
+    @property
+    def parameterstring(self):
+        """Reassembled parameter string"""
+        if self.parameters:
+            parameters = ';'.join('%s=%s' % (key, value)
+                                  for key, value in self.parameters.iteritems())
+            return parameters
+        else:
+            return self.parameters
+
+    def join(self, otherurl):
+        """Join this url to a possibly relative URL to form an absolute
+        interpretation of the latter."""
+        return uri(urlparse.urljoin(str(self), str(otherurl)))
+
+    def unsplit(self):
+        """String version of URL without parameters"""
+        components = (self.scheme, self.netloc, self.path, self.querystring,
+                      self.fragment)
+        return urlparse.urlunsplit(components)
+
+    def geturl(self):
+        components = (self.scheme, self.netloc, self.path,
+                      self.parameterstring, self.querystring, self.fragment)
+        return urlparse.urlunparse(components)
+
+    def __str__(self):
+        return self.geturl()
-- 
1.7.4.1

[PATCH 2/3] Clean up how we handle unpack/fetch deps from SRC_URI

[Chris Larson]

From: Chris Larson <chris_larson@mentor.com>

Signed-off-by: Chris Larson <chris_larson@mentor.com>
---
 meta/classes/base.bbclass |   56 +++++++++++++++++++++++---------------------
 1 files changed, 29 insertions(+), 27 deletions(-)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 9c9fc7c..0a1aebf 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -254,9 +254,37 @@ do_build () {
 	:
 }
 
+
+def set_uri_depends(d):
+    import oe.fetch
+
+    unpack_depends = set()
+    fetch_depends = set()
+    for uri in oe.fetch.src_uri(d):
+	if uri.scheme == 'git':
+	    fetch_depends.add('git-native')
+	elif uri.scheme == 'hg':
+	    fetch_depends.add('mercurial-native')
+	elif uri.scheme == 'osc':
+	    fetch_depends.add('osc-native')
+
+	root, ext = os.path.splitext(uri.path)
+	if ext == '.xz':
+	    unpack_depends.add('xz-native')
+
+    fdepends = d.getVarFlag('do_fetch', 'depends') or ''
+    fdepends += ' '.join('%s:do_populate_sysroot' % f for f in fetch_depends)
+    d.setVarFlag('do_fetch', 'depends', fdepends)
+
+    udepends = d.getVarFlag('do_unpack', 'depends') or ''
+    udepends += ' '.join('%s:do_populate_sysroot' % u for u in unpack_depends)
+    d.setVarFlag('do_unpack', 'depends', udepends)
+
 python () {
     import exceptions, string
 
+    set_uri_depends(d)
+
     # If PRINC is set, try and increase the PR value by the amount specified
     princ = bb.data.getVar('PRINC', d, True)
     if princ:
@@ -341,25 +369,6 @@ python () {
     if use_nls != None:
         bb.data.setVar('USE_NLS', use_nls, d)
 
-    # Git packages should DEPEND on git-native
-    srcuri = bb.data.getVar('SRC_URI', d, 1)
-    if "git://" in srcuri:
-        depends = bb.data.getVarFlag('do_fetch', 'depends', d) or ""
-        depends = depends + " git-native:do_populate_sysroot"
-        bb.data.setVarFlag('do_fetch', 'depends', depends, d)
-
-    # Mercurial packages should DEPEND on mercurial-native
-    elif "hg://" in srcuri:
-        depends = bb.data.getVarFlag('do_fetch', 'depends', d) or ""
-        depends = depends + " mercurial-native:do_populate_sysroot"
-        bb.data.setVarFlag('do_fetch', 'depends', depends, d)
-
-    # OSC packages should DEPEND on osc-native
-    elif "osc://" in srcuri:
-        depends = bb.data.getVarFlag('do_fetch', 'depends', d) or ""
-        depends = depends + " osc-native:do_populate_sysroot"
-        bb.data.setVarFlag('do_fetch', 'depends', depends, d)
-
     # bb.utils.sha256_file() will fail if hashlib isn't present, so we fallback
     # on shasum-native.  We need to ensure that it is staged before we fetch.
     if bb.data.getVar('PN', d, True) != "shasum-native":
@@ -370,13 +379,6 @@ python () {
             depends = depends + " shasum-native:do_populate_sysroot"
             bb.data.setVarFlag('do_fetch', 'depends', depends, d)
 
-    # *.xz should depends on xz-native for unpacking
-    # Not endswith because of "*.patch.xz;patch=1". Need bb.decodeurl in future
-    if '.xz' in srcuri:
-        depends = bb.data.getVarFlag('do_unpack', 'depends', d) or ""
-        depends = depends + " xz-native:do_populate_sysroot"
-        bb.data.setVarFlag('do_unpack', 'depends', depends, d)
-
     # 'multimachine' handling
     mach_arch = bb.data.getVar('MACHINE_ARCH', d, 1)
     pkg_arch = bb.data.getVar('PACKAGE_ARCH', d, 1)
@@ -432,7 +434,7 @@ def check_gcc3(data):
 	for gcc3 in gcc3_versions.split():
 		if check_app_exists(gcc3, data):
 			return gcc3
-	
+
 	return False
 
 addtask cleanall after do_clean
-- 
1.7.4.1

[PATCH 3/3] base: when using zip files, depend on unzip-native

[Chris Larson]

From: Chris Larson <chris_larson@mentor.com>

Signed-off-by: Chris Larson <chris_larson@mentor.com>
---
 meta/classes/base.bbclass |    4 +++-
 1 files changed, 3 insertions(+), 1 deletions(-)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 0a1aebf..7a11c9a 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -269,7 +269,9 @@ def set_uri_depends(d):
 	    fetch_depends.add('osc-native')
 
 	root, ext = os.path.splitext(uri.path)
-	if ext == '.xz':
+	if ext == '.zip':
+	    unpack_depends.add('unzip-native')
+	elif ext == '.xz':
 	    unpack_depends.add('xz-native')
 
     fdepends = d.getVarFlag('do_fetch', 'depends') or ''
-- 
1.7.4.1

Re: [PATCH 0/3] Add oe.fetch, rework fetch/unpack deps from SRC_URI

[Chris Larson]

On Tue, Apr 5, 2011 at 1:39 PM, Chris Larson <kergoth@gmail.com> wrote:
> This was intended to clean up and consolidate the code in base.bbclass which
> sets up dependencies conditionally based upon the contents of SRC_URI.  To get
> there, I decided to incorporate and rewrite some Uri handling code I had
> written previously.  This is much more convenient to use than urldata and
> SRC_URI, and the caching can be easily re-implemented if we move it into
> bitbake at some point (e.g. create an lru cache decorator, inject its
> construction/sync into the standard caching spots where cache/signature code
> caching is done, if necessary).  Finally, this makes .zip files pull in
> unzip-native.

Ugh, ignore this patch, at least for now.  The url handling code has
some problems with our nonstandard url parameters. *rolls eyes and
mutters*.  Will send the unpack/patch deps cleanup separately.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics