[PATCH 0/2] bitbake-layers: add a ability to query layer dependencies from layer index

[PATCH 0/2] bitbake-layers: add a ability to query layer dependencies from layer index

[Chong Lu]

$ bitbake-layers show-layer-deps errorlayer,meta-openstack
Parsing recipes..done.
Layer                 Dependencies          Git repository                                           Subdirectory
===================================================================================================================
meta-openstack        meta-ruby             git://git.openembedded.org/meta-openembedded             meta-ruby
meta-openstack        meta-networking       git://git.openembedded.org/meta-openembedded             meta-networking
meta-openstack        meta-virtualization   git://git.yoctoproject.org/meta-virtualization           
meta-openstack        openembedded-core     git://git.openembedded.org/openembedded-core             meta
meta-openstack        meta-oe               git://git.openembedded.org/meta-openembedded             meta-oe
WARNING: errorlayer is not found in layer index.
Cloning into '/buildarea2/clu1/source/poky/meta-cloud-services'...
Add "meta-openstack" to conf/bblayers.conf
Cloning into '/buildarea2/clu1/source/poky/meta-openembedded'...
Add "meta-ruby" to conf/bblayers.conf
Add "meta-networking" to conf/bblayers.conf
Cloning into '/buildarea2/clu1/source/poky/meta-virtualization'...
Add "meta-virtualization" to conf/bblayers.conf
Add "meta-oe" to conf/bblayers.conf

$ cat conf/bblayers.conf 
# LAYER_CONF_VERSION is increased each time build/conf/bblayers.conf
# changes incompatibly
LCONF_VERSION = "6"

BBPATH = "${TOPDIR}"
BBFILES ?= ""

BBLAYERS ?= " \
  /buildarea2/clu1/source/poky/meta \
  /buildarea2/clu1/source/poky/meta-yocto \
  /buildarea2/clu1/source/poky/meta-yocto-bsp \
  /buildarea2/clu1/source/poky/meta-cloud-services/meta-openstack \
  /buildarea2/clu1/source/poky/meta-openembedded/meta-ruby \
  /buildarea2/clu1/source/poky/meta-openembedded/meta-networking \
  /buildarea2/clu1/source/poky/meta-virtualization \
  /buildarea2/clu1/source/poky/meta-openembedded/meta-oe \
  "

The following changes since commit 35c9fa0588ed8e88b541a6c80cc1517324616cea:

  maintainers: Update for non-maintained recipes (2015-01-20 21:39:41 +0000)

are available in the git repository at:

  git://git.pokylinux.org/poky-contrib chonglu/layerindex
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=chonglu/layerindex

Chong Lu (2):
  bitbake.conf: Add two variables for layer index
  bitbake-layers: add a ability to query layer dependencies from layer
    index

 bitbake/bin/bitbake-layers | 174 +++++++++++++++++++++++++++++++++++++++++++++
 meta/conf/bitbake.conf     |   6 ++
 2 files changed, 180 insertions(+)

-- 
1.9.1

[PATCH 1/2] bitbake.conf: Add two variables for layer index

[Chong Lu]

Add BITBAKE_LAYERINDEX_URL variable that bitbake-layers can use to find layer index.
Add LAYER_FETCH_DIR variable that bitbake-layers can use to specify fetch directory.

[YOCTO #5348]

Signed-off-by: Chong Lu <Chong.Lu@windriver.com>
---
 meta/conf/bitbake.conf | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index d22e9e8..a8f4dc9 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -549,6 +549,12 @@ SELECTED_OPTIMIZATION[vardeps] += "FULL_OPTIMIZATION DEBUG_OPTIMIZATION"
 BUILD_OPTIMIZATION = "-O2 -pipe"
 
 ##################################################################
+# The OE layer index.
+##################################################################
+BITBAKE_LAYERINDEX_URL ??= "http://layers.openembedded.org/layerindex/api/"
+LAYER_FETCH_DIR ??= "${COREBASE}"
+
+##################################################################
 # Download locations and utilities.
 ##################################################################
 
-- 
1.9.1

[PATCH 2/2] bitbake-layers: add a ability to query layer dependencies from layer index

[Chong Lu]

Add a command to query layer dependencies from layer index. Fetch layer and its
dependency layers and add them into conf/bblayers.conf.

[YOCTO #5348]

Signed-off-by: Chong Lu <Chong.Lu@windriver.com>
---
 bitbake/bin/bitbake-layers | 174 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 174 insertions(+)

diff --git a/bitbake/bin/bitbake-layers b/bitbake/bin/bitbake-layers
index 9879498..0b00e4c 100755
--- a/bitbake/bin/bitbake-layers
+++ b/bitbake/bin/bitbake-layers
@@ -27,6 +27,8 @@ import sys
 import fnmatch
 from collections import defaultdict
 import re
+import httplib, urlparse, json
+import subprocess
 
 bindir = os.path.dirname(__file__)
 topdir = os.path.dirname(bindir)
@@ -157,6 +159,178 @@ usage: remove-layer <layerdir>
                 sys.stderr.write("No layers matching %s found in BBLAYERS\n" % item)
 
 
+    def get_json_data(self, apiurl):
+        proxy_settings = os.environ.get("http_proxy", None)
+        conn = None
+        _parsedurl = urlparse.urlparse(apiurl)
+        path = _parsedurl.path
+        query = _parsedurl.query
+        def parse_url(url):
+            parsedurl = urlparse.urlparse(url)
+            try:
+                (host, port) = parsedurl.netloc.split(":")
+            except ValueError:
+                host = parsedurl.netloc
+                port = None
+
+            if port is None:
+                port = 80
+            else:
+                port = int(port)
+            return (host, port)
+
+        if proxy_settings is None:
+            host, port = parse_url(apiurl)
+            conn = httplib.HTTPConnection(host, port)
+            conn.request("GET", path + "?" + query)
+        else:
+            host, port = parse_url(proxy_settings)
+            conn = httplib.HTTPConnection(host, port)
+            conn.request("GET", apiurl)
+
+        r = conn.getresponse()
+        if r.status != 200:
+            raise Exception("Failed to read " + path + ": %d %s" % (r.status, r.reason))
+        return json.loads(r.read())
+
+
+    def get_layer_deps(self, layername, layeritems, layerbranches, layerdependencies, selfname=False):
+        def layeritems_info_id(items_name, layeritems):
+            litems_id = ""
+            for li in layeritems:
+                if li['name'] == items_name:
+                    litems_id = li['id']
+                    break
+            if litems_id:
+                return litems_id
+
+        def layerbranches_info(items_id, layerbranches):
+            lbranch = {}
+            for lb in layerbranches:
+                # branch is master.
+                if lb['layer'] == items_id and lb['branch'] == 1:
+                    lbranch['id'] = lb['id']
+                    lbranch['vcs_subdir'] = lb['vcs_subdir']
+                    break
+            if not lbranch['id']:
+                logger.error("The id of layerBranches is not found.")
+                return
+            else:
+                return lbranch
+
+        def layerdependencies_info(lb_id, layerdependencies):
+            ld_deps = []
+            for ld in layerdependencies:
+                if ld['layerbranch'] == lb_id and not ld['dependency'] in ld_deps:
+                    ld_deps.append(ld['dependency'])
+            if not ld_deps:
+                logger.error("The dependency of layerDependencies is not found.")
+                return
+            else:
+                return ld_deps
+
+        def layeritems_info_name_subdir(items_id, layeritems):
+            litems = {}
+            for li in layeritems:
+                if li['id'] == items_id:
+                    litems['vcs_url'] = li['vcs_url']
+                    litems['name'] = li['name']
+                    break
+            return litems
+
+        if selfname:
+            selfid = layeritems_info_id(layername, layeritems)
+            selfsubdir = layerbranches_info(selfid, layerbranches)['vcs_subdir']
+            selfurl = layeritems_info_name_subdir(selfid, layeritems)['vcs_url']
+            if selfsubdir and selfurl:
+                return selfurl, selfsubdir
+            else:
+                logger.error("Can NOT get %s git repo and subdir" % layername)
+                return
+        ldict = {}
+        itemsid = layeritems_info_id(layername, layeritems)
+        if not itemsid:
+            return layername, None
+        lbid = layerbranches_info(itemsid, layerbranches)['id']
+        for dependency in layerdependencies_info(lbid, layerdependencies):
+            lname = layeritems_info_name_subdir(dependency, layeritems)['name']
+            lurl = layeritems_info_name_subdir(dependency, layeritems)['vcs_url']
+            lsubdir = layerbranches_info(dependency, layerbranches)['vcs_subdir']
+            ldict[lname] = lurl, lsubdir
+        return None, ldict
+
+
+    def get_fetch_layer(self, fetchdir, url, subdir):
+        layername = self.get_layer_name(url)
+        if os.path.splitext(layername)[1] == '.git':
+            layername = os.path.splitext(layername)[0]
+        repodir = os.path.join(fetchdir, layername)
+        layerdir = os.path.join(repodir, subdir)
+        if not os.path.exists(repodir):
+            logger.plain("Cloning into '%s'..." % repodir)
+            result = subprocess.call('git clone -q %s %s' % (url, repodir), shell = True)
+            if result:
+                logger.error("Failed to download %s" % url)
+            else:
+                return layername, layerdir
+        elif os.path.exists(layerdir):
+            return layername, layerdir
+        else:
+            logger.error("%s is not in %s" % (url, subdir))
+
+
+    def do_show_layer_deps(self, layernames):
+        """Find layer dependencies from layer index. Fetch it and its dependency layers. Add them to conf/bblayers.conf.
+
+usage: show-layer-deps <layername,...>
+"""
+        if not layernames:
+            sys.stderr.write("Please specify layer name.\n")
+            return
+        self.init_bbhandler()
+        apiurl = self.bbhandler.config_data.getVar('BITBAKE_LAYERINDEX_URL', True)
+        if not apiurl:
+            logger.error("Can NOT get BITBAKE_LAYERINDEX_URL.")
+        apilinks = self.get_json_data(apiurl)
+        layeritems = self.get_json_data(apilinks['layerItems'])
+        layerbranches = self.get_json_data(apilinks['layerBranches'])
+        layerdependencies = self.get_json_data(apilinks['layerDependencies'])
+        valuenames = []
+        invaluenames = []
+        repourls = []
+        display = True
+        for layername in layernames.split(','):
+            if not layername == "meta":
+                invaluename, layerdict = self.get_layer_deps(layername, layeritems, layerbranches, layerdependencies)
+                if layerdict:
+                    repourls.append(self.get_layer_deps(layername, layeritems, layerbranches, layerdependencies, selfname=True))
+                    for layer in layerdict:
+                        if display:
+                            logger.plain("%s  %s  %s  %s" % ("Layer".ljust(20), "Dependencies".ljust(20), "Git repository".ljust(55), "Subdirectory"))
+                            logger.plain('=' * 115)
+                            display = False
+                        logger.plain("%s  %s  %s  %s" % (layername.ljust(20), layer.ljust(20), layerdict[layer][0].ljust(55), layerdict[layer][1]))
+                        if not layer == "openembedded-core" and not (layerdict[layer][0], layerdict[layer][1]) in repourls:
+                            repourls.append((layerdict[layer][0], layerdict[layer][1]))
+                if invaluename and not invaluename in invaluenames:
+                    invaluenames.append(invaluename)
+        for invaluename in invaluenames:
+            logger.warn("%s is not found in layer index." % invaluename)
+        fetchdir = self.bbhandler.config_data.getVar('LAYER_FETCH_DIR', True)
+        if not fetchdir:
+            logger.error("Can NOT get LAYER_FETCH_DIR.")
+            return
+        if not os.path.exists(fetchdir):
+            os.makedirs(fetchdir)
+        for repourl, subdir in repourls:
+            name, layerdir = self.get_fetch_layer(fetchdir, repourl, subdir)
+            if subdir:
+                logger.plain("Add \"%s\" to conf/bblayers.conf" % subdir)
+            else:
+                logger.plain("Add \"%s\" to conf/bblayers.conf" % name)
+            self.do_add_layer(layerdir)
+
+
     def version_str(self, pe, pv, pr = None):
         verstr = "%s" % pv
         if pr:
-- 
1.9.1

Re: [PATCH 0/2] bitbake-layers: add a ability to query layer dependencies from layer index

[Trevor Woerner]

On 01/21/15 03:02, Chong Lu wrote:
> $ bitbake-layers show-layer-deps errorlayer,meta-openstack
> Parsing recipes..done.
> Layer                 Dependencies          Git repository                                           Subdirectory
> ===================================================================================================================
> meta-openstack        meta-ruby             git://git.openembedded.org/meta-openembedded             meta-ruby
> meta-openstack        meta-networking       git://git.openembedded.org/meta-openembedded             meta-networking
> meta-openstack        meta-virtualization   git://git.yoctoproject.org/meta-virtualization           
> meta-openstack        openembedded-core     git://git.openembedded.org/openembedded-core             meta
> meta-openstack        meta-oe               git://git.openembedded.org/meta-openembedded             meta-oe
> WARNING: errorlayer is not found in layer index.
> Cloning into '/buildarea2/clu1/source/poky/meta-cloud-services'...
> Add "meta-openstack" to conf/bblayers.conf
> Cloning into '/buildarea2/clu1/source/poky/meta-openembedded'...
> Add "meta-ruby" to conf/bblayers.conf
> Add "meta-networking" to conf/bblayers.conf
> Cloning into '/buildarea2/clu1/source/poky/meta-virtualization'...
> Add "meta-virtualization" to conf/bblayers.conf
> Add "meta-oe" to conf/bblayers.conf

I wouldn't expect a sub-command called "show-layer-deps" to go ahead and
add the shown dependencies to my configuration. Maybe either an explicit
switch or separate "add-layer-deps" command should be added instead?

Re: [PATCH 0/2] bitbake-layers: add a ability to query layer dependencies from layer index

[Chong Lu]

On 01/22/2015 02:19 AM, Trevor Woerner wrote:
> On 01/21/15 03:02, Chong Lu wrote:
>> $ bitbake-layers show-layer-deps errorlayer,meta-openstack
>> Parsing recipes..done.
>> Layer                 Dependencies          Git repository                                           Subdirectory
>> ===================================================================================================================
>> meta-openstack        meta-ruby             git://git.openembedded.org/meta-openembedded             meta-ruby
>> meta-openstack        meta-networking       git://git.openembedded.org/meta-openembedded             meta-networking
>> meta-openstack        meta-virtualization   git://git.yoctoproject.org/meta-virtualization
>> meta-openstack        openembedded-core     git://git.openembedded.org/openembedded-core             meta
>> meta-openstack        meta-oe               git://git.openembedded.org/meta-openembedded             meta-oe
>> WARNING: errorlayer is not found in layer index.
>> Cloning into '/buildarea2/clu1/source/poky/meta-cloud-services'...
>> Add "meta-openstack" to conf/bblayers.conf
>> Cloning into '/buildarea2/clu1/source/poky/meta-openembedded'...
>> Add "meta-ruby" to conf/bblayers.conf
>> Add "meta-networking" to conf/bblayers.conf
>> Cloning into '/buildarea2/clu1/source/poky/meta-virtualization'...
>> Add "meta-virtualization" to conf/bblayers.conf
>> Add "meta-oe" to conf/bblayers.conf
> I wouldn't expect a sub-command called "show-layer-deps" to go ahead and
> add the shown dependencies to my configuration. Maybe either an explicit
> switch or separate "add-layer-deps" command should be added instead?
>

OK, Thanks for your suggestion, I will add two options for fetch and add 
layers.

Best Regards
Chong