[PATCH V2 0/1] bitbake-layers: add a ability to query layer dependencies from layer index

[PATCH V2 0/1] bitbake-layers: add a ability to query layer dependencies from layer index

[Chong Lu]

Change since V1:
Extend bitbake-layers to add this ability.

Add meta-openstack to BBLAYERS.

$ bitbake-layers show-layerdependencies
Layer                 Dependencies          Git repository                                      Subdirectory
===============================================================================================================
meta-yocto            openembedded-core     git://git.openembedded.org/openembedded-core        meta
meta-yocto-bsp        openembedded-core     git://git.openembedded.org/openembedded-core        meta
meta-openstack        meta-ruby             git://git.openembedded.org/meta-openembedded        meta-ruby
meta-openstack        meta-networking       git://git.openembedded.org/meta-openembedded        meta-networking
meta-openstack        meta-virtualization   git://git.yoctoproject.org/meta-virtualization      
meta-openstack        openembedded-core     git://git.openembedded.org/openembedded-core        meta
meta-openstack        meta-oe               git://git.openembedded.org/meta-openembedded        meta-oe

The following changes since commit 2674ddb4b0c0645c91d1794cbd9166418b984351:

  dev-manual: Some minor fixes to some text. (2015-01-06 14:27:03 +0000)

are available in the git repository at:

  git://git.pokylinux.org/poky-contrib chonglu/layerindex
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=chonglu/layerindex

Chong Lu (1):
  bitbake-layers: add a ability to query layer dependencies from layer
    index

 bitbake/bin/bitbake-layers | 116 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 116 insertions(+)

-- 
1.9.1

[PATCH V2 1/1] bitbake-layers: add a ability to query layer dependencies from layer index

[Chong Lu]

It maybe depends on other layers when one layer is added to BBLAYERS. If define
LAYERDEPENDS variable in this layer, we will get error from bitbake. But
sometimes, we don't have defined. Add a mechanism to extend bitbake-layers and
it will query layer dependencies from layer index
(http://layers.openembedded.org/layerindex/api/)
Use `bitbake-layers show-layer-dependencies' to find out dependencies and update
bblayers.conf as appropriate.

[YOCTO #5348]

Signed-off-by: Chong Lu <Chong.Lu@windriver.com>
---
 bitbake/bin/bitbake-layers | 116 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 116 insertions(+)

diff --git a/bitbake/bin/bitbake-layers b/bitbake/bin/bitbake-layers
index 9879498..0756095 100755
--- a/bitbake/bin/bitbake-layers
+++ b/bitbake/bin/bitbake-layers
@@ -27,6 +27,7 @@ import sys
 import fnmatch
 from collections import defaultdict
 import re
+import httplib, urlparse, json
 
 bindir = os.path.dirname(__file__)
 topdir = os.path.dirname(bindir)
@@ -157,6 +158,121 @@ usage: remove-layer <layerdir>
                 sys.stderr.write("No layers matching %s found in BBLAYERS\n" % item)
 
 
+    def get_json_data(self, apiurl):
+        proxy_settings = os.environ.get("http_proxy", None)
+        conn = None
+        _parsedurl = urlparse.urlparse(apiurl)
+        path = _parsedurl.path
+        query = _parsedurl.query
+        def parse_url(url):
+            parsedurl = urlparse.urlparse(url)
+            try:
+                (host, port) = parsedurl.netloc.split(":")
+            except ValueError:
+                host = parsedurl.netloc
+                port = None
+
+            if port is None:
+                port = 80
+            else:
+                port = int(port)
+            return (host, port)
+
+        if proxy_settings is None:
+            host, port = parse_url(apiurl)
+            conn = httplib.HTTPConnection(host, port)
+            conn.request("GET", path + "?" + query)
+        else:
+            host, port = parse_url(proxy_settings)
+            conn = httplib.HTTPConnection(host, port)
+            conn.request("GET", apiurl)
+
+        r = conn.getresponse()
+        if r.status != 200:
+            raise Exception("Failed to read " + path + ": %d %s" % (r.status, r.reason))
+        return json.loads(r.read())
+
+
+    def get_layer_deps(self, layername, layeritems, layerbranches, layerdependencies):
+        def layeritems_info_id(items_name, layeritems):
+            litems_id = ""
+            for li in layeritems:
+                if li['name'] == items_name:
+                    litems_id = li['id']
+                    break
+            return litems_id
+
+        def layerbranches_info(items_id, layerbranches):
+            lbranch = {}
+            for lb in layerbranches:
+                # branch is master.
+                if lb['layer'] == items_id and lb['branch'] == 1:
+                    lbranch['id'] = lb['id']
+                    lbranch['vcs_subdir'] = lb['vcs_subdir']
+                    break
+            if not lbranch['id']:
+                logger.plain("The id of layerBranches is not found.")
+                return
+            else:
+                return lbranch
+
+        def layerdependencies_info(lb_id, layerdependencies):
+            ld_deps = []
+            for ld in layerdependencies:
+                if ld['layerbranch'] == lb_id and not ld['dependency'] in ld_deps:
+                    ld_deps.append(ld['dependency'])
+            if not ld_deps:
+                logger.plain("The dependency of layerDependencies is not found.")
+                return
+            else:
+                return ld_deps
+
+        def layeritems_info_name_subdir(items_id, layeritems):
+            litems = {}
+            for li in layeritems:
+                if li['id'] == items_id:
+                    litems['vcs_url'] = li['vcs_url']
+                    litems['name'] = li['name']
+                    break
+            return litems
+
+        itemsid = layeritems_info_id(layername, layeritems)
+        if not itemsid:
+            return
+        lbid = layerbranches_info(itemsid, layerbranches)['id']
+        ldict = {}
+        for dependency in layerdependencies_info(lbid, layerdependencies):
+            lname = layeritems_info_name_subdir(dependency, layeritems)['name']
+            lurl = layeritems_info_name_subdir(dependency, layeritems)['vcs_url']
+            lsubdir = layerbranches_info(dependency, layerbranches)['vcs_subdir']
+            ldict[lname] = lurl, lsubdir
+        return ldict
+
+
+    def do_show_layer_dependencies(self, args):
+        """Find layer dependencies from layer index"""
+        apilinks = self.get_json_data(apiurl = "http://layers.openembedded.org/layerindex/api/")
+        layeritems = self.get_json_data(apilinks['layerItems'])
+        layerbranches = self.get_json_data(apilinks['layerBranches'])
+        layerdependencies = self.get_json_data(apilinks['layerDependencies'])
+        self.init_bbhandler(config_only = True)
+        bblayers_conf = os.path.join('conf', 'bblayers.conf')
+        logger.plain("%s  %s  %s  %s" % ("Layer".ljust(20), "Dependencies".ljust(20), "Git repository".ljust(50), "Subdirectory"))
+        logger.plain('=' * 111)
+        if not os.path.exists(bblayers_conf):
+            sys.stderr.write("Unable to find bblayers.conf\n")
+            return
+        for layerdir in self.bblayers:
+            layername = self.get_layer_name(layerdir)
+            if not layername == "meta":
+                layerdict = self.get_layer_deps(layername, layeritems, layerbranches, layerdependencies)
+                if layerdict:
+                    for layer in layerdict:
+                        logger.plain("%s  %s  %s  %s" % (layername.ljust(20), layer.ljust(20), layerdict[layer][0].ljust(50), layerdict[layer][1]))
+                else:
+                    logger.plain("%s" % layername)
+
+
     def version_str(self, pe, pv, pr = None):
         verstr = "%s" % pv
         if pr:
-- 
1.9.1

Re: [PATCH V2 1/1] bitbake-layers: add a ability to query layer dependencies from layer index

[Paul Eggleton]

Hi Chong,

On Monday 12 January 2015 14:48:42 Chong Lu wrote:
> It maybe depends on other layers when one layer is added to BBLAYERS. If
> define LAYERDEPENDS variable in this layer, we will get error from bitbake.
> But sometimes, we don't have defined. Add a mechanism to extend
> bitbake-layers and it will query layer dependencies from layer index
> (http://layers.openembedded.org/layerindex/api/)
> Use `bitbake-layers show-layer-dependencies' to find out dependencies and
> update bblayers.conf as appropriate.

OK, so this is actually a patch against bitbake and thus should be sent to the 
bitbake mailing list instead.

Having said that, I have looked over the code. This is definitely a step in the 
right direction, however I would make the following comments:

1) Since this is for bitbake we can't really have hardcoded references to the 
OE layer index in the code. I'd suggest we have bitbake-layers query a 
variable e.g. BITBAKE_LAYERINDEX_URL, and then separately patch OE to set that 
from conf/bitbake.conf. (It should also give a sensible error if the variable 
isn't set for some reason.)

2) I think this could be a bit more practical in terms of usage. As described 
in the bug I was thinking of a command for which you would specify the name of 
a layer you wanted to use, and then bitbake-layers would call the layer index, 
find the specified layer and all of its dependencies, fetch them, then add them 
to bblayers.conf. This kind of does the first part but leaves the rest to the 
user - I think we can go a bit further than that. FWIW as you may have seen I 
did just add some code to make clean modifications to bblayers.conf which may 
be useful for part of this.

For #2 there may be a slight problem to solve, which is that whatever 
equivalent of OE-Core you have on your local disk needs to be considered 
equivalent to the "oe-core" layer in the layer index so you don't end up 
adding that. Toaster has solved this same issue, but in a way that won't work 
here, so we might have to figure out an alternative solution.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH V2 1/1] bitbake-layers: add a ability to query layer dependencies from layer index

[Mario Domenech Goulart]

Hi,

On Mon, 12 Jan 2015 14:48:42 +0800 Chong Lu <Chong.Lu@windriver.com> wrote:

> It maybe depends on other layers when one layer is added to BBLAYERS. If define
> LAYERDEPENDS variable in this layer, we will get error from bitbake. But
> sometimes, we don't have defined. Add a mechanism to extend bitbake-layers and
> it will query layer dependencies from layer index
> (http://layers.openembedded.org/layerindex/api/)
> Use `bitbake-layers show-layer-dependencies' to find out dependencies and update
> bblayers.conf as appropriate.
>
> [YOCTO #5348]
>
> Signed-off-by: Chong Lu <Chong.Lu@windriver.com>
> ---
>  bitbake/bin/bitbake-layers | 116 +++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 116 insertions(+)
>
> diff --git a/bitbake/bin/bitbake-layers b/bitbake/bin/bitbake-layers
> index 9879498..0756095 100755
> --- a/bitbake/bin/bitbake-layers
> +++ b/bitbake/bin/bitbake-layers
> @@ -27,6 +27,7 @@ import sys
>  import fnmatch
>  from collections import defaultdict
>  import re
> +import httplib, urlparse, json
>  
>  bindir = os.path.dirname(__file__)
>  topdir = os.path.dirname(bindir)
> @@ -157,6 +158,121 @@ usage: remove-layer <layerdir>
>                  sys.stderr.write("No layers matching %s found in BBLAYERS\n" % item)
>  
>  
> +    def get_json_data(self, apiurl):
> +        proxy_settings = os.environ.get("http_proxy", None)
> +        conn = None
> +        _parsedurl = urlparse.urlparse(apiurl)
> +        path = _parsedurl.path
> +        query = _parsedurl.query
> +        def parse_url(url):
> +            parsedurl = urlparse.urlparse(url)
> +            try:
> +                (host, port) = parsedurl.netloc.split(":")
> +            except ValueError:
> +                host = parsedurl.netloc
> +                port = None
> +
> +            if port is None:
> +                port = 80
> +            else:
> +                port = int(port)
> +            return (host, port)
> +
> +        if proxy_settings is None:
> +            host, port = parse_url(apiurl)
> +            conn = httplib.HTTPConnection(host, port)
> +            conn.request("GET", path + "?" + query)
> +        else:
> +            host, port = parse_url(proxy_settings)
> +            conn = httplib.HTTPConnection(host, port)
> +            conn.request("GET", apiurl)
> +
> +        r = conn.getresponse()
> +        if r.status != 200:
> +            raise Exception("Failed to read " + path + ": %d %s" % (r.status, r.reason))
> +        return json.loads(r.read())
> +
> +
> +    def get_layer_deps(self, layername, layeritems, layerbranches, layerdependencies):
> +        def layeritems_info_id(items_name, layeritems):
> +            litems_id = ""
> +            for li in layeritems:
> +                if li['name'] == items_name:
> +                    litems_id = li['id']
> +                    break
> +            return litems_id
> +
> +        def layerbranches_info(items_id, layerbranches):
> +            lbranch = {}
> +            for lb in layerbranches:
> +                # branch is master.
> +                if lb['layer'] == items_id and lb['branch'] == 1:
> +                    lbranch['id'] = lb['id']
> +                    lbranch['vcs_subdir'] = lb['vcs_subdir']
> +                    break
> +            if not lbranch['id']:
> +                logger.plain("The id of layerBranches is not found.")
> +                return
> +            else:
> +                return lbranch
> +
> +        def layerdependencies_info(lb_id, layerdependencies):
> +            ld_deps = []
> +            for ld in layerdependencies:
> +                if ld['layerbranch'] == lb_id and not ld['dependency'] in ld_deps:
> +                    ld_deps.append(ld['dependency'])
> +            if not ld_deps:
> +                logger.plain("The dependency of layerDependencies is not found.")
> +                return
> +            else:
> +                return ld_deps
> +
> +        def layeritems_info_name_subdir(items_id, layeritems):
> +            litems = {}
> +            for li in layeritems:
> +                if li['id'] == items_id:
> +                    litems['vcs_url'] = li['vcs_url']
> +                    litems['name'] = li['name']
> +                    break
> +            return litems
> +
> +        itemsid = layeritems_info_id(layername, layeritems)
> +        if not itemsid:
> +            return
> +        lbid = layerbranches_info(itemsid, layerbranches)['id']
> +        ldict = {}
> +        for dependency in layerdependencies_info(lbid, layerdependencies):
> +            lname = layeritems_info_name_subdir(dependency, layeritems)['name']
> +            lurl = layeritems_info_name_subdir(dependency, layeritems)['vcs_url']
> +            lsubdir = layerbranches_info(dependency, layerbranches)['vcs_subdir']
> +            ldict[lname] = lurl, lsubdir
> +        return ldict
> +
> +
> +    def do_show_layer_dependencies(self, args):
> +        """Find layer dependencies from layer index"""
> +        apilinks = self.get_json_data(apiurl = "http://layers.openembedded.org/layerindex/api/")
> +        layeritems = self.get_json_data(apilinks['layerItems'])
> +        layerbranches = self.get_json_data(apilinks['layerBranches'])
> +        layerdependencies = self.get_json_data(apilinks['layerDependencies'])

This is subject to race conditions.  There's no guarantee with regard to
atomicity between requests.


> +        self.init_bbhandler(config_only = True)
> +        bblayers_conf = os.path.join('conf', 'bblayers.conf')
> +        logger.plain("%s  %s  %s  %s" % ("Layer".ljust(20), "Dependencies".ljust(20), "Git repository".ljust(50), "Subdirectory"))
> +        logger.plain('=' * 111)
> +        if not os.path.exists(bblayers_conf):
> +            sys.stderr.write("Unable to find bblayers.conf\n")
> +            return
> +        for layerdir in self.bblayers:
> +            layername = self.get_layer_name(layerdir)
> +            if not layername == "meta":
> +                layerdict = self.get_layer_deps(layername, layeritems, layerbranches, layerdependencies)
> +                if layerdict:
> +                    for layer in layerdict:
> +                        logger.plain("%s  %s  %s  %s" % (layername.ljust(20), layer.ljust(20), layerdict[layer][0].ljust(50), layerdict[layer][1]))
> +                else:
> +                    logger.plain("%s" % layername)
> +
> +
>      def version_str(self, pe, pv, pr = None):
>          verstr = "%s" % pv
>          if pr:
> -- 
> 1.9.1

Best wishes.
Mario
-- 
http://www.ossystems.com.br

Re: [PATCH V2 1/1] bitbake-layers: add a ability to query layer dependencies from layer index

[Chong Lu]

On 01/15/2015 10:20 PM, Paul Eggleton wrote:
> Hi Chong,
>
> On Monday 12 January 2015 14:48:42 Chong Lu wrote:
>> It maybe depends on other layers when one layer is added to BBLAYERS. If
>> define LAYERDEPENDS variable in this layer, we will get error from bitbake.
>> But sometimes, we don't have defined. Add a mechanism to extend
>> bitbake-layers and it will query layer dependencies from layer index
>> (http://layers.openembedded.org/layerindex/api/)
>> Use `bitbake-layers show-layer-dependencies' to find out dependencies and
>> update bblayers.conf as appropriate.
> OK, so this is actually a patch against bitbake and thus should be sent to the
> bitbake mailing list instead.
>
> Having said that, I have looked over the code. This is definitely a step in the
> right direction, however I would make the following comments:
>
> 1) Since this is for bitbake we can't really have hardcoded references to the
> OE layer index in the code. I'd suggest we have bitbake-layers query a
> variable e.g. BITBAKE_LAYERINDEX_URL, and then separately patch OE to set that
> from conf/bitbake.conf. (It should also give a sensible error if the variable
> isn't set for some reason.)
>
> 2) I think this could be a bit more practical in terms of usage. As described
> in the bug I was thinking of a command for which you would specify the name of
> a layer you wanted to use, and then bitbake-layers would call the layer index,
> find the specified layer and all of its dependencies, fetch them, then add them
> to bblayers.conf. This kind of does the first part but leaves the rest to the
> user - I think we can go a bit further than that. FWIW as you may have seen I
> did just add some code to make clean modifications to bblayers.conf which may
> be useful for part of this.
>
> For #2 there may be a slight problem to solve, which is that whatever
> equivalent of OE-Core you have on your local disk needs to be considered
> equivalent to the "oe-core" layer in the layer index so you don't end up
> adding that. Toaster has solved this same issue, but in a way that won't work
> here, so we might have to figure out an alternative solution.
>
> Cheers,
> Paul
>

OK. I will modify this and resend one to bitbake mail list.

Best Regards
Chong