* [PATCH 0/2] bitbake-layers: add a ability to query layer dependencies from layer index
@ 2015-01-21 8:02 Chong Lu
2015-01-21 8:02 ` [PATCH 1/2] bitbake.conf: Add two variables for " Chong Lu
` (2 more replies)
0 siblings, 3 replies; 5+ messages in thread
From: Chong Lu @ 2015-01-21 8:02 UTC (permalink / raw)
To: bitbake-devel
$ bitbake-layers show-layer-deps errorlayer,meta-openstack
Parsing recipes..done.
Layer Dependencies Git repository Subdirectory
===================================================================================================================
meta-openstack meta-ruby git://git.openembedded.org/meta-openembedded meta-ruby
meta-openstack meta-networking git://git.openembedded.org/meta-openembedded meta-networking
meta-openstack meta-virtualization git://git.yoctoproject.org/meta-virtualization
meta-openstack openembedded-core git://git.openembedded.org/openembedded-core meta
meta-openstack meta-oe git://git.openembedded.org/meta-openembedded meta-oe
WARNING: errorlayer is not found in layer index.
Cloning into '/buildarea2/clu1/source/poky/meta-cloud-services'...
Add "meta-openstack" to conf/bblayers.conf
Cloning into '/buildarea2/clu1/source/poky/meta-openembedded'...
Add "meta-ruby" to conf/bblayers.conf
Add "meta-networking" to conf/bblayers.conf
Cloning into '/buildarea2/clu1/source/poky/meta-virtualization'...
Add "meta-virtualization" to conf/bblayers.conf
Add "meta-oe" to conf/bblayers.conf
$ cat conf/bblayers.conf
# LAYER_CONF_VERSION is increased each time build/conf/bblayers.conf
# changes incompatibly
LCONF_VERSION = "6"
BBPATH = "${TOPDIR}"
BBFILES ?= ""
BBLAYERS ?= " \
/buildarea2/clu1/source/poky/meta \
/buildarea2/clu1/source/poky/meta-yocto \
/buildarea2/clu1/source/poky/meta-yocto-bsp \
/buildarea2/clu1/source/poky/meta-cloud-services/meta-openstack \
/buildarea2/clu1/source/poky/meta-openembedded/meta-ruby \
/buildarea2/clu1/source/poky/meta-openembedded/meta-networking \
/buildarea2/clu1/source/poky/meta-virtualization \
/buildarea2/clu1/source/poky/meta-openembedded/meta-oe \
"
The following changes since commit 35c9fa0588ed8e88b541a6c80cc1517324616cea:
maintainers: Update for non-maintained recipes (2015-01-20 21:39:41 +0000)
are available in the git repository at:
git://git.pokylinux.org/poky-contrib chonglu/layerindex
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=chonglu/layerindex
Chong Lu (2):
bitbake.conf: Add two variables for layer index
bitbake-layers: add a ability to query layer dependencies from layer
index
bitbake/bin/bitbake-layers | 174 +++++++++++++++++++++++++++++++++++++++++++++
meta/conf/bitbake.conf | 6 ++
2 files changed, 180 insertions(+)
--
1.9.1
^ permalink raw reply [flat|nested] 5+ messages in thread* [PATCH 1/2] bitbake.conf: Add two variables for layer index 2015-01-21 8:02 [PATCH 0/2] bitbake-layers: add a ability to query layer dependencies from layer index Chong Lu @ 2015-01-21 8:02 ` Chong Lu 2015-01-21 8:02 ` [PATCH 2/2] bitbake-layers: add a ability to query layer dependencies from " Chong Lu 2015-01-21 18:19 ` [PATCH 0/2] " Trevor Woerner 2 siblings, 0 replies; 5+ messages in thread From: Chong Lu @ 2015-01-21 8:02 UTC (permalink / raw) To: bitbake-devel Add BITBAKE_LAYERINDEX_URL variable that bitbake-layers can use to find layer index. Add LAYER_FETCH_DIR variable that bitbake-layers can use to specify fetch directory. [YOCTO #5348] Signed-off-by: Chong Lu <Chong.Lu@windriver.com> --- meta/conf/bitbake.conf | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf index d22e9e8..a8f4dc9 100644 --- a/meta/conf/bitbake.conf +++ b/meta/conf/bitbake.conf @@ -549,6 +549,12 @@ SELECTED_OPTIMIZATION[vardeps] += "FULL_OPTIMIZATION DEBUG_OPTIMIZATION" BUILD_OPTIMIZATION = "-O2 -pipe" ################################################################## +# The OE layer index. +################################################################## +BITBAKE_LAYERINDEX_URL ??= "http://layers.openembedded.org/layerindex/api/" +LAYER_FETCH_DIR ??= "${COREBASE}" + +################################################################## # Download locations and utilities. ################################################################## -- 1.9.1 ^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH 2/2] bitbake-layers: add a ability to query layer dependencies from layer index 2015-01-21 8:02 [PATCH 0/2] bitbake-layers: add a ability to query layer dependencies from layer index Chong Lu 2015-01-21 8:02 ` [PATCH 1/2] bitbake.conf: Add two variables for " Chong Lu @ 2015-01-21 8:02 ` Chong Lu 2015-01-21 18:19 ` [PATCH 0/2] " Trevor Woerner 2 siblings, 0 replies; 5+ messages in thread From: Chong Lu @ 2015-01-21 8:02 UTC (permalink / raw) To: bitbake-devel Add a command to query layer dependencies from layer index. Fetch layer and its dependency layers and add them into conf/bblayers.conf. [YOCTO #5348] Signed-off-by: Chong Lu <Chong.Lu@windriver.com> --- bitbake/bin/bitbake-layers | 174 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 174 insertions(+) diff --git a/bitbake/bin/bitbake-layers b/bitbake/bin/bitbake-layers index 9879498..0b00e4c 100755 --- a/bitbake/bin/bitbake-layers +++ b/bitbake/bin/bitbake-layers @@ -27,6 +27,8 @@ import sys import fnmatch from collections import defaultdict import re +import httplib, urlparse, json +import subprocess bindir = os.path.dirname(__file__) topdir = os.path.dirname(bindir) @@ -157,6 +159,178 @@ usage: remove-layer <layerdir> sys.stderr.write("No layers matching %s found in BBLAYERS\n" % item) + def get_json_data(self, apiurl): + proxy_settings = os.environ.get("http_proxy", None) + conn = None + _parsedurl = urlparse.urlparse(apiurl) + path = _parsedurl.path + query = _parsedurl.query + def parse_url(url): + parsedurl = urlparse.urlparse(url) + try: + (host, port) = parsedurl.netloc.split(":") + except ValueError: + host = parsedurl.netloc + port = None + + if port is None: + port = 80 + else: + port = int(port) + return (host, port) + + if proxy_settings is None: + host, port = parse_url(apiurl) + conn = httplib.HTTPConnection(host, port) + conn.request("GET", path + "?" + query) + else: + host, port = parse_url(proxy_settings) + conn = httplib.HTTPConnection(host, port) + conn.request("GET", apiurl) + + r = conn.getresponse() + if r.status != 200: + raise Exception("Failed to read " + path + ": %d %s" % (r.status, r.reason)) + return json.loads(r.read()) + + + def get_layer_deps(self, layername, layeritems, layerbranches, layerdependencies, selfname=False): + def layeritems_info_id(items_name, layeritems): + litems_id = "" + for li in layeritems: + if li['name'] == items_name: + litems_id = li['id'] + break + if litems_id: + return litems_id + + def layerbranches_info(items_id, layerbranches): + lbranch = {} + for lb in layerbranches: + # branch is master. + if lb['layer'] == items_id and lb['branch'] == 1: + lbranch['id'] = lb['id'] + lbranch['vcs_subdir'] = lb['vcs_subdir'] + break + if not lbranch['id']: + logger.error("The id of layerBranches is not found.") + return + else: + return lbranch + + def layerdependencies_info(lb_id, layerdependencies): + ld_deps = [] + for ld in layerdependencies: + if ld['layerbranch'] == lb_id and not ld['dependency'] in ld_deps: + ld_deps.append(ld['dependency']) + if not ld_deps: + logger.error("The dependency of layerDependencies is not found.") + return + else: + return ld_deps + + def layeritems_info_name_subdir(items_id, layeritems): + litems = {} + for li in layeritems: + if li['id'] == items_id: + litems['vcs_url'] = li['vcs_url'] + litems['name'] = li['name'] + break + return litems + + if selfname: + selfid = layeritems_info_id(layername, layeritems) + selfsubdir = layerbranches_info(selfid, layerbranches)['vcs_subdir'] + selfurl = layeritems_info_name_subdir(selfid, layeritems)['vcs_url'] + if selfsubdir and selfurl: + return selfurl, selfsubdir + else: + logger.error("Can NOT get %s git repo and subdir" % layername) + return + ldict = {} + itemsid = layeritems_info_id(layername, layeritems) + if not itemsid: + return layername, None + lbid = layerbranches_info(itemsid, layerbranches)['id'] + for dependency in layerdependencies_info(lbid, layerdependencies): + lname = layeritems_info_name_subdir(dependency, layeritems)['name'] + lurl = layeritems_info_name_subdir(dependency, layeritems)['vcs_url'] + lsubdir = layerbranches_info(dependency, layerbranches)['vcs_subdir'] + ldict[lname] = lurl, lsubdir + return None, ldict + + + def get_fetch_layer(self, fetchdir, url, subdir): + layername = self.get_layer_name(url) + if os.path.splitext(layername)[1] == '.git': + layername = os.path.splitext(layername)[0] + repodir = os.path.join(fetchdir, layername) + layerdir = os.path.join(repodir, subdir) + if not os.path.exists(repodir): + logger.plain("Cloning into '%s'..." % repodir) + result = subprocess.call('git clone -q %s %s' % (url, repodir), shell = True) + if result: + logger.error("Failed to download %s" % url) + else: + return layername, layerdir + elif os.path.exists(layerdir): + return layername, layerdir + else: + logger.error("%s is not in %s" % (url, subdir)) + + + def do_show_layer_deps(self, layernames): + """Find layer dependencies from layer index. Fetch it and its dependency layers. Add them to conf/bblayers.conf. + +usage: show-layer-deps <layername,...> +""" + if not layernames: + sys.stderr.write("Please specify layer name.\n") + return + self.init_bbhandler() + apiurl = self.bbhandler.config_data.getVar('BITBAKE_LAYERINDEX_URL', True) + if not apiurl: + logger.error("Can NOT get BITBAKE_LAYERINDEX_URL.") + apilinks = self.get_json_data(apiurl) + layeritems = self.get_json_data(apilinks['layerItems']) + layerbranches = self.get_json_data(apilinks['layerBranches']) + layerdependencies = self.get_json_data(apilinks['layerDependencies']) + valuenames = [] + invaluenames = [] + repourls = [] + display = True + for layername in layernames.split(','): + if not layername == "meta": + invaluename, layerdict = self.get_layer_deps(layername, layeritems, layerbranches, layerdependencies) + if layerdict: + repourls.append(self.get_layer_deps(layername, layeritems, layerbranches, layerdependencies, selfname=True)) + for layer in layerdict: + if display: + logger.plain("%s %s %s %s" % ("Layer".ljust(20), "Dependencies".ljust(20), "Git repository".ljust(55), "Subdirectory")) + logger.plain('=' * 115) + display = False + logger.plain("%s %s %s %s" % (layername.ljust(20), layer.ljust(20), layerdict[layer][0].ljust(55), layerdict[layer][1])) + if not layer == "openembedded-core" and not (layerdict[layer][0], layerdict[layer][1]) in repourls: + repourls.append((layerdict[layer][0], layerdict[layer][1])) + if invaluename and not invaluename in invaluenames: + invaluenames.append(invaluename) + for invaluename in invaluenames: + logger.warn("%s is not found in layer index." % invaluename) + fetchdir = self.bbhandler.config_data.getVar('LAYER_FETCH_DIR', True) + if not fetchdir: + logger.error("Can NOT get LAYER_FETCH_DIR.") + return + if not os.path.exists(fetchdir): + os.makedirs(fetchdir) + for repourl, subdir in repourls: + name, layerdir = self.get_fetch_layer(fetchdir, repourl, subdir) + if subdir: + logger.plain("Add \"%s\" to conf/bblayers.conf" % subdir) + else: + logger.plain("Add \"%s\" to conf/bblayers.conf" % name) + self.do_add_layer(layerdir) + + def version_str(self, pe, pv, pr = None): verstr = "%s" % pv if pr: -- 1.9.1 ^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [PATCH 0/2] bitbake-layers: add a ability to query layer dependencies from layer index 2015-01-21 8:02 [PATCH 0/2] bitbake-layers: add a ability to query layer dependencies from layer index Chong Lu 2015-01-21 8:02 ` [PATCH 1/2] bitbake.conf: Add two variables for " Chong Lu 2015-01-21 8:02 ` [PATCH 2/2] bitbake-layers: add a ability to query layer dependencies from " Chong Lu @ 2015-01-21 18:19 ` Trevor Woerner 2015-01-22 1:46 ` Chong Lu 2 siblings, 1 reply; 5+ messages in thread From: Trevor Woerner @ 2015-01-21 18:19 UTC (permalink / raw) To: Chong Lu, bitbake-devel On 01/21/15 03:02, Chong Lu wrote: > $ bitbake-layers show-layer-deps errorlayer,meta-openstack > Parsing recipes..done. > Layer Dependencies Git repository Subdirectory > =================================================================================================================== > meta-openstack meta-ruby git://git.openembedded.org/meta-openembedded meta-ruby > meta-openstack meta-networking git://git.openembedded.org/meta-openembedded meta-networking > meta-openstack meta-virtualization git://git.yoctoproject.org/meta-virtualization > meta-openstack openembedded-core git://git.openembedded.org/openembedded-core meta > meta-openstack meta-oe git://git.openembedded.org/meta-openembedded meta-oe > WARNING: errorlayer is not found in layer index. > Cloning into '/buildarea2/clu1/source/poky/meta-cloud-services'... > Add "meta-openstack" to conf/bblayers.conf > Cloning into '/buildarea2/clu1/source/poky/meta-openembedded'... > Add "meta-ruby" to conf/bblayers.conf > Add "meta-networking" to conf/bblayers.conf > Cloning into '/buildarea2/clu1/source/poky/meta-virtualization'... > Add "meta-virtualization" to conf/bblayers.conf > Add "meta-oe" to conf/bblayers.conf I wouldn't expect a sub-command called "show-layer-deps" to go ahead and add the shown dependencies to my configuration. Maybe either an explicit switch or separate "add-layer-deps" command should be added instead? ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH 0/2] bitbake-layers: add a ability to query layer dependencies from layer index 2015-01-21 18:19 ` [PATCH 0/2] " Trevor Woerner @ 2015-01-22 1:46 ` Chong Lu 0 siblings, 0 replies; 5+ messages in thread From: Chong Lu @ 2015-01-22 1:46 UTC (permalink / raw) To: Trevor Woerner, bitbake-devel On 01/22/2015 02:19 AM, Trevor Woerner wrote: > On 01/21/15 03:02, Chong Lu wrote: >> $ bitbake-layers show-layer-deps errorlayer,meta-openstack >> Parsing recipes..done. >> Layer Dependencies Git repository Subdirectory >> =================================================================================================================== >> meta-openstack meta-ruby git://git.openembedded.org/meta-openembedded meta-ruby >> meta-openstack meta-networking git://git.openembedded.org/meta-openembedded meta-networking >> meta-openstack meta-virtualization git://git.yoctoproject.org/meta-virtualization >> meta-openstack openembedded-core git://git.openembedded.org/openembedded-core meta >> meta-openstack meta-oe git://git.openembedded.org/meta-openembedded meta-oe >> WARNING: errorlayer is not found in layer index. >> Cloning into '/buildarea2/clu1/source/poky/meta-cloud-services'... >> Add "meta-openstack" to conf/bblayers.conf >> Cloning into '/buildarea2/clu1/source/poky/meta-openembedded'... >> Add "meta-ruby" to conf/bblayers.conf >> Add "meta-networking" to conf/bblayers.conf >> Cloning into '/buildarea2/clu1/source/poky/meta-virtualization'... >> Add "meta-virtualization" to conf/bblayers.conf >> Add "meta-oe" to conf/bblayers.conf > I wouldn't expect a sub-command called "show-layer-deps" to go ahead and > add the shown dependencies to my configuration. Maybe either an explicit > switch or separate "add-layer-deps" command should be added instead? > OK, Thanks for your suggestion, I will add two options for fetch and add layers. Best Regards Chong ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2015-01-22 1:46 UTC | newest] Thread overview: 5+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2015-01-21 8:02 [PATCH 0/2] bitbake-layers: add a ability to query layer dependencies from layer index Chong Lu 2015-01-21 8:02 ` [PATCH 1/2] bitbake.conf: Add two variables for " Chong Lu 2015-01-21 8:02 ` [PATCH 2/2] bitbake-layers: add a ability to query layer dependencies from " Chong Lu 2015-01-21 18:19 ` [PATCH 0/2] " Trevor Woerner 2015-01-22 1:46 ` Chong Lu
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.