* [PATCH 1/2] bitbake.conf, collections.inc: Add COLLECTIONS mechanism w/ a sane default.
@ 2009-04-02 0:27 Chris Larson
2009-04-02 0:27 ` [PATCH 2/2] collections.inc: add support for collection archives Chris Larson
` (3 more replies)
0 siblings, 4 replies; 9+ messages in thread
From: Chris Larson @ 2009-04-02 0:27 UTC (permalink / raw)
To: openembedded-devel
Added code which takes a list of directories in COLLECTIONS, in priority
order (highest to lowest), and uses those to populate BBFILES,
BBFILE_COLLECTIONS, BBFILE_PATTERN_*, and BBFILE_PRIORITY_*.
The default COLLECTIONS is based on BBPATH, so you can now specify a sane
BBPATH, not bother setting up the collections variables or BBFILES, and
still likely be able to do a build.
Signed-off-by: Chris Larson <clarson@mvista.com>
---
conf/bitbake.conf | 1 +
conf/collections.inc | 55 ++++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 56 insertions(+), 0 deletions(-)
create mode 100644 conf/collections.inc
diff --git a/conf/bitbake.conf b/conf/bitbake.conf
index 7c159af..134525a 100644
--- a/conf/bitbake.conf
+++ b/conf/bitbake.conf
@@ -620,6 +620,7 @@ OVERRIDES = "local:${MACHINE}:${DISTRO}:${TARGET_OS}:${TARGET_ARCH}:build-${BUIL
# Include the rest of the config files.
##################################################################
+require conf/collections.inc
include conf/site.conf
include conf/auto.conf
include conf/local.conf
diff --git a/conf/collections.inc b/conf/collections.inc
new file mode 100644
index 0000000..9355b00
--- /dev/null
+++ b/conf/collections.inc
@@ -0,0 +1,55 @@
+# Take a list of directories in COLLECTIONS, in priority order (highest to
+# lowest), and use those to populate BBFILES, BBFILE_COLLECTIONS,
+# BBFILE_PATTERN_*, and BBFILE_PRIORITY_*.
+
+COLLECTIONS = "${@' '.join(d.getVar('BBPATH', 1).split(':'))}"
+
+def collections_setup(d):
+ """ Populate collection and bbfiles metadata from the COLLECTIONS var. """
+ import bb
+ import os
+ from itertools import izip, chain
+ from glob import glob
+
+ def setifunset(k, v):
+ if d.getVar(k, 0) is None:
+ d.setVar(k, v)
+
+ collections = d.getVar("COLLECTIONS", 1)
+ if not collections:
+ return
+ globbed = (glob(path) for path in collections.split())
+ collections = list(chain(*globbed))
+
+ collectionmap = {}
+ namemap = {}
+ for collection in collections:
+ basename = os.path.basename(collection).split(os.path.extsep)[0]
+ if namemap.get(basename):
+ basename = "%s-%s" % (basename, hash(collection))
+ namemap[basename] = collection
+ collectionmap[collection] = basename
+
+ for (collection, priority) in izip(collections, xrange(len(collections), 0, -1)):
+ if not os.path.exists(collection):
+ bb.fatal("Collection %s does not exist" % collection)
+
+ name = collectionmap[collection]
+ if not name:
+ bb.fatal("Unable to determine collection name for %s" % collection)
+
+ setifunset("BBFILE_PATTERN_%s" % name, "^%s/" % collection)
+ setifunset("BBFILE_PRIORITY_%s" % name, str(priority))
+
+ setifunset("BBFILE_COLLECTIONS", " ".join(collectionmap.values()))
+ setifunset("BBFILES", " ".join(collectionmap.keys()))
+
+addhandler collections_eh
+python collections_eh () {
+ from bb.event import getName
+
+ if getName(e) == "ConfigParsed":
+ collections_setup(e.data)
+
+ return NotHandled
+}
--
1.5.2.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH 2/2] collections.inc: add support for collection archives.
2009-04-02 0:27 [PATCH 1/2] bitbake.conf, collections.inc: Add COLLECTIONS mechanism w/ a sane default Chris Larson
@ 2009-04-02 0:27 ` Chris Larson
2009-04-02 10:07 ` [PATCH 1/2] bitbake.conf, collections.inc: Add COLLECTIONS mechanism w/ a sane default Graeme Gregory
` (2 subsequent siblings)
3 siblings, 0 replies; 9+ messages in thread
From: Chris Larson @ 2009-04-02 0:27 UTC (permalink / raw)
To: openembedded-devel
From: Chris Larson <clarson@kergoth.com>
Adds the ability to specify an archive in COLLECTIONS. Any archives of a
supported format (default tar.gz/tgz/tar.X, tar.bz2/tbz/tbz2, zip/jar) will be
unpacked into COLLECTIONS_UNPACKDIR and used from there. The default value of COLLECTIONS_UNPACKDIR is ${TMPDIR}/collections.
Signed-off-by: Chris Larson <clarson@kergoth.com>
---
conf/collections.inc | 73 ++++++++++++++++++++++++++++++++++++++++++++++++++
1 files changed, 73 insertions(+), 0 deletions(-)
diff --git a/conf/collections.inc b/conf/collections.inc
index 9355b00..e692811 100644
--- a/conf/collections.inc
+++ b/conf/collections.inc
@@ -1,8 +1,72 @@
# Take a list of directories in COLLECTIONS, in priority order (highest to
# lowest), and use those to populate BBFILES, BBFILE_COLLECTIONS,
# BBFILE_PATTERN_*, and BBFILE_PRIORITY_*.
+#
+# Specifying an archive in COLLECTIONS is also supported. Any archives of a
+# supported format will be unpacked into COLLECTIONS_UNPACKDIR and used from
+# there.
COLLECTIONS = "${@' '.join(d.getVar('BBPATH', 1).split(':'))}"
+COLLECTIONS_UNPACKDIR = "${TMPDIR}/collections"
+
+def collection_unpack(collection, name, d):
+ """ Unpack a collection archive and return the path to it. """
+ import bb
+ import os
+ from md5 import md5
+
+ handlers = {
+ ("tar"): "tar x --no-same-owner -f %s",
+ ("tar.gz", "tgz", "tar.Z"): "tar xz --no-same-owner -f %s",
+ ("tar.bz2", "tbz", "tbz2"): "tar xj --no-same-owner -f %s",
+ ("zip", "jar"): "unzip -q -o %s",
+ }
+
+ outpath = os.path.join(d.getVar("COLLECTIONS_UNPACKDIR", 1), name)
+
+ try:
+ collectiondata = open(collection, "r").read()
+ except IOError:
+ bb.fatal("Unable to open %s to calculate md5 sum" % collection)
+
+ md5obj = md5()
+ md5obj.update(collectiondata)
+ md5sum = md5obj.hexdigest()
+
+ md5file = os.path.join(outpath, "md5")
+ if os.path.exists(md5file):
+ try:
+ oldmd5sum = open(md5file).read()
+ except IOError:
+ pass
+ else:
+ if oldmd5sum == md5sum:
+ bb.debug(1, "Using existing %s for collection %s" % (outpath, name))
+ return outpath
+
+ bb.note("Removing old unpacked collection at %s" % outpath)
+ os.system("rm -rf %s" % outpath)
+
+ try:
+ cmd = (cmd for (exts, cmd) in handlers.iteritems()
+ for e in exts
+ if collection.endswith(e)).next()
+ cmd = "cd %s && PATH=\"%s\" %s" % (outpath, d.getVar("PATH", 1), cmd)
+ except StopIteration:
+ bb.fatal("Unable to find unpack handler for %s" % collection)
+
+ if not os.path.isdir(outpath):
+ os.makedirs(outpath)
+
+ bb.note("Unpacking %s to %s/" % (collection, outpath))
+ ret = os.system(cmd % collection)
+ if ret != 0:
+ bb.fatal("Unable to unpack %s" % collection)
+
+ md5out = open(md5file, "w")
+ md5out.write(md5sum)
+ md5out.close()
+ return outpath
def collections_setup(d):
""" Populate collection and bbfiles metadata from the COLLECTIONS var. """
@@ -38,6 +102,15 @@ def collections_setup(d):
if not name:
bb.fatal("Unable to determine collection name for %s" % collection)
+ if not os.path.isdir(collection):
+ del collectionmap[collection]
+ unpacked = collection_unpack(collection, name, d)
+ if unpacked:
+ collection = unpacked
+ collectionmap[collection] = name
+ else:
+ bb.fatal("Unable to unpack collection %s" % collection)
+
setifunset("BBFILE_PATTERN_%s" % name, "^%s/" % collection)
setifunset("BBFILE_PRIORITY_%s" % name, str(priority))
--
1.5.2.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH 1/2] bitbake.conf, collections.inc: Add COLLECTIONS mechanism w/ a sane default.
2009-04-02 0:27 [PATCH 1/2] bitbake.conf, collections.inc: Add COLLECTIONS mechanism w/ a sane default Chris Larson
2009-04-02 0:27 ` [PATCH 2/2] collections.inc: add support for collection archives Chris Larson
@ 2009-04-02 10:07 ` Graeme Gregory
2009-04-02 19:56 ` Denys Dmytriyenko
2009-04-03 6:45 ` Holger Schurig
2009-04-03 8:54 ` Holger Schurig
3 siblings, 1 reply; 9+ messages in thread
From: Graeme Gregory @ 2009-04-02 10:07 UTC (permalink / raw)
To: openembedded-devel
Chris Larson wrote:
> Added code which takes a list of directories in COLLECTIONS, in priority
> order (highest to lowest), and uses those to populate BBFILES,
> BBFILE_COLLECTIONS, BBFILE_PATTERN_*, and BBFILE_PRIORITY_*.
>
> The default COLLECTIONS is based on BBPATH, so you can now specify a sane
> BBPATH, not bother setting up the collections variables or BBFILES, and
> still likely be able to do a build.
>
>
I haven't had time to test the patch, but I really like the idea of
this. Also the archives part as well. This will make setup so much
easier for people wanting to overlay. And improve the ability for OE to
be shipped in setup scripts.
Graeme
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH 1/2] bitbake.conf, collections.inc: Add COLLECTIONS mechanism w/ a sane default.
2009-04-02 10:07 ` [PATCH 1/2] bitbake.conf, collections.inc: Add COLLECTIONS mechanism w/ a sane default Graeme Gregory
@ 2009-04-02 19:56 ` Denys Dmytriyenko
2009-04-02 20:28 ` Otavio Salvador
0 siblings, 1 reply; 9+ messages in thread
From: Denys Dmytriyenko @ 2009-04-02 19:56 UTC (permalink / raw)
To: openembedded-devel
On Thu, Apr 02, 2009 at 11:07:40AM +0100, Graeme Gregory wrote:
> Chris Larson wrote:
> > Added code which takes a list of directories in COLLECTIONS, in priority
> > order (highest to lowest), and uses those to populate BBFILES,
> > BBFILE_COLLECTIONS, BBFILE_PATTERN_*, and BBFILE_PRIORITY_*.
> >
> > The default COLLECTIONS is based on BBPATH, so you can now specify a sane
> > BBPATH, not bother setting up the collections variables or BBFILES, and
> > still likely be able to do a build.
> >
> >
> I haven't had time to test the patch, but I really like the idea of
> this. Also the archives part as well. This will make setup so much
> easier for people wanting to overlay. And improve the ability for OE to
> be shipped in setup scripts.
+1 - support the idea!
I'll try to test the patches on my overlay...
--
Denys
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH 1/2] bitbake.conf, collections.inc: Add COLLECTIONS mechanism w/ a sane default.
2009-04-02 19:56 ` Denys Dmytriyenko
@ 2009-04-02 20:28 ` Otavio Salvador
0 siblings, 0 replies; 9+ messages in thread
From: Otavio Salvador @ 2009-04-02 20:28 UTC (permalink / raw)
To: openembedded-devel
Denys Dmytriyenko <denis@denix.org> writes:
> On Thu, Apr 02, 2009 at 11:07:40AM +0100, Graeme Gregory wrote:
>> Chris Larson wrote:
>> > Added code which takes a list of directories in COLLECTIONS, in priority
>> > order (highest to lowest), and uses those to populate BBFILES,
>> > BBFILE_COLLECTIONS, BBFILE_PATTERN_*, and BBFILE_PRIORITY_*.
>> >
>> > The default COLLECTIONS is based on BBPATH, so you can now specify a sane
>> > BBPATH, not bother setting up the collections variables or BBFILES, and
>> > still likely be able to do a build.
>> >
>> >
>> I haven't had time to test the patch, but I really like the idea of
>> this. Also the archives part as well. This will make setup so much
>> easier for people wanting to overlay. And improve the ability for OE to
>> be shipped in setup scripts.
>
> +1 - support the idea!
>
> I'll try to test the patches on my overlay...
+1
--
Otavio Salvador O.S. Systems
E-mail: otavio@ossystems.com.br http://www.ossystems.com.br
Mobile: +55 53 9981-7854 http://projetos.ossystems.com.br
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH 1/2] bitbake.conf, collections.inc: Add COLLECTIONS mechanism w/ a sane default.
2009-04-02 0:27 [PATCH 1/2] bitbake.conf, collections.inc: Add COLLECTIONS mechanism w/ a sane default Chris Larson
2009-04-02 0:27 ` [PATCH 2/2] collections.inc: add support for collection archives Chris Larson
2009-04-02 10:07 ` [PATCH 1/2] bitbake.conf, collections.inc: Add COLLECTIONS mechanism w/ a sane default Graeme Gregory
@ 2009-04-03 6:45 ` Holger Schurig
2009-04-03 15:19 ` Chris Larson
2009-04-03 8:54 ` Holger Schurig
3 siblings, 1 reply; 9+ messages in thread
From: Holger Schurig @ 2009-04-03 6:45 UTC (permalink / raw)
To: openembedded-devel; +Cc: openembedded-devel
> +# Take a list of directories in COLLECTIONS, in priority order (highest to
> +# lowest), and use those to populate BBFILES, BBFILE_COLLECTIONS,
> +# BBFILE_PATTERN_*, and BBFILE_PRIORITY_*.
> +
I like this idea, but I think that this *.inc file
needs a larger comment block, e.g. not only stating
what it does, but also why that would be handy. A
typical use case or maybe a problem-solution section.
I think when more of the *.inc and *.bbclass files get
such in-file comments it'll be easier for people skimming
throught the code to grasp the big picture.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH 1/2] bitbake.conf, collections.inc: Add COLLECTIONS mechanism w/ a sane default.
2009-04-02 0:27 [PATCH 1/2] bitbake.conf, collections.inc: Add COLLECTIONS mechanism w/ a sane default Chris Larson
` (2 preceding siblings ...)
2009-04-03 6:45 ` Holger Schurig
@ 2009-04-03 8:54 ` Holger Schurig
2009-04-03 15:17 ` Chris Larson
3 siblings, 1 reply; 9+ messages in thread
From: Holger Schurig @ 2009-04-03 8:54 UTC (permalink / raw)
To: openembedded-devel; +Cc: openembedded-devel
On Thursday 02 April 2009 02:27:17 Chris Larson wrote:
> COLLECTIONS = "${@' '.join(d.getVar('BBPATH', 1).split(':'))}"
Why do we need a new variable here? Can't we just use the
equivalent BBPATH.split() ?? Or is there a use-case where it
makes sense to have BBPATH != COLLECTIONS ?
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH 1/2] bitbake.conf, collections.inc: Add COLLECTIONS mechanism w/ a sane default.
2009-04-03 8:54 ` Holger Schurig
@ 2009-04-03 15:17 ` Chris Larson
0 siblings, 0 replies; 9+ messages in thread
From: Chris Larson @ 2009-04-03 15:17 UTC (permalink / raw)
To: openembedded-devel; +Cc: openembedded-devel
On Fri, Apr 3, 2009 at 1:54 AM, Holger Schurig
<hs4233@mail.mn-solutions.de> wrote:
> On Thursday 02 April 2009 02:27:17 Chris Larson wrote:
>> COLLECTIONS = "${@' '.join(d.getVar('BBPATH', 1).split(':'))}"
>
> Why do we need a new variable here? Can't we just use the
> equivalent BBPATH.split() ?? Or is there a use-case where it
> makes sense to have BBPATH != COLLECTIONS ?
Yes, there are a number of possible cases. For one, consider someone
with a ~/oe, under which they have multiple work areas, each with
their own trees.. say, poky, a local company specific metadata repo,
and upstream oe. Also consider that ~/oe is in BBPATH, so that
~/oe/conf/site.conf can be used for those trees. This is exactly the
setup I have on a few of my workstations. If I were to let it use the
BBPATH based setup always, I'd always have *all* the recipes from
*all* those trees being parsed. Not optimal. In addition, the
followup patch to facilitate archive based collections wouldn't be
feasable in that case.
--
Chris Larson
clarson at kergoth dot com
clarson at mvista dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Software Engineer
MontaVista Software, Inc.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH 1/2] bitbake.conf, collections.inc: Add COLLECTIONS mechanism w/ a sane default.
2009-04-03 6:45 ` Holger Schurig
@ 2009-04-03 15:19 ` Chris Larson
0 siblings, 0 replies; 9+ messages in thread
From: Chris Larson @ 2009-04-03 15:19 UTC (permalink / raw)
To: openembedded-devel; +Cc: openembedded-devel
On Thu, Apr 2, 2009 at 11:45 PM, Holger Schurig
<hs4233@mail.mn-solutions.de> wrote:
>> +# Take a list of directories in COLLECTIONS, in priority order (highest to
>> +# lowest), and use those to populate BBFILES, BBFILE_COLLECTIONS,
>> +# BBFILE_PATTERN_*, and BBFILE_PRIORITY_*.
>> +
>
> I like this idea, but I think that this *.inc file
> needs a larger comment block, e.g. not only stating
> what it does, but also why that would be handy. A
> typical use case or maybe a problem-solution section.
>
> I think when more of the *.inc and *.bbclass files get
> such in-file comments it'll be easier for people skimming
> throught the code to grasp the big picture.
Arguably, it could apply to everyone's use cases. It's more
convenient to use than BBFILES even when you aren't relying on
bitbake's collections support to make it prefer the same recipe in one
collection over another, imo.
--
Chris Larson
clarson at kergoth dot com
clarson at mvista dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Software Engineer
MontaVista Software, Inc.
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2009-04-03 15:22 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2009-04-02 0:27 [PATCH 1/2] bitbake.conf, collections.inc: Add COLLECTIONS mechanism w/ a sane default Chris Larson
2009-04-02 0:27 ` [PATCH 2/2] collections.inc: add support for collection archives Chris Larson
2009-04-02 10:07 ` [PATCH 1/2] bitbake.conf, collections.inc: Add COLLECTIONS mechanism w/ a sane default Graeme Gregory
2009-04-02 19:56 ` Denys Dmytriyenko
2009-04-02 20:28 ` Otavio Salvador
2009-04-03 6:45 ` Holger Schurig
2009-04-03 15:19 ` Chris Larson
2009-04-03 8:54 ` Holger Schurig
2009-04-03 15:17 ` Chris Larson
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox