* [PATCH] parse/cache/cooker: Preserve order in the file inclusion list
@ 2012-11-19 15:01 Richard Purdie
2012-11-19 15:09 ` Martin Jansa
0 siblings, 1 reply; 3+ messages in thread
From: Richard Purdie @ 2012-11-19 15:01 UTC (permalink / raw)
To: bitbake-devel
The data returned by get_file_depends() may me used in contexts like
checksums where order is important. The current usage of sets means
that some of the checksums can change in circumstances they should not.
This patch changes to use lists, thereby removing the problem.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
diff --git a/bitbake/lib/bb/cache.py b/bitbake/lib/bb/cache.py
index 619b9ee..a3c073a 100644
--- a/bitbake/lib/bb/cache.py
+++ b/bitbake/lib/bb/cache.py
@@ -405,12 +405,12 @@ class Cache(object):
"""Parse the specified filename, returning the recipe information"""
infos = []
datastores = cls.load_bbfile(filename, appends, configdata)
- depends = set()
+ depends = []
for variant, data in sorted(datastores.iteritems(),
key=lambda i: i[0],
reverse=True):
virtualfn = cls.realfn2virtual(filename, variant)
- depends |= (data.getVar("__depends", False) or set())
+ depends = depends + (data.getVar("__depends", False) or [])
if depends and not variant:
data.setVar("__depends", depends)
diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index e299059..98d6957 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -692,8 +692,8 @@ class BBCooker:
# Generate a list of parsed configuration files by searching the files
# listed in the __depends and __base_depends variables with a .conf suffix.
conffiles = []
- dep_files = self.configuration.data.getVar('__depends') or set()
- dep_files.union(self.configuration.data.getVar('__base_depends') or set())
+ dep_files = self.configuration.data.getVar('__base_depends') or []
+ dep_files = dep_files + (self.configuration.data.getVar('__depends') or [])
for f in dep_files:
if f[0].endswith(".conf"):
diff --git a/bitbake/lib/bb/parse/__init__.py b/bitbake/lib/bb/parse/__init__.py
index 7b9c47e..4293d09c 100644
--- a/bitbake/lib/bb/parse/__init__.py
+++ b/bitbake/lib/bb/parse/__init__.py
@@ -73,8 +73,7 @@ def update_mtime(f):
def mark_dependency(d, f):
if f.startswith('./'):
f = "%s/%s" % (os.getcwd(), f[2:])
- deps = d.getVar('__depends') or set()
- deps.update([(f, cached_mtime(f))])
+ deps = (d.getVar('__depends') or []) + [(f, cached_mtime(f))]
d.setVar('__depends', deps)
def supports(fn, data):
@@ -134,8 +133,8 @@ def vars_from_file(mypkg, d):
def get_file_depends(d):
'''Return the dependent files'''
dep_files = []
- depends = d.getVar('__depends', True) or set()
- depends = depends.union(d.getVar('__base_depends', True) or set())
+ depends = d.getVar('__base_depends', True) or []
+ depends = depends + (d.getVar('__depends', True) or [])
for (fn, _) in depends:
dep_files.append(os.path.abspath(fn))
return " ".join(dep_files)
^ permalink raw reply related [flat|nested] 3+ messages in thread
* Re: [PATCH] parse/cache/cooker: Preserve order in the file inclusion list
2012-11-19 15:01 [PATCH] parse/cache/cooker: Preserve order in the file inclusion list Richard Purdie
@ 2012-11-19 15:09 ` Martin Jansa
2012-11-19 15:20 ` Richard Purdie
0 siblings, 1 reply; 3+ messages in thread
From: Martin Jansa @ 2012-11-19 15:09 UTC (permalink / raw)
To: Richard Purdie; +Cc: bitbake-devel
[-- Attachment #1: Type: text/plain, Size: 4528 bytes --]
On Mon, Nov 19, 2012 at 03:01:20PM +0000, Richard Purdie wrote:
> The data returned by get_file_depends() may me used in contexts like
> checksums where order is important. The current usage of sets means
> that some of the checksums can change in circumstances they should not.
>
> This patch changes to use lists, thereby removing the problem.
Should this cover also BBINCLUDED variable in parsing cache?
Sometimes I see all recipes reparsed without any reason (no change in
metadata).
Comparing output of bitbake -p (with added patch to dump parse data,
shows order of BBINCLUDED changing.
e.g. from yesterday:
OE om-gta02@shr ~/shr-core $ diff 20121118.BBINCLUDED 20121119.BBINCLUDED
12c12
< /OE/shr-core/openembedded-core/meta/conf/machine/include/tune-arm920t.inc
---
> /OE/shr-core/openembedded-core/meta/classes/mirrors.bbclass
45a46
> /OE/shr-core/meta-smartphone/meta-shr/conf/distro/shr.conf
51c52
< /OE/shr-core/openembedded-core/meta/classes/mirrors.bbclass
---
> /OE/shr-core/openembedded-core/meta/conf/machine/include/tune-arm920t.inc
54d54
< /OE/shr-core/meta-smartphone/meta-shr/conf/distro/shr.conf
60d59
< /OE/shr-core/openembedded-core/meta/conf/distro/include/tcmode-default.inc
62a62
> /OE/shr-core/openembedded-core/meta/conf/distro/include/tcmode-default.inc
Cheers,
>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
> diff --git a/bitbake/lib/bb/cache.py b/bitbake/lib/bb/cache.py
> index 619b9ee..a3c073a 100644
> --- a/bitbake/lib/bb/cache.py
> +++ b/bitbake/lib/bb/cache.py
> @@ -405,12 +405,12 @@ class Cache(object):
> """Parse the specified filename, returning the recipe information"""
> infos = []
> datastores = cls.load_bbfile(filename, appends, configdata)
> - depends = set()
> + depends = []
> for variant, data in sorted(datastores.iteritems(),
> key=lambda i: i[0],
> reverse=True):
> virtualfn = cls.realfn2virtual(filename, variant)
> - depends |= (data.getVar("__depends", False) or set())
> + depends = depends + (data.getVar("__depends", False) or [])
> if depends and not variant:
> data.setVar("__depends", depends)
>
> diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
> index e299059..98d6957 100644
> --- a/bitbake/lib/bb/cooker.py
> +++ b/bitbake/lib/bb/cooker.py
> @@ -692,8 +692,8 @@ class BBCooker:
> # Generate a list of parsed configuration files by searching the files
> # listed in the __depends and __base_depends variables with a .conf suffix.
> conffiles = []
> - dep_files = self.configuration.data.getVar('__depends') or set()
> - dep_files.union(self.configuration.data.getVar('__base_depends') or set())
> + dep_files = self.configuration.data.getVar('__base_depends') or []
> + dep_files = dep_files + (self.configuration.data.getVar('__depends') or [])
>
> for f in dep_files:
> if f[0].endswith(".conf"):
> diff --git a/bitbake/lib/bb/parse/__init__.py b/bitbake/lib/bb/parse/__init__.py
> index 7b9c47e..4293d09c 100644
> --- a/bitbake/lib/bb/parse/__init__.py
> +++ b/bitbake/lib/bb/parse/__init__.py
> @@ -73,8 +73,7 @@ def update_mtime(f):
> def mark_dependency(d, f):
> if f.startswith('./'):
> f = "%s/%s" % (os.getcwd(), f[2:])
> - deps = d.getVar('__depends') or set()
> - deps.update([(f, cached_mtime(f))])
> + deps = (d.getVar('__depends') or []) + [(f, cached_mtime(f))]
> d.setVar('__depends', deps)
>
> def supports(fn, data):
> @@ -134,8 +133,8 @@ def vars_from_file(mypkg, d):
> def get_file_depends(d):
> '''Return the dependent files'''
> dep_files = []
> - depends = d.getVar('__depends', True) or set()
> - depends = depends.union(d.getVar('__base_depends', True) or set())
> + depends = d.getVar('__base_depends', True) or []
> + depends = depends + (d.getVar('__depends', True) or [])
> for (fn, _) in depends:
> dep_files.append(os.path.abspath(fn))
> return " ".join(dep_files)
>
>
>
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/bitbake-devel
--
Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [PATCH] parse/cache/cooker: Preserve order in the file inclusion list
2012-11-19 15:09 ` Martin Jansa
@ 2012-11-19 15:20 ` Richard Purdie
0 siblings, 0 replies; 3+ messages in thread
From: Richard Purdie @ 2012-11-19 15:20 UTC (permalink / raw)
To: Martin Jansa; +Cc: bitbake-devel
On Mon, 2012-11-19 at 16:09 +0100, Martin Jansa wrote:
> On Mon, Nov 19, 2012 at 03:01:20PM +0000, Richard Purdie wrote:
> > The data returned by get_file_depends() may me used in contexts like
> > checksums where order is important. The current usage of sets means
> > that some of the checksums can change in circumstances they should not.
> >
> > This patch changes to use lists, thereby removing the problem.
>
> Should this cover also BBINCLUDED variable in parsing cache?
>
> Sometimes I see all recipes reparsed without any reason (no change in
> metadata).
>
> Comparing output of bitbake -p (with added patch to dump parse data,
> shows order of BBINCLUDED changing.
Yes, this is exactly why I'm proposing this patch.
Cheers,
Richard
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2012-11-19 15:34 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-11-19 15:01 [PATCH] parse/cache/cooker: Preserve order in the file inclusion list Richard Purdie
2012-11-19 15:09 ` Martin Jansa
2012-11-19 15:20 ` Richard Purdie
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.