* [PATCH 1/9] classes/buildhistory: remove obsolete flat package history feature
2012-08-02 9:23 [PATCH 0/9] buildhistory improvements Paul Eggleton
@ 2012-08-02 9:23 ` Paul Eggleton
2012-08-02 9:23 ` [PATCH 2/9] classes/buildhistory: ensure old package info is removed Paul Eggleton
` (9 subsequent siblings)
10 siblings, 0 replies; 17+ messages in thread
From: Paul Eggleton @ 2012-08-02 9:23 UTC (permalink / raw)
To: openembedded-core
Remove the ability to set BUILDHISTORY_KEEP_VERSIONS = "1" to save the
package history as flat versioned files rather than relying on git to
keep previous versions of the package information. git has proven to
work quite well in this capacity.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/classes/buildhistory.bbclass | 43 +++++++------------------------------
1 file changed, 8 insertions(+), 35 deletions(-)
diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index ddb76e8..f5494cd 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -156,7 +156,6 @@ python buildhistory_emit_pkghistory() {
rcpinfo.depends = sortlist(squashspaces(d.getVar('DEPENDS', True) or ""))
rcpinfo.packages = packages
write_recipehistory(rcpinfo, d)
- write_latestlink(None, pe, pv, pr, d)
# Apparently the version can be different on a per-package basis (see Python)
pkgdest = d.getVar('PKGDEST', True)
@@ -199,8 +198,6 @@ python buildhistory_emit_pkghistory() {
pkginfo.filelist = " ".join(filelist)
write_pkghistory(pkginfo, d)
-
- write_latestlink(pkg, pe, pv, pr, d)
}
@@ -212,8 +209,8 @@ def write_recipehistory(rcpinfo, d):
if not os.path.exists(pkghistdir):
os.makedirs(pkghistdir)
- verfile = os.path.join(pkghistdir, "%s:%s-%s" % (rcpinfo.pe, rcpinfo.pv, rcpinfo.pr))
- f = open(verfile, "w")
+ infofile = os.path.join(pkghistdir, "latest")
+ f = open(infofile, "w")
try:
if rcpinfo.pe != "0":
f.write("PE = %s\n" % rcpinfo.pe)
@@ -226,16 +223,16 @@ def write_recipehistory(rcpinfo, d):
def write_pkghistory(pkginfo, d):
- bb.debug(2, "Writing package history")
+ bb.debug(2, "Writing package history for package %s" % pkginfo.name)
pkghistdir = d.getVar('BUILDHISTORY_DIR_PACKAGE', True)
- verpath = os.path.join(pkghistdir, pkginfo.name)
- if not os.path.exists(verpath):
- os.makedirs(verpath)
+ pkgpath = os.path.join(pkghistdir, pkginfo.name)
+ if not os.path.exists(pkgpath):
+ os.makedirs(pkgpath)
- verfile = os.path.join(verpath, "%s:%s-%s" % (pkginfo.pe, pkginfo.pv, pkginfo.pr))
- f = open(verfile, "w")
+ infofile = os.path.join(pkgpath, "latest")
+ f = open(infofile, "w")
try:
if pkginfo.pe != "0":
f.write("PE = %s\n" % pkginfo.pe)
@@ -250,30 +247,6 @@ def write_pkghistory(pkginfo, d):
f.close()
-def write_latestlink(pkg, pe, pv, pr, d):
- import shutil
-
- pkghistdir = d.getVar('BUILDHISTORY_DIR_PACKAGE', True)
-
- def rm_link(path):
- try:
- os.unlink(path)
- except OSError:
- return
-
- if pkg:
- filedir = os.path.join(pkghistdir, pkg)
- else:
- filedir = pkghistdir
- latest_file = os.path.join(filedir, "latest")
- ver_file = os.path.join(filedir, "%s:%s-%s" % (pe, pv, pr))
- rm_link(latest_file)
- if d.getVar('BUILDHISTORY_KEEP_VERSIONS', True) == '1':
- shutil.copy(ver_file, latest_file)
- else:
- shutil.move(ver_file, latest_file)
-
-
buildhistory_get_image_installed() {
# Anything requiring the use of the packaging system should be done in here
# in case the packaging files are going to be removed for this image
--
1.7.9.5
^ permalink raw reply related [flat|nested] 17+ messages in thread* [PATCH 2/9] classes/buildhistory: ensure old package info is removed
2012-08-02 9:23 [PATCH 0/9] buildhistory improvements Paul Eggleton
2012-08-02 9:23 ` [PATCH 1/9] classes/buildhistory: remove obsolete flat package history feature Paul Eggleton
@ 2012-08-02 9:23 ` Paul Eggleton
2012-08-02 9:23 ` [PATCH 3/9] classes/buildhistory: remove unused functions Paul Eggleton
` (8 subsequent siblings)
10 siblings, 0 replies; 17+ messages in thread
From: Paul Eggleton @ 2012-08-02 9:23 UTC (permalink / raw)
To: openembedded-core
If a package is removed from PACKAGES, ensure that the package info file
and directory are removed from buildhistory so that we don't have stale
data lying around.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/classes/buildhistory.bbclass | 18 ++++++++++++++----
1 file changed, 14 insertions(+), 4 deletions(-)
diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index f5494cd..c3555da 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -149,6 +149,19 @@ python buildhistory_emit_pkghistory() {
pr = d.getVar('PR', True)
packages = squashspaces(d.getVar('PACKAGES', True))
+ packagelist = packages.split()
+ if not os.path.exists(pkghistdir):
+ os.makedirs(pkghistdir)
+ else:
+ # Remove files for packages that no longer exist
+ for item in os.listdir(pkghistdir):
+ if item != "latest":
+ if item not in packagelist:
+ subdir = os.path.join(pkghistdir, item)
+ for subfile in os.listdir(subdir):
+ os.unlink(os.path.join(subdir, subfile))
+ os.rmdir(subdir)
+
rcpinfo = RecipeInfo(pn)
rcpinfo.pe = pe
rcpinfo.pv = pv
@@ -159,7 +172,7 @@ python buildhistory_emit_pkghistory() {
# Apparently the version can be different on a per-package basis (see Python)
pkgdest = d.getVar('PKGDEST', True)
- for pkg in packages.split():
+ for pkg in packagelist:
pe = getpkgvar(pkg, 'PE') or "0"
pv = getpkgvar(pkg, 'PV')
pr = getpkgvar(pkg, 'PR')
@@ -206,9 +219,6 @@ def write_recipehistory(rcpinfo, d):
pkghistdir = d.getVar('BUILDHISTORY_DIR_PACKAGE', True)
- if not os.path.exists(pkghistdir):
- os.makedirs(pkghistdir)
-
infofile = os.path.join(pkghistdir, "latest")
f = open(infofile, "w")
try:
--
1.7.9.5
^ permalink raw reply related [flat|nested] 17+ messages in thread* [PATCH 3/9] classes/buildhistory: remove unused functions
2012-08-02 9:23 [PATCH 0/9] buildhistory improvements Paul Eggleton
2012-08-02 9:23 ` [PATCH 1/9] classes/buildhistory: remove obsolete flat package history feature Paul Eggleton
2012-08-02 9:23 ` [PATCH 2/9] classes/buildhistory: ensure old package info is removed Paul Eggleton
@ 2012-08-02 9:23 ` Paul Eggleton
2012-08-02 9:23 ` [PATCH 4/9] classes/buildhistory: save preinst/postinst/prerm/postrm Paul Eggleton
` (7 subsequent siblings)
10 siblings, 0 replies; 17+ messages in thread
From: Paul Eggleton @ 2012-08-02 9:23 UTC (permalink / raw)
To: openembedded-core
getlastrecipeversion and readRecipeInfo weren't called anywhere.
Remove them so we don't have to keep them up-to-date with future
additions to the recipe info file.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/classes/buildhistory.bbclass | 29 -----------------------------
1 file changed, 29 deletions(-)
diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index c3555da..510a6df 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -68,28 +68,6 @@ python buildhistory_emit_pkghistory() {
return val
- def readRecipeInfo(pn, histfile):
- rcpinfo = RecipeInfo(pn)
- f = open(histfile, "r")
- try:
- for line in f:
- lns = line.split('=')
- name = lns[0].strip()
- value = lns[1].strip(" \t\r\n").strip('"')
- if name == "PE":
- rcpinfo.pe = value
- elif name == "PV":
- rcpinfo.pv = value
- elif name == "PR":
- rcpinfo.pr = value
- elif name == "DEPENDS":
- rcpinfo.depends = value
- elif name == "PACKAGES":
- rcpinfo.packages = value
- finally:
- f.close()
- return rcpinfo
-
def readPackageInfo(pkg, histfile):
pkginfo = PackageInfo(pkg)
f = open(histfile, "r")
@@ -118,13 +96,6 @@ python buildhistory_emit_pkghistory() {
f.close()
return pkginfo
- def getlastrecipeversion(pn):
- try:
- histfile = os.path.join(pkghistdir, "latest")
- return readRecipeInfo(pn, histfile)
- except EnvironmentError:
- return None
-
def getlastpkgversion(pkg):
try:
histfile = os.path.join(pkghistdir, pkg, "latest")
--
1.7.9.5
^ permalink raw reply related [flat|nested] 17+ messages in thread* [PATCH 4/9] classes/buildhistory: save preinst/postinst/prerm/postrm
2012-08-02 9:23 [PATCH 0/9] buildhistory improvements Paul Eggleton
` (2 preceding siblings ...)
2012-08-02 9:23 ` [PATCH 3/9] classes/buildhistory: remove unused functions Paul Eggleton
@ 2012-08-02 9:23 ` Paul Eggleton
2012-08-02 9:23 ` [PATCH 5/9] classes/buildhistory: record PKG/PKGE/PKGV/PKGR Paul Eggleton
` (6 subsequent siblings)
10 siblings, 0 replies; 17+ messages in thread
From: Paul Eggleton @ 2012-08-02 9:23 UTC (permalink / raw)
To: openembedded-core
Write the value of these package script variables into the packageinfo
so that any changes to them can be tracked (in separate files since they
are multi-line).
Inspired by an earlier patch from Andreas Müller <schnitzeltony@googlemail.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/classes/buildhistory.bbclass | 24 ++++++++++------
meta/lib/oe/buildhistory_analysis.py | 50 +++++++++++++++++++++++++++++++++-
2 files changed, 65 insertions(+), 9 deletions(-)
diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index 510a6df..200d031 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -57,6 +57,8 @@ python buildhistory_emit_pkghistory() {
self.rrecommends = ""
self.files = ""
self.filelist = ""
+ # Variables that need to be written to their own separate file
+ self.filevars = dict.fromkeys(['pkg_preinst', 'pkg_postinst', 'pkg_prerm', 'pkg_postrm'])
# Should check PACKAGES here to see if anything removed
@@ -167,6 +169,8 @@ python buildhistory_emit_pkghistory() {
pkginfo.rdepends = sortpkglist(squashspaces(getpkgvar(pkg, 'RDEPENDS') or ""))
pkginfo.rrecommends = sortpkglist(squashspaces(getpkgvar(pkg, 'RRECOMMENDS') or ""))
pkginfo.files = squashspaces(getpkgvar(pkg, 'FILES') or "")
+ for filevar in pkginfo.filevars:
+ pkginfo.filevars[filevar] = getpkgvar(pkg, filevar)
# Gather information about packaged files
pkgdestpkg = os.path.join(pkgdest, pkg)
@@ -191,16 +195,13 @@ def write_recipehistory(rcpinfo, d):
pkghistdir = d.getVar('BUILDHISTORY_DIR_PACKAGE', True)
infofile = os.path.join(pkghistdir, "latest")
- f = open(infofile, "w")
- try:
+ with open(infofile, "w") as f:
if rcpinfo.pe != "0":
f.write("PE = %s\n" % rcpinfo.pe)
f.write("PV = %s\n" % rcpinfo.pv)
f.write("PR = %s\n" % rcpinfo.pr)
f.write("DEPENDS = %s\n" % rcpinfo.depends)
f.write("PACKAGES = %s\n" % rcpinfo.packages)
- finally:
- f.close()
def write_pkghistory(pkginfo, d):
@@ -213,8 +214,7 @@ def write_pkghistory(pkginfo, d):
os.makedirs(pkgpath)
infofile = os.path.join(pkgpath, "latest")
- f = open(infofile, "w")
- try:
+ with open(infofile, "w") as f:
if pkginfo.pe != "0":
f.write("PE = %s\n" % pkginfo.pe)
f.write("PV = %s\n" % pkginfo.pv)
@@ -224,8 +224,16 @@ def write_pkghistory(pkginfo, d):
f.write("PKGSIZE = %d\n" % pkginfo.size)
f.write("FILES = %s\n" % pkginfo.files)
f.write("FILELIST = %s\n" % pkginfo.filelist)
- finally:
- f.close()
+
+ for filevar in pkginfo.filevars:
+ filevarpath = os.path.join(pkgpath, "latest.%s" % filevar)
+ val = pkginfo.filevars[filevar]
+ if val:
+ with open(filevarpath, "w") as f:
+ f.write(val)
+ else:
+ if os.path.exists(filevarpath):
+ os.unlink(filevarpath)
buildhistory_get_image_installed() {
diff --git a/meta/lib/oe/buildhistory_analysis.py b/meta/lib/oe/buildhistory_analysis.py
index 29dc4a9..6c6a085 100644
--- a/meta/lib/oe/buildhistory_analysis.py
+++ b/meta/lib/oe/buildhistory_analysis.py
@@ -90,6 +90,18 @@ class ChangeRecord:
else:
percentchg = 100
out = '%s changed from %s to %s (%s%d%%)' % (self.fieldname, self.oldvalue or "''", self.newvalue or "''", '+' if percentchg > 0 else '', percentchg)
+ elif self.fieldname in ['pkg_preinst', 'pkg_postinst', 'pkg_prerm', 'pkg_postrm']:
+ if self.oldvalue and self.newvalue:
+ out = '%s changed:\n ' % self.fieldname
+ elif self.newvalue:
+ out = '%s added:\n ' % self.fieldname
+ elif self.oldvalue:
+ out = '%s cleared:\n ' % self.fieldname
+ alines = self.oldvalue.splitlines()
+ blines = self.newvalue.splitlines()
+ diff = difflib.unified_diff(alines, blines, self.fieldname, self.fieldname, lineterm='')
+ out += '\n '.join(list(diff)[2:])
+ out += '\n --'
elif self.fieldname in img_monitor_files:
if outer:
prefix = 'Changes to %s ' % self.path
@@ -330,7 +342,12 @@ def process_changes(repopath, revision1, revision2 = 'HEAD', report_all = False)
for d in diff.iter_change_type('M'):
path = os.path.dirname(d.a_blob.path)
if path.startswith('packages/'):
- changes.extend(compare_dict_blobs(path, d.a_blob, d.b_blob, report_all))
+ filename = os.path.basename(d.a_blob.path)
+ if filename == 'latest':
+ changes.extend(compare_dict_blobs(path, d.a_blob, d.b_blob, report_all))
+ elif filename.startswith('latest.'):
+ chg = ChangeRecord(path, filename, d.a_blob.data_stream.read(), d.b_blob.data_stream.read(), True)
+ changes.append(chg)
elif path.startswith('images/'):
filename = os.path.basename(d.a_blob.path)
if filename in img_monitor_files:
@@ -356,6 +373,37 @@ def process_changes(repopath, revision1, revision2 = 'HEAD', report_all = False)
elif filename == 'image-info.txt':
changes.extend(compare_dict_blobs(path, d.a_blob, d.b_blob, report_all))
+ # Look for added preinst/postinst/prerm/postrm
+ # (without reporting newly added recipes)
+ addedpkgs = []
+ addedchanges = []
+ for d in diff.iter_change_type('A'):
+ path = os.path.dirname(d.b_blob.path)
+ if path.startswith('packages/'):
+ filename = os.path.basename(d.b_blob.path)
+ if filename == 'latest':
+ addedpkgs.append(path)
+ elif filename.startswith('latest.'):
+ chg = ChangeRecord(path, filename[7:], '', d.b_blob.data_stream.read(), True)
+ addedchanges.append(chg)
+ for chg in addedchanges:
+ found = False
+ for pkg in addedpkgs:
+ if chg.path.startswith(pkg):
+ found = True
+ break
+ if not found:
+ changes.append(chg)
+
+ # Look for cleared preinst/postinst/prerm/postrm
+ for d in diff.iter_change_type('D'):
+ path = os.path.dirname(d.a_blob.path)
+ if path.startswith('packages/'):
+ filename = os.path.basename(d.a_blob.path)
+ if filename != 'latest' and filename.startswith('latest.'):
+ chg = ChangeRecord(path, filename[7:], d.a_blob.data_stream.read(), '', True)
+ changes.append(chg)
+
# Link related changes
for chg in changes:
if chg.monitored:
--
1.7.9.5
^ permalink raw reply related [flat|nested] 17+ messages in thread* [PATCH 5/9] classes/buildhistory: record PKG/PKGE/PKGV/PKGR
2012-08-02 9:23 [PATCH 0/9] buildhistory improvements Paul Eggleton
` (3 preceding siblings ...)
2012-08-02 9:23 ` [PATCH 4/9] classes/buildhistory: save preinst/postinst/prerm/postrm Paul Eggleton
@ 2012-08-02 9:23 ` Paul Eggleton
2012-08-02 9:23 ` [PATCH 6/9] scripts: add buildhistory-tag script Paul Eggleton
` (5 subsequent siblings)
10 siblings, 0 replies; 17+ messages in thread
From: Paul Eggleton @ 2012-08-02 9:23 UTC (permalink / raw)
To: openembedded-core
Save PKG (the actual output package name, which is often different due
to debian renaming), and PKGE/PKGV/PKGR (which may be manipulated in
certain special cases e.g. gitpkgv.bbclass in meta-oe, the
external-sourcery-toolchain recipe, etc.) Note that these are only
written when they are different from the normal package name in the
case of PKG, or PE/PV/PR for the other variables.
Also, use PKGE/PKGV/PKGR instead of PE/PV/PR when comparing package
versions since these actually represent the version that the package
manager sees.
Implements [YOCTO #2787].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/classes/buildhistory.bbclass | 66 +++++++++++++++++++++++++---------
meta/lib/oe/buildhistory_analysis.py | 22 +++++++++++-
2 files changed, 71 insertions(+), 17 deletions(-)
diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index 200d031..76648c9 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -51,6 +51,11 @@ python buildhistory_emit_pkghistory() {
self.pe = "0"
self.pv = "0"
self.pr = "r0"
+ # pkg/pkge/pkgv/pkgr should be empty because we want to be able to default them
+ self.pkg = ""
+ self.pkge = ""
+ self.pkgv = ""
+ self.pkgr = ""
self.size = 0
self.depends = ""
self.rdepends = ""
@@ -72,8 +77,7 @@ python buildhistory_emit_pkghistory() {
def readPackageInfo(pkg, histfile):
pkginfo = PackageInfo(pkg)
- f = open(histfile, "r")
- try:
+ with open(histfile, "r") as f:
for line in f:
lns = line.split('=')
name = lns[0].strip()
@@ -84,6 +88,14 @@ python buildhistory_emit_pkghistory() {
pkginfo.pv = value
elif name == "PR":
pkginfo.pr = value
+ elif name == "PKG":
+ pkginfo.pkg = value
+ elif name == "PKGE":
+ pkginfo.pkge = value
+ elif name == "PKGV":
+ pkginfo.pkgv = value
+ elif name == "PKGR":
+ pkginfo.pkgr = value
elif name == "RDEPENDS":
pkginfo.rdepends = value
elif name == "RRECOMMENDS":
@@ -94,8 +106,15 @@ python buildhistory_emit_pkghistory() {
pkginfo.files = value
elif name == "FILELIST":
pkginfo.filelist = value
- finally:
- f.close()
+ # Apply defaults
+ if not pkginfo.pkg:
+ pkginfo.pkg = pkginfo.name
+ if not pkginfo.pkge:
+ pkginfo.pkge = pkginfo.pe
+ if not pkginfo.pkgv:
+ pkginfo.pkgv = pkginfo.pv
+ if not pkginfo.pkgr:
+ pkginfo.pkgr = pkginfo.pr
return pkginfo
def getlastpkgversion(pkg):
@@ -143,29 +162,33 @@ python buildhistory_emit_pkghistory() {
rcpinfo.packages = packages
write_recipehistory(rcpinfo, d)
- # Apparently the version can be different on a per-package basis (see Python)
pkgdest = d.getVar('PKGDEST', True)
for pkg in packagelist:
- pe = getpkgvar(pkg, 'PE') or "0"
- pv = getpkgvar(pkg, 'PV')
- pr = getpkgvar(pkg, 'PR')
+ pkge = getpkgvar(pkg, 'PKGE') or "0"
+ pkgv = getpkgvar(pkg, 'PKGV')
+ pkgr = getpkgvar(pkg, 'PKGR')
#
# Find out what the last version was
# Make sure the version did not decrease
#
lastversion = getlastpkgversion(pkg)
if lastversion:
- last_pe = lastversion.pe
- last_pv = lastversion.pv
- last_pr = lastversion.pr
- r = bb.utils.vercmp((pe, pv, pr), (last_pe, last_pv, last_pr))
+ last_pkge = lastversion.pkge
+ last_pkgv = lastversion.pkgv
+ last_pkgr = lastversion.pkgr
+ r = bb.utils.vercmp((pkge, pkgv, pkgr), (last_pkge, last_pkgv, last_pkgr))
if r < 0:
- bb.error("Package version for package %s went backwards which would break package feeds from (%s:%s-%s to %s:%s-%s)" % (pkg, last_pe, last_pv, last_pr, pe, pv, pr))
+ bb.error("Package version for package %s went backwards which would break package feeds from (%s:%s-%s to %s:%s-%s)" % (pkg, last_pkge, last_pkgv, last_pkgr, pkge, pkgv, pkgr))
pkginfo = PackageInfo(pkg)
- pkginfo.pe = pe
- pkginfo.pv = pv
- pkginfo.pr = pr
+ # Apparently the version can be different on a per-package basis (see Python)
+ pkginfo.pe = getpkgvar(pkg, 'PE') or "0"
+ pkginfo.pv = getpkgvar(pkg, 'PV')
+ pkginfo.pr = getpkgvar(pkg, 'PR')
+ pkginfo.pkg = getpkgvar(pkg, 'PKG') or pkg
+ pkginfo.pkge = pkge
+ pkginfo.pkgv = pkgv
+ pkginfo.pkgr = pkgr
pkginfo.rdepends = sortpkglist(squashspaces(getpkgvar(pkg, 'RDEPENDS') or ""))
pkginfo.rrecommends = sortpkglist(squashspaces(getpkgvar(pkg, 'RRECOMMENDS') or ""))
pkginfo.files = squashspaces(getpkgvar(pkg, 'FILES') or "")
@@ -219,6 +242,17 @@ def write_pkghistory(pkginfo, d):
f.write("PE = %s\n" % pkginfo.pe)
f.write("PV = %s\n" % pkginfo.pv)
f.write("PR = %s\n" % pkginfo.pr)
+
+ pkgvars = {}
+ pkgvars['PKG'] = pkginfo.pkg if pkginfo.pkg != pkginfo.name else ''
+ pkgvars['PKGE'] = pkginfo.pkge if pkginfo.pkge != pkginfo.pe else ''
+ pkgvars['PKGV'] = pkginfo.pkgv if pkginfo.pkgv != pkginfo.pv else ''
+ pkgvars['PKGR'] = pkginfo.pkgr if pkginfo.pkgr != pkginfo.pr else ''
+ for pkgvar in pkgvars:
+ val = pkgvars[pkgvar]
+ if val:
+ f.write("%s = %s\n" % (pkgvar, val))
+
f.write("RDEPENDS = %s\n" % pkginfo.rdepends)
f.write("RRECOMMENDS = %s\n" % pkginfo.rrecommends)
f.write("PKGSIZE = %d\n" % pkginfo.size)
diff --git a/meta/lib/oe/buildhistory_analysis.py b/meta/lib/oe/buildhistory_analysis.py
index 6c6a085..55bd7b7 100644
--- a/meta/lib/oe/buildhistory_analysis.py
+++ b/meta/lib/oe/buildhistory_analysis.py
@@ -19,9 +19,10 @@ import bb.utils
# How to display fields
list_fields = ['DEPENDS', 'RDEPENDS', 'RRECOMMENDS', 'FILES', 'FILELIST', 'USER_CLASSES', 'IMAGE_CLASSES', 'IMAGE_FEATURES', 'IMAGE_LINGUAS', 'IMAGE_INSTALL', 'BAD_RECOMMENDATIONS']
list_order_fields = ['PACKAGES']
+defaultval_fields = ['PKG', 'PKGE', 'PKGV', 'PKGR']
numeric_fields = ['PKGSIZE', 'IMAGESIZE']
# Fields to monitor
-monitor_fields = ['RDEPENDS', 'RRECOMMENDS', 'PACKAGES', 'FILELIST', 'PKGSIZE', 'IMAGESIZE']
+monitor_fields = ['RDEPENDS', 'RRECOMMENDS', 'PACKAGES', 'FILELIST', 'PKGSIZE', 'IMAGESIZE', 'PKG', 'PKGE', 'PKGV', 'PKGR']
# Percentage change to alert for numeric fields
monitor_numeric_threshold = 20
# Image files to monitor (note that image-info.txt is handled separately)
@@ -90,6 +91,10 @@ class ChangeRecord:
else:
percentchg = 100
out = '%s changed from %s to %s (%s%d%%)' % (self.fieldname, self.oldvalue or "''", self.newvalue or "''", '+' if percentchg > 0 else '', percentchg)
+ elif self.fieldname in defaultval_fields:
+ out = '%s changed from %s to %s' % (self.fieldname, self.oldvalue, self.newvalue)
+ if self.fieldname == 'PKG' and '[default]' in self.newvalue:
+ out += ' - may indicate debian renaming failure'
elif self.fieldname in ['pkg_preinst', 'pkg_postinst', 'pkg_prerm', 'pkg_postrm']:
if self.oldvalue and self.newvalue:
out = '%s changed:\n ' % self.fieldname
@@ -299,6 +304,14 @@ def compare_dict_blobs(path, ablob, bblob, report_all):
adict = blob_to_dict(ablob)
bdict = blob_to_dict(bblob)
+ defaultvals = {}
+ defaultvals['PKG'] = os.path.basename(path)
+ defaultvals['PKGE'] = adict.get('PE', '0')
+ defaultvals['PKGV'] = adict.get('PV', '')
+ defaultvals['PKGR'] = adict.get('PR', '')
+ for key in defaultvals:
+ defaultvals[key] = '%s [default]' % defaultvals[key]
+
changes = []
keys = list(set(adict.keys()) | set(bdict.keys()))
for key in keys:
@@ -327,6 +340,13 @@ def compare_dict_blobs(path, ablob, bblob, report_all):
blist.sort()
if ' '.join(alist) == ' '.join(blist):
continue
+
+ if key in defaultval_fields:
+ if not astr:
+ astr = defaultvals[key]
+ elif not bstr:
+ bstr = defaultvals[key]
+
chg = ChangeRecord(path, key, astr, bstr, key in monitor_fields)
changes.append(chg)
return changes
--
1.7.9.5
^ permalink raw reply related [flat|nested] 17+ messages in thread* [PATCH 6/9] scripts: add buildhistory-tag script
2012-08-02 9:23 [PATCH 0/9] buildhistory improvements Paul Eggleton
` (4 preceding siblings ...)
2012-08-02 9:23 ` [PATCH 5/9] classes/buildhistory: record PKG/PKGE/PKGV/PKGR Paul Eggleton
@ 2012-08-02 9:23 ` Paul Eggleton
2012-08-02 10:18 ` Burton, Ross
2012-08-02 9:23 ` [PATCH 7/9] buildhistory_analysis: tidy up duplicate split_version function Paul Eggleton
` (4 subsequent siblings)
10 siblings, 1 reply; 17+ messages in thread
From: Paul Eggleton @ 2012-08-02 9:23 UTC (permalink / raw)
To: openembedded-core
This trivial script determines BUILDHISTORY_DIR using bitbake and then
passes the command line options to "git tag" on the buildhistory
repository. This is useful for setting a tag before making some changes;
even if the build takes a number of executions of bitbake (resulting in
multiple commits to the buildhistory repository) you can still do the
overall comparison easily.
Suggested by Ross Burton <ross.burton@intel.com>.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/buildhistory-tag | 42 ++++++++++++++++++++++++++++++++++++++++++
1 file changed, 42 insertions(+)
create mode 100755 scripts/buildhistory-tag
diff --git a/scripts/buildhistory-tag b/scripts/buildhistory-tag
new file mode 100755
index 0000000..da90bcf
--- /dev/null
+++ b/scripts/buildhistory-tag
@@ -0,0 +1,42 @@
+#!/bin/sh
+
+# buildhistory-tag
+#
+# Copyright (C) 2012 Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+
+if [ -z $1 ] ; then
+ echo "Please specify at least the name of the tag to set"
+ exit 1
+fi
+
+BUILDHISTORY_DIR=`bitbake -e | grep '^BUILDHISTORY_DIR=' | sed -e 's/^[A-Z_]*=//' -e 's/"//g'`
+if [ -z "$BUILDHISTORY_DIR" ]; then
+ echo "Unable to determine BUILDHISTORY_DIR"
+ exit 1
+fi
+
+if [ ! -d "$BUILDHISTORY_DIR" ]; then
+ echo "BUILDHISTORY_DIR $BUILDHISTORY_DIR does not exist"
+ exit 1
+fi
+
+if [ ! -d "$BUILDHISTORY_DIR/.git" ]; then
+ echo "BUILDHISTORY_DIR $BUILDHISTORY_DIR exists but is not a git repository"
+ exit 1
+fi
+
+(cd $BUILDHISTORY_DIR; git tag $@)
--
1.7.9.5
^ permalink raw reply related [flat|nested] 17+ messages in thread* Re: [PATCH 6/9] scripts: add buildhistory-tag script
2012-08-02 9:23 ` [PATCH 6/9] scripts: add buildhistory-tag script Paul Eggleton
@ 2012-08-02 10:18 ` Burton, Ross
2012-08-02 10:49 ` Paul Eggleton
0 siblings, 1 reply; 17+ messages in thread
From: Burton, Ross @ 2012-08-02 10:18 UTC (permalink / raw)
To: Patches and discussions about the oe-core layer
On 2 August 2012 10:23, Paul Eggleton <paul.eggleton@linux.intel.com> wrote:
> This trivial script determines BUILDHISTORY_DIR using bitbake and then
> passes the command line options to "git tag" on the buildhistory
> repository. This is useful for setting a tag before making some changes;
> even if the build takes a number of executions of bitbake (resulting in
> multiple commits to the buildhistory repository) you can still do the
> overall comparison easily.
>
> Suggested by Ross Burton <ross.burton@intel.com>.
I've since changed my local script to be "buildhistory" that just runs
git "$@", so "buildhistory log" etc work nicely.
Ross
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH 6/9] scripts: add buildhistory-tag script
2012-08-02 10:18 ` Burton, Ross
@ 2012-08-02 10:49 ` Paul Eggleton
2012-08-02 11:12 ` Burton, Ross
0 siblings, 1 reply; 17+ messages in thread
From: Paul Eggleton @ 2012-08-02 10:49 UTC (permalink / raw)
To: Burton, Ross; +Cc: openembedded-core
On Thursday 02 August 2012 11:18:20 Burton, Ross wrote:
> On 2 August 2012 10:23, Paul Eggleton <paul.eggleton@linux.intel.com> wrote:
> > This trivial script determines BUILDHISTORY_DIR using bitbake and then
> > passes the command line options to "git tag" on the buildhistory
> > repository. This is useful for setting a tag before making some changes;
> > even if the build takes a number of executions of bitbake (resulting in
> > multiple commits to the buildhistory repository) you can still do the
> > overall comparison easily.
> >
> > Suggested by Ross Burton <ross.burton@intel.com>.
>
> I've since changed my local script to be "buildhistory" that just runs
> git "$@", so "buildhistory log" etc work nicely.
Interesting - sounds like it would be useful, but I wonder how it would then
fit in with buildhistory-diff as a separate script (particularly with diff as a
valid option to "buildhistory")...
Cheers,
Paul
--
Paul Eggleton
Intel Open Source Technology Centre
^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH 7/9] buildhistory_analysis: tidy up duplicate split_version function
2012-08-02 9:23 [PATCH 0/9] buildhistory improvements Paul Eggleton
` (5 preceding siblings ...)
2012-08-02 9:23 ` [PATCH 6/9] scripts: add buildhistory-tag script Paul Eggleton
@ 2012-08-02 9:23 ` Paul Eggleton
2012-08-02 9:23 ` [PATCH 8/9] buildhistory_analysis: ignore removal of self-dependencies Paul Eggleton
` (3 subsequent siblings)
10 siblings, 0 replies; 17+ messages in thread
From: Paul Eggleton @ 2012-08-02 9:23 UTC (permalink / raw)
To: openembedded-core
This function is now provided by bb.utils and since we have now bumped
the minimum bitbake version, we can switch to that one instead.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/lib/oe/buildhistory_analysis.py | 20 +-------------------
1 file changed, 1 insertion(+), 19 deletions(-)
diff --git a/meta/lib/oe/buildhistory_analysis.py b/meta/lib/oe/buildhistory_analysis.py
index 55bd7b7..a1a0408 100644
--- a/meta/lib/oe/buildhistory_analysis.py
+++ b/meta/lib/oe/buildhistory_analysis.py
@@ -261,24 +261,6 @@ def compare_lists(alines, blines):
return filechanges
-def split_version(s):
- """Split a version string into its constituent parts (PE, PV, PR)
- FIXME: this is a duplicate of a new function in bitbake/lib/bb/utils -
- we should switch to that once we can bump the minimum bitbake version
- """
- s = s.strip(" <>=")
- e = 0
- if s.count(':'):
- e = int(s.split(":")[0])
- s = s.split(":")[1]
- r = ""
- if s.count('-'):
- r = s.rsplit("-", 1)[1]
- s = s.rsplit("-", 1)[0]
- v = s
- return (e, v, r)
-
-
def compare_pkg_lists(astr, bstr):
depvera = bb.utils.explode_dep_versions(astr)
depverb = bb.utils.explode_dep_versions(bstr)
@@ -290,7 +272,7 @@ def compare_pkg_lists(astr, bstr):
dva = depvera[k]
dvb = depverb[k]
if dva and dvb and dva != dvb:
- if bb.utils.vercmp(split_version(dva), split_version(dvb)) < 0:
+ if bb.utils.vercmp(bb.utils.split_version(dva), bb.utils.split_version(dvb)) < 0:
remove.append(k)
for k in remove:
--
1.7.9.5
^ permalink raw reply related [flat|nested] 17+ messages in thread* [PATCH 8/9] buildhistory_analysis: ignore removal of self-dependencies
2012-08-02 9:23 [PATCH 0/9] buildhistory improvements Paul Eggleton
` (6 preceding siblings ...)
2012-08-02 9:23 ` [PATCH 7/9] buildhistory_analysis: tidy up duplicate split_version function Paul Eggleton
@ 2012-08-02 9:23 ` Paul Eggleton
2012-08-02 9:23 ` [PATCH 9/9] classes/buildhistory: save metadata revisions Paul Eggleton
` (2 subsequent siblings)
10 siblings, 0 replies; 17+ messages in thread
From: Paul Eggleton @ 2012-08-02 9:23 UTC (permalink / raw)
To: openembedded-core
The recent removal of self-dependencies for dbg/dev packages produced
a fair amount of noise, so filter it out since we know this is a
reasonable change.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/lib/oe/buildhistory_analysis.py | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/meta/lib/oe/buildhistory_analysis.py b/meta/lib/oe/buildhistory_analysis.py
index a1a0408..a5da7d9 100644
--- a/meta/lib/oe/buildhistory_analysis.py
+++ b/meta/lib/oe/buildhistory_analysis.py
@@ -286,8 +286,9 @@ def compare_dict_blobs(path, ablob, bblob, report_all):
adict = blob_to_dict(ablob)
bdict = blob_to_dict(bblob)
+ pkgname = os.path.basename(path)
defaultvals = {}
- defaultvals['PKG'] = os.path.basename(path)
+ defaultvals['PKG'] = pkgname
defaultvals['PKGE'] = adict.get('PE', '0')
defaultvals['PKGV'] = adict.get('PV', '')
defaultvals['PKGR'] = adict.get('PR', '')
@@ -320,6 +321,9 @@ def compare_dict_blobs(path, ablob, bblob, report_all):
alist.sort()
blist = bstr.split()
blist.sort()
+ # We don't care about the removal of self-dependencies
+ if pkgname in alist and not pkgname in blist:
+ alist.remove(pkgname)
if ' '.join(alist) == ' '.join(blist):
continue
--
1.7.9.5
^ permalink raw reply related [flat|nested] 17+ messages in thread* [PATCH 9/9] classes/buildhistory: save metadata revisions
2012-08-02 9:23 [PATCH 0/9] buildhistory improvements Paul Eggleton
` (7 preceding siblings ...)
2012-08-02 9:23 ` [PATCH 8/9] buildhistory_analysis: ignore removal of self-dependencies Paul Eggleton
@ 2012-08-02 9:23 ` Paul Eggleton
2012-08-02 11:27 ` [PATCH 0/9] buildhistory improvements Burton, Ross
2012-08-06 14:29 ` Richard Purdie
10 siblings, 0 replies; 17+ messages in thread
From: Paul Eggleton @ 2012-08-02 9:23 UTC (permalink / raw)
To: openembedded-core
Always write the metadata revisions for each layer into a machine-
readable "metadata-revs" file so that you can potentially link changes
in the output back to changes in the metadata. (Unlike the existing
similar build-id file, this is not specific to image changes.)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/classes/buildhistory.bbclass | 23 +++++++++++++++++++----
1 file changed, 19 insertions(+), 4 deletions(-)
diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index 76648c9..e121088 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -354,6 +354,15 @@ def buildhistory_get_layers(d):
layertext = "Configured metadata layers:\n%s\n" % '\n'.join(get_layers_branch_rev(d))
return layertext
+def buildhistory_get_metadata_revs(d):
+ # We want an easily machine-readable format here, so get_layers_branch_rev isn't quite what we want
+ layers = (d.getVar("BBLAYERS", True) or "").split()
+ medadata_revs = ["%-17s = %s:%s" % (os.path.basename(i), \
+ base_get_metadata_git_branch(i, None).strip(), \
+ base_get_metadata_git_revision(i, None)) \
+ for i in layers]
+ return '\n'.join(medadata_revs)
+
def squashspaces(string):
import re
@@ -382,19 +391,25 @@ buildhistory_commit() {
return
fi
+ # Create a machine-readable list of metadata revisions for each layer
+ cat > ${BUILDHISTORY_DIR}/metadata-revs <<END
+${@buildhistory_get_metadata_revs(d)}
+END
+
( cd ${BUILDHISTORY_DIR}/
# Initialise the repo if necessary
if [ ! -d .git ] ; then
git init -q
fi
- # Ensure there are new/changed files to commit
- repostatus=`git status --porcelain`
+ # Check if there are new/changed files to commit (other than metadata-revs)
+ repostatus=`git status --porcelain | grep -v " metadata-revs$"`
if [ "$repostatus" != "" ] ; then
- git add ${BUILDHISTORY_DIR}/*
+ git add .
HOSTNAME=`hostname 2>/dev/null || echo unknown`
# porcelain output looks like "?? packages/foo/bar"
+ # Ensure we commit metadata-revs with the first commit
for entry in `echo "$repostatus" | awk '{print $2}' | awk -F/ '{print $1}' | sort | uniq` ; do
- git commit ${BUILDHISTORY_DIR}/$entry -m "$entry: Build ${BUILDNAME} of ${DISTRO} ${DISTRO_VERSION} for machine ${MACHINE} on $HOSTNAME" --author "${BUILDHISTORY_COMMIT_AUTHOR}" > /dev/null
+ git commit $entry metadata-revs -m "$entry: Build ${BUILDNAME} of ${DISTRO} ${DISTRO_VERSION} for machine ${MACHINE} on $HOSTNAME" --author "${BUILDHISTORY_COMMIT_AUTHOR}" > /dev/null
done
if [ "${BUILDHISTORY_PUSH_REPO}" != "" ] ; then
git push -q ${BUILDHISTORY_PUSH_REPO}
--
1.7.9.5
^ permalink raw reply related [flat|nested] 17+ messages in thread* Re: [PATCH 0/9] buildhistory improvements
2012-08-02 9:23 [PATCH 0/9] buildhistory improvements Paul Eggleton
` (8 preceding siblings ...)
2012-08-02 9:23 ` [PATCH 9/9] classes/buildhistory: save metadata revisions Paul Eggleton
@ 2012-08-02 11:27 ` Burton, Ross
2012-08-02 12:58 ` Martin Jansa
2012-08-06 14:29 ` Richard Purdie
10 siblings, 1 reply; 17+ messages in thread
From: Burton, Ross @ 2012-08-02 11:27 UTC (permalink / raw)
To: Patches and discussions about the oe-core layer
On 2 August 2012 10:23, Paul Eggleton <paul.eggleton@linux.intel.com> wrote:
> A bunch of improvements for buildhistory, implementing some feature
> requests and todo items of my own, as well as tidying up some of the
> code.
I've been using this branch for an hour or so whilst reviewing
changes, the new features are useful and it didn't break.
Acked-by: Ross Burton <ross.burton@intel.com>
Ross
^ permalink raw reply [flat|nested] 17+ messages in thread* Re: [PATCH 0/9] buildhistory improvements
2012-08-02 11:27 ` [PATCH 0/9] buildhistory improvements Burton, Ross
@ 2012-08-02 12:58 ` Martin Jansa
2012-08-02 16:14 ` Andreas Müller
0 siblings, 1 reply; 17+ messages in thread
From: Martin Jansa @ 2012-08-02 12:58 UTC (permalink / raw)
To: Patches and discussions about the oe-core layer
[-- Attachment #1: Type: text/plain, Size: 810 bytes --]
On Thu, Aug 02, 2012 at 12:27:05PM +0100, Burton, Ross wrote:
> On 2 August 2012 10:23, Paul Eggleton <paul.eggleton@linux.intel.com> wrote:
> > A bunch of improvements for buildhistory, implementing some feature
> > requests and todo items of my own, as well as tidying up some of the
> > code.
>
> I've been using this branch for an hour or so whilst reviewing
> changes, the new features are useful and it didn't break.
>
> Acked-by: Ross Burton <ross.burton@intel.com>
My tests also look good, some interesting (expected) changes recorded
here
http://git.shr-project.org/git/?p=buildhistory.git;a=commitdiff;h=2fe33b15dcec9eb69f9075f138cea2405c061b00
Thanks!
Acked-by: Martin Jansa <Martin.Jansa@gmail.com>
Cheers,
--
Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH 0/9] buildhistory improvements
2012-08-02 12:58 ` Martin Jansa
@ 2012-08-02 16:14 ` Andreas Müller
0 siblings, 0 replies; 17+ messages in thread
From: Andreas Müller @ 2012-08-02 16:14 UTC (permalink / raw)
To: Patches and discussions about the oe-core layer
On Thu, Aug 2, 2012 at 2:58 PM, Martin Jansa <martin.jansa@gmail.com> wrote:
> On Thu, Aug 02, 2012 at 12:27:05PM +0100, Burton, Ross wrote:
>> On 2 August 2012 10:23, Paul Eggleton <paul.eggleton@linux.intel.com> wrote:
>> > A bunch of improvements for buildhistory, implementing some feature
>> > requests and todo items of my own, as well as tidying up some of the
>> > code.
>>
>> I've been using this branch for an hour or so whilst reviewing
>> changes, the new features are useful and it didn't break.
>>
>> Acked-by: Ross Burton <ross.burton@intel.com>
>
> My tests also look good, some interesting (expected) changes recorded
> here
>
> http://git.shr-project.org/git/?p=buildhistory.git;a=commitdiff;h=2fe33b15dcec9eb69f9075f138cea2405c061b00
>
> Thanks!
>
> Acked-by: Martin Jansa <Martin.Jansa@gmail.com>
>
> Cheers,
>
> --
> Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com
Although I have no time for testing at the moment: thanks for doing this
Andreas
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH 0/9] buildhistory improvements
2012-08-02 9:23 [PATCH 0/9] buildhistory improvements Paul Eggleton
` (9 preceding siblings ...)
2012-08-02 11:27 ` [PATCH 0/9] buildhistory improvements Burton, Ross
@ 2012-08-06 14:29 ` Richard Purdie
10 siblings, 0 replies; 17+ messages in thread
From: Richard Purdie @ 2012-08-06 14:29 UTC (permalink / raw)
To: Patches and discussions about the oe-core layer
On Thu, 2012-08-02 at 10:23 +0100, Paul Eggleton wrote:
> A bunch of improvements for buildhistory, implementing some feature
> requests and todo items of my own, as well as tidying up some of the
> code.
>
>
> The following changes since commit 404d2d490fc347203e89d274530c17fb5f0aa20f:
>
> gcc-configure-target: Set native-system-header-dir for target gcc (2012-08-01 23:11:09 +0100)
>
> are available in the git repository at:
>
> git://git.openembedded.org/openembedded-core-contrib paule/buildhistory-fixes7
> http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/buildhistory-fixes7
>
> Paul Eggleton (9):
> classes/buildhistory: remove obsolete flat package history feature
> classes/buildhistory: ensure old package info is removed
> classes/buildhistory: remove unused functions
> classes/buildhistory: save preinst/postinst/prerm/postrm
> classes/buildhistory: record PKG/PKGE/PKGV/PKGR
> buildhistory_analysis: tidy up duplicate split_version function
> buildhistory_analysis: ignore removal of self-dependencies
> classes/buildhistory: save metadata revisions
Merged to master apart from:
scripts: add buildhistory-tag script
which seems to need more discussion/work.
Cheers,
Richard
^ permalink raw reply [flat|nested] 17+ messages in thread