[yocto autobuilder][PATCH 0/3] Fix buildlogger

[yocto autobuilder][PATCH 0/3] Fix buildlogger

[Joshua Lock]

Since build 999 we haven't seen any builds logged to the BuildLogs wiki page,
that turns out to be because the JSON dicts keys are strings and thus the sort
order isn't as expected, i.e. 999 > 1001.

This series tidies up the buildlogger some and includes a fix for this issue by
converting the keys of the dicts to a list of ints and sorting that to determine
the most recent build.

The following changes since commit ba01fe19440a91091fe15544ecc2809a79f19401:

  nightly-qa-systemd.conf: Add create_eventlog property (2016-08-31 21:22:11 +0100)

are available in the git repository at:

  git://git.yoctoproject.org/yocto-autobuilder joshuagl/buildlogger
  http://git.yoctoproject.org/cgit.cgi/yocto-autobuilder/log/?h=joshuagl/buildlogger

Joshua Lock (3):
  buildlogger: remove old, incomplete, comment
  buildlogger: don't hardcode YP AB in generated URL
  buildlogger: fix sorting of recent builds

 bin/buildlogger              | 23 +++++++++++++----------
 etc/buildlogger.conf.example |  1 +
 2 files changed, 14 insertions(+), 10 deletions(-)

-- 
2.7.4

[yocto autobuilder][PATCH 1/3] buildlogger: remove old, incomplete, comment

[Joshua Lock]

The comment isn't complete and doesn't make much sense.

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
---
 bin/buildlogger | 6 ------
 1 file changed, 6 deletions(-)

diff --git a/bin/buildlogger b/bin/buildlogger
index 97f67ec..7a529c0 100755
--- a/bin/buildlogger
+++ b/bin/buildlogger
@@ -11,12 +11,6 @@ __maintainer__ = "Joshua Lock"
 __email__ = "joshua.g.lock@intel.com"
 '''
 
-# We'd probably benefit from using some caching, but first we'd need the AB API
-# to include
-#
-# We can set repo url, branch & commit for a bunch of repositorys.
-# Do they all get built for nightly?
-
 import codecs
 try:
     import configparser
-- 
2.7.4

[yocto autobuilder][PATCH 2/3] buildlogger: don't hardcode YP AB in generated URL

[Joshua Lock]

When generating text for a BuildLog wiki entry we were hard-coding
the Yocto Project AB URL in the entry's title, rather than that
add a new variable to buildlogger.conf and use the value of this
variable to construct the URL in the link.

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
---
 bin/buildlogger              | 7 +++++--
 etc/buildlogger.conf.example | 1 +
 2 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/bin/buildlogger b/bin/buildlogger
index 7a529c0..1b31281 100755
--- a/bin/buildlogger
+++ b/bin/buildlogger
@@ -30,6 +30,7 @@ except ImportError:
     sys.exit(1)
 
 abapi = ''
+abwww = ''
 # Wiki editing params
 un = ''
 pw = ''
@@ -49,6 +50,7 @@ def load_config(configfile):
     global title
     global wikiapi
     global abapi
+    global abwww
     success = False
 
     if os.path.exists(configfile):
@@ -59,6 +61,7 @@ def load_config(configfile):
             pw = config.get('wiki', 'password')
             title = config.get('wiki', 'pagetitle')
             wikiapi = config.get('wiki', 'apiuri')
+            abwww = config.get('builder', 'wwwuri')
             abapi = config.get('builder', 'apiuri')
             success = True
         except configparser.Error as ex:
@@ -195,8 +198,8 @@ def ab_last_build_to_entry(build_json, build_id):
         if prop[0] == 'commit_poky':
             chash = prop[1]
 
-    urlfmt = 'https://autobuilder.yoctoproject.org/main/builders/%s/builds/%s/'
-    url = urlfmt % (builder, buildid)
+    urlfmt = '%s/builders/%s/builds/%s/'
+    url = urlfmt % (abwww, builder, buildid)
     sectionfmt = '==[%s %s %s - %s %s]=='
     section_title = sectionfmt % (url, builder, buildid, buildbranch, chash)
     summaryfmt = 'Adding new BuildLog entry for build %s (%s)'
diff --git a/etc/buildlogger.conf.example b/etc/buildlogger.conf.example
index 459961c..330af56 100644
--- a/etc/buildlogger.conf.example
+++ b/etc/buildlogger.conf.example
@@ -5,4 +5,5 @@ pagetitle = BuildLogPageTitle
 apiuri = https://wiki.yoctoproject.org/wiki/api.php
 
 [builder]
+wwwuri = https://autobuilder.yoctoproject.org/main
 apiuri = https://autobuilder.yoctoproject.org/main/json/builders/nightly/builds/_all
-- 
2.7.4

[yocto autobuilder][PATCH 3/3] buildlogger: fix sorting of recent builds

[Joshua Lock]

Recent builds, >999, on the YP AutoBuilder were not being logged.
This is because we sort the builds based on a string representation
of their build number, yet string 1001 < string 999.

Resolve this by converting the build id's to an int, sorting them
and then determining which is the highest/most recent build id.

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
---
 bin/buildlogger | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/bin/buildlogger b/bin/buildlogger
index 1b31281..912f419 100755
--- a/bin/buildlogger
+++ b/bin/buildlogger
@@ -219,7 +219,7 @@ def write_last_build(buildid):
 
 # Read last logged buildid from a file
 def read_last_build():
-    last_build = ''
+    last_build = '0'
     try:
         with open(cachefile, 'r') as fi:
             last_build = fi.readline()
@@ -254,7 +254,13 @@ def watch_for_builds(configfile):
             print("Failed to decode JSON: %s" % str(e))
             continue
 
-        last_build = sorted(build_json.keys())[-1]
+        # The keys of the JSON dict are unicode strings, which makes
+        # sorting to find the most recent difficult (i.e. for a string type
+        # 999 > 1001). Convert them to ints for a simpler sort.
+        buildids = map(int, build_json.keys())
+        buildids.sort()
+        last_build = str(buildids[-1])
+
         # If a new build is detected, post a new entry to the BuildLog
         if last_build != last_logged:
             new_entry, summary = ab_last_build_to_entry(build_json, last_build)
-- 
2.7.4

Re: [yocto autobuilder][PATCH 0/3] Fix buildlogger

[Beth 'pidge' Flanagan]

merged into master. Thanks!

-b

On Wed, 2016-09-07 at 16:33 +0100, Joshua Lock wrote:
> Since build 999 we haven't seen any builds logged to the BuildLogs
> wiki page,
> that turns out to be because the JSON dicts keys are strings and thus
> the sort
> order isn't as expected, i.e. 999 > 1001.
> 
> This series tidies up the buildlogger some and includes a fix for
> this issue by
> converting the keys of the dicts to a list of ints and sorting that
> to determine
> the most recent build.
> 
> The following changes since commit
> ba01fe19440a91091fe15544ecc2809a79f19401:
> 
>   nightly-qa-systemd.conf: Add create_eventlog property (2016-08-31
> 21:22:11 +0100)
> 
> are available in the git repository at:
> 
>   git://git.yoctoproject.org/yocto-autobuilder joshuagl/buildlogger
>   http://git.yoctoproject.org/cgit.cgi/yocto-autobuilder/log/?h=joshu
> agl/buildlogger
> 
> Joshua Lock (3):
>   buildlogger: remove old, incomplete, comment
>   buildlogger: don't hardcode YP AB in generated URL
>   buildlogger: fix sorting of recent builds
> 
>  bin/buildlogger              | 23 +++++++++++++----------
>  etc/buildlogger.conf.example |  1 +
>  2 files changed, 14 insertions(+), 10 deletions(-)
>