* [PATCH 0/2] toaster: cummulative 28072017 patch
@ 2017-07-29 0:14 David Reyna
2017-07-29 0:14 ` [PATCH 1/2] toaster: enable remote HTTP API for status aggregation David Reyna
2017-07-29 0:14 ` [PATCH 2/2] toaster: move to new bitbake xmlrpc default David Reyna
0 siblings, 2 replies; 3+ messages in thread
From: David Reyna @ 2017-07-29 0:14 UTC (permalink / raw)
To: bitbake-devel
From: David Reyna <David.Reyna@windriver.com>
WARNING: there is one new file added with this patch:
create mode 100644 bitbake/lib/toaster/toastergui/templates/health.html
* IMPORTANT: This fixes the fatal regressions to Toaster from xmlrpc changes in bitbake.
This also addresses a new race condition that Toaster suddenly started hitting where
the 'bitbake.lock' existed but was not populated with the server info for period of time
(~ second).
toaster: move to new bitbake xmlrpc default
* This provides a health ping URL, plus adds a JSON API for fetching
Toaster status for multi-instance Toaster aggregators
toaster: enable remote HTTP API for status aggregation
The following changes since commit 3c1c4ceaa470957fe5633d65215f165a38d61be7:
yocto-project-qs, ref-manual: Upgraded tar to 1.27 from 1.24 (2017-07-27 23:00:23 +0100)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib dreyna/submit/dreyna/toaster/cummulative_280717_patch
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=dreyna/submit/dreyna/toaster/cummulative_280717_patch
David Reyna (2):
toaster: enable remote HTTP API for status aggregation
toaster: move to new bitbake xmlrpc default
lib/toaster/bldcontrol/bbcontroller.py | 4 +-
lib/toaster/bldcontrol/localhostbecontroller.py | 19 ++++--
lib/toaster/orm/models.py | 2 +-
lib/toaster/toastergui/templates/health.html | 6 ++
lib/toaster/toastergui/urls.py | 7 +-
lib/toaster/toastergui/views.py | 86 ++++++++++++++++++++++++-
lib/toaster/toastermain/urls.py | 5 +-
7 files changed, 118 insertions(+), 11 deletions(-)
create mode 100644 bitbake/lib/toaster/toastergui/templates/health.html
--
1.9.1
^ permalink raw reply [flat|nested] 3+ messages in thread
* [PATCH 1/2] toaster: enable remote HTTP API for status aggregation
2017-07-29 0:14 [PATCH 0/2] toaster: cummulative 28072017 patch David Reyna
@ 2017-07-29 0:14 ` David Reyna
2017-07-29 0:14 ` [PATCH 2/2] toaster: move to new bitbake xmlrpc default David Reyna
1 sibling, 0 replies; 3+ messages in thread
From: David Reyna @ 2017-07-29 0:14 UTC (permalink / raw)
To: bitbake-devel
From: David Reyna <David.Reyna@windriver.com>
Add support for Toaster aggregators with a set of api links that
return JSON data for (a) builds in progress, (b) builds completed,
(c) specific build data, and (d) an is-alive health ping link.
[YOCTO #11794]
Signed-off-by: David Reyna <David.Reyna@windriver.com>
---
lib/toaster/toastergui/templates/health.html | 6 ++
lib/toaster/toastergui/urls.py | 7 ++-
lib/toaster/toastergui/views.py | 86 +++++++++++++++++++++++++++-
lib/toaster/toastermain/urls.py | 5 +-
4 files changed, 100 insertions(+), 4 deletions(-)
create mode 100644 bitbake/lib/toaster/toastergui/templates/health.html
diff --git a/lib/toaster/toastergui/templates/health.html b/lib/toaster/toastergui/templates/health.html
new file mode 100644
index 0000000..f17fdbc
--- /dev/null
+++ b/lib/toaster/toastergui/templates/health.html
@@ -0,0 +1,6 @@
+<!DOCTYPE html>
+<html lang="en">
+ <head><title>Toaster Health</title></head>
+ <body>Ok</body>
+</html>
+
diff --git a/lib/toaster/toastergui/urls.py b/lib/toaster/toastergui/urls.py
index 6aebc3f..3ad5566 100644
--- a/lib/toaster/toastergui/urls.py
+++ b/lib/toaster/toastergui/urls.py
@@ -244,6 +244,11 @@ urlpatterns = [
url(r'^mostrecentbuilds$', widgets.MostRecentBuildsView.as_view(),
name='most_recent_builds'),
- # default redirection
+ # JSON data for aggregators
+ url(r'^api/builds$', views.json_builds, name='json_builds'),
+ url(r'^api/building$', views.json_building, name='json_building'),
+ url(r'^api/build/(?P<build_id>\d+)$', views.json_build, name='json_build'),
+
+ # default redirection
url(r'^$', RedirectView.as_view(url='landing', permanent=True)),
]
diff --git a/lib/toaster/toastergui/views.py b/lib/toaster/toastergui/views.py
index 5720b9d..334bb4a 100755
--- a/lib/toaster/toastergui/views.py
+++ b/lib/toaster/toastergui/views.py
@@ -35,7 +35,7 @@ from orm.models import BitbakeVersion, CustomImageRecipe
from django.core.urlresolvers import reverse, resolve
from django.core.exceptions import MultipleObjectsReturned, ObjectDoesNotExist
from django.core.paginator import Paginator, EmptyPage, PageNotAnInteger
-from django.http import HttpResponseNotFound
+from django.http import HttpResponseNotFound, JsonResponse
from django.utils import timezone
from datetime import timedelta, datetime
from toastergui.templatetags.projecttags import json as jsonfilter
@@ -1256,6 +1256,89 @@ def managedcontextprocessor(request):
}
return ret
+# REST-based API calls to return build/building status to external Toaster
+# managers and aggregators via JSON
+
+def _json_build_status(build_id,extend):
+ build_stat = None
+ try:
+ build = Build.objects.get( pk = build_id )
+ build_stat = {}
+ build_stat['id'] = build.id
+ build_stat['name'] = build.build_name
+ build_stat['machine'] = build.machine
+ build_stat['distro'] = build.distro
+ build_stat['start'] = build.started_on
+ # look up target name
+ target= Target.objects.get( build = build )
+ if target:
+ if target.task:
+ build_stat['target'] = '%s:%s' % (target.target,target.task)
+ else:
+ build_stat['target'] = '%s' % (target.target)
+ else:
+ build_stat['target'] = ''
+ # look up project name
+ project = Project.objects.get( build = build )
+ if project:
+ build_stat['project'] = project.name
+ else:
+ build_stat['project'] = ''
+ if Build.IN_PROGRESS == build.outcome:
+ now = timezone.now()
+ timediff = now - build.started_on
+ build_stat['seconds']='%.3f' % timediff.total_seconds()
+ build_stat['clone']='%d:%d' % (build.repos_cloned,build.repos_to_clone)
+ build_stat['parse']='%d:%d' % (build.recipes_parsed,build.recipes_to_parse)
+ tf = Task.objects.filter(build = build)
+ tfc = tf.count()
+ if tfc > 0:
+ tfd = tf.exclude(order__isnull=True).count()
+ else:
+ tfd = 0
+ build_stat['task']='%d:%d' % (tfd,tfc)
+ else:
+ build_stat['outcome'] = build.get_outcome_text()
+ timediff = build.completed_on - build.started_on
+ build_stat['seconds']='%.3f' % timediff.total_seconds()
+ build_stat['stop'] = build.completed_on
+ messages = LogMessage.objects.all().filter(build = build)
+ errors = len(messages.filter(level=LogMessage.ERROR) |
+ messages.filter(level=LogMessage.EXCEPTION) |
+ messages.filter(level=LogMessage.CRITICAL))
+ build_stat['errors'] = errors
+ warnings = len(messages.filter(level=LogMessage.WARNING))
+ build_stat['warnings'] = warnings
+ if extend:
+ build_stat['cooker_log'] = build.cooker_log_path
+ except Exception as e:
+ build_state = str(e)
+ return build_stat
+
+def json_builds(request):
+ build_table = []
+ builds = []
+ try:
+ builds = Build.objects.exclude(outcome=Build.IN_PROGRESS).order_by("-started_on")
+ for build in builds:
+ build_table.append(_json_build_status(build.id,False))
+ except Exception as e:
+ build_table = str(e)
+ return JsonResponse({'builds' : build_table, 'count' : len(builds)})
+
+def json_building(request):
+ build_table = []
+ builds = []
+ try:
+ builds = Build.objects.filter(outcome=Build.IN_PROGRESS).order_by("-started_on")
+ for build in builds:
+ build_table.append(_json_build_status(build.id,False))
+ except Exception as e:
+ build_table = str(e)
+ return JsonResponse({'building' : build_table, 'count' : len(builds)})
+
+def json_build(request,build_id):
+ return JsonResponse({'build' : _json_build_status(build_id,True)})
import toastermain.settings
@@ -1694,3 +1777,4 @@ if True:
return render(request, "unavailable_artifact.html")
except (ObjectDoesNotExist, IOError):
return render(request, "unavailable_artifact.html")
+
diff --git a/lib/toaster/toastermain/urls.py b/lib/toaster/toastermain/urls.py
index bb32559..6750bdf 100644
--- a/lib/toaster/toastermain/urls.py
+++ b/lib/toaster/toastermain/urls.py
@@ -20,9 +20,8 @@
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
from django.conf.urls import patterns, include, url
-from django.views.generic import RedirectView
+from django.views.generic import RedirectView, TemplateView
from django.views.decorators.cache import never_cache
-
import bldcollector.views
import logging
@@ -46,6 +45,8 @@ urlpatterns = [
# in the future.
url(r'^orm/eventfile$', bldcollector.views.eventfile),
+ url(r'^health$', TemplateView.as_view(template_name="health.html"), name='Toaster Health'),
+
# if no application is selected, we have the magic toastergui app here
url(r'^$', never_cache(RedirectView.as_view(url='/toastergui/', permanent=True))),
]
--
1.9.1
^ permalink raw reply related [flat|nested] 3+ messages in thread
* [PATCH 2/2] toaster: move to new bitbake xmlrpc default
2017-07-29 0:14 [PATCH 0/2] toaster: cummulative 28072017 patch David Reyna
2017-07-29 0:14 ` [PATCH 1/2] toaster: enable remote HTTP API for status aggregation David Reyna
@ 2017-07-29 0:14 ` David Reyna
1 sibling, 0 replies; 3+ messages in thread
From: David Reyna @ 2017-07-29 0:14 UTC (permalink / raw)
To: bitbake-devel
From: David Reyna <David.Reyna@windriver.com>
The bitbake option "-t SERVERTYPE" was deprecated and can be
removed since the desired XMLRPC listener now the default.
The bitbake server port cannot be "-1" anymore and must be
explicitly passed.
There is a race condition for the bblock file to not only
be created but to actually be populated, so a delay test loop
is required (usually only one pass).
The 'xmlrpcclient' is now the class that allows Toaster to for
example kill builds.
The events for populating the recipe parsing now either show only
the final result or are skipped entiredly, so the progress
calculator needs to be changed to not block on the parsing phase.
[YOCTO #11851]
Signed-off-by: David Reyna <David.Reyna@windriver.com>
---
lib/toaster/bldcontrol/bbcontroller.py | 4 ++--
lib/toaster/bldcontrol/localhostbecontroller.py | 19 +++++++++++++++----
lib/toaster/orm/models.py | 2 +-
3 files changed, 18 insertions(+), 7 deletions(-)
diff --git a/lib/toaster/bldcontrol/bbcontroller.py b/lib/toaster/bldcontrol/bbcontroller.py
index 912f67b..5195600 100644
--- a/lib/toaster/bldcontrol/bbcontroller.py
+++ b/lib/toaster/bldcontrol/bbcontroller.py
@@ -37,8 +37,8 @@ class BitbakeController(object):
"""
def __init__(self, be):
- import bb.server.xmlrpc
- self.connection = bb.server.xmlrpc._create_server(be.bbaddress,
+ import bb.server.xmlrpcclient
+ self.connection = bb.server.xmlrpcclient._create_server(be.bbaddress,
int(be.bbport))[0]
def _runCommand(self, command):
diff --git a/lib/toaster/bldcontrol/localhostbecontroller.py b/lib/toaster/bldcontrol/localhostbecontroller.py
index 2916246..6142f7e 100644
--- a/lib/toaster/bldcontrol/localhostbecontroller.py
+++ b/lib/toaster/bldcontrol/localhostbecontroller.py
@@ -24,6 +24,7 @@ import os
import sys
import re
import shutil
+import time
from django.db import transaction
from django.db.models import Q
from bldcontrol.models import BuildEnvironment, BRLayer, BRVariable, BRTarget, BRBitbake
@@ -331,12 +332,22 @@ class LocalhostBEController(BuildEnvironmentController):
bitbake = os.path.join(self.pokydirname, 'bitbake', 'bin', 'bitbake')
toasterlayers = os.path.join(builddir,"conf/toaster-bblayers.conf")
self._shellcmd('bash -c \"source %s %s; BITBAKE_UI="knotty" %s --read %s --read %s '
- '--server-only -t xmlrpc -B 0.0.0.0:0\"' % (oe_init,
+ '--server-only -B 0.0.0.0:0\"' % (oe_init,
builddir, bitbake, confpath, toasterlayers), self.be.sourcedir)
# read port number from bitbake.lock
self.be.bbport = ""
bblock = os.path.join(builddir, 'bitbake.lock')
+ # allow 10 seconds for bb lock file to appear but also be populated
+ for lock_check in range(10):
+ if not os.path.exists(bblock):
+ logger.debug("localhostbecontroller: waiting for bblock file to appear")
+ time.sleep(1)
+ continue
+ if 10 < os.stat(bblock).st_size:
+ break
+ logger.debug("localhostbecontroller: waiting for bblock content to appear")
+ time.sleep(1)
with open(bblock) as fplock:
for line in fplock:
if ":" in line:
@@ -365,10 +376,10 @@ class LocalhostBEController(BuildEnvironmentController):
log = os.path.join(builddir, 'toaster_ui.log')
local_bitbake = os.path.join(os.path.dirname(os.getenv('BBBASEDIR')),
'bitbake')
- self._shellcmd(['bash -c \"(TOASTER_BRBE="%s" BBSERVER="0.0.0.0:-1" '
+ self._shellcmd(['bash -c \"(TOASTER_BRBE="%s" BBSERVER="0.0.0.0:%s" '
'%s %s -u toasterui --token="" >>%s 2>&1;'
- 'BITBAKE_UI="knotty" BBSERVER=0.0.0.0:-1 %s -m)&\"' \
- % (brbe, local_bitbake, bbtargets, log, bitbake)],
+ 'BITBAKE_UI="knotty" BBSERVER=0.0.0.0:%s %s -m)&\"' \
+ % (brbe, self.be.bbport, local_bitbake, bbtargets, log, self.be.bbport, bitbake)],
builddir, nowait=True)
logger.debug('localhostbecontroller: Build launched, exiting. '
diff --git a/lib/toaster/orm/models.py b/lib/toaster/orm/models.py
index 05cc5a8..7aaebed 100644
--- a/lib/toaster/orm/models.py
+++ b/lib/toaster/orm/models.py
@@ -451,7 +451,7 @@ class Build(models.Model):
recipes_to_parse = models.IntegerField(default=1)
# number of recipes parsed so far for this build
- recipes_parsed = models.IntegerField(default=0)
+ recipes_parsed = models.IntegerField(default=1)
# number of repos to clone for this build
repos_to_clone = models.IntegerField(default=1)
--
1.9.1
^ permalink raw reply related [flat|nested] 3+ messages in thread
end of thread, other threads:[~2017-07-29 0:22 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2017-07-29 0:14 [PATCH 0/2] toaster: cummulative 28072017 patch David Reyna
2017-07-29 0:14 ` [PATCH 1/2] toaster: enable remote HTTP API for status aggregation David Reyna
2017-07-29 0:14 ` [PATCH 2/2] toaster: move to new bitbake xmlrpc default David Reyna
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.