* [PATCH 1/3] toaster: Read correct cooker log path from toasterui
2015-09-11 20:57 [PATCH 0/3] Read correct cooker log brian avery
@ 2015-09-11 20:57 ` brian avery
2015-09-11 20:57 ` [PATCH 2/3] toaster: Avoid unnecessary local to local copy of cooker log brian avery
2015-09-11 20:57 ` [PATCH 3/3] toaster: Don't def a function for each call to build_artifact() brian avery
2 siblings, 0 replies; 4+ messages in thread
From: brian avery @ 2015-09-11 20:57 UTC (permalink / raw)
To: bitbake-devel
From: Elliot Smith <elliot.smith@intel.com>
The BB_CONSOLELOG variable changes by the time we read it in
BuildInfoHelper. This means that the log file location we
are using is incorrect, so the links to the cooker logs don't
work.
Instead, read it at the point when the BuildStarted event occurs
in toasterui. The BB_CONSOLELOG variable has the correct value
here, so pass that to BuildInfoHelper.
[YOCTO #8209]
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
---
lib/bb/ui/buildinfohelper.py | 8 ++++----
lib/bb/ui/toasterui.py | 2 +-
2 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/lib/bb/ui/buildinfohelper.py b/lib/bb/ui/buildinfohelper.py
index 2d1ed51..6e313fe 100644
--- a/lib/bb/ui/buildinfohelper.py
+++ b/lib/bb/ui/buildinfohelper.py
@@ -704,7 +704,7 @@ class BuildInfoHelper(object):
## methods to convert event/external info into objects that the ORM layer uses
- def _get_build_information(self):
+ def _get_build_information(self, consolelogfile):
build_info = {}
# Generate an identifier for each new build
@@ -713,7 +713,7 @@ class BuildInfoHelper(object):
build_info['distro_version'] = self.server.runCommand(["getVariable", "DISTRO_VERSION"])[0]
build_info['started_on'] = timezone.now()
build_info['completed_on'] = timezone.now()
- build_info['cooker_log_path'] = self.server.runCommand(["getVariable", "BB_CONSOLELOG"])[0]
+ build_info['cooker_log_path'] = consolelogfile
build_info['build_name'] = self.server.runCommand(["getVariable", "BUILDNAME"])[0]
build_info['bitbake_version'] = self.server.runCommand(["getVariable", "BB_VERSION"])[0]
@@ -847,9 +847,9 @@ class BuildInfoHelper(object):
logger.warn("buildinfohelper: cannot identify layer exception:%s ", nee)
- def store_started_build(self, event):
+ def store_started_build(self, event, consolelogfile):
assert '_pkgs' in vars(event)
- build_information = self._get_build_information()
+ build_information = self._get_build_information(consolelogfile)
build_obj = self.orm_wrapper.create_build_object(build_information, self.brbe, self.project)
diff --git a/lib/bb/ui/toasterui.py b/lib/bb/ui/toasterui.py
index 9c7e87d..e0c278b 100644
--- a/lib/bb/ui/toasterui.py
+++ b/lib/bb/ui/toasterui.py
@@ -126,7 +126,7 @@ def main(server, eventHandler, params ):
# the code will look into the protected variables of the event; no easy way around this
if isinstance(event, bb.event.BuildStarted):
- buildinfohelper.store_started_build(event)
+ buildinfohelper.store_started_build(event, consolelogfile)
if isinstance(event, (bb.build.TaskStarted, bb.build.TaskSucceeded, bb.build.TaskFailedSilent)):
buildinfohelper.update_and_store_task(event)
--
1.9.1
^ permalink raw reply related [flat|nested] 4+ messages in thread* [PATCH 2/3] toaster: Avoid unnecessary local to local copy of cooker log
2015-09-11 20:57 [PATCH 0/3] Read correct cooker log brian avery
2015-09-11 20:57 ` [PATCH 1/3] toaster: Read correct cooker log path from toasterui brian avery
@ 2015-09-11 20:57 ` brian avery
2015-09-11 20:57 ` [PATCH 3/3] toaster: Don't def a function for each call to build_artifact() brian avery
2 siblings, 0 replies; 4+ messages in thread
From: brian avery @ 2015-09-11 20:57 UTC (permalink / raw)
To: bitbake-devel
From: Elliot Smith <elliot.smith@intel.com>
The cooker log is copied from its original (bitbake) location
to an artifact directory chosen by the user (chosen when
checksettings.py runs as part of the managed mode;
it's one of the annoying questions you're asked at startup).
The copy happens as part of the runbuilds script run, which
is started in a loop from the toaster startup script
in managed mode.
When a user requests the log for a build via toaster, they
are getting the log which has been copied to the artifact
directory, not the original bitbake log.
This works for the managed case, where the runbuilds command is
running in a loop and copying log files for completed builds to
the artifact directory. However, in analysis mode, there are two
problems:
1. checksettings isn't run, so the artifacts directory isn't
set. toaster is then unable to figure out where the log files
should have been copied to.
2. The log files aren't copied to the artifacts directory
anyway, as runbuilds isn't running in analysis mode.
To fix this, just point the user to the local bitbake log file
in its original location. This avoids the copy step, and means
we can remove a whole question from the toaster startup sequence,
as we no longer need an artifact directory.
The only downside to this is that it's not appropriate for
remote bitbake servers. We will need to revisit this and
possibly reinstate the copy step once we have to reconcile
local and remote builds and make their logs available in
the same way.
[YOCTO #8209]
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
---
.../management/commands/checksettings.py | 22 --------------------
.../bldcontrol/management/commands/runbuilds.py | 24 ++--------------------
lib/toaster/toastergui/views.py | 13 ++++++------
3 files changed, 8 insertions(+), 51 deletions(-)
diff --git a/lib/toaster/bldcontrol/management/commands/checksettings.py b/lib/toaster/bldcontrol/management/commands/checksettings.py
index 3ccc7c6..b2c573c 100644
--- a/lib/toaster/bldcontrol/management/commands/checksettings.py
+++ b/lib/toaster/bldcontrol/management/commands/checksettings.py
@@ -64,27 +64,6 @@ class Command(NoArgsCommand):
return ""
return DN(self._find_first_path_for_file(DN(self.guesspath), "bblayers.conf", 4))
-
- def _verify_artifact_storage_dir(self):
- # verify that we have a settings for downloading artifacts
- while ToasterSetting.objects.filter(name="ARTIFACTS_STORAGE_DIR").count() == 0:
- guessedpath = os.getcwd() + "/toaster_build_artifacts/"
- print("\nToaster needs to know in which directory it can download build log files and other artifacts.\nToaster suggests \"%s\"." % guessedpath)
- artifacts_storage_dir = raw_input("Press Enter to select \"%s\" or type the full path to a different directory: " % guessedpath)
- if len(artifacts_storage_dir) == 0:
- artifacts_storage_dir = guessedpath
- if len(artifacts_storage_dir) > 0 and artifacts_storage_dir.startswith("/"):
- try:
- os.makedirs(artifacts_storage_dir)
- except OSError as ose:
- if "File exists" in str(ose):
- pass
- else:
- raise ose
- ToasterSetting.objects.create(name="ARTIFACTS_STORAGE_DIR", value=artifacts_storage_dir)
- return 0
-
-
def _verify_build_environment(self):
# refuse to start if we have no build environments
while BuildEnvironment.objects.count() == 0:
@@ -239,7 +218,6 @@ class Command(NoArgsCommand):
def handle_noargs(self, **options):
retval = 0
- retval += self._verify_artifact_storage_dir()
retval += self._verify_build_environment()
retval += self._verify_default_settings()
retval += self._verify_builds_in_progress()
diff --git a/lib/toaster/bldcontrol/management/commands/runbuilds.py b/lib/toaster/bldcontrol/management/commands/runbuilds.py
index c3e9b74..718e144 100644
--- a/lib/toaster/bldcontrol/management/commands/runbuilds.py
+++ b/lib/toaster/bldcontrol/management/commands/runbuilds.py
@@ -77,31 +77,11 @@ class Command(NoArgsCommand):
bec.be.save()
def archive(self):
- ''' archives data from the builds '''
- artifact_storage_dir = ToasterSetting.objects.get(name="ARTIFACTS_STORAGE_DIR").value
for br in BuildRequest.objects.filter(state = BuildRequest.REQ_ARCHIVE):
- # save cooker log
if br.build == None:
br.state = BuildRequest.REQ_FAILED
- br.save()
- continue
- build_artifact_storage_dir = os.path.join(artifact_storage_dir, "%d" % br.build.pk)
- try:
- os.makedirs(build_artifact_storage_dir)
- except OSError as ose:
- if "File exists" in str(ose):
- pass
- else:
- raise ose
-
- file_name = os.path.join(build_artifact_storage_dir, "cooker_log.txt")
- try:
- with open(file_name, "w") as f:
- f.write(br.environment.get_artifact(br.build.cooker_log_path).read())
- except IOError:
- os.unlink(file_name)
-
- br.state = BuildRequest.REQ_COMPLETED
+ else:
+ br.state = BuildRequest.REQ_COMPLETED
br.save()
def cleanup(self):
diff --git a/lib/toaster/toastergui/views.py b/lib/toaster/toastergui/views.py
index 4e8f69e..784272f 100755
--- a/lib/toaster/toastergui/views.py
+++ b/lib/toaster/toastergui/views.py
@@ -2710,7 +2710,6 @@ if True:
def build_artifact(request, build_id, artifact_type, artifact_id):
if artifact_type in ["cookerlog"]:
- # these artifacts are saved after building, so they are on the server itself
def _mimetype_for_artifact(path):
try:
import magic
@@ -2741,16 +2740,16 @@ if True:
except ImportError:
return "binary/octet-stream"
try:
- # match code with runbuilds.Command.archive()
- build_artifact_storage_dir = os.path.join(ToasterSetting.objects.get(name="ARTIFACTS_STORAGE_DIR").value, "%d" % int(build_id))
- file_name = os.path.join(build_artifact_storage_dir, "cooker_log.txt")
-
+ build = Build.objects.get(pk = build_id)
+ file_name = build.cooker_log_path
fsock = open(file_name, "r")
- content_type=_mimetype_for_artifact(file_name)
+ content_type = _mimetype_for_artifact(file_name)
response = HttpResponse(fsock, content_type = content_type)
- response['Content-Disposition'] = 'attachment; filename=' + os.path.basename(file_name)
+ disposition = 'attachment; filename=cooker.log'
+ response['Content-Disposition'] = disposition
+
return response
except IOError:
context = {
--
1.9.1
^ permalink raw reply related [flat|nested] 4+ messages in thread* [PATCH 3/3] toaster: Don't def a function for each call to build_artifact()
2015-09-11 20:57 [PATCH 0/3] Read correct cooker log brian avery
2015-09-11 20:57 ` [PATCH 1/3] toaster: Read correct cooker log path from toasterui brian avery
2015-09-11 20:57 ` [PATCH 2/3] toaster: Avoid unnecessary local to local copy of cooker log brian avery
@ 2015-09-11 20:57 ` brian avery
2 siblings, 0 replies; 4+ messages in thread
From: brian avery @ 2015-09-11 20:57 UTC (permalink / raw)
To: bitbake-devel
From: Elliot Smith <elliot.smith@intel.com>
Cache the mimetype object and only define the function for
getting a mimetype once.
Also ensure that filemagic is listed as a requirement of
toaster. Doing this also means we can remove the code
which tries multiple different "magic" libraries, as we know
we have the right version available.
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
---
lib/toaster/toastergui/views.py | 42 ++++++++++-------------------------------
toaster-requirements.txt | 1 +
2 files changed, 11 insertions(+), 32 deletions(-)
diff --git a/lib/toaster/toastergui/views.py b/lib/toaster/toastergui/views.py
index 784272f..e918b05 100755
--- a/lib/toaster/toastergui/views.py
+++ b/lib/toaster/toastergui/views.py
@@ -47,10 +47,19 @@ import json
from os.path import dirname
import itertools
+import magic
import logging
logger = logging.getLogger("toaster")
+class MimeTypeFinder(object):
+ _magic = magic.Magic(flags = magic.MAGIC_MIME_TYPE)
+
+ # returns the mimetype for a file path
+ @classmethod
+ def get_mimetype(self, path):
+ return self._magic.id_filename(path)
+
# all new sessions should come through the landing page;
# determine in which mode we are running in, and redirect appropriately
def landing(request):
@@ -68,8 +77,6 @@ def landing(request):
return render(request, 'landing.html', context)
-
-
# returns a list for most recent builds;
def _get_latest_builds(prj=None):
queryset = Build.objects.all()
@@ -2710,40 +2717,11 @@ if True:
def build_artifact(request, build_id, artifact_type, artifact_id):
if artifact_type in ["cookerlog"]:
- def _mimetype_for_artifact(path):
- try:
- import magic
-
- # fair warning: this is a mess; there are multiple competing and incompatible
- # magic modules floating around, so we try some of the most common combinations
-
- try: # we try ubuntu's python-magic 5.4
- m = magic.open(magic.MAGIC_MIME_TYPE)
- m.load()
- return m.file(path)
- except AttributeError:
- pass
-
- try: # we try python-magic 0.4.6
- m = magic.Magic(magic.MAGIC_MIME)
- return m.from_file(path)
- except AttributeError:
- pass
-
- try: # we try pip filemagic 1.6
- m = magic.Magic(flags=magic.MAGIC_MIME_TYPE)
- return m.id_filename(path)
- except AttributeError:
- pass
-
- return "binary/octet-stream"
- except ImportError:
- return "binary/octet-stream"
try:
build = Build.objects.get(pk = build_id)
file_name = build.cooker_log_path
fsock = open(file_name, "r")
- content_type = _mimetype_for_artifact(file_name)
+ content_type = MimeTypeFinder.get_mimetype(file_name)
response = HttpResponse(fsock, content_type = content_type)
diff --git a/toaster-requirements.txt b/toaster-requirements.txt
index 19b5293..1d92d5e 100644
--- a/toaster-requirements.txt
+++ b/toaster-requirements.txt
@@ -2,3 +2,4 @@ Django==1.6
South==0.8.4
argparse==1.2.1
wsgiref==0.1.2
+filemagic==1.6
--
1.9.1
^ permalink raw reply related [flat|nested] 4+ messages in thread