[PATCH 0/2] Fix cooker log download button

[PATCH 0/2] Fix cooker log download button

[Ed Bartosh]

* Retrieve a build's log file location directly from the file system,
as the one which is available at the time of the BuildStarted event
is incorrect.

* Hide the "Download build log" button for builds which fail early
(e.g. due to recipe parsing) and consequently have no cooker_log_path.


The following changes since commit 34399e5950509ffe978d9e6a31e91193d1bdcf0a:

  toaster: orm Machines filter don't pass self in as parameter (2015-10-06 20:08:50 -0700)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib ed/toaster/submit/elliot/toaster/wrong_log-8373
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/toaster/submit/elliot/toaster/wrong_log-8373

Elliot Smith (2):
  toaster: Manually retrieve log file location from filesystem
  toaster: Hide "Download build log" button if log doesn't exist

 bitbake/lib/bb/ui/toasterui.py                     | 31 +++++++++++++++++++++-
 .../toastergui/templates/builddashboard.html       |  4 ++-
 2 files changed, 33 insertions(+), 2 deletions(-)

--
Ed

[PATCH 1/2] toaster: Manually retrieve log file location from filesystem

[Ed Bartosh]

From: Elliot Smith <elliot.smith@intel.com>

The log file location reported by the BB_CONSOLELOG variable
does not point to the log location for the current build at
the time when the BuildStarted event is fired. It
actually points to the location where the next build will
log to. This means that the log file paths associated with
a build in the cooker_log_path field are incorrect, with
the result that the "Download build log" button doesn't work.

Instead, when a build starts, get the latest-dated log file
and associate it with the build.

An issue explaining why this is a problem, plus steps to
demonstrate it:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=8411

This patch is a temporary workaround for issue 8411,
as discussed in
https://bugzilla.yoctoproject.org/show_bug.cgi?id=8373#c2
It fixes 8373 for now, but should be properly fixed if
bitbake can provide the correct log location at the point
when BuildStarted is fired.

[YOCTO #8373]

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/bb/ui/toasterui.py | 31 ++++++++++++++++++++++++++++++-
 1 file changed, 30 insertions(+), 1 deletion(-)

diff --git a/bitbake/lib/bb/ui/toasterui.py b/bitbake/lib/bb/ui/toasterui.py
index dbe0d09..e3d3b8f 100644
--- a/bitbake/lib/bb/ui/toasterui.py
+++ b/bitbake/lib/bb/ui/toasterui.py
@@ -33,6 +33,7 @@ from bb.ui.buildinfohelper import BuildInfoHelper
 import bb.msg
 import logging
 import os
+import glob
 
 # pylint: disable=invalid-name
 # module properties for UI modules are read by bitbake and the contract should not be broken
@@ -61,6 +62,33 @@ def _log_settings_from_server(server):
         raise BaseException(error)
     return includelogs, loglines, consolelogfile
 
+# TODO this is a work-around for the bug mentioned in
+# https://bugzilla.yoctoproject.org/show_bug.cgi?id=8373#c2
+# - the log file location reported by BB_CONSOLELOG at the point
+# when the BuildStarted event occurs does not correspond to the
+# actual log file location;
+# as a work-around, we get the latest-dated log file in the same
+# directory as is in the path stored in BB_CONSOLELOG
+#
+# consolelogfile: log file location reported by BB_CONSOLELOG
+# at the point when a build starts, which isn't the actual log location;
+# the assumption is that the real log file will be in the same
+# directory as the reported log file
+#
+# returns the latest-dated *.log file in the same directory
+# as consolelogfile; if there are no *.log files, this returns
+# consolelogfile
+def _get_real_log_file(consolelogfile):
+    log_file_dir = os.path.dirname(consolelogfile)
+    log_file_pattern = os.path.join(log_file_dir, "*.log")
+    log_files = glob.glob(log_file_pattern)
+    log_files = filter(os.path.isfile, log_files)
+    log_files.sort(key=os.path.getmtime)
+
+    if len(log_files) > 0:
+        return log_files[-1]
+    else:
+        return consolelogfile
 
 def main(server, eventHandler, params ):
     helper = uihelper.BBUIHelper()
@@ -126,7 +154,8 @@ def main(server, eventHandler, params ):
             # the code will look into the protected variables of the event; no easy way around this
 
             if isinstance(event, bb.event.BuildStarted):
-                buildinfohelper.store_started_build(event, consolelogfile)
+                real_consolelogfile = _get_real_log_file(consolelogfile)
+                buildinfohelper.store_started_build(event, real_consolelogfile)
 
             if isinstance(event, (bb.build.TaskStarted, bb.build.TaskSucceeded, bb.build.TaskFailedSilent)):
                 buildinfohelper.update_and_store_task(event)
-- 
2.1.4

[PATCH 2/2] toaster: Hide "Download build log" button if log doesn't exist

[Ed Bartosh]

From: Elliot Smith <elliot.smith@intel.com>

Our builds pages show all builds, but also include build requests
which may have resulted in a build failure, before the build
started (e.g. at the recipe parsing stage).

In such cases, the BuildStarted event is not captured by Toaster,
so we have no idea where the log file for the failed build is.

The result is that a build is shown by the Toaster UI /builds/ pages,
but it is really a pretend build which never went beyond being a
build request, and which has no associated log file. In turn, this
breaks the "Download build log" button on the build dashboard,
as there's no log file associated with the build.

Fix this by hiding the "Download build log" button for builds
which don't have a cooker_log_path.

[YOCTO #8373]

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/toaster/toastergui/templates/builddashboard.html | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/bitbake/lib/toaster/toastergui/templates/builddashboard.html b/bitbake/lib/toaster/toastergui/templates/builddashboard.html
index bab8e38..aa99134 100644
--- a/bitbake/lib/toaster/toastergui/templates/builddashboard.html
+++ b/bitbake/lib/toaster/toastergui/templates/builddashboard.html
@@ -37,7 +37,9 @@
     <span > <i class="icon-warning-sign yellow"></i><strong><a href="#warnings" class="warning show-warnings"> {{build.warnings.count}} warning{{build.warnings.count|pluralize}}</a></strong></span>
 {% endif %}
             <span class="pull-right">Build time: <a href="{% url 'buildtime' build.pk %}">{{ build.timespent_seconds|sectohms }}</a>
-            <a class="btn {%if build.outcome == build.SUCCEEDED%}btn-success{%else%}btn-danger{%endif%} pull-right log" href="{% url 'build_artifact' build.id "cookerlog" build.id %}">Download build log</a>
+            {% if build.cooker_log_path %}
+                <a class="btn {%if build.outcome == build.SUCCEEDED%}btn-success{%else%}btn-danger{%endif%} pull-right log" href="{% url 'build_artifact' build.id "cookerlog" build.id %}">Download build log</a>
+            {% endif %}
             </span>
 
 {%endif%}
-- 
2.1.4

Re: [PATCH 1/2] toaster: Manually retrieve log file location from filesystem

[Richard Purdie]

On Wed, 2015-10-07 at 13:20 +0300, Ed Bartosh wrote:
> From: Elliot Smith <elliot.smith@intel.com>
> 
> The log file location reported by the BB_CONSOLELOG variable
> does not point to the log location for the current build at
> the time when the BuildStarted event is fired. It
> actually points to the location where the next build will
> log to. This means that the log file paths associated with
> a build in the cooker_log_path field are incorrect, with
> the result that the "Download build log" button doesn't work.
> 
> Instead, when a build starts, get the latest-dated log file
> and associate it with the build.
> 
> An issue explaining why this is a problem, plus steps to
> demonstrate it:
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=8411

Rather than some of the proposed workarounds, I'd rather like to try and
fix this properly. I've put some quick analysis into the above bug and I
think it shouldn't be too difficult to fix. If anyone wants to test and
send a patch I'd be more than happy for that as I'm travelling at the
moment (doing this form the airport) but I'll try and take a look later
today if I get the change.

Cheers,

Richard