[Buildroot] [PATCH v2 1/3] autobuild-run: create reason file on build failures

[Buildroot] [PATCH v2 1/3] autobuild-run: create reason file on build failures

[Atharva Lele]

When a build fails, we calculate the reason of failure on
the server side as well as client side. To solve this redundancy
we only calculate on the client side and submit a file which
contains the reason of failure.

Signed-off-by: Atharva Lele <itsatharva@gmail.com>
---
NOTE: This series depends on work from builder-class series

Changes v1 -> v2:
  - Split commit into two, one for adding reason file and one for
    accounting for reproducibility failures (next patch).
---
 scripts/autobuild-run | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/scripts/autobuild-run b/scripts/autobuild-run
index 92afb26..c18d8aa 100755
--- a/scripts/autobuild-run
+++ b/scripts/autobuild-run
@@ -596,6 +596,11 @@ class Builder:
             # not found
             return None
 
+        reason = get_failure_reason()
+        if reason:
+            with open(os.path.join(self.resultdir, "reason"), "w+") as reasonf:
+                reasonf.write("-".join(reason))
+
         def extract_end_log(resultfile):
             """Save the last part of the build log, starting from the failed package"""
 
@@ -604,7 +609,6 @@ class Builder:
                                 (os.path.join(self.outputdir, "logfile"), resultfile)],
                                 shell=True)
 
-            reason = get_failure_reason()
             if not reason:
                 extract_last_500_lines()
             else:
@@ -628,7 +632,6 @@ class Builder:
         def copy_config_log_files():
             """Recursively copy any config.log files from the failing package"""
 
-            reason = get_failure_reason()
             if not reason:
                 return
 
-- 
2.20.1

[Buildroot] [PATCH v2 2/3] autobuild-run: account for reproducibility failures when creating the reason file

[Atharva Lele]

Signed-off-by: Atharva Lele <itsatharva@gmail.com>
---
Changes v1 -> v2:
  - Split into separate commit for reproducibility failures
---
 scripts/autobuild-run | 33 +++++++++++++++++++++++----------
 1 file changed, 23 insertions(+), 10 deletions(-)

diff --git a/scripts/autobuild-run b/scripts/autobuild-run
index c18d8aa..6068376 100755
--- a/scripts/autobuild-run
+++ b/scripts/autobuild-run
@@ -582,16 +582,26 @@ class Builder:
             return
 
         def get_failure_reason():
-            # Output is a tuple (package, version), or None.
-            lastlines = decode_bytes(subprocess.Popen(
-                ["tail", "-n", "3", os.path.join(self.outputdir, "logfile")],
-                stdout=subprocess.PIPE).communicate()[0]).splitlines()
+            # Output is a tuple (package, version), or None in case of package failure
+            # Output is "reproducible" in case of reproducibility failure
+
+            reproducible_results = os.path.join(self.resultdir, "reproducible_results")
+            if os.path.exists(reproducible_results):
+                if os.stat(reproducible_results).st_size > 0:
+                    reason = "reproducible"
+                    return reason
+                else:
+                    return None
+            else:
+                lastlines = decode_bytes(subprocess.Popen(
+                    ["tail", "-n", "3", os.path.join(self.outputdir, "logfile")],
+                    stdout=subprocess.PIPE).communicate()[0]).splitlines()
 
-            regexp = re.compile(r'make: \*\*\* .*/(?:build|toolchain)/([^/]*)/')
-            for line in lastlines:
-                m = regexp.search(line)
-                if m:
-                    return m.group(1).rsplit('-', 1)
+                regexp = re.compile(r'make: \*\*\* .*/(?:build|toolchain)/([^/]*)/')
+                for line in lastlines:
+                    m = regexp.search(line)
+                    if m:
+                        return m.group(1).rsplit('-', 1)
 
             # not found
             return None
@@ -599,7 +609,10 @@ class Builder:
         reason = get_failure_reason()
         if reason:
             with open(os.path.join(self.resultdir, "reason"), "w+") as reasonf:
-                reasonf.write("-".join(reason))
+                if reason == "reproducible":
+                    reasonf.write(reason)
+                else:
+                    reasonf.write("-".join(reason))
 
         def extract_end_log(resultfile):
             """Save the last part of the build log, starting from the failed package"""
-- 
2.20.1

[Buildroot] [PATCH v2 3/3] web/import.inc.php: support reading failure reason from reason file

[Atharva Lele]

Signed-off-by: Atharva Lele <itsatharva@gmail.com>
---
NOTE: This patch is untested.
---
 web/import.inc.php | 26 +++++++++++++++-----------
 1 file changed, 15 insertions(+), 11 deletions(-)

diff --git a/web/import.inc.php b/web/import.inc.php
index a9b368f..6f878e0 100644
--- a/web/import.inc.php
+++ b/web/import.inc.php
@@ -230,17 +230,21 @@ function import_result($buildid, $filename)
     if ($status == 0)
       $reason = "none";
     else {
-	$tmp = Array();
-	exec("tail -3 " . $thisbuildfinaldir . "build-end.log | grep -v '\[_all\]' | grep 'make.*: \*\*\*' | sed 's,.*\[\([^\]*\)\] Error.*,\\1,' | sed 's,.*/build/\([^/]*\)/.*,\\1,'", $tmp);
-	if (trim($tmp[0]))
-	  $reason = $tmp[0];
-	else {
-	  exec("tail -1 " . $thisbuildfinaldir . "build-time.log | grep :start: | cut -d':' -f4", $tmp);
-	  if (trim($tmp[0]))
-	    $reason = trim($tmp[0]);
-	  else
-	    $reason = "unknown";
-	}
+      if (file_exists($thisbuildfinaldir . "reason"))
+        $reason = trim(file_get_contents($thisbuildfinaldir . "reason", "r"));
+      else {
+        $tmp = Array();
+        exec("tail -3 " . $thisbuildfinaldir . "build-end.log | grep -v '\[_all\]' | grep 'make.*: \*\*\*' | sed 's,.*\[\([^\]*\)\] Error.*,\\1,' | sed 's,.*/build/\([^/]*\)/.*,\\1,'", $tmp);
+        if (trim($tmp[0]))
+          $reason = $tmp[0];
+        else {
+          exec("tail -1 " . $thisbuildfinaldir . "build-time.log | grep :start: | cut -d':' -f4", $tmp);
+          if (trim($tmp[0]))
+            $reason = trim($tmp[0]);
+          else
+            $reason = "unknown";
+        }
+      }
     }
 
     /* Compress files that are typically too large and infrequently
-- 
2.20.1

[Buildroot] [PATCH v2 1/3] autobuild-run: create reason file on build failures

[Thomas Petazzoni]

Hello Atharva,

Thanks for the second iteration. One question below.

On Sun, 23 Jun 2019 00:11:16 +0530
Atharva Lele <itsatharva@gmail.com> wrote:

> +        reason = get_failure_reason()
> +        if reason:
> +            with open(os.path.join(self.resultdir, "reason"), "w+") as reasonf:
> +                reasonf.write("-".join(reason))

Why do we have this "-".join(reason) instead of just writing the
contents of the "reason" variable to the file ?

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH v2 1/3] autobuild-run: create reason file on build failures

[Atharva Lele]

On Sat, Jun 22, 2019 at 1:50 PM Thomas Petazzoni <
thomas.petazzoni@bootlin.com> wrote:

>
> Why do we have this "-".join(reason) instead of just writing the
> contents of the "reason" variable to the file ?
>

The reason to do that is to achieve the same formatting for the failure
reason
as the PHP script that is "package-version"

Regards,
Atharva Lele
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20190622/2507a460/attachment.html>