[Buildroot] [PATCH 0/2] [AUTOBUILD] Store package files list in results (branch yem/files-list)

[Buildroot] [PATCH 0/2] [AUTOBUILD] Store package files list in results (branch yem/files-list)

[Yann E. MORIN]

Hello All!

This small series against the autobuilder code, stores the packages
files lists in autobuild results, so that it is easier to do a quick
analysis about the origin of files in a build.

Regards,
Yann E. MORIN.


The following changes since commit e90dc8753f9deaca9ecb41a5874d7a559ee2557c

  utils/daily-mail: sort overall summary in global e-mail by branch name (2019-03-15 15:19:36 +0100)


are available in the git repository at:

  /home/ymorin/cache/upstream/buildroot/buildroot-test

for you to fetch changes up to 1414cf2855df133bd02c1a32d8ebe3e632f37734

  [AUTOBUILD] autobuild-run: also copy packages files lists if they exist (2019-03-31 19:30:26 +0200)


----------------------------------------------------------------
Yann E. MORIN (2):
      [AUTOBUILD] autobuild-run: add helper function to copy-if-exists
      [AUTOBUILD] autobuild-run: also copy packages files lists if they exist

 scripts/autobuild-run | 17 +++++++++++------
 1 file changed, 11 insertions(+), 6 deletions(-)

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 1/2] [AUTOBUILD] autobuild-run: add helper function to copy-if-exists

[Yann E. MORIN]

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 scripts/autobuild-run | 14 ++++++++------
 1 file changed, 8 insertions(+), 6 deletions(-)

diff --git a/scripts/autobuild-run b/scripts/autobuild-run
index 3d2e99a..b65d8fb 100755
--- a/scripts/autobuild-run
+++ b/scripts/autobuild-run
@@ -478,12 +478,14 @@ def send_results(result, **kwargs):
                     os.path.join(resultdir, "defconfig"))
     shutil.copyfile(os.path.join(outputdir, "branch"),
                     os.path.join(resultdir, "branch"))
-    if os.path.exists(os.path.join(outputdir, "build", "build-time.log")):
-        shutil.copyfile(os.path.join(outputdir, "build", "build-time.log"),
-                        os.path.join(resultdir, "build-time.log"))
-    if os.path.exists(os.path.join(outputdir, "legal-info", "manifest.csv")):
-        shutil.copyfile(os.path.join(outputdir, "legal-info", "manifest.csv"),
-                        os.path.join(resultdir, "licenses-manifest.csv"))
+
+    def copy_if_exists(dir, src, dst=None):
+        if os.path.exists(os.path.join(outputdir, d, src)):
+            shutil.copyfile(os.path.join(outputdir, d, src),
+                            os.path.join(resultdir, src if dst is None else dst))
+
+    copy_if_exists("build", "build-time.log")
+    copy_if_exists("legal-info", "manifest.csv", "licenses-manifest.csv")
 
     subprocess.call(["git log -n 1 --pretty=format:%%H > %s" % \
                      os.path.join(resultdir, "gitid")],
-- 
2.14.1

[Buildroot] [PATCH 2/2] [AUTOBUILD] autobuild-run: also copy packages files lists if they exist

[Yann E. MORIN]

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 scripts/autobuild-run | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/scripts/autobuild-run b/scripts/autobuild-run
index b65d8fb..3523772 100755
--- a/scripts/autobuild-run
+++ b/scripts/autobuild-run
@@ -485,6 +485,9 @@ def send_results(result, **kwargs):
                             os.path.join(resultdir, src if dst is None else dst))
 
     copy_if_exists("build", "build-time.log")
+    copy_if_exists("build", "packages-file-list.txt")
+    copy_if_exists("build", "packages-file-list-host.txt")
+    copy_if_exists("build", "packages-file-list-staging.txt")
     copy_if_exists("legal-info", "manifest.csv", "licenses-manifest.csv")
 
     subprocess.call(["git log -n 1 --pretty=format:%%H > %s" % \
-- 
2.14.1

[Buildroot] [PATCH 0/2] [AUTOBUILD] Store package files list in results (branch yem/files-list)

[Thomas Petazzoni]

Hello Yann,

On Sun, 31 Mar 2019 19:48:20 +0200
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> Yann E. MORIN (2):
>       [AUTOBUILD] autobuild-run: add helper function to copy-if-exists
>       [AUTOBUILD] autobuild-run: also copy packages files lists if they exist

Both applied. I hope those additional files will not be too big: we
keep all those files for all build results, so it can add up quite a
bit over time. The build results currently take 64 GB of disk space,
and require 1303600 files.

Thanks,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 0/2] [AUTOBUILD] Store package files list in results (branch yem/files-list)

[Yann E. MORIN]

Thomas, All,

On 2019-04-03 23:13 +0200, Thomas Petazzoni spake thusly:
> On Sun, 31 Mar 2019 19:48:20 +0200
> "Yann E. MORIN" <yann.morin.1998@free.fr> wrote:
> 
> > Yann E. MORIN (2):
> >       [AUTOBUILD] autobuild-run: add helper function to copy-if-exists
> >       [AUTOBUILD] autobuild-run: also copy packages files lists if they exist
> 
> Both applied. I hope those additional files will not be too big: we
> keep all those files for all build results, so it can add up quite a
> bit over time. The build results currently take 64 GB of disk space,
> and require 1303600 files.

I would expect those files to be of about the same size as the build.log
snippet we keep.

If it turns out to be a problem, then we can restrict that to only failed
builds.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 0/2] [AUTOBUILD] Store package files list in results (branch yem/files-list)

[Thomas Petazzoni]

On Wed, 3 Apr 2019 23:30:12 +0200
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> I would expect those files to be of about the same size as the build.log
> snippet we keep.
> 
> If it turns out to be a problem, then we can restrict that to only failed
> builds.

It is true that for failed builds, not much of the information we keep
is interesting. Perhaps just the .config/defconfig are interesting. The
build.log and other files are mainly useful to investigate issues.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com