[yocto-autobuilder][PATCH 0/4] autobuilder: make toaster eventlog creation optional

[yocto-autobuilder][PATCH 0/4] autobuilder: make toaster eventlog creation optional

[Bill Randle]

Currently, a toaster eventlog is generated for most all machine specific
images (e.g., nightly-arm, nightly-x86, etc.) and placed in the top-level
build directory. Having it in the top level dir causes inotify overflow in
bitbake, due to the large number of changes to the file.

This set of patches (1) make generation of the eventlog optional, by
having a user selection in the Forced Build web pages, (2) places the
eventlog in build/tmp/log, instead of build and (3) adds a timestamp to
the eventlog filename since multiple event logs can be generated during
a multi-image buildset and otherwise only the last one would otherwise
be saved.

[YOCTO #9884]

Bill Randle (4):
  remote_kick.py: add default entry for new 'create_eventlog' property
  BuildImages.py: build toaster eventlog only when requested
  UploadToasterEventlog.py: handle multiple eventlog files
  buildset-config.controller: add new 'create_eventlog' forced build
    property

 bin/remote_kick.py                                 |  1 +
 buildset-config.controller/nightly-arm-lsb.conf    |  4 ++++
 buildset-config.controller/nightly-arm.conf        |  4 ++++
 buildset-config.controller/nightly-arm64.conf      |  4 ++++
 buildset-config.controller/nightly-mips-lsb.conf   |  4 ++++
 buildset-config.controller/nightly-mips.conf       |  4 ++++
 buildset-config.controller/nightly-mips64.conf     |  4 ++++
 buildset-config.controller/nightly-no-x11.conf     |  4 ++++
 buildset-config.controller/nightly-ppc-lsb.conf    |  4 ++++
 buildset-config.controller/nightly-ppc.conf        |  4 ++++
 buildset-config.controller/nightly-x32.conf        |  4 ++++
 buildset-config.controller/nightly-x86-64-lsb.conf |  4 ++++
 buildset-config.controller/nightly-x86-64.conf     |  4 ++++
 buildset-config.controller/nightly-x86-lsb.conf    |  4 ++++
 buildset-config.controller/nightly-x86.conf        |  4 ++++
 buildset-config.controller/nightly.conf            |  7 ++++++-
 .../autobuilder/buildsteps/BuildImages.py          | 24 +++++++++++++++++-----
 .../buildsteps/UploadToasterEventlog.py            | 16 +++++++++++-----
 18 files changed, 93 insertions(+), 11 deletions(-)

-- 
2.5.5

[yocto-autobuilder][PATCH 1/4] remote_kick.py: add default entry for new 'create_eventlog' property

[Bill Randle]

Default eventlog creating to False when starting an AB build session
from the command line.

[YOCTO #9884]

Signed-off-by: Bill Randle <william.c.randle@intel.com>
---
 bin/remote_kick.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/bin/remote_kick.py b/bin/remote_kick.py
index 20a7b7e..7320162 100644
--- a/bin/remote_kick.py
+++ b/bin/remote_kick.py
@@ -44,6 +44,7 @@ class RemoteBuild():
         kw['branch_eclipse-poky-luna'] = 'luna-master'
         kw['branch_eclipse-poky-mars'] = 'mars-master'
         kw['branch_oecore'] = 'master'
+        kw['custom_create_eventlog'] = 'False'
         kw['custom_deploy_artifacts'] = 'False'
         kw['custom_is_milestone'] = 'False'
         kw['custom_milestone_number'] = ''
-- 
2.5.5

[yocto-autobuilder][PATCH 2/4] BuildImages.py: build toaster eventlog only when requested

[Bill Randle]

Qualify creation of the toaster eventlog based on the setting of the new
'custom_create_eventlog' property. Place the eventlog in build/tmp/log,
instead of build and add a timestamp to the eventlog filename since
multiple event logs can be generated during a multi-image buildset and
otherwise only the last one would otherwise be saved.

[YOCTO #9884]

Signed-off-by: Bill Randle <william.c.randle@intel.com>
---
 .../autobuilder/buildsteps/BuildImages.py          | 24 +++++++++++++++++-----
 1 file changed, 19 insertions(+), 5 deletions(-)

diff --git a/lib/python2.7/site-packages/autobuilder/buildsteps/BuildImages.py b/lib/python2.7/site-packages/autobuilder/buildsteps/BuildImages.py
index 7ef8aab..9ea350f 100644
--- a/lib/python2.7/site-packages/autobuilder/buildsteps/BuildImages.py
+++ b/lib/python2.7/site-packages/autobuilder/buildsteps/BuildImages.py
@@ -15,7 +15,7 @@ from buildbot.steps.shell import ShellCommand
 from buildbot.process.buildstep import LogLineObserver
 from distutils.version import StrictVersion
 from buildbot.status.results import SUCCESS, SKIPPED
-import os
+import os, datetime
 
 from lib.buildsteps import BitbakeShellCommand
 
@@ -47,8 +47,23 @@ class BuildImages(BitbakeShellCommand):
             self.deploycheck = self.getProperty('custom_deploy_artifacts')
         except:
             self.deploycheck = "True"
+
+        # the eventlog capability exists only in bitbake 1.25 and newer
+        self.create_eventlog = "False"
+        if self.getProperty('bitbakeversion') \
+           and StrictVersion(self.getProperty('bitbakeversion')) >= StrictVersion("1.25"):
+            try:
+                self.create_eventlog = self.getProperty("custom_create_eventlog")
+                if self.create_eventlog == "True":
+                    timestamp = datetime.datetime.now().strftime("%Y%m%d%H%M%S")
+                    self.eventlog_name = "tmp/log/bitbake_eventlog-%s.json" % timestamp
+            except:
+                pass
+
         if self.images == "#TOASTER":
-            bitbakeflags = "-k -w 'bitbake_eventlog.json' "
+            bitbakeflags = "-k "
+            if self.create_eventlog == "True":
+                bitbakeflags += "-w '" + self.eventlog_name + "' "
             self.images=self.getProperty("custom_images")
             self.command = ". ./oe-init-build-env; bitbake " + bitbakeflags + self.images
             self.description = ["Building " + str(self.images)]
@@ -130,9 +145,8 @@ class BuildImages(BitbakeShellCommand):
                 self.description = ["Skipping Build. No Images to be built"]
             else:
                 bitbakeflags = "-k "
-                # -w only exists in bitbake 1.25 and newer, use distroversion string and make sure we're on poky >1.7
-                if self.getProperty('bitbakeversion') and StrictVersion(self.getProperty('bitbakeversion')) >= StrictVersion("1.25"):
-                    bitbakeflags += "-w 'bitbake_eventlog.json' "
+                if self.create_eventlog == "True":
+                    bitbakeflags += "-w '" + self.eventlog_name + "' "
                 if self.minnowExists is None or self.minnowExists == "True":
                     self.command = ". ./oe-init-build-env; bitbake " + bitbakeflags + self.images
                     self.description = ["Building " + str(self.images)]
-- 
2.5.5

[yocto-autobuilder][PATCH 3/4] UploadToasterEventlog.py: handle multiple eventlog files

[Bill Randle]

Now that multiple eventlog files can be generated, upload thyem all. Also,
only upload files in user requested an eventlog be generated.

[YOCTO #9884]

Signed-off-by: Bill Randle <william.c.randle@intel.com>
---
 .../autobuilder/buildsteps/UploadToasterEventlog.py      | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

diff --git a/lib/python2.7/site-packages/autobuilder/buildsteps/UploadToasterEventlog.py b/lib/python2.7/site-packages/autobuilder/buildsteps/UploadToasterEventlog.py
index ed1cb1c..0fbb899 100644
--- a/lib/python2.7/site-packages/autobuilder/buildsteps/UploadToasterEventlog.py
+++ b/lib/python2.7/site-packages/autobuilder/buildsteps/UploadToasterEventlog.py
@@ -34,17 +34,23 @@ class UploadToasterEventlog(ShellCommand):
         self.workerdir=os.path.join(os.path.join(YOCTO_ABBASE, "yocto-worker"))
         self.buildername=self.getProperty("buildername")
         self.layerversion_core = self.getProperty("layerversion_core")
-        self.filepath=os.path.join(os.path.join(
-                            self.workerdir, self.buildername),
-                            "build/build/bitbake_eventlog.json")
+        try:
+            self.create_eventlog = self.getProperty("custom_create_eventlog")
+        except:
+            self.create_eventlog = "False"
 
-        if int(self.layerversion_core) < 5:
+        if int(self.layerversion_core) < 5 or self.create_eventlog == "False":
             self.command = "echo 'Skipping Step.'"
         else:
             if os.environ.get('TOASTER_UPLOAD_URL') is not None:
+                self.filepath = os.path.join(os.path.join(
+                                self.workerdir, self.buildername),
+                                "build/build/tmp/log/bitbake_eventlog-*.json")
                 self.toasterserver = str(os.environ.get('TOASTER_UPLOAD_URL'))
 
-                self.command = "curl -F eventlog=@" + self.filepath + " " + self.toasterserver
+                self.command = "for fn in %s; do " % self.filepath
+                self.command += "curl -F eventlog=@$fn " + self.toasterserver + ";"
+                self.command += "done"
             else:
                 self.command = "echo environment var TOASTER_UPLOADURL not set - Step is noop"
             self.description = ["Uploading toaster data"]
-- 
2.5.5

[yocto-autobuilder][PATCH 4/4] buildset-config.controller: add new 'create_eventlog' forced build property

[Bill Randle]

Add a new 'create_eventlog' custom property to the configurations that
could potentially create a toaster event log. In the case of "nightly.conf",
ensure the property is passed to the Triggered builds.

[YOCTO #9884]

Signed-off-by: Bill Randle <william.c.randle@intel.com>
---
 buildset-config.controller/nightly-arm-lsb.conf    | 4 ++++
 buildset-config.controller/nightly-arm.conf        | 4 ++++
 buildset-config.controller/nightly-arm64.conf      | 4 ++++
 buildset-config.controller/nightly-mips-lsb.conf   | 4 ++++
 buildset-config.controller/nightly-mips.conf       | 4 ++++
 buildset-config.controller/nightly-mips64.conf     | 4 ++++
 buildset-config.controller/nightly-no-x11.conf     | 4 ++++
 buildset-config.controller/nightly-ppc-lsb.conf    | 4 ++++
 buildset-config.controller/nightly-ppc.conf        | 4 ++++
 buildset-config.controller/nightly-x32.conf        | 4 ++++
 buildset-config.controller/nightly-x86-64-lsb.conf | 4 ++++
 buildset-config.controller/nightly-x86-64.conf     | 4 ++++
 buildset-config.controller/nightly-x86-lsb.conf    | 4 ++++
 buildset-config.controller/nightly-x86.conf        | 4 ++++
 buildset-config.controller/nightly.conf            | 7 ++++++-
 15 files changed, 62 insertions(+), 1 deletion(-)

diff --git a/buildset-config.controller/nightly-arm-lsb.conf b/buildset-config.controller/nightly-arm-lsb.conf
index eb8d3eb..f94db21 100644
--- a/buildset-config.controller/nightly-arm-lsb.conf
+++ b/buildset-config.controller/nightly-arm-lsb.conf
@@ -10,6 +10,10 @@ repos: [{'poky':
         {'meta-qt3':
             {'repourl':'git://git.yoctoproject.org/meta-qt3',
              'branch':'master'}}]
+props: [{'create_eventlog':{'prop_type':'ChoiceStringParameter',
+                       'choices': ['False', 'True'],
+                       'name': 'create_eventlog',
+                       'label':'<hr>Create Toaster event log as part of image creation?:'}}]
 steps: [{'SetDest':{}},
         {'CheckOutLayers': {}},
         {'RunPreamble': {}},
diff --git a/buildset-config.controller/nightly-arm.conf b/buildset-config.controller/nightly-arm.conf
index 2e6c829..f4c3fb8 100644
--- a/buildset-config.controller/nightly-arm.conf
+++ b/buildset-config.controller/nightly-arm.conf
@@ -4,6 +4,10 @@ repos: [{'poky':
             {'repourl':'git://git.yoctoproject.org/poky',
              'layerversion':{'core':'meta', 'yoctobsp':'meta-yocto-bsp', 'yocto':'meta-yocto', 'poky':'meta-poky'},
              'branch':'master'}}]
+props: [{'create_eventlog':{'prop_type':'ChoiceStringParameter',
+                       'choices': ['False', 'True'],
+                       'name': 'create_eventlog',
+                       'label':'<hr>Create Toaster event log as part of image creation?:'}}]
 steps: [{'SetDest':{}},
         {'CheckOutLayers': {}},
         {'RunPreamble': {}},
diff --git a/buildset-config.controller/nightly-arm64.conf b/buildset-config.controller/nightly-arm64.conf
index 32a5492..3986c93 100644
--- a/buildset-config.controller/nightly-arm64.conf
+++ b/buildset-config.controller/nightly-arm64.conf
@@ -4,6 +4,10 @@ repos: [{'poky':
             {'repourl':'git://git.yoctoproject.org/poky',
                 'layerversion':{'core':'meta', 'yoctobsp':'meta-yocto-bsp', 'yocto':'meta-yocto', 'poky':'meta-poky'},
              'branch':'master'}}]
+props: [{'create_eventlog':{'prop_type':'ChoiceStringParameter',
+                       'choices': ['False', 'True'],
+                       'name': 'create_eventlog',
+                       'label':'<hr>Create Toaster event log as part of image creation?:'}}]
 steps: [{'SetDest':{}},
         {'CheckOutLayers': {}},
         {'RunPreamble': {}},
diff --git a/buildset-config.controller/nightly-mips-lsb.conf b/buildset-config.controller/nightly-mips-lsb.conf
index 81faaad..65fb4ec 100644
--- a/buildset-config.controller/nightly-mips-lsb.conf
+++ b/buildset-config.controller/nightly-mips-lsb.conf
@@ -10,6 +10,10 @@ repos: [{'poky':
         {'meta-qt3':
             {'repourl':'git://git.yoctoproject.org/meta-qt3',
              'branch':'master'}}]
+props: [{'create_eventlog':{'prop_type':'ChoiceStringParameter',
+                       'choices': ['False', 'True'],
+                       'name': 'create_eventlog',
+                       'label':'<hr>Create Toaster event log as part of image creation?:'}}]
 steps: [{'SetDest':{}},
         {'CheckOutLayers': {}},
         {'RunPreamble': {}},
diff --git a/buildset-config.controller/nightly-mips.conf b/buildset-config.controller/nightly-mips.conf
index e419ce4..1dbad7e 100644
--- a/buildset-config.controller/nightly-mips.conf
+++ b/buildset-config.controller/nightly-mips.conf
@@ -4,6 +4,10 @@ repos: [{'poky':
             {'repourl':'git://git.yoctoproject.org/poky',
              'layerversion':{'core':'meta', 'yoctobsp':'meta-yocto-bsp', 'yocto':'meta-yocto', 'poky':'meta-poky'},
              'branch':'master'}}]
+props: [{'create_eventlog':{'prop_type':'ChoiceStringParameter',
+                       'choices': ['False', 'True'],
+                       'name': 'create_eventlog',
+                       'label':'<hr>Create Toaster event log as part of image creation?:'}}]
 steps: [{'SetDest':{}},
         {'CheckOutLayers': {}},
         {'RunPreamble': {}},
diff --git a/buildset-config.controller/nightly-mips64.conf b/buildset-config.controller/nightly-mips64.conf
index 13f8573..73cb577 100644
--- a/buildset-config.controller/nightly-mips64.conf
+++ b/buildset-config.controller/nightly-mips64.conf
@@ -4,6 +4,10 @@ repos: [{'poky':
             {'repourl':'git://git.yoctoproject.org/poky',
              'layerversion':{'core':'meta', 'yoctobsp':'meta-yocto-bsp', 'yocto':'meta-yocto', 'poky':'meta-poky'},
              'branch':'master'}}]
+props: [{'create_eventlog':{'prop_type':'ChoiceStringParameter',
+                       'choices': ['False', 'True'],
+                       'name': 'create_eventlog',
+                       'label':'<hr>Create Toaster event log as part of image creation?:'}}]
 steps: [{'SetDest':{}},
         {'CheckOutLayers': {}},
         {'RunPreamble': {}},
diff --git a/buildset-config.controller/nightly-no-x11.conf b/buildset-config.controller/nightly-no-x11.conf
index 4ef1b35..063ae89 100644
--- a/buildset-config.controller/nightly-no-x11.conf
+++ b/buildset-config.controller/nightly-no-x11.conf
@@ -4,6 +4,10 @@ repos: [{'poky':
             {'repourl':'git://git.yoctoproject.org/poky',
              'layerversion':{'core':'meta', 'yoctobsp':'meta-yocto-bsp', 'yocto':'meta-yocto', 'poky':'meta-poky'},
              'branch':'master'}}]
+props: [{'create_eventlog':{'prop_type':'ChoiceStringParameter',
+                       'choices': ['False', 'True'],
+                       'name': 'create_eventlog',
+                       'label':'<hr>Create Toaster event log as part of image creation?:'}}]
 steps: [{'SetDest':{}},
         {'CheckOutLayers': {}},
         {'RunPreamble': {}},
diff --git a/buildset-config.controller/nightly-ppc-lsb.conf b/buildset-config.controller/nightly-ppc-lsb.conf
index 7cf05c1..5944240 100644
--- a/buildset-config.controller/nightly-ppc-lsb.conf
+++ b/buildset-config.controller/nightly-ppc-lsb.conf
@@ -10,6 +10,10 @@ repos: [{'poky':
         {'meta-qt3':
             {'repourl':'git://git.yoctoproject.org/meta-qt3',
              'branch':'master'}}]
+props: [{'create_eventlog':{'prop_type':'ChoiceStringParameter',
+                       'choices': ['False', 'True'],
+                       'name': 'create_eventlog',
+                       'label':'<hr>Create Toaster event log as part of image creation?:'}}]
 steps: [{'SetDest':{}},
         {'CheckOutLayers': {}},
         {'RunPreamble': {}},
diff --git a/buildset-config.controller/nightly-ppc.conf b/buildset-config.controller/nightly-ppc.conf
index 16413c3..1e26307 100644
--- a/buildset-config.controller/nightly-ppc.conf
+++ b/buildset-config.controller/nightly-ppc.conf
@@ -4,6 +4,10 @@ repos: [{'poky':
             {'repourl':'git://git.yoctoproject.org/poky',
              'layerversion':{'core':'meta', 'yoctobsp':'meta-yocto-bsp', 'yocto':'meta-yocto', 'poky':'meta-poky'},
              'branch':'master'}}]
+props: [{'create_eventlog':{'prop_type':'ChoiceStringParameter',
+                       'choices': ['False', 'True'],
+                       'name': 'create_eventlog',
+                       'label':'<hr>Create Toaster event log as part of image creation?:'}}]
 steps: [{'SetDest':{}},
         {'CheckOutLayers': {}},
         {'RunPreamble': {}},
diff --git a/buildset-config.controller/nightly-x32.conf b/buildset-config.controller/nightly-x32.conf
index 2ec5b87..e811a30 100644
--- a/buildset-config.controller/nightly-x32.conf
+++ b/buildset-config.controller/nightly-x32.conf
@@ -4,6 +4,10 @@ repos: [{'poky':
             {'repourl':'git://git.yoctoproject.org/poky',
              'layerversion':{'core':'meta', 'yoctobsp':'meta-yocto-bsp', 'yocto':'meta-yocto', 'poky':'meta-poky'},
              'branch':'master'}}]
+props: [{'create_eventlog':{'prop_type':'ChoiceStringParameter',
+                       'choices': ['False', 'True'],
+                       'name': 'create_eventlog',
+                       'label':'<hr>Create Toaster event log as part of image creation?:'}}]
 steps: [{'SetDest':{}},
         {'CheckOutLayers': {}},
         {'RunPreamble': {}},
diff --git a/buildset-config.controller/nightly-x86-64-lsb.conf b/buildset-config.controller/nightly-x86-64-lsb.conf
index b0bb0a5..784f23d 100644
--- a/buildset-config.controller/nightly-x86-64-lsb.conf
+++ b/buildset-config.controller/nightly-x86-64-lsb.conf
@@ -10,6 +10,10 @@ repos: [{'poky':
         {'meta-qt3':
             {'repourl':'git://git.yoctoproject.org/meta-qt3',
              'branch':'master'}}]
+props: [{'create_eventlog':{'prop_type':'ChoiceStringParameter',
+                       'choices': ['False', 'True'],
+                       'name': 'create_eventlog',
+                       'label':'<hr>Create Toaster event log as part of image creation?:'}}]
 steps: [{'SetDest':{}},
         {'CheckOutLayers': {}},
         {'RunPreamble': {}},
diff --git a/buildset-config.controller/nightly-x86-64.conf b/buildset-config.controller/nightly-x86-64.conf
index 92e5866..93df577 100644
--- a/buildset-config.controller/nightly-x86-64.conf
+++ b/buildset-config.controller/nightly-x86-64.conf
@@ -4,6 +4,10 @@ repos: [{'poky':
             {'repourl':'git://git.yoctoproject.org/poky',
              'layerversion':{'core':'meta', 'yoctobsp':'meta-yocto-bsp', 'yocto':'meta-yocto', 'poky':'meta-poky'},
              'branch':'master'}}]
+props: [{'create_eventlog':{'prop_type':'ChoiceStringParameter',
+                       'choices': ['False', 'True'],
+                       'name': 'create_eventlog',
+                       'label':'<hr>Create Toaster event log as part of image creation?:'}}]
 steps: [{'SetDest':{}},
         {'CheckOutLayers': {}},
         {'RunPreamble': {}},
diff --git a/buildset-config.controller/nightly-x86-lsb.conf b/buildset-config.controller/nightly-x86-lsb.conf
index 34b6780..1381b4c 100644
--- a/buildset-config.controller/nightly-x86-lsb.conf
+++ b/buildset-config.controller/nightly-x86-lsb.conf
@@ -10,6 +10,10 @@ repos: [{'poky':
         {'meta-qt3':
             {'repourl':'git://git.yoctoproject.org/meta-qt3',
              'branch':'master'}}]
+props: [{'create_eventlog':{'prop_type':'ChoiceStringParameter',
+                       'choices': ['False', 'True'],
+                       'name': 'create_eventlog',
+                       'label':'<hr>Create Toaster event log as part of image creation?:'}}]
 steps: [{'SetDest':{}},
         {'CheckOutLayers': {}},
         {'RunPreamble': {}},
diff --git a/buildset-config.controller/nightly-x86.conf b/buildset-config.controller/nightly-x86.conf
index a5c9f0a..07fb34c 100644
--- a/buildset-config.controller/nightly-x86.conf
+++ b/buildset-config.controller/nightly-x86.conf
@@ -4,6 +4,10 @@ repos: [{'poky':
             {'repourl':'git://git.yoctoproject.org/poky',
              'layerversion':{'core':'meta', 'yoctobsp':'meta-yocto-bsp', 'yocto':'meta-yocto', 'poky':'meta-poky'},
              'branch':'master'}}]
+props: [{'create_eventlog':{'prop_type':'ChoiceStringParameter',
+                       'choices': ['False', 'True'],
+                       'name': 'create_eventlog',
+                       'label':'<hr>Create Toaster event log as part of image creation?:'}}]
 steps: [{'SetDest':{}},
         {'CheckOutLayers': {}},
         {'RunPreamble': {}},
diff --git a/buildset-config.controller/nightly.conf b/buildset-config.controller/nightly.conf
index 2ac83dc..0f9dd4e 100644
--- a/buildset-config.controller/nightly.conf
+++ b/buildset-config.controller/nightly.conf
@@ -74,6 +74,10 @@ props: [{'release_me':{'prop_type':'ChoiceStringParameter',
                        'name': 'prefered_kernel',
                        'default': 'linux-yocto',
                        'label':'<hr><h3> PREFERRED_PROVIDER_virtual/kernel (default is "linux-yocto"):</h3>'}},
+        {'create_eventlog':{'prop_type':'ChoiceStringParameter',
+                       'choices': ['False', 'True'],
+                       'name': 'create_eventlog',
+                       'label':'<hr><h3> Should a Toaster event log be created as part of the image creation?:</h3>'}},
         {'deploy_artifacts':{'prop_type':'ChoiceStringParameter',
                        'choices': ['True', 'False'],
                        'name': 'deploy_artifacts',
@@ -87,7 +91,8 @@ steps: [{'SetDest':{}},
                             'atextprepend': 'SOURCE_MIRROR_FETCH = "1"\n'}},
         {'CreateBBLayersConf': {'buildprovider' : 'yocto'}},
         {'BuildImages': {'images': 'universe -c fetch'}},
-        {'TriggerBuilds': {'schedulerName': 'main-build', 'waitForFinish': 'True',
+        {'TriggerBuilds': {'copy_properties': ['custom_create_eventlog'],
+                           'schedulerName': 'main-build', 'waitForFinish': 'True',
                            'schedulerNames': {'nightly-arm': {}, 'nightly-arm-lsb': {},
                            'nightly-mips': {}, 'nightly-mips-lsb': {},
                            'nightly-multilib': {}, 'nightly-x32': {},
-- 
2.5.5

Re: [yocto-autobuilder][PATCH 2/4] BuildImages.py: build toaster eventlog only when requested

[Joshua G Lock]

On Thu, 2016-07-14 at 10:05 -0700, Bill Randle wrote:
> Qualify creation of the toaster eventlog based on the setting of the
> new
> 'custom_create_eventlog' property. Place the eventlog in
> build/tmp/log,
> instead of build and add a timestamp to the eventlog filename since
> multiple event logs can be generated during a multi-image buildset
> and
> otherwise only the last one would otherwise be saved.
> 
> [YOCTO #9884]
> 
> Signed-off-by: Bill Randle <william.c.randle@intel.com>
> ---
>  .../autobuilder/buildsteps/BuildImages.py          | 24
> +++++++++++++++++-----
>  1 file changed, 19 insertions(+), 5 deletions(-)
> 
> diff --git a/lib/python2.7/site-
> packages/autobuilder/buildsteps/BuildImages.py b/lib/python2.7/site-
> packages/autobuilder/buildsteps/BuildImages.py
> index 7ef8aab..9ea350f 100644
> --- a/lib/python2.7/site-
> packages/autobuilder/buildsteps/BuildImages.py
> +++ b/lib/python2.7/site-
> packages/autobuilder/buildsteps/BuildImages.py
> @@ -15,7 +15,7 @@ from buildbot.steps.shell import ShellCommand
>  from buildbot.process.buildstep import LogLineObserver
>  from distutils.version import StrictVersion
>  from buildbot.status.results import SUCCESS, SKIPPED
> -import os
> +import os, datetime
>  
>  from lib.buildsteps import BitbakeShellCommand
>  
> @@ -47,8 +47,23 @@ class BuildImages(BitbakeShellCommand):
>              self.deploycheck =
> self.getProperty('custom_deploy_artifacts')
>          except:
>              self.deploycheck = "True"
> +
> +        # the eventlog capability exists only in bitbake 1.25 and
> newer
> +        self.create_eventlog = "False"
> +        if self.getProperty('bitbakeversion') \
> +           and StrictVersion(self.getProperty('bitbakeversion')) >=
> StrictVersion("1.25"):
> +            try:
> +                self.create_eventlog =
> self.getProperty("custom_create_eventlog")
> +                if self.create_eventlog == "True":
> +                    timestamp =
> datetime.datetime.now().strftime("%Y%m%d%H%M%S")
> +                    self.eventlog_name = "tmp/log/bitbake_eventlog-
> %s.json" % timestamp
> +            except:
> +                pass

Do we really want to catch all exceptions here and ignore them? What
exception might be thrown by the above code?

> +
>          if self.images == "#TOASTER":
> -            bitbakeflags = "-k -w 'bitbake_eventlog.json' "
> +            bitbakeflags = "-k "
> +            if self.create_eventlog == "True":
> +                bitbakeflags += "-w '" + self.eventlog_name + "' "
>              self.images=self.getProperty("custom_images")
>              self.command = ". ./oe-init-build-env; bitbake " +
> bitbakeflags + self.images
>              self.description = ["Building " + str(self.images)]
> @@ -130,9 +145,8 @@ class BuildImages(BitbakeShellCommand):
>                  self.description = ["Skipping Build. No Images to be
> built"]
>              else:
>                  bitbakeflags = "-k "
> -                # -w only exists in bitbake 1.25 and newer, use
> distroversion string and make sure we're on poky >1.7
> -                if self.getProperty('bitbakeversion') and
> StrictVersion(self.getProperty('bitbakeversion')) >=
> StrictVersion("1.25"):
> -                    bitbakeflags += "-w 'bitbake_eventlog.json' "
> +                if self.create_eventlog == "True":
> +                    bitbakeflags += "-w '" + self.eventlog_name + "'
> "
>                  if self.minnowExists is None or self.minnowExists ==
> "True":
>                      self.command = ". ./oe-init-build-env; bitbake "
> + bitbakeflags + self.images
>                      self.description = ["Building " +
> str(self.images)]

Re: [yocto-autobuilder][PATCH 2/4] BuildImages.py: build toaster eventlog only when requested

[Randle, William C]

[-- Attachment #1: Type: text/plain, Size: 3881 bytes --]

On Fri, 2016-07-15 at 22:37 +0100, Joshua G Lock wrote:

On Thu, 2016-07-14 at 10:05 -0700, Bill Randle wrote:


Qualify creation of the toaster eventlog based on the setting of the
new
'custom_create_eventlog' property. Place the eventlog in
build/tmp/log,
instead of build and add a timestamp to the eventlog filename since
multiple event logs can be generated during a multi-image buildset
and
otherwise only the last one would otherwise be saved.

[YOCTO #9884]

Signed-off-by: Bill Randle <william.c.randle@intel.com<mailto:william.c.randle@intel.com>>
---
 .../autobuilder/buildsteps/BuildImages.py          | 24
+++++++++++++++++-----
 1 file changed, 19 insertions(+), 5 deletions(-)

diff --git a/lib/python2.7/site-
packages/autobuilder/buildsteps/BuildImages.py b/lib/python2.7/site-
packages/autobuilder/buildsteps/BuildImages.py
index 7ef8aab..9ea350f 100644
--- a/lib/python2.7/site-
packages/autobuilder/buildsteps/BuildImages.py
+++ b/lib/python2.7/site-
packages/autobuilder/buildsteps/BuildImages.py
@@ -15,7 +15,7 @@ from buildbot.steps.shell import ShellCommand
 from buildbot.process.buildstep import LogLineObserver
 from distutils.version import StrictVersion
 from buildbot.status.results import SUCCESS, SKIPPED
-import os
+import os, datetime

 from lib.buildsteps import BitbakeShellCommand

@@ -47,8 +47,23 @@ class BuildImages(BitbakeShellCommand):
             self.deploycheck =
self.getProperty('custom_deploy_artifacts')
         except:
             self.deploycheck = "True"
+
+        # the eventlog capability exists only in bitbake 1.25 and
newer
+        self.create_eventlog = "False"
+        if self.getProperty('bitbakeversion') \
+           and StrictVersion(self.getProperty('bitbakeversion')) >=
StrictVersion("1.25"):
+            try:
+                self.create_eventlog =
self.getProperty("custom_create_eventlog")
+                if self.create_eventlog == "True":
+                    timestamp =
datetime.datetime.now().strftime("%Y%m%d%H%M%S")
+                    self.eventlog_name = "tmp/log/bitbake_eventlog-
%s.json" % timestamp
+            except:
+                pass



Do we really want to catch all exceptions here and ignore them? What
exception might be thrown by the above code?



The only exception I know of would be the case where the property is not defined because the code was invoked from a page that did not set it (not all pages do).



+
         if self.images == "#TOASTER":
-            bitbakeflags = "-k -w 'bitbake_eventlog.json' "
+            bitbakeflags = "-k "
+            if self.create_eventlog == "True":
+                bitbakeflags += "-w '" + self.eventlog_name + "' "
             self.images=self.getProperty("custom_images")
             self.command = ". ./oe-init-build-env; bitbake " +
bitbakeflags + self.images
             self.description = ["Building " + str(self.images)]
@@ -130,9 +145,8 @@ class BuildImages(BitbakeShellCommand):
                 self.description = ["Skipping Build. No Images to be
built"]
             else:
                 bitbakeflags = "-k "
-                # -w only exists in bitbake 1.25 and newer, use
distroversion string and make sure we're on poky >1.7
-                if self.getProperty('bitbakeversion') and
StrictVersion(self.getProperty('bitbakeversion')) >=
StrictVersion("1.25"):
-                    bitbakeflags += "-w 'bitbake_eventlog.json' "
+                if self.create_eventlog == "True":
+                    bitbakeflags += "-w '" + self.eventlog_name + "'
"
                 if self.minnowExists is None or self.minnowExists ==
"True":
                     self.command = ". ./oe-init-build-env; bitbake "
+ bitbakeflags + self.images
                     self.description = ["Building " +
str(self.images)]


[-- Attachment #2: Type: text/html, Size: 7279 bytes --]

Re: [yocto-autobuilder][PATCH 2/4] BuildImages.py: build toaster eventlog only when requested

[Randle, William C]

[-- Attachment #1: Type: text/plain, Size: 4451 bytes --]

On Fri, 2016-07-15 at 16:54 -0700, Randle, William C wrote:
On Fri, 2016-07-15 at 22:37 +0100, Joshua G Lock wrote:

On Thu, 2016-07-14 at 10:05 -0700, Bill Randle wrote:


Qualify creation of the toaster eventlog based on the setting of the
new
'custom_create_eventlog' property. Place the eventlog in
build/tmp/log,
instead of build and add a timestamp to the eventlog filename since
multiple event logs can be generated during a multi-image buildset
and
otherwise only the last one would otherwise be saved.

[YOCTO #9884]

Signed-off-by: Bill Randle <william.c.randle@intel.com<mailto:william.c.randle@intel.com>>
---
 .../autobuilder/buildsteps/BuildImages.py          | 24
+++++++++++++++++-----
 1 file changed, 19 insertions(+), 5 deletions(-)

diff --git a/lib/python2.7/site-
packages/autobuilder/buildsteps/BuildImages.py b/lib/python2.7/site-
packages/autobuilder/buildsteps/BuildImages.py
index 7ef8aab..9ea350f 100644
--- a/lib/python2.7/site-
packages/autobuilder/buildsteps/BuildImages.py
+++ b/lib/python2.7/site-
packages/autobuilder/buildsteps/BuildImages.py
@@ -15,7 +15,7 @@ from buildbot.steps.shell import ShellCommand
 from buildbot.process.buildstep import LogLineObserver
 from distutils.version import StrictVersion
 from buildbot.status.results import SUCCESS, SKIPPED
-import os
+import os, datetime

 from lib.buildsteps import BitbakeShellCommand

@@ -47,8 +47,23 @@ class BuildImages(BitbakeShellCommand):
             self.deploycheck =
self.getProperty('custom_deploy_artifacts')
         except:
             self.deploycheck = "True"
+
+        # the eventlog capability exists only in bitbake 1.25 and
newer
+        self.create_eventlog = "False"
+        if self.getProperty('bitbakeversion') \
+           and StrictVersion(self.getProperty('bitbakeversion')) >=
StrictVersion("1.25"):
+            try:
+                self.create_eventlog =
self.getProperty("custom_create_eventlog")
+                if self.create_eventlog == "True":
+                    timestamp =
datetime.datetime.now().strftime("%Y%m%d%H%M%S")
+                    self.eventlog_name = "tmp/log/bitbake_eventlog-
%s.json" % timestamp
+            except:
+                pass



Do we really want to catch all exceptions here and ignore them? What
exception might be thrown by the above code?



The only exception I know of would be the case where the property is not defined because the code was invoked from a page that did not set it (not all pages do).


To clarify a bit more, if you look at the full source file (BuildImages.py), you will see that standard way to check a property is:
    try:
       somevar = self.getProperty("propname")
    except:
       somevar = default_value

In the added code, it actually sets the default value first:
    self.create_eventlog = "False"
because there are several checks made. Since the default is already set, there's no reason to set it again in the except caluse, hence I just did a 'pass'.

    -Bill



+
         if self.images == "#TOASTER":
-            bitbakeflags = "-k -w 'bitbake_eventlog.json' "
+            bitbakeflags = "-k "
+            if self.create_eventlog == "True":
+                bitbakeflags += "-w '" + self.eventlog_name + "' "
             self.images=self.getProperty("custom_images")
             self.command = ". ./oe-init-build-env; bitbake " +
bitbakeflags + self.images
             self.description = ["Building " + str(self.images)]
@@ -130,9 +145,8 @@ class BuildImages(BitbakeShellCommand):
                 self.description = ["Skipping Build. No Images to be
built"]
             else:
                 bitbakeflags = "-k "
-                # -w only exists in bitbake 1.25 and newer, use
distroversion string and make sure we're on poky >1.7
-                if self.getProperty('bitbakeversion') and
StrictVersion(self.getProperty('bitbakeversion')) >=
StrictVersion("1.25"):
-                    bitbakeflags += "-w 'bitbake_eventlog.json' "
+                if self.create_eventlog == "True":
+                    bitbakeflags += "-w '" + self.eventlog_name + "'
"
                 if self.minnowExists is None or self.minnowExists ==
"True":
                     self.command = ". ./oe-init-build-env; bitbake "
+ bitbakeflags + self.images
                     self.description = ["Building " +
str(self.images)]


[-- Attachment #2: Type: text/html, Size: 8209 bytes --]

Re: [yocto-autobuilder][PATCH 0/4] autobuilder: make toaster eventlog creation optional

[Beth 'pidge' Flanagan]

Merged into master.

-b

On Thu, 2016-07-14 at 10:05 -0700, Bill Randle wrote:
> Currently, a toaster eventlog is generated for most all machine
> specific
> images (e.g., nightly-arm, nightly-x86, etc.) and placed in the top-
> level
> build directory. Having it in the top level dir causes inotify
> overflow in
> bitbake, due to the large number of changes to the file.
> 
> This set of patches (1) make generation of the eventlog optional, by
> having a user selection in the Forced Build web pages, (2) places the
> eventlog in build/tmp/log, instead of build and (3) adds a timestamp
> to
> the eventlog filename since multiple event logs can be generated
> during
> a multi-image buildset and otherwise only the last one would
> otherwise
> be saved.
> 
> [YOCTO #9884]
> 
> Bill Randle (4):
>   remote_kick.py: add default entry for new 'create_eventlog'
> property
>   BuildImages.py: build toaster eventlog only when requested
>   UploadToasterEventlog.py: handle multiple eventlog files
>   buildset-config.controller: add new 'create_eventlog' forced build
>     property
> 
>  bin/remote_kick.py                                 |  1 +
>  buildset-config.controller/nightly-arm-lsb.conf    |  4 ++++
>  buildset-config.controller/nightly-arm.conf        |  4 ++++
>  buildset-config.controller/nightly-arm64.conf      |  4 ++++
>  buildset-config.controller/nightly-mips-lsb.conf   |  4 ++++
>  buildset-config.controller/nightly-mips.conf       |  4 ++++
>  buildset-config.controller/nightly-mips64.conf     |  4 ++++
>  buildset-config.controller/nightly-no-x11.conf     |  4 ++++
>  buildset-config.controller/nightly-ppc-lsb.conf    |  4 ++++
>  buildset-config.controller/nightly-ppc.conf        |  4 ++++
>  buildset-config.controller/nightly-x32.conf        |  4 ++++
>  buildset-config.controller/nightly-x86-64-lsb.conf |  4 ++++
>  buildset-config.controller/nightly-x86-64.conf     |  4 ++++
>  buildset-config.controller/nightly-x86-lsb.conf    |  4 ++++
>  buildset-config.controller/nightly-x86.conf        |  4 ++++
>  buildset-config.controller/nightly.conf            |  7 ++++++-
>  .../autobuilder/buildsteps/BuildImages.py          | 24
> +++++++++++++++++-----
>  .../buildsteps/UploadToasterEventlog.py            | 16 +++++++++++-
> ----
>  18 files changed, 93 insertions(+), 11 deletions(-)
> 
-- 
Beth 'pidge' Flanagan <pidge@toganlabs.com>
toganlabs.com