[[PATCH][yocto-autobuilder] 1/2] buildsteps/CreateAutoConf.py: Add support for specify MACHINE by build prop

[[PATCH][yocto-autobuilder] 1/2] buildsteps/CreateAutoConf.py: Add support for specify MACHINE by build prop

[Aníbal Limón]

Sometimes is a good idea to could specify a machine to build via a build
property.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 .../site-packages/autobuilder/buildsteps/CreateAutoConf.py          | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/lib/python2.7/site-packages/autobuilder/buildsteps/CreateAutoConf.py b/lib/python2.7/site-packages/autobuilder/buildsteps/CreateAutoConf.py
index 021f542..7915dd5 100644
--- a/lib/python2.7/site-packages/autobuilder/buildsteps/CreateAutoConf.py
+++ b/lib/python2.7/site-packages/autobuilder/buildsteps/CreateAutoConf.py
@@ -174,7 +174,11 @@ class CreateAutoConf(ShellCommand):
                     fout = fout + 'baselib = \\042${@d.getVar(\\047BASE_LIB_tune-\\047 + (d.getVar(\\047DEFAULTTUNE\\047, True) or \\047INVALID\\047), True) or \\047lib\\047}\\042 \n'
             if self.distro == "poky-rt":
                 fout = fout + 'PREFERRED_PROVIDER_virtual/kernel="linux-yocto-rt" \n'
-            machine=self.machine
+            machine=self.getProperty('custom_machine')
+            if machine:
+                self.machine = machine
+            else:
+                machine = self.machine
             if layerversion is not None and int(layerversion) > 1:
                 if self.machine == "atom-pc":
                     machine = self.machine.replace("atom-pc", "genericx86")
-- 
2.1.4

[[PATCH][yocto-autobuilder] 2/2] buildset-config.yocto-qa: Add nightly-oe-build-perf-test buildset

[Aníbal Limón]

From: Monserrat Sedeno <monserratx.sedeno.bustos.intel.com>

The new nightly-oe-build-perf-test buildset executes the
build-perf-test-wrapper.sh into a worker and publish the results to
specified folder by AB config.

When the performance script finish it sends an notifcation email to
yocto alias users, the worker machine needs to configure the email
addresses for yocto alias.

[YOCTO #9377]

Signed-off-by: Monserrat Sedeno <monserratx.sedeno.bustos.intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 .../nightly-oe-build-perf-test.conf                | 28 ++++++++
 config/autobuilder.conf.example                    |  8 +++
 .../autobuilder/buildsteps/RunOeBuildPerfTest.py   | 78 ++++++++++++++++++++++
 .../autobuilder/buildsteps/SendOePerfEmail.py      | 64 ++++++++++++++++++
 .../buildsteps/TarballOeBuildPerfTest.py           | 33 +++++++++
 lib/python2.7/site-packages/autobuilder/config.py  |  2 +-
 6 files changed, 212 insertions(+), 1 deletion(-)
 create mode 100644 buildset-config.yocto-qa/nightly-oe-build-perf-test.conf
 create mode 100644 lib/python2.7/site-packages/autobuilder/buildsteps/RunOeBuildPerfTest.py
 create mode 100644 lib/python2.7/site-packages/autobuilder/buildsteps/SendOePerfEmail.py
 create mode 100644 lib/python2.7/site-packages/autobuilder/buildsteps/TarballOeBuildPerfTest.py

diff --git a/buildset-config.yocto-qa/nightly-oe-build-perf-test.conf b/buildset-config.yocto-qa/nightly-oe-build-perf-test.conf
new file mode 100644
index 0000000..999036d
--- /dev/null
+++ b/buildset-config.yocto-qa/nightly-oe-build-perf-test.conf
@@ -0,0 +1,28 @@
+[nightly-oe-build-perf-test]
+builders: 'example-worker'
+repos: [{'poky':
+            {'repourl':'git://git.yoctoproject.org/poky',
+             'layerversion':{'core':'meta', 'yoctobsp':'meta-yocto-bsp', 'yocto':'meta-yocto', 'poky':'meta-poky'},
+             'branch':'master'}}]
+props: [{'machine':{'prop_type':'ChoiceStringParameter',
+                       'choices': ['qemux86', 'qemuarm', 'qemumips', 'qemuppc', 'qemux86-64', 'qemuarm64', 'qemumips64'],
+                       'name': 'machine',
+                       'label':'<hr>Selects machine for performance measurement.'}},
+        {'create_eventlog':{'prop_type':'ChoiceStringParameter',
+                       'choices': ['False', 'True'],
+                       'name': 'create_eventlog',
+                       'label':'<hr>Create Toaster event log as part of image creation?:'}}]
+steps: [{'SetDest':{}},
+        {'CheckOutLayers': {}},
+        {'RunPreamble': {}},
+        {'GetDistroVersion' : {'distro': 'poky'}},
+        {'CreateAutoConf': {'machine': 'qemux86-64', 'SDKMACHINE' : 'i686',
+                            'distro': 'poky', 'buildhistory' : True,
+                            'atextappend' : 'CONNECTIVITY_CHECK_URIS = ""'}},
+        {'CreateBBLayersConf': {'buildprovider' : 'yocto'}},
+        {'SyncPersistDB' : {'distro' : 'poky'}},
+        {'GetBitbakeVersion': {}},
+        {'RunOeBuildPerfTest': {}},
+        {'TarballOeBuildPerfTest': {}},
+        {'SendOePerfEmail': {}},
+        ]
diff --git a/config/autobuilder.conf.example b/config/autobuilder.conf.example
index 2e42013..2ee11e6 100644
--- a/config/autobuilder.conf.example
+++ b/config/autobuilder.conf.example
@@ -87,3 +87,11 @@ QA_MAIL_SIG = "Multiline\nSig\nLine"
 
 [Buildlogger]
 BUILDLOG_TO_WIKI = False
+
+[Performance]
+PERFORMANCE_PUBLISH_DIR = "/tmp/yocto-autobuilder/performance"
+PERFORMANCE_SEND_EMAIL = "False"
+PERFORMANCE_MAIL_TO = "root@localhost otherperson@localhost"
+PERFORMANCE_MAIL_CC = "buildcc@localhost"
+PERFORMANCE_MAIL_BCC = "buildbcc@localhost"
+PERFORMANCE_MAIL_SIG = "Multiline\nSig\nLine"
diff --git a/lib/python2.7/site-packages/autobuilder/buildsteps/RunOeBuildPerfTest.py b/lib/python2.7/site-packages/autobuilder/buildsteps/RunOeBuildPerfTest.py
new file mode 100644
index 0000000..dcc17dd
--- /dev/null
+++ b/lib/python2.7/site-packages/autobuilder/buildsteps/RunOeBuildPerfTest.py
@@ -0,0 +1,78 @@
+import os, datetime, subprocess
+from buildbot.steps.shell import ShellCommand
+from buildbot.process.buildstep import LogLineObserver
+
+from autobuilder.config import PERFORMANCE_PUBLISH_DIR
+
+class OeBuildPerfTestLogLineObserver(LogLineObserver):
+    """
+        Scans lines in order to save the oe-buil-perf-test command
+        output.
+    """
+
+    def _handleLine(self, line):
+        if not hasattr(self.step, 'oe_build_perf_test_output'):
+            self.step.oe_build_perf_test_output = ""
+            self.step.oe_build_perf_test_match = False
+
+        # Search for ### Shell environment set up for builds. ### to start
+        # capturing.
+        if not self.step.oe_build_perf_test_match and line.startswith('###'):
+            self.step.oe_build_perf_test_match = True
+
+        if self.step.oe_build_perf_test_match:
+            self.step.oe_build_perf_test_output += line + '\n'
+
+    def outLineReceived(self, line):
+        self._handleLine(line)
+
+    def errLineReceived(self, line):
+        self._handleLine(line)
+
+class RunOeBuildPerfTest(ShellCommand):
+    flunkOnFailure = True
+    name = "Running oe-build-perf-test"
+
+    def __init__(self, factory, argdict=None, **kwargs):
+        self.tests = None
+        self.factory = factory
+        for k, v in argdict.iteritems():
+                setattr(self, k, v)
+        self.description = "Running oe-build-perf-test"
+        self.timeout = 100000
+        kwargs['timeout']=self.timeout
+        ShellCommand.__init__(self, **kwargs)
+
+        self.stdio_observer = OeBuildPerfTestLogLineObserver()
+        self.addLogObserver('stdio', self.stdio_observer)
+
+    def start(self):
+        branch = self.getProperty("branch")
+        revision = self.getProperty("got_revision")[0:10] # small rev
+        machine = self.getProperty("MACHINE")
+
+        # for oe-build-perf-test
+        timestamp = datetime.datetime.now().strftime("%Y%m%d%H%M%S")
+        results_basename =  'results-%s-%s-%s' % (branch, revision, timestamp)
+        results_dir = os.path.join(PERFORMANCE_PUBLISH_DIR, results_basename)
+        lock_file = 'oe-build-perf.lock'
+        globalres_log = os.path.join(PERFORMANCE_PUBLISH_DIR, 'globalres.log')
+        git_dir = os.path.join(PERFORMANCE_PUBLISH_DIR, 'git')
+        
+        self.setProperty("oe_perf_globalres_log", globalres_log, "RunOeBuildPerfTest")
+        self.setProperty("oe_perf_results_dir", results_dir, "RunOeBuildPerfTest")
+
+        self.command = "mkdir -p %s; " % (results_dir)
+
+        self.command += ". ./oe-init-build-env; " 
+        self.command += """ oe-build-perf-test --out-dir "%s" \
+--globalres-file "%s" --lock-file "%s" --commit-results "%s" --commit-results-branch "{tester_host}/{git_branch}/%s" \
+--commit-results-tag "{tester_host}/{git_branch}/%s/{git_commit_count}-g{git_commit}/{tag_num}"
+""" % (results_dir, globalres_log, lock_file, git_dir, machine, machine)
+
+
+        ShellCommand.start(self)
+
+    def commandComplete(self, cmd):
+        self.setProperty("oe_build_perf_test_output",
+                self.oe_build_perf_test_output, "RunOeBuildPerfTest")
diff --git a/lib/python2.7/site-packages/autobuilder/buildsteps/SendOePerfEmail.py b/lib/python2.7/site-packages/autobuilder/buildsteps/SendOePerfEmail.py
new file mode 100644
index 0000000..6054397
--- /dev/null
+++ b/lib/python2.7/site-packages/autobuilder/buildsteps/SendOePerfEmail.py
@@ -0,0 +1,64 @@
+from buildbot.steps.shell import ShellCommand
+from buildbot.process.buildstep import SKIPPED
+import os, subprocess
+
+class SendOePerfEmail(ShellCommand):
+    haltOnFailure = False
+    flunkOnFailure = True
+    name = "SendOePerfEmail"
+    description = ["Sending Performance Email"]
+    def __init__(self, factory, argdict=None, **kwargs):
+        self.factory = factory
+        description = ["Sending alert emails"]
+        for k, v in argdict.iteritems():
+            setattr(self, k, v)
+        # Timeout needs to be passed to LoggingBuildStep as a kwarg
+        self.timeout = 100000
+        kwargs['timeout']=self.timeout
+        ShellCommand.__init__(self, **kwargs)
+
+    def start(self):
+        if not os.environ.get("PERFORMANCE_SEND_EMAIL") == "True":
+            return SKIPPED
+
+        branch = self.getProperty("branch")
+        oe_build_perf_test_output = self.getProperty("oe_build_perf_test_output")
+        mailto = ""
+        mailcc = ""
+        mailbcc = ""
+        mailsig = ""
+        if os.environ.get('PERFORMANCE_MAIL_TO'):
+            mailto = os.environ.get('PERFORMANCE_MAIL_TO')
+        if os.environ.get('PERFORMANCE_MAIL_CC'):
+            mailcc = os.environ.get('PERFORMANCE_MAIL_CC')
+        if os.environ.get('PERFORMANCE_MAIL_BCC'):
+            mailbcc = os.environ.get('PERFORMANCE_MAIL_BCC')
+        if os.environ.get('PERFORMANCE_MAIL_SIG'):
+            mailsig = os.environ.get('PERFORMANCE_MAIL_SIG')
+
+        archive_dir = self.getProperty("oe_perf_archive_dir")
+        globalres_log = self.getProperty("oe_perf_globalres_log")
+        email_base = '''
+Running on %s \n
+
+%s
+
+-----------------\n\n Global results file \n\n""$read_file"" \n 
+-----------------\n\n Archive results in %s \n''' % (oe_build_perf_test_output,
+        os.uname()[1], archive_dir)
+
+        mailsubject = "Build Performance Report - %s branch" % (branch)
+        email_header = ""
+        if mailto is not None and mailto is not "":
+            email_header += "To: " + mailto + "\n"
+        if mailcc is not None and mailcc is not "":
+            email_header += "Cc: " + mailcc + "\n"
+        if mailbcc is not None and mailbcc is not "":
+            email_header += "Bcc: " + mailbcc + "\n"
+        
+        email_header += "Subject: " + mailsubject + "\n"
+        mailcmd = 'read_file=`cat %s`;' %(globalres_log)
+        mailcmd += ' echo "' + email_header + '\n' + email_base + '\n' + mailsig + '"  | sendmail -t;'
+        self.command = mailcmd
+
+        ShellCommand.start(self)
diff --git a/lib/python2.7/site-packages/autobuilder/buildsteps/TarballOeBuildPerfTest.py b/lib/python2.7/site-packages/autobuilder/buildsteps/TarballOeBuildPerfTest.py
new file mode 100644
index 0000000..ca60c93
--- /dev/null
+++ b/lib/python2.7/site-packages/autobuilder/buildsteps/TarballOeBuildPerfTest.py
@@ -0,0 +1,33 @@
+import os, datetime, subprocess
+from buildbot.steps.shell import ShellCommand
+from autobuilder.config import PERFORMANCE_PUBLISH_DIR
+
+class TarballOeBuildPerfTest(ShellCommand):
+    flunkOnFailure = True
+    name = "Tarball oe-build-perf-test results"
+
+    def __init__(self, factory, argdict=None, **kwargs):
+        self.tests = None
+        self.factory = factory
+        for k, v in argdict.iteritems():
+                setattr(self, k, v)
+        self.description = "Running tarball oe-build-perf-test results"
+        self.timeout = 100000
+        kwargs['timeout']=self.timeout
+        ShellCommand.__init__(self, **kwargs)
+  
+    def start(self):
+        results_dir = self.getProperty("oe_perf_results_dir")
+
+        results_basename = os.path.basename(results_dir)
+        archive_dir = os.path.join(PERFORMANCE_PUBLISH_DIR, 'archives')
+        archive_file = os.path.join(archive_dir, results_basename + '.tar.gz')
+
+        self.setProperty("oe_perf_archive_dir", archive_dir, "TarballOeBuildPerfTest")
+        self.setProperty("oe_perf_archive_file", archive_dir, "TarballOeBuildPerfTest")
+
+        self.command = "mkdir -p %s; " % archive_dir
+        self.command += "tar -czf \"%s\" \"%s\"" % (archive_file, results_dir)
+                
+        ShellCommand.start(self)
+
diff --git a/lib/python2.7/site-packages/autobuilder/config.py b/lib/python2.7/site-packages/autobuilder/config.py
index eced0c7..9d945b1 100644
--- a/lib/python2.7/site-packages/autobuilder/config.py
+++ b/lib/python2.7/site-packages/autobuilder/config.py
@@ -22,4 +22,4 @@ X8664TC_PUBLISH_DIR = os.environ.get("X8664TC_PUBLISH_DIR")
 RPM_PUBLISH_DIR = os.environ.get("RPM_PUBLISH_DIR")
 IPK_PUBLISH_DIR = os.environ.get("IPK_PUBLISH_DIR")
 DEB_PUBLISH_DIR = os.environ.get("DEB_PUBLISH_DIR")
-
+PERFORMANCE_PUBLISH_DIR = os.environ.get("PERFORMANCE_PUBLISH_DIR")
-- 
2.1.4

Re: [[PATCH][yocto-autobuilder] 1/2] buildsteps/CreateAutoConf.py: Add support for specify MACHINE by build prop

[Joshua Lock]

On Mon, 2017-01-16 at 10:25 -0600, Aníbal Limón wrote:
> Sometimes is a good idea to could specify a machine to build via a
> build
> property.

Why's that? I can guess but the commit log should really tell me what
the motivation for the change is.

> 
> Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
> ---
>  .../site-
> packages/autobuilder/buildsteps/CreateAutoConf.py          | 6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
> 
> diff --git a/lib/python2.7/site-
> packages/autobuilder/buildsteps/CreateAutoConf.py
> b/lib/python2.7/site-
> packages/autobuilder/buildsteps/CreateAutoConf.py
> index 021f542..7915dd5 100644
> --- a/lib/python2.7/site-
> packages/autobuilder/buildsteps/CreateAutoConf.py
> +++ b/lib/python2.7/site-
> packages/autobuilder/buildsteps/CreateAutoConf.py
> @@ -174,7 +174,11 @@ class CreateAutoConf(ShellCommand):
>                      fout = fout + 'baselib = \\042${@d.getVar(\\047B
> ASE_LIB_tune-\\047 + (d.getVar(\\047DEFAULTTUNE\\047, True) or
> \\047INVALID\\047), True) or \\047lib\\047}\\042 \n'
>              if self.distro == "poky-rt":
>                  fout = fout +
> 'PREFERRED_PROVIDER_virtual/kernel="linux-yocto-rt" \n'
> -            machine=self.machine
> +            machine=self.getProperty('custom_machine')
> +            if machine:
> +                self.machine = machine
> +            else:
> +                machine = self.machine
>              if layerversion is not None and int(layerversion) > 1:
>                  if self.machine == "atom-pc":
>                      machine = self.machine.replace("atom-pc",
> "genericx86")

Re: [[PATCH][yocto-autobuilder] 1/2] buildsteps/CreateAutoConf.py: Add support for specify MACHINE by build prop

[Aníbal Limón]

[-- Attachment #1: Type: text/plain, Size: 2177 bytes --]



On 01/18/2017 12:13 PM, Joshua Lock wrote:
> On Mon, 2017-01-16 at 10:25 -0600, Aníbal Limón wrote:
>> Sometimes is a good idea to could specify a machine to build via a
>> build
>> property.
> 
> Why's that? I can guess but the commit log should really tell me what
> the motivation for the change is.

In the performance buildset patch [1] there is a build property to
specify the machine via the web interface, without this change the
machine is overwrite to predefined in the buildset.

Cheers,
	alimon

[1]
http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder/tree/buildset-config.yocto-qa/nightly-oe-build-perf-test.conf?h=contrib/alimon/devel&id=6c25a37cd801f2cdecc21c76499625a117e08ff7#n7

> 
>>
>> Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
>> ---
>>  .../site-
>> packages/autobuilder/buildsteps/CreateAutoConf.py          | 6 +++++-
>>  1 file changed, 5 insertions(+), 1 deletion(-)
>>
>> diff --git a/lib/python2.7/site-
>> packages/autobuilder/buildsteps/CreateAutoConf.py
>> b/lib/python2.7/site-
>> packages/autobuilder/buildsteps/CreateAutoConf.py
>> index 021f542..7915dd5 100644
>> --- a/lib/python2.7/site-
>> packages/autobuilder/buildsteps/CreateAutoConf.py
>> +++ b/lib/python2.7/site-
>> packages/autobuilder/buildsteps/CreateAutoConf.py
>> @@ -174,7 +174,11 @@ class CreateAutoConf(ShellCommand):
>>                      fout = fout + 'baselib = \\042${@d.getVar(\\047B
>> ASE_LIB_tune-\\047 + (d.getVar(\\047DEFAULTTUNE\\047, True) or
>> \\047INVALID\\047), True) or \\047lib\\047}\\042 \n'
>>              if self.distro == "poky-rt":
>>                  fout = fout +
>> 'PREFERRED_PROVIDER_virtual/kernel="linux-yocto-rt" \n'
>> -            machine=self.machine
>> +            machine=self.getProperty('custom_machine')
>> +            if machine:
>> +                self.machine = machine
>> +            else:
>> +                machine = self.machine
>>              if layerversion is not None and int(layerversion) > 1:
>>                  if self.machine == "atom-pc":
>>                      machine = self.machine.replace("atom-pc",
>> "genericx86")


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [[PATCH][yocto-autobuilder] 1/2] buildsteps/CreateAutoConf.py: Add support for specify MACHINE by build prop

[Beth 'pidge' Flanagan]

On Wed, 2017-01-18 at 18:13 +0000, Joshua Lock wrote:
> On Mon, 2017-01-16 at 10:25 -0600, Aníbal Limón wrote:
> > 
> > Sometimes is a good idea to could specify a machine to build via a
> > build
> > property.
> Why's that? I can guess but the commit log should really tell me what
> the motivation for the change is.

I agree. Please submit a v2 with a bit better of a commit log.

-b

> 
> > 
> > 
> > Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
> > ---
> >  .../site-
> > packages/autobuilder/buildsteps/CreateAutoConf.py          | 6
> > +++++-
> >  1 file changed, 5 insertions(+), 1 deletion(-)
> > 
> > diff --git a/lib/python2.7/site-
> > packages/autobuilder/buildsteps/CreateAutoConf.py
> > b/lib/python2.7/site-
> > packages/autobuilder/buildsteps/CreateAutoConf.py
> > index 021f542..7915dd5 100644
> > --- a/lib/python2.7/site-
> > packages/autobuilder/buildsteps/CreateAutoConf.py
> > +++ b/lib/python2.7/site-
> > packages/autobuilder/buildsteps/CreateAutoConf.py
> > @@ -174,7 +174,11 @@ class CreateAutoConf(ShellCommand):
> >                      fout = fout + 'baselib = \\042${@d.getVar(\\04
> > 7B
> > ASE_LIB_tune-\\047 + (d.getVar(\\047DEFAULTTUNE\\047, True) or
> > \\047INVALID\\047), True) or \\047lib\\047}\\042 \n'
> >              if self.distro == "poky-rt":
> >                  fout = fout +
> > 'PREFERRED_PROVIDER_virtual/kernel="linux-yocto-rt" \n'
> > -            machine=self.machine
> > +            machine=self.getProperty('custom_machine')
> > +            if machine:
> > +                self.machine = machine
> > +            else:
> > +                machine = self.machine
> >              if layerversion is not None and int(layerversion) > 1:
> >                  if self.machine == "atom-pc":
> >                      machine = self.machine.replace("atom-pc",
> > "genericx86")