[yocto-autobuilder][PATCHv2 1/5] buildsteps/BuildImages: Setproperty BUILDIMAGES

[yocto-autobuilder][PATCHv2 1/5] buildsteps/BuildImages: Setproperty BUILDIMAGES

[Aníbal Limón]

In certain situations is need to know what images was built for
example the DAFT executes flashing over certain image.

[YOCTO #10604]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 lib/python2.7/site-packages/autobuilder/buildsteps/BuildImages.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/lib/python2.7/site-packages/autobuilder/buildsteps/BuildImages.py b/lib/python2.7/site-packages/autobuilder/buildsteps/BuildImages.py
index 30f7a18..7262fd3 100644
--- a/lib/python2.7/site-packages/autobuilder/buildsteps/BuildImages.py
+++ b/lib/python2.7/site-packages/autobuilder/buildsteps/BuildImages.py
@@ -146,6 +146,9 @@ class BuildImages(BitbakeShellCommand):
                 if self.minnowExists is None or self.minnowExists == "True":
                     self.command = ". ./oe-init-build-env; bitbake " + bitbakeflags + self.images
                     self.description = ["Building " + str(self.images)]
+        
+        self.setProperty("BUILDIMAGES", self.images)
+
         ShellCommand.start(self)
 
     def describe(self, done=False):
-- 
2.1.4

[yocto-autobuilder][PATCHv2 2/5] buildsteps/RunSanityTests: Add support for simpleremote target

[Aníbal Limón]

The testimage class supports a simpleremote target specifying the
server and target ip to run tests over SSH.

The main usage of this will in conjunction with the DAFT to make
tests on real HW.

[YOCTO #10604]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 .../site-packages/autobuilder/buildsteps/RunSanityTests.py        | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/lib/python2.7/site-packages/autobuilder/buildsteps/RunSanityTests.py b/lib/python2.7/site-packages/autobuilder/buildsteps/RunSanityTests.py
index 6fa03bc..0022cb9 100644
--- a/lib/python2.7/site-packages/autobuilder/buildsteps/RunSanityTests.py
+++ b/lib/python2.7/site-packages/autobuilder/buildsteps/RunSanityTests.py
@@ -60,6 +60,14 @@ class RunSanityTests(BitbakeShellCommand):
         elif layerversion > 1:
             self.command = self.command + "echo 'INHERIT += \"testimage\"' >> ./conf/auto.conf;"
             self.command = self.command + "echo 'TEST_QEMUBOOT_TIMEOUT = \"1500\"' >> ./conf/auto.conf;"
+
+            target_ip = self.getProperty('target_ip')
+            server_ip = self.getProperty('server_ip')
+            if target_ip and server_ip:
+                self.command = self.command +  "echo 'TEST_TARGET=\"simpleremote\"' >> ./conf/auto.conf;"
+                self.command = self.command +  "echo 'TEST_TARGET_IP=\"%s\"' >> ./conf/auto.conf;" % target_ip
+                self.command = self.command +  "echo 'TEST_SERVER_IP=\"%s\"' >> ./conf/auto.conf;" % server_ip
+
             if self.suites:
                 if layerversion > 3:
                     self.command = self.command + "echo 'TEST_SUITES = \"" + self.suites + "\"'" + " >> ./conf/auto.conf;"
-- 
2.1.4

[yocto-autobuilder][PATCHv2 3/5] autobuilder/buildsteps: Add DaftGetDevices step

[Aníbal Limón]

This buildstep will copy the devices configuration from the
worker and read it, in order to get daft ip an ssh port, to
use later in RunSanityTests

[YOCTO #10604]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Monserrat Sedeno <monserratx.sedeno.bustos.intel.com>
Signed-off-by: Edwin Plauchu <edwin.plauchu.camacho@linux.intel.com>
---
 config/autobuilder.conf.example                    |   3 +
 .../autobuilder/buildsteps/DaftGetDevices.py       |  44 ++++++++
 lib/python2.7/site-packages/autobuilder/config.py  |   1 +
 .../site-packages/autobuilder/lib/daft.py          | 115 +++++++++++++++++++++
 4 files changed, 163 insertions(+)
 create mode 100644 lib/python2.7/site-packages/autobuilder/buildsteps/DaftGetDevices.py
 create mode 100644 lib/python2.7/site-packages/autobuilder/lib/daft.py

diff --git a/config/autobuilder.conf.example b/config/autobuilder.conf.example
index 2ee11e6..e5ec16b 100644
--- a/config/autobuilder.conf.example
+++ b/config/autobuilder.conf.example
@@ -95,3 +95,6 @@ PERFORMANCE_MAIL_TO = "root@localhost otherperson@localhost"
 PERFORMANCE_MAIL_CC = "buildcc@localhost"
 PERFORMANCE_MAIL_BCC = "buildbcc@localhost"
 PERFORMANCE_MAIL_SIG = "Multiline\nSig\nLine"
+
+[Daft]
+DAFT_WORKER_DEVICES_CFG = "/etc/daft/devices.cfg"
diff --git a/lib/python2.7/site-packages/autobuilder/buildsteps/DaftGetDevices.py b/lib/python2.7/site-packages/autobuilder/buildsteps/DaftGetDevices.py
new file mode 100644
index 0000000..133a4d6
--- /dev/null
+++ b/lib/python2.7/site-packages/autobuilder/buildsteps/DaftGetDevices.py
@@ -0,0 +1,44 @@
+import os
+from buildbot.steps.transfer import FileUpload
+from buildbot.process.buildstep import BuildStep
+
+from lib.daft import DeployScanner
+from autobuilder.config import DAFT_WORKER_DEVICES_CFG
+
+class DaftGetDevices(FileUpload):
+    haltOnFailure = True
+
+    name = "DaftGetDevices"
+
+    def __init__(self, factory, argdict=None, **kwargs):
+        self.tests = None
+        self.factory = factory
+        for k, v in argdict.iteritems():
+            setattr(self, k, v)
+        self.description = "Getting devices configuration"
+        self.timeout = 100000
+        kwargs['timeout']=self.timeout
+
+        super(DaftGetDevices, self).__init__(DAFT_WORKER_DEVICES_CFG,
+                os.path.join('/tmp', 'devices.cfg'))
+
+    def finished(self, result):
+        if self.cmd:
+            ds = DeployScanner(devsconf_file = self.masterdest)
+            devices = ds()
+
+            found = False
+            dut_label = self.getProperty('custom_dut')
+            for d in devices:
+                if d['dut_label'] == dut_label:
+                    self.setProperty('dut_name', d['dut_label'], 'DaftGetDevices')
+                    self.setProperty('server_ip', d['server_address'], 'DaftGetDevices')
+                    target_ip = "%s:%s" % (d['ctrl_address'], d['dut_sshport'])
+                    self.setProperty('target_ip', target_ip, 'DaftGetDevices')
+
+                    found = True
+
+            if not found:
+                 return BuildStep.finished(self, FAILURE)
+
+        return super(DaftGetDevices, self).finished(result)
diff --git a/lib/python2.7/site-packages/autobuilder/config.py b/lib/python2.7/site-packages/autobuilder/config.py
index 9d945b1..5bcf6c6 100644
--- a/lib/python2.7/site-packages/autobuilder/config.py
+++ b/lib/python2.7/site-packages/autobuilder/config.py
@@ -23,3 +23,4 @@ RPM_PUBLISH_DIR = os.environ.get("RPM_PUBLISH_DIR")
 IPK_PUBLISH_DIR = os.environ.get("IPK_PUBLISH_DIR")
 DEB_PUBLISH_DIR = os.environ.get("DEB_PUBLISH_DIR")
 PERFORMANCE_PUBLISH_DIR = os.environ.get("PERFORMANCE_PUBLISH_DIR")
+DAFT_WORKER_DEVICES_CFG = os.environ.get("DAFT_WORKER_DEVICES_CFG")
diff --git a/lib/python2.7/site-packages/autobuilder/lib/daft.py b/lib/python2.7/site-packages/autobuilder/lib/daft.py
new file mode 100644
index 0000000..eef4bab
--- /dev/null
+++ b/lib/python2.7/site-packages/autobuilder/lib/daft.py
@@ -0,0 +1,115 @@
+import os
+import ConfigParser as configparser
+
+class DeployScanner(object):
+    '''
+    In charge of scanning deployed daft bbb devices
+    '''
+
+    __MAGIC_SERVER_ADDRESS = '192.168.30.1' # DAFT uses this address for this internal network
+    __MAGIC_SSH_PORT = 2233 # it is hardcoded as per DAFT implementation manual
+
+    def  __init__(self, *args, **kwargs):
+        self.devsconf_fp = kwargs.get('devsconf_file', None)
+        if not self.devsconf_fp:
+            raise Exception('not fed devsconf file')
+        self.ign_leases = kwargs.get('ign_leases', True)
+        if not self.ign_leases:
+            self.leases_file_path =  kwargs.get('leases_file', None)
+            if not self.leases_file_path:
+                raise Exception('not fed leases file')
+
+    def __call__(self):
+        '''
+        Creates relation of deployed devices
+        Returns:
+            List of dictionaries containing info about devices deployed.
+        '''
+        def create_relation(i,j):
+            r = []
+            for conf in i:
+                for active in j:
+                    if conf['bb_ip'] == active['ip']:
+                        r.append({
+                            'dut_label': conf['device'].lower(),
+                            'dut_family': conf['device_type'].lower(),
+                            'dut_sshport': str(self.__MAGIC_SSH_PORT),
+                            'ctrl_address': conf['bb_ip'],
+                            'server_address': self.__MAGIC_SERVER_ADDRESS
+                        })
+            return r
+
+        def slack_relation(i):
+            r = []
+            for conf in i:
+                r.append({
+                    'dut_label': conf['device'],
+                    'dut_family': conf['device_type'],
+                    'dut_sshport': str(self.__MAGIC_SSH_PORT),
+                    'ctrl_address': conf['bb_ip'],
+                    'server_address': self.__MAGIC_SERVER_ADDRESS
+                })
+            return r
+
+        fc = self.__fetch_confs()
+        if not fc:
+            raise Exception('There are no configurations as per BBB devices')
+
+        if self.ign_leases:
+            # Devices that nobody knows if were deployed
+            return slack_relation(fc)
+
+        als = self.__active_leases()
+        if not als:
+            raise Exception('DHCP server has not registered any host yet')
+
+        return create_relation(fc, als)
+
+    def __fetch_confs(self):
+        '''
+        Read and parse BBB configuration file and return result as dictionary
+        '''
+        config = configparser.SafeConfigParser()
+        config.read(self.devsconf_fp)
+        configurations = []
+        for device in config.sections():
+            device_config = dict(config.items(device))
+            device_config["device"] = device
+            device_config["device_type"] = device.rstrip('1234567890_')
+            configurations.append(device_config)
+        return configurations
+
+    def __active_leases(self):
+        """
+        Read the active leases from dnsmasq leases file and return a list of
+        active leases as dictionaries.
+        Args:
+            file_name (str): Path to leases file, e.g. /path/to/file/dnsmasq.leases
+        Returns:
+            List of dictionaries containing the active leases.
+            The dictionaries have the following format:
+            {
+                "mac": "device_mac_address",
+                "ip": "device_ip_address",
+                "hostname": "device_host_name",
+                "client_id": "client_id_or_*_if_unset"
+            }
+        """
+        with open(self.leases_file_path) as lease_file:
+            leases = lease_file.readlines()
+
+        leases_list = []
+
+        # dnsmasq.leases contains rows with the following format:
+        # <lease_expiry_time_as_epoch_format> <mac> <ip> <hostname> <domain>
+        # See:
+        #http://lists.thekelleys.org.uk/pipermail/dnsmasq-discuss/2005q1/000143.html
+        for lease in leases:
+            lease = lease.split()
+            leases_list.append({
+                "mac": lease[1],
+                "ip": lease[2],
+                "hostname": lease[3],
+                "client_id": lease[4],
+            })
+        return leases_list
-- 
2.1.4

[yocto-autobuilder][PATCHv2 4/5] autobuilder/buildsteps: Add DAFTFlash buildstep

[Aníbal Limón]

This will execute daft flash cycle without reboot and prepare the dut
for sanity tests.

[YOCTO #10604]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Monserrat Sedeno <monserratx.sedeno.bustos.intel.com>
Signed-off-by: Edwin Plauchu <edwin.plauchu.camacho@linux.intel.com>
---
 config/autobuilder.conf.example                    |  1 +
 .../autobuilder/buildsteps/DaftFlash.py            | 42 ++++++++++++++++++++++
 lib/python2.7/site-packages/autobuilder/config.py  |  1 +
 3 files changed, 44 insertions(+)
 create mode 100644 lib/python2.7/site-packages/autobuilder/buildsteps/DaftFlash.py

diff --git a/config/autobuilder.conf.example b/config/autobuilder.conf.example
index e5ec16b..2e992cf 100644
--- a/config/autobuilder.conf.example
+++ b/config/autobuilder.conf.example
@@ -98,3 +98,4 @@ PERFORMANCE_MAIL_SIG = "Multiline\nSig\nLine"
 
 [Daft]
 DAFT_WORKER_DEVICES_CFG = "/etc/daft/devices.cfg"
+DAFT_WORKER_WORKSPACE = "/home/ab/workspace"
diff --git a/lib/python2.7/site-packages/autobuilder/buildsteps/DaftFlash.py b/lib/python2.7/site-packages/autobuilder/buildsteps/DaftFlash.py
new file mode 100644
index 0000000..d5f6de8
--- /dev/null
+++ b/lib/python2.7/site-packages/autobuilder/buildsteps/DaftFlash.py
@@ -0,0 +1,42 @@
+import os
+
+from autobuilder.config import DAFT_WORKER_WORKSPACE
+from buildbot.steps.shell import ShellCommand
+
+class DaftFlash(ShellCommand):
+    haltOnFailure = True
+
+    name = "DaftFlash"
+
+    def __init__(self, factory, argdict=None, **kwargs):
+        self.tests = None
+        self.factory = factory
+        for k, v in argdict.iteritems():
+            setattr(self, k, v)
+        self.description = "DUT Flashing"
+        self.timeout = 100000
+        kwargs['timeout']=self.timeout
+        ShellCommand.__init__(self, **kwargs)
+
+    def start(self):
+        workspace_dir = DAFT_WORKER_WORKSPACE
+
+        dut_name = self.getProperty('dut_name')
+        workdir = self.getProperty('workdir')
+        image = self.getProperty('BUILDIMAGES')
+        machine = self.getProperty('MACHINE')
+
+        image_path = os.path.join(workdir, 'build', 'build', 'tmp', 'deploy',
+                'images', machine, '%s-%s.hddimg' % (image, machine))
+        image_name = os.path.basename(image_path)
+
+        # XXX: DAFT needs to have the image in a shared workspace 
+        self.command = "cp %s %s; " % (image_path, workspace_dir)
+
+        self.command += "cd %s; " % (workspace_dir)
+
+        # XXX: DAFT needs to be executed by root user
+        daft_cmd = "daft --setout %s %s" % (dut_name, image_name)
+        self.command += "sudo su -l -c 'cd {} && {}'; ".format(workspace_dir, daft_cmd)
+
+        ShellCommand.start(self)
diff --git a/lib/python2.7/site-packages/autobuilder/config.py b/lib/python2.7/site-packages/autobuilder/config.py
index 5bcf6c6..fcc08f0 100644
--- a/lib/python2.7/site-packages/autobuilder/config.py
+++ b/lib/python2.7/site-packages/autobuilder/config.py
@@ -24,3 +24,4 @@ IPK_PUBLISH_DIR = os.environ.get("IPK_PUBLISH_DIR")
 DEB_PUBLISH_DIR = os.environ.get("DEB_PUBLISH_DIR")
 PERFORMANCE_PUBLISH_DIR = os.environ.get("PERFORMANCE_PUBLISH_DIR")
 DAFT_WORKER_DEVICES_CFG = os.environ.get("DAFT_WORKER_DEVICES_CFG")
+DAFT_WORKER_WORKSPACE = os.environ.get("DAFT_WORKER_WORKSPACE")
-- 
2.1.4

[yocto-autobuilder][PATCHv2 5/5] buildset-config.yocto-qa: Add intel-corei7-64-daft buildset

[Aníbal Limón]

This buildset is for support running sanity tests over a DUT like
Minnowboard for the moment only builds core-image-sato, there is a need
to make it configurable.

Also the listo of the available devices are fixed into the UI as
parameters.

[YOCTO #10604]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 buildset-config.yocto-qa/intel-corei7-64-daft.conf | 59 ++++++++++++++++++++++
 1 file changed, 59 insertions(+)
 create mode 100644 buildset-config.yocto-qa/intel-corei7-64-daft.conf

diff --git a/buildset-config.yocto-qa/intel-corei7-64-daft.conf b/buildset-config.yocto-qa/intel-corei7-64-daft.conf
new file mode 100644
index 0000000..cb3bcda
--- /dev/null
+++ b/buildset-config.yocto-qa/intel-corei7-64-daft.conf
@@ -0,0 +1,59 @@
+[intel-corei7-64-daft]
+builders: 'example-worker'
+repos: [{'poky':
+            {'repourl':'git://git.yoctoproject.org/poky',
+             'layerversion':{'core':'meta', 'yoctobsp':'meta-yocto-bsp', 'yocto':'meta-yocto', 'poky':'meta-poky'},
+             'branch':'master'}},
+        {'meta-intel':
+            {'repourl':'git://git.yoctoproject.org/meta-intel',
+             'layerversion':{'intel':'meta-intel'},
+             'branch':'master'}}]
+props: [{'kmeta':{'prop_type':'StringParameter',
+                       'size': 15,
+                       'name': 'kmeta',
+                       'default': '',
+                       'label':'<hr><h3> branch for KMETA_${MACHINE} (default is blank):</h3>'}},
+        {'srcrev_meta':{'prop_type':'StringParameter',
+                       'size': 15,
+                       'name': 'srcrev_meta',
+                       'default': '${AUTOREV}',
+                       'label':'<hr><h3> SRCREV for KMETA_${MACHINE} (default is ${AUTOREV}):<br></h3>'}},
+        {'srcuri_meta':{'prop_type':'StringParameter',
+                       'size': 50,
+                       'name': 'srcuri_meta',
+                       'default': '',
+                       'label':'<hr><h3> SRC_URI_pn-linux-yocto KMETA append. (default is blank).<br> This should be constructed as *just* the git uri portion of the string. <br>(git://git.yoctoproject.org/linux-yocto-3.19.git</h3>'}},
+        {'kbranch':{'prop_type':'StringParameter',
+                       'size': 15,
+                       'name': 'kbranch',
+                       'default': '',
+                       'label':'<hr><h3> branch for KBRANCH_${MACHINE} (default is blank):</h3>'}},
+        {'srcrev_machine':{'prop_type':'StringParameter',
+                       'size': 15,
+                       'name': 'srcrev_machine',
+                       'default': '${AUTOREV}',
+                       'label':'<hr><h3> SRCREV for KBRANCH. (default is ${AUTOREV}):<br></h3>'}},
+        {'srcuri_machine':{'prop_type':'StringParameter',
+                       'size': 50,
+                       'name': 'srcuri_machine',
+                       'default': '',
+                       'label':'<hr><h3> SRC_URI_pn-linux-yocto KBRANCH append. (default is blank).<br> This should be constructed as *just* the git uri portion of the string. <br>(git://git.yoctoproject.org/linux-yocto-3.19.git</h3>'}},
+	{'dut':{'prop_type':'ChoiceStringParameter',
+		   'choices': ['Minnowboard1'],
+		   'name': 'dut',
+		   'label':'<hr><h3>Selects DUT for flashing.<br></h3>'}}]
+steps: [{'SetDest':{}},
+        {'CheckOutLayers': {}},
+        {'RunPreamble': {}},
+        {'CheckBSPExists': {}},
+        {'GetDistroVersion' : {'distro': 'poky'}},
+        {'CreateAutoConf': {'machine': 'intel-corei7-64', 'SDKMACHINE' : 'x86_64', 
+                            'distro': 'poky'}},
+        {'CreateBBLayersConf': {'buildprovider' : 'yocto',
+                                'bsplayer': True, 'bspprovider': 'intel',
+                                'layerdirs': ['meta-intel', 'meta-intel/meta-tlk']}},
+        {'AddKernelProps': {}},
+        {'DaftGetDevices': {}},
+        {'BuildImages': {'images': 'core-image-sato'}},
+        {'DaftFlash':{}},
+        {'RunSanityTests': {'images': 'core-image-sato'}},]
-- 
2.1.4

Re: [yocto-autobuilder][PATCHv2 4/5] autobuilder/buildsteps: Add DAFTFlash buildstep

[Aníbal Limón]

[-- Attachment #1: Type: text/plain, Size: 3735 bytes --]

This v2 fixes an small issue on variable name, DAFT_WORKER_WORKSPACE_DIR
-> DAFT_WORKER_WORKSPACE.

Cheers,
Anibal

On 06/06/2017 10:46 AM, Aníbal Limón wrote:
> This will execute daft flash cycle without reboot and prepare the dut
> for sanity tests.
> 
> [YOCTO #10604]
> 
> Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
> Signed-off-by: Monserrat Sedeno <monserratx.sedeno.bustos.intel.com>
> Signed-off-by: Edwin Plauchu <edwin.plauchu.camacho@linux.intel.com>
> ---
>  config/autobuilder.conf.example                    |  1 +
>  .../autobuilder/buildsteps/DaftFlash.py            | 42 ++++++++++++++++++++++
>  lib/python2.7/site-packages/autobuilder/config.py  |  1 +
>  3 files changed, 44 insertions(+)
>  create mode 100644 lib/python2.7/site-packages/autobuilder/buildsteps/DaftFlash.py
> 
> diff --git a/config/autobuilder.conf.example b/config/autobuilder.conf.example
> index e5ec16b..2e992cf 100644
> --- a/config/autobuilder.conf.example
> +++ b/config/autobuilder.conf.example
> @@ -98,3 +98,4 @@ PERFORMANCE_MAIL_SIG = "Multiline\nSig\nLine"
>  
>  [Daft]
>  DAFT_WORKER_DEVICES_CFG = "/etc/daft/devices.cfg"
> +DAFT_WORKER_WORKSPACE = "/home/ab/workspace"
> diff --git a/lib/python2.7/site-packages/autobuilder/buildsteps/DaftFlash.py b/lib/python2.7/site-packages/autobuilder/buildsteps/DaftFlash.py
> new file mode 100644
> index 0000000..d5f6de8
> --- /dev/null
> +++ b/lib/python2.7/site-packages/autobuilder/buildsteps/DaftFlash.py
> @@ -0,0 +1,42 @@
> +import os
> +
> +from autobuilder.config import DAFT_WORKER_WORKSPACE
> +from buildbot.steps.shell import ShellCommand
> +
> +class DaftFlash(ShellCommand):
> +    haltOnFailure = True
> +
> +    name = "DaftFlash"
> +
> +    def __init__(self, factory, argdict=None, **kwargs):
> +        self.tests = None
> +        self.factory = factory
> +        for k, v in argdict.iteritems():
> +            setattr(self, k, v)
> +        self.description = "DUT Flashing"
> +        self.timeout = 100000
> +        kwargs['timeout']=self.timeout
> +        ShellCommand.__init__(self, **kwargs)
> +
> +    def start(self):
> +        workspace_dir = DAFT_WORKER_WORKSPACE
> +
> +        dut_name = self.getProperty('dut_name')
> +        workdir = self.getProperty('workdir')
> +        image = self.getProperty('BUILDIMAGES')
> +        machine = self.getProperty('MACHINE')
> +
> +        image_path = os.path.join(workdir, 'build', 'build', 'tmp', 'deploy',
> +                'images', machine, '%s-%s.hddimg' % (image, machine))
> +        image_name = os.path.basename(image_path)
> +
> +        # XXX: DAFT needs to have the image in a shared workspace 
> +        self.command = "cp %s %s; " % (image_path, workspace_dir)
> +
> +        self.command += "cd %s; " % (workspace_dir)
> +
> +        # XXX: DAFT needs to be executed by root user
> +        daft_cmd = "daft --setout %s %s" % (dut_name, image_name)
> +        self.command += "sudo su -l -c 'cd {} && {}'; ".format(workspace_dir, daft_cmd)
> +
> +        ShellCommand.start(self)
> diff --git a/lib/python2.7/site-packages/autobuilder/config.py b/lib/python2.7/site-packages/autobuilder/config.py
> index 5bcf6c6..fcc08f0 100644
> --- a/lib/python2.7/site-packages/autobuilder/config.py
> +++ b/lib/python2.7/site-packages/autobuilder/config.py
> @@ -24,3 +24,4 @@ IPK_PUBLISH_DIR = os.environ.get("IPK_PUBLISH_DIR")
>  DEB_PUBLISH_DIR = os.environ.get("DEB_PUBLISH_DIR")
>  PERFORMANCE_PUBLISH_DIR = os.environ.get("PERFORMANCE_PUBLISH_DIR")
>  DAFT_WORKER_DEVICES_CFG = os.environ.get("DAFT_WORKER_DEVICES_CFG")
> +DAFT_WORKER_WORKSPACE = os.environ.get("DAFT_WORKER_WORKSPACE")
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [yocto-autobuilder][PATCHv2 1/5] buildsteps/BuildImages: Setproperty BUILDIMAGES

[Joshua Lock]

On Tue, 2017-06-06 at 10:46 -0500, Aníbal Limón wrote:
> In certain situations is need to know what images was built for
> example the DAFT executes flashing over certain image.
> 
> [YOCTO #10604]
> 
> Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
> ---
>  lib/python2.7/site-packages/autobuilder/buildsteps/BuildImages.py |
> 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/lib/python2.7/site-
> packages/autobuilder/buildsteps/BuildImages.py b/lib/python2.7/site-
> packages/autobuilder/buildsteps/BuildImages.py
> index 30f7a18..7262fd3 100644
> --- a/lib/python2.7/site-
> packages/autobuilder/buildsteps/BuildImages.py
> +++ b/lib/python2.7/site-
> packages/autobuilder/buildsteps/BuildImages.py
> @@ -146,6 +146,9 @@ class BuildImages(BitbakeShellCommand):
>                  if self.minnowExists is None or self.minnowExists ==
> "True":
>                      self.command = ". ./oe-init-build-env; bitbake "
> + bitbakeflags + self.images
>                      self.description = ["Building " +
> str(self.images)]
> +        
> +        self.setProperty("BUILDIMAGES", self.images)

Generally speaking caps case property names should only be used when
they map to a variable in OE. BuildImages would work.

> +
>          ShellCommand.start(self)
>  
>      def describe(self, done=False):

Re: [yocto-autobuilder][PATCHv2 3/5] autobuilder/buildsteps: Add DaftGetDevices step

[Joshua Lock]

On Tue, 2017-06-06 at 10:46 -0500, Aníbal Limón wrote:
> This buildstep will copy the devices configuration from the
> worker and read it, in order to get daft ip an ssh port, to
> use later in RunSanityTests
> 
> [YOCTO #10604]
> 
> Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
> Signed-off-by: Monserrat Sedeno <monserratx.sedeno.bustos.intel.com>

Typo in the email address here.

> Signed-off-by: Edwin Plauchu <edwin.plauchu.camacho@linux.intel.com>
> ---
>  config/autobuilder.conf.example                    |   3 +
>  .../autobuilder/buildsteps/DaftGetDevices.py       |  44 ++++++++
>  lib/python2.7/site-packages/autobuilder/config.py  |   1 +
>  .../site-packages/autobuilder/lib/daft.py          | 115
> +++++++++++++++++++++
>  4 files changed, 163 insertions(+)
>  create mode 100644 lib/python2.7/site-
> packages/autobuilder/buildsteps/DaftGetDevices.py
>  create mode 100644 lib/python2.7/site-
> packages/autobuilder/lib/daft.py
> 
> diff --git a/config/autobuilder.conf.example
> b/config/autobuilder.conf.example
> index 2ee11e6..e5ec16b 100644
> --- a/config/autobuilder.conf.example
> +++ b/config/autobuilder.conf.example
> @@ -95,3 +95,6 @@ PERFORMANCE_MAIL_TO = "root@localhost otherperson@l
> ocalhost"
>  PERFORMANCE_MAIL_CC = "buildcc@localhost"
>  PERFORMANCE_MAIL_BCC = "buildbcc@localhost"
>  PERFORMANCE_MAIL_SIG = "Multiline\nSig\nLine"
> +
> +[Daft]
> +DAFT_WORKER_DEVICES_CFG = "/etc/daft/devices.cfg"
> diff --git a/lib/python2.7/site-
> packages/autobuilder/buildsteps/DaftGetDevices.py
> b/lib/python2.7/site-
> packages/autobuilder/buildsteps/DaftGetDevices.py
> new file mode 100644
> index 0000000..133a4d6
> --- /dev/null
> +++ b/lib/python2.7/site-
> packages/autobuilder/buildsteps/DaftGetDevices.py
> @@ -0,0 +1,44 @@
> +import os
> +from buildbot.steps.transfer import FileUpload
> +from buildbot.process.buildstep import BuildStep
> +
> +from lib.daft import DeployScanner
> +from autobuilder.config import DAFT_WORKER_DEVICES_CFG
> +
> +class DaftGetDevices(FileUpload):
> +    haltOnFailure = True
> +
> +    name = "DaftGetDevices"
> +
> +    def __init__(self, factory, argdict=None, **kwargs):
> +        self.tests = None
> +        self.factory = factory
> +        for k, v in argdict.iteritems():
> +            setattr(self, k, v)
> +        self.description = "Getting devices configuration"
> +        self.timeout = 100000
> +        kwargs['timeout']=self.timeout
> +
> +        super(DaftGetDevices,
> self).__init__(DAFT_WORKER_DEVICES_CFG,
> +                os.path.join('/tmp', 'devices.cfg'))
> +
> +    def finished(self, result):
> +        if self.cmd:
> +            ds = DeployScanner(devsconf_file = self.masterdest)
> +            devices = ds()

I'm not at all fond of this. It would be easier to follow this code if
this were an explicit method.

> +
> +            found = False
> +            dut_label = self.getProperty('custom_dut')
> +            for d in devices:
> +                if d['dut_label'] == dut_label:
> +                    self.setProperty('dut_name', d['dut_label'],
> 'DaftGetDevices')
> +                    self.setProperty('server_ip',
> d['server_address'], 'DaftGetDevices')
> +                    target_ip = "%s:%s" % (d['ctrl_address'],
> d['dut_sshport'])
> +                    self.setProperty('target_ip', target_ip,
> 'DaftGetDevices')
> +
> +                    found = True
> +
> +            if not found:
> +                 return BuildStep.finished(self, FAILURE)
> +
> +        return super(DaftGetDevices, self).finished(result)
> diff --git a/lib/python2.7/site-packages/autobuilder/config.py
> b/lib/python2.7/site-packages/autobuilder/config.py
> index 9d945b1..5bcf6c6 100644
> --- a/lib/python2.7/site-packages/autobuilder/config.py
> +++ b/lib/python2.7/site-packages/autobuilder/config.py
> @@ -23,3 +23,4 @@ RPM_PUBLISH_DIR = os.environ.get("RPM_PUBLISH_DIR")
>  IPK_PUBLISH_DIR = os.environ.get("IPK_PUBLISH_DIR")
>  DEB_PUBLISH_DIR = os.environ.get("DEB_PUBLISH_DIR")
>  PERFORMANCE_PUBLISH_DIR = os.environ.get("PERFORMANCE_PUBLISH_DIR")
> +DAFT_WORKER_DEVICES_CFG = os.environ.get("DAFT_WORKER_DEVICES_CFG")
> diff --git a/lib/python2.7/site-packages/autobuilder/lib/daft.py
> b/lib/python2.7/site-packages/autobuilder/lib/daft.py
> new file mode 100644
> index 0000000..eef4bab
> --- /dev/null
> +++ b/lib/python2.7/site-packages/autobuilder/lib/daft.py
> @@ -0,0 +1,115 @@
> +import os
> +import ConfigParser as configparser
> +
> +class DeployScanner(object):
> +    '''
> +    In charge of scanning deployed daft bbb devices
> +    '''
> +
> +    __MAGIC_SERVER_ADDRESS = '192.168.30.1' # DAFT uses this address
> for this internal network
> +    __MAGIC_SSH_PORT = 2233 # it is hardcoded as per DAFT
> implementation manual
> +
> +    def  __init__(self, *args, **kwargs):
> +        self.devsconf_fp = kwargs.get('devsconf_file', None)
> +        if not self.devsconf_fp:
> +            raise Exception('not fed devsconf file')
> +        self.ign_leases = kwargs.get('ign_leases', True)
> +        if not self.ign_leases:
> +            self.leases_file_path =  kwargs.get('leases_file', None)
> +            if not self.leases_file_path:
> +                raise Exception('not fed leases file')
> +
> +    def __call__(self):
> +        '''
> +        Creates relation of deployed devices
> +        Returns:
> +            List of dictionaries containing info about devices
> deployed.
> +        '''
> +        def create_relation(i,j):
> +            r = []

Please use more verbose variable names for clarity. 

> +            for conf in i:
> +                for active in j:
> +                    if conf['bb_ip'] == active['ip']:
> +                        r.append({
> +                            'dut_label': conf['device'].lower(),
> +                            'dut_family':
> conf['device_type'].lower(),
> +                            'dut_sshport':
> str(self.__MAGIC_SSH_PORT),
> +                            'ctrl_address': conf['bb_ip'],
> +                            'server_address':
> self.__MAGIC_SERVER_ADDRESS
> +                        })
> +            return r
> +
> +        def slack_relation(i):
> +            r = []
> +            for conf in i:

Please use more verbose variable names for clarity. 

> +                r.append({
> +                    'dut_label': conf['device'],
> +                    'dut_family': conf['device_type'],
> +                    'dut_sshport': str(self.__MAGIC_SSH_PORT),
> +                    'ctrl_address': conf['bb_ip'],
> +                    'server_address': self.__MAGIC_SERVER_ADDRESS
> +                })
> +            return r
> +
> +        fc = self.__fetch_confs()
> +        if not fc:
> +            raise Exception('There are no configurations as per BBB
> devices')
> +
> +        if self.ign_leases:
> +            # Devices that nobody knows if were deployed
> +            return slack_relation(fc)
> +
> +        als = self.__active_leases()
> +        if not als:
> +            raise Exception('DHCP server has not registered any host
> yet')
> +
> +        return create_relation(fc, als)
> +
> +    def __fetch_confs(self):
> +        '''
> +        Read and parse BBB configuration file and return result as
> dictionary
> +        '''
> +        config = configparser.SafeConfigParser()
> +        config.read(self.devsconf_fp)
> +        configurations = []
> +        for device in config.sections():
> +            device_config = dict(config.items(device))
> +            device_config["device"] = device
> +            device_config["device_type"] =
> device.rstrip('1234567890_')
> +            configurations.append(device_config)
> +        return configurations
> +
> +    def __active_leases(self):
> +        """
> +        Read the active leases from dnsmasq leases file and return a
> list of
> +        active leases as dictionaries.
> +        Args:
> +            file_name (str): Path to leases file, e.g.
> /path/to/file/dnsmasq.leases
> +        Returns:
> +            List of dictionaries containing the active leases.
> +            The dictionaries have the following format:
> +            {
> +                "mac": "device_mac_address",
> +                "ip": "device_ip_address",
> +                "hostname": "device_host_name",
> +                "client_id": "client_id_or_*_if_unset"
> +            }
> +        """
> +        with open(self.leases_file_path) as lease_file:
> +            leases = lease_file.readlines()
> +
> +        leases_list = []
> +
> +        # dnsmasq.leases contains rows with the following format:
> +        # <lease_expiry_time_as_epoch_format> <mac> <ip> <hostname>
> <domain>
> +        # See:
> +        #http://lists.thekelleys.org.uk/pipermail/dnsmasq-discuss/20
> 05q1/000143.html
> +        for lease in leases:
> +            lease = lease.split()
> +            leases_list.append({
> +                "mac": lease[1],
> +                "ip": lease[2],
> +                "hostname": lease[3],
> +                "client_id": lease[4],
> +            })
> +        return leases_list

Re: [yocto-autobuilder][PATCHv2 4/5] autobuilder/buildsteps: Add DAFTFlash buildstep

[Joshua Lock]

On Tue, 2017-06-06 at 10:46 -0500, Aníbal Limón wrote:
> This will execute daft flash cycle without reboot and prepare the dut
> for sanity tests.
> 
> [YOCTO #10604]
> 
> Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
> Signed-off-by: Monserrat Sedeno <monserratx.sedeno.bustos.intel.com>

Typo in the email address here.

> Signed-off-by: Edwin Plauchu <edwin.plauchu.camacho@linux.intel.com>
> ---
>  config/autobuilder.conf.example                    |  1 +
>  .../autobuilder/buildsteps/DaftFlash.py            | 42
> ++++++++++++++++++++++
>  lib/python2.7/site-packages/autobuilder/config.py  |  1 +
>  3 files changed, 44 insertions(+)
>  create mode 100644 lib/python2.7/site-
> packages/autobuilder/buildsteps/DaftFlash.py
> 
> diff --git a/config/autobuilder.conf.example
> b/config/autobuilder.conf.example
> index e5ec16b..2e992cf 100644
> --- a/config/autobuilder.conf.example
> +++ b/config/autobuilder.conf.example
> @@ -98,3 +98,4 @@ PERFORMANCE_MAIL_SIG = "Multiline\nSig\nLine"
>  
>  [Daft]
>  DAFT_WORKER_DEVICES_CFG = "/etc/daft/devices.cfg"
> +DAFT_WORKER_WORKSPACE = "/home/ab/workspace"
> diff --git a/lib/python2.7/site-
> packages/autobuilder/buildsteps/DaftFlash.py b/lib/python2.7/site-
> packages/autobuilder/buildsteps/DaftFlash.py
> new file mode 100644
> index 0000000..d5f6de8
> --- /dev/null
> +++ b/lib/python2.7/site-packages/autobuilder/buildsteps/DaftFlash.py
> @@ -0,0 +1,42 @@
> +import os
> +
> +from autobuilder.config import DAFT_WORKER_WORKSPACE
> +from buildbot.steps.shell import ShellCommand
> +
> +class DaftFlash(ShellCommand):
> +    haltOnFailure = True
> +
> +    name = "DaftFlash"
> +
> +    def __init__(self, factory, argdict=None, **kwargs):
> +        self.tests = None
> +        self.factory = factory
> +        for k, v in argdict.iteritems():
> +            setattr(self, k, v)
> +        self.description = "DUT Flashing"
> +        self.timeout = 100000
> +        kwargs['timeout']=self.timeout
> +        ShellCommand.__init__(self, **kwargs)
> +
> +    def start(self):
> +        workspace_dir = DAFT_WORKER_WORKSPACE
> +
> +        dut_name = self.getProperty('dut_name')
> +        workdir = self.getProperty('workdir')
> +        image = self.getProperty('BUILDIMAGES')

Remember to fix this when you rename the property.

> +        machine = self.getProperty('MACHINE')
> +
> +        image_path = os.path.join(workdir, 'build', 'build', 'tmp',
> 'deploy',
> +                'images', machine, '%s-%s.hddimg' % (image,
> machine))
> +        image_name = os.path.basename(image_path)
> +
> +        # XXX: DAFT needs to have the image in a shared workspace 
> +        self.command = "cp %s %s; " % (image_path, workspace_dir)
> +
> +        self.command += "cd %s; " % (workspace_dir)
> +
> +        # XXX: DAFT needs to be executed by root user

Why's that? Is that an intrinsic requirement of DAFT or is it because
DAFT wants to use certain devices that are typically owned by root? Can
we instead make the requirement that the user running daft has
permissions and groups correctly configured to access these devices?

> +        daft_cmd = "daft --setout %s %s" % (dut_name, image_name)
> +        self.command += "sudo su -l -c 'cd {} && {}';
> ".format(workspace_dir, daft_cmd)
> +
> +        ShellCommand.start(self)
> diff --git a/lib/python2.7/site-packages/autobuilder/config.py
> b/lib/python2.7/site-packages/autobuilder/config.py
> index 5bcf6c6..fcc08f0 100644
> --- a/lib/python2.7/site-packages/autobuilder/config.py
> +++ b/lib/python2.7/site-packages/autobuilder/config.py
> @@ -24,3 +24,4 @@ IPK_PUBLISH_DIR = os.environ.get("IPK_PUBLISH_DIR")
>  DEB_PUBLISH_DIR = os.environ.get("DEB_PUBLISH_DIR")
>  PERFORMANCE_PUBLISH_DIR = os.environ.get("PERFORMANCE_PUBLISH_DIR")
>  DAFT_WORKER_DEVICES_CFG = os.environ.get("DAFT_WORKER_DEVICES_CFG")
> +DAFT_WORKER_WORKSPACE = os.environ.get("DAFT_WORKER_WORKSPACE")

Re: [yocto-autobuilder][PATCHv2 3/5] autobuilder/buildsteps: Add DaftGetDevices step

[Aníbal Limón]

On 07/03/2017 09:03 AM, Joshua Lock wrote:
> On Tue, 2017-06-06 at 10:46 -0500, Aníbal Limón wrote:
>> This buildstep will copy the devices configuration from the
>> worker and read it, in order to get daft ip an ssh port, to
>> use later in RunSanityTests
>>
>> [YOCTO #10604]
>>
>> Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
>> Signed-off-by: Monserrat Sedeno <monserratx.sedeno.bustos.intel.com>
> 
> Typo in the email address here.
> 
>> Signed-off-by: Edwin Plauchu <edwin.plauchu.camacho@linux.intel.com>
>> ---
>>  config/autobuilder.conf.example                    |   3 +
>>  .../autobuilder/buildsteps/DaftGetDevices.py       |  44 ++++++++
>>  lib/python2.7/site-packages/autobuilder/config.py  |   1 +
>>  .../site-packages/autobuilder/lib/daft.py          | 115
>> +++++++++++++++++++++
>>  4 files changed, 163 insertions(+)
>>  create mode 100644 lib/python2.7/site-
>> packages/autobuilder/buildsteps/DaftGetDevices.py
>>  create mode 100644 lib/python2.7/site-
>> packages/autobuilder/lib/daft.py
>>
>> diff --git a/config/autobuilder.conf.example
>> b/config/autobuilder.conf.example
>> index 2ee11e6..e5ec16b 100644
>> --- a/config/autobuilder.conf.example
>> +++ b/config/autobuilder.conf.example
>> @@ -95,3 +95,6 @@ PERFORMANCE_MAIL_TO = "root@localhost otherperson@l
>> ocalhost"
>>  PERFORMANCE_MAIL_CC = "buildcc@localhost"
>>  PERFORMANCE_MAIL_BCC = "buildbcc@localhost"
>>  PERFORMANCE_MAIL_SIG = "Multiline\nSig\nLine"
>> +
>> +[Daft]
>> +DAFT_WORKER_DEVICES_CFG = "/etc/daft/devices.cfg"
>> diff --git a/lib/python2.7/site-
>> packages/autobuilder/buildsteps/DaftGetDevices.py
>> b/lib/python2.7/site-
>> packages/autobuilder/buildsteps/DaftGetDevices.py
>> new file mode 100644
>> index 0000000..133a4d6
>> --- /dev/null
>> +++ b/lib/python2.7/site-
>> packages/autobuilder/buildsteps/DaftGetDevices.py
>> @@ -0,0 +1,44 @@
>> +import os
>> +from buildbot.steps.transfer import FileUpload
>> +from buildbot.process.buildstep import BuildStep
>> +
>> +from lib.daft import DeployScanner
>> +from autobuilder.config import DAFT_WORKER_DEVICES_CFG
>> +
>> +class DaftGetDevices(FileUpload):
>> +    haltOnFailure = True
>> +
>> +    name = "DaftGetDevices"
>> +
>> +    def __init__(self, factory, argdict=None, **kwargs):
>> +        self.tests = None
>> +        self.factory = factory
>> +        for k, v in argdict.iteritems():
>> +            setattr(self, k, v)
>> +        self.description = "Getting devices configuration"
>> +        self.timeout = 100000
>> +        kwargs['timeout']=self.timeout
>> +
>> +        super(DaftGetDevices,
>> self).__init__(DAFT_WORKER_DEVICES_CFG,
>> +                os.path.join('/tmp', 'devices.cfg'))
>> +
>> +    def finished(self, result):
>> +        if self.cmd:
>> +            ds = DeployScanner(devsconf_file = self.masterdest)
>> +            devices = ds()
> 
> I'm not at all fond of this. It would be easier to follow this code if
> this were an explicit method.

This code was original created by Edwin, i will change the code to add
better clarity.

> 
>> +
>> +            found = False
>> +            dut_label = self.getProperty('custom_dut')
>> +            for d in devices:
>> +                if d['dut_label'] == dut_label:
>> +                    self.setProperty('dut_name', d['dut_label'],
>> 'DaftGetDevices')
>> +                    self.setProperty('server_ip',
>> d['server_address'], 'DaftGetDevices')
>> +                    target_ip = "%s:%s" % (d['ctrl_address'],
>> d['dut_sshport'])
>> +                    self.setProperty('target_ip', target_ip,
>> 'DaftGetDevices')
>> +
>> +                    found = True
>> +
>> +            if not found:
>> +                 return BuildStep.finished(self, FAILURE)
>> +
>> +        return super(DaftGetDevices, self).finished(result)
>> diff --git a/lib/python2.7/site-packages/autobuilder/config.py
>> b/lib/python2.7/site-packages/autobuilder/config.py
>> index 9d945b1..5bcf6c6 100644
>> --- a/lib/python2.7/site-packages/autobuilder/config.py
>> +++ b/lib/python2.7/site-packages/autobuilder/config.py
>> @@ -23,3 +23,4 @@ RPM_PUBLISH_DIR = os.environ.get("RPM_PUBLISH_DIR")
>>  IPK_PUBLISH_DIR = os.environ.get("IPK_PUBLISH_DIR")
>>  DEB_PUBLISH_DIR = os.environ.get("DEB_PUBLISH_DIR")
>>  PERFORMANCE_PUBLISH_DIR = os.environ.get("PERFORMANCE_PUBLISH_DIR")
>> +DAFT_WORKER_DEVICES_CFG = os.environ.get("DAFT_WORKER_DEVICES_CFG")
>> diff --git a/lib/python2.7/site-packages/autobuilder/lib/daft.py
>> b/lib/python2.7/site-packages/autobuilder/lib/daft.py
>> new file mode 100644
>> index 0000000..eef4bab
>> --- /dev/null
>> +++ b/lib/python2.7/site-packages/autobuilder/lib/daft.py
>> @@ -0,0 +1,115 @@
>> +import os
>> +import ConfigParser as configparser
>> +
>> +class DeployScanner(object):
>> +    '''
>> +    In charge of scanning deployed daft bbb devices
>> +    '''
>> +
>> +    __MAGIC_SERVER_ADDRESS = '192.168.30.1' # DAFT uses this address
>> for this internal network
>> +    __MAGIC_SSH_PORT = 2233 # it is hardcoded as per DAFT
>> implementation manual
>> +
>> +    def  __init__(self, *args, **kwargs):
>> +        self.devsconf_fp = kwargs.get('devsconf_file', None)
>> +        if not self.devsconf_fp:
>> +            raise Exception('not fed devsconf file')
>> +        self.ign_leases = kwargs.get('ign_leases', True)
>> +        if not self.ign_leases:
>> +            self.leases_file_path =  kwargs.get('leases_file', None)
>> +            if not self.leases_file_path:
>> +                raise Exception('not fed leases file')
>> +
>> +    def __call__(self):
>> +        '''
>> +        Creates relation of deployed devices
>> +        Returns:
>> +            List of dictionaries containing info about devices
>> deployed.
>> +        '''
>> +        def create_relation(i,j):
>> +            r = []
> 
> Please use more verbose variable names for clarity. 

Agree,

> 
>> +            for conf in i:
>> +                for active in j:
>> +                    if conf['bb_ip'] == active['ip']:
>> +                        r.append({
>> +                            'dut_label': conf['device'].lower(),
>> +                            'dut_family':
>> conf['device_type'].lower(),
>> +                            'dut_sshport':
>> str(self.__MAGIC_SSH_PORT),
>> +                            'ctrl_address': conf['bb_ip'],
>> +                            'server_address':
>> self.__MAGIC_SERVER_ADDRESS
>> +                        })
>> +            return r
>> +
>> +        def slack_relation(i):
>> +            r = []
>> +            for conf in i:
> 
> Please use more verbose variable names for clarity. 

Agree,

Anibal

> 
>> +                r.append({
>> +                    'dut_label': conf['device'],
>> +                    'dut_family': conf['device_type'],
>> +                    'dut_sshport': str(self.__MAGIC_SSH_PORT),
>> +                    'ctrl_address': conf['bb_ip'],
>> +                    'server_address': self.__MAGIC_SERVER_ADDRESS
>> +                })
>> +            return r
>> +
>> +        fc = self.__fetch_confs()
>> +        if not fc:
>> +            raise Exception('There are no configurations as per BBB
>> devices')
>> +
>> +        if self.ign_leases:
>> +            # Devices that nobody knows if were deployed
>> +            return slack_relation(fc)
>> +
>> +        als = self.__active_leases()
>> +        if not als:
>> +            raise Exception('DHCP server has not registered any host
>> yet')
>> +
>> +        return create_relation(fc, als)
>> +
>> +    def __fetch_confs(self):
>> +        '''
>> +        Read and parse BBB configuration file and return result as
>> dictionary
>> +        '''
>> +        config = configparser.SafeConfigParser()
>> +        config.read(self.devsconf_fp)
>> +        configurations = []
>> +        for device in config.sections():
>> +            device_config = dict(config.items(device))
>> +            device_config["device"] = device
>> +            device_config["device_type"] =
>> device.rstrip('1234567890_')
>> +            configurations.append(device_config)
>> +        return configurations
>> +
>> +    def __active_leases(self):
>> +        """
>> +        Read the active leases from dnsmasq leases file and return a
>> list of
>> +        active leases as dictionaries.
>> +        Args:
>> +            file_name (str): Path to leases file, e.g.
>> /path/to/file/dnsmasq.leases
>> +        Returns:
>> +            List of dictionaries containing the active leases.
>> +            The dictionaries have the following format:
>> +            {
>> +                "mac": "device_mac_address",
>> +                "ip": "device_ip_address",
>> +                "hostname": "device_host_name",
>> +                "client_id": "client_id_or_*_if_unset"
>> +            }
>> +        """
>> +        with open(self.leases_file_path) as lease_file:
>> +            leases = lease_file.readlines()
>> +
>> +        leases_list = []
>> +
>> +        # dnsmasq.leases contains rows with the following format:
>> +        # <lease_expiry_time_as_epoch_format> <mac> <ip> <hostname>
>> <domain>
>> +        # See:
>> +        #http://lists.thekelleys.org.uk/pipermail/dnsmasq-discuss/20
>> 05q1/000143.html
>> +        for lease in leases:
>> +            lease = lease.split()
>> +            leases_list.append({
>> +                "mac": lease[1],
>> +                "ip": lease[2],
>> +                "hostname": lease[3],
>> +                "client_id": lease[4],
>> +            })
>> +        return leases_list

Re: [yocto-autobuilder][PATCHv2 1/5] buildsteps/BuildImages: Setproperty BUILDIMAGES

[Aníbal Limón]

On 07/03/2017 09:02 AM, Joshua Lock wrote:
> On Tue, 2017-06-06 at 10:46 -0500, Aníbal Limón wrote:
>> In certain situations is need to know what images was built for
>> example the DAFT executes flashing over certain image.
>>
>> [YOCTO #10604]
>>
>> Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
>> ---
>>  lib/python2.7/site-packages/autobuilder/buildsteps/BuildImages.py |
>> 3 +++
>>  1 file changed, 3 insertions(+)
>>
>> diff --git a/lib/python2.7/site-
>> packages/autobuilder/buildsteps/BuildImages.py b/lib/python2.7/site-
>> packages/autobuilder/buildsteps/BuildImages.py
>> index 30f7a18..7262fd3 100644
>> --- a/lib/python2.7/site-
>> packages/autobuilder/buildsteps/BuildImages.py
>> +++ b/lib/python2.7/site-
>> packages/autobuilder/buildsteps/BuildImages.py
>> @@ -146,6 +146,9 @@ class BuildImages(BitbakeShellCommand):
>>                  if self.minnowExists is None or self.minnowExists ==
>> "True":
>>                      self.command = ". ./oe-init-build-env; bitbake "
>> + bitbakeflags + self.images
>>                      self.description = ["Building " +
>> str(self.images)]
>> +        
>> +        self.setProperty("BUILDIMAGES", self.images)
> 
> Generally speaking caps case property names should only be used when
> they map to a variable in OE. BuildImages would work.

Agree,

Anibal
> 
>> +
>>          ShellCommand.start(self)
>>  
>>      def describe(self, done=False):

Re: [yocto-autobuilder][PATCHv2 4/5] autobuilder/buildsteps: Add DAFTFlash buildstep

[Aníbal Limón]

On 07/03/2017 09:03 AM, Joshua Lock wrote:
> On Tue, 2017-06-06 at 10:46 -0500, Aníbal Limón wrote:
>> This will execute daft flash cycle without reboot and prepare the dut
>> for sanity tests.
>>
>> [YOCTO #10604]
>>
>> Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
>> Signed-off-by: Monserrat Sedeno <monserratx.sedeno.bustos.intel.com>
> 
> Typo in the email address here.
> 
>> Signed-off-by: Edwin Plauchu <edwin.plauchu.camacho@linux.intel.com>
>> ---
>>  config/autobuilder.conf.example                    |  1 +
>>  .../autobuilder/buildsteps/DaftFlash.py            | 42
>> ++++++++++++++++++++++
>>  lib/python2.7/site-packages/autobuilder/config.py  |  1 +
>>  3 files changed, 44 insertions(+)
>>  create mode 100644 lib/python2.7/site-
>> packages/autobuilder/buildsteps/DaftFlash.py
>>
>> diff --git a/config/autobuilder.conf.example
>> b/config/autobuilder.conf.example
>> index e5ec16b..2e992cf 100644
>> --- a/config/autobuilder.conf.example
>> +++ b/config/autobuilder.conf.example
>> @@ -98,3 +98,4 @@ PERFORMANCE_MAIL_SIG = "Multiline\nSig\nLine"
>>  
>>  [Daft]
>>  DAFT_WORKER_DEVICES_CFG = "/etc/daft/devices.cfg"
>> +DAFT_WORKER_WORKSPACE = "/home/ab/workspace"
>> diff --git a/lib/python2.7/site-
>> packages/autobuilder/buildsteps/DaftFlash.py b/lib/python2.7/site-
>> packages/autobuilder/buildsteps/DaftFlash.py
>> new file mode 100644
>> index 0000000..d5f6de8
>> --- /dev/null
>> +++ b/lib/python2.7/site-packages/autobuilder/buildsteps/DaftFlash.py
>> @@ -0,0 +1,42 @@
>> +import os
>> +
>> +from autobuilder.config import DAFT_WORKER_WORKSPACE
>> +from buildbot.steps.shell import ShellCommand
>> +
>> +class DaftFlash(ShellCommand):
>> +    haltOnFailure = True
>> +
>> +    name = "DaftFlash"
>> +
>> +    def __init__(self, factory, argdict=None, **kwargs):
>> +        self.tests = None
>> +        self.factory = factory
>> +        for k, v in argdict.iteritems():
>> +            setattr(self, k, v)
>> +        self.description = "DUT Flashing"
>> +        self.timeout = 100000
>> +        kwargs['timeout']=self.timeout
>> +        ShellCommand.__init__(self, **kwargs)
>> +
>> +    def start(self):
>> +        workspace_dir = DAFT_WORKER_WORKSPACE
>> +
>> +        dut_name = self.getProperty('dut_name')
>> +        workdir = self.getProperty('workdir')
>> +        image = self.getProperty('BUILDIMAGES')
> 
> Remember to fix this when you rename the property.
> 
>> +        machine = self.getProperty('MACHINE')
>> +
>> +        image_path = os.path.join(workdir, 'build', 'build', 'tmp',
>> 'deploy',
>> +                'images', machine, '%s-%s.hddimg' % (image,
>> machine))
>> +        image_name = os.path.basename(image_path)
>> +
>> +        # XXX: DAFT needs to have the image in a shared workspace 
>> +        self.command = "cp %s %s; " % (image_path, workspace_dir)
>> +
>> +        self.command += "cd %s; " % (workspace_dir)
>> +
>> +        # XXX: DAFT needs to be executed by root user
> 
> Why's that? Is that an intrinsic requirement of DAFT or is it because
> DAFT wants to use certain devices that are typically owned by root? Can
> we instead make the requirement that the user running daft has
> permissions and groups correctly configured to access these devices?

I don't have too much insight about this security issue, i think there
is because the deployment resources files are on /root directory, so i
agree that this needs to be changed but i don't know how much effort
will require.

I'm adding Jair since he knows more about this situation, and Miko that
as far as i know he works on refkit QA.

Cheers,
Anibal


> 
>> +        daft_cmd = "daft --setout %s %s" % (dut_name, image_name)
>> +        self.command += "sudo su -l -c 'cd {} && {}';
>> ".format(workspace_dir, daft_cmd)
>> +
>> +        ShellCommand.start(self)
>> diff --git a/lib/python2.7/site-packages/autobuilder/config.py
>> b/lib/python2.7/site-packages/autobuilder/config.py
>> index 5bcf6c6..fcc08f0 100644
>> --- a/lib/python2.7/site-packages/autobuilder/config.py
>> +++ b/lib/python2.7/site-packages/autobuilder/config.py
>> @@ -24,3 +24,4 @@ IPK_PUBLISH_DIR = os.environ.get("IPK_PUBLISH_DIR")
>>  DEB_PUBLISH_DIR = os.environ.get("DEB_PUBLISH_DIR")
>>  PERFORMANCE_PUBLISH_DIR = os.environ.get("PERFORMANCE_PUBLISH_DIR")
>>  DAFT_WORKER_DEVICES_CFG = os.environ.get("DAFT_WORKER_DEVICES_CFG")
>> +DAFT_WORKER_WORKSPACE = os.environ.get("DAFT_WORKER_WORKSPACE")

Re: [yocto-autobuilder][PATCHv2 4/5] autobuilder/buildsteps: Add DAFTFlash buildstep

[Ylinen, Mikko]

[-- Attachment #1: Type: text/plain, Size: 3912 bytes --]

On Mon, Jul 3, 2017 at 7:37 PM, Aníbal Limón <anibal.limon@linux.intel.com>
wrote:

>
>
> On 07/03/2017 09:03 AM, Joshua Lock wrote:
> > On Tue, 2017-06-06 at 10:46 -0500, Aníbal Limón wrote:
> >> This will execute daft flash cycle without reboot and prepare the dut
> >> for sanity tests.
> >>
> >> [YOCTO #10604]
> >>
> >> Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
> >> Signed-off-by: Monserrat Sedeno <monserratx.sedeno.bustos.intel.com>
> >
> > Typo in the email address here.
> >
> >> Signed-off-by: Edwin Plauchu <edwin.plauchu.camacho@linux.intel.com>
> >> ---
> >>  config/autobuilder.conf.example                    |  1 +
> >>  .../autobuilder/buildsteps/DaftFlash.py            | 42
> >> ++++++++++++++++++++++
> >>  lib/python2.7/site-packages/autobuilder/config.py  |  1 +
> >>  3 files changed, 44 insertions(+)
> >>  create mode 100644 lib/python2.7/site-
> >> packages/autobuilder/buildsteps/DaftFlash.py
> >>
> >> diff --git a/config/autobuilder.conf.example
> >> b/config/autobuilder.conf.example
> >> index e5ec16b..2e992cf 100644
> >> --- a/config/autobuilder.conf.example
> >> +++ b/config/autobuilder.conf.example
> >> @@ -98,3 +98,4 @@ PERFORMANCE_MAIL_SIG = "Multiline\nSig\nLine"
> >>
> >>  [Daft]
> >>  DAFT_WORKER_DEVICES_CFG = "/etc/daft/devices.cfg"
> >> +DAFT_WORKER_WORKSPACE = "/home/ab/workspace"
> >> diff --git a/lib/python2.7/site-
> >> packages/autobuilder/buildsteps/DaftFlash.py b/lib/python2.7/site-
> >> packages/autobuilder/buildsteps/DaftFlash.py
> >> new file mode 100644
> >> index 0000000..d5f6de8
> >> --- /dev/null
> >> +++ b/lib/python2.7/site-packages/autobuilder/buildsteps/DaftFlash.py
> >> @@ -0,0 +1,42 @@
> >> +import os
> >> +
> >> +from autobuilder.config import DAFT_WORKER_WORKSPACE
> >> +from buildbot.steps.shell import ShellCommand
> >> +
> >> +class DaftFlash(ShellCommand):
> >> +    haltOnFailure = True
> >> +
> >> +    name = "DaftFlash"
> >> +
> >> +    def __init__(self, factory, argdict=None, **kwargs):
> >> +        self.tests = None
> >> +        self.factory = factory
> >> +        for k, v in argdict.iteritems():
> >> +            setattr(self, k, v)
> >> +        self.description = "DUT Flashing"
> >> +        self.timeout = 100000
> >> +        kwargs['timeout']=self.timeout
> >> +        ShellCommand.__init__(self, **kwargs)
> >> +
> >> +    def start(self):
> >> +        workspace_dir = DAFT_WORKER_WORKSPACE
> >> +
> >> +        dut_name = self.getProperty('dut_name')
> >> +        workdir = self.getProperty('workdir')
> >> +        image = self.getProperty('BUILDIMAGES')
> >
> > Remember to fix this when you rename the property.
> >
> >> +        machine = self.getProperty('MACHINE')
> >> +
> >> +        image_path = os.path.join(workdir, 'build', 'build', 'tmp',
> >> 'deploy',
> >> +                'images', machine, '%s-%s.hddimg' % (image,
> >> machine))
> >> +        image_name = os.path.basename(image_path)
> >> +
> >> +        # XXX: DAFT needs to have the image in a shared workspace
> >> +        self.command = "cp %s %s; " % (image_path, workspace_dir)
> >> +
> >> +        self.command += "cd %s; " % (workspace_dir)
> >> +
> >> +        # XXX: DAFT needs to be executed by root user
> >
> > Why's that? Is that an intrinsic requirement of DAFT or is it because
> > DAFT wants to use certain devices that are typically owned by root? Can
> > we instead make the requirement that the user running daft has
> > permissions and groups correctly configured to access these devices?
>
> I don't have too much insight about this security issue, i think there
> is because the deployment resources files are on /root directory, so i
> agree that this needs to be changed but i don't know how much effort
> will require.
>
>
DAFT is not run as root (by Refkit CI at least).

-- Mikko

[-- Attachment #2: Type: text/html, Size: 5520 bytes --]

Re: [yocto-autobuilder][PATCHv2 4/5] autobuilder/buildsteps: Add DAFTFlash buildstep

[Gonzalez Plascencia, Jair De Jesus]

[-- Attachment #1: Type: text/plain, Size: 4703 bytes --]

It seems to be a configuration issue. Currently, I’m working on the test harness we’ll use to expand our DAFT setup, and as part of it, I’ve updated the BeagleBone filesystem. Once it is deployed, the AB should be able to call DAFT without using the root user.

Regards,
Jair

From: Ylinen, Mikko [mailto:mikko.ylinen@intel.com]
Sent: Monday, July 3, 2017 1:23 PM
To: Aníbal Limón <anibal.limon@linux.intel.com>
Cc: Joshua Lock <joshua.g.lock@linux.intel.com>; yocto@yoctoproject.org; Sedeno Bustos, MonserratX <monserratx.sedeno.bustos@intel.com>; juan.m.cruz.alcaraz@linux.intel.com; jose.perez.carranza@linux.intel.com; Gonzalez Plascencia, Jair De Jesus <jair.de.jesus.gonzalez.plascencia@intel.com>
Subject: Re: [yocto-autobuilder][PATCHv2 4/5] autobuilder/buildsteps: Add DAFTFlash buildstep



On Mon, Jul 3, 2017 at 7:37 PM, Aníbal Limón <anibal.limon@linux.intel.com<mailto:anibal.limon@linux.intel.com>> wrote:


On 07/03/2017 09:03 AM, Joshua Lock wrote:
> On Tue, 2017-06-06 at 10:46 -0500, Aníbal Limón wrote:
>> This will execute daft flash cycle without reboot and prepare the dut
>> for sanity tests.
>>
>> [YOCTO #10604]
>>
>> Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com<mailto:anibal.limon@linux.intel.com>>
>> Signed-off-by: Monserrat Sedeno <monserratx.sedeno.bustos.intel.com<http://monserratx.sedeno.bustos.intel.com>>
>
> Typo in the email address here.
>
>> Signed-off-by: Edwin Plauchu <edwin.plauchu.camacho@linux.intel.com<mailto:edwin.plauchu.camacho@linux.intel.com>>
>> ---
>>  config/autobuilder.conf.example                    |  1 +
>>  .../autobuilder/buildsteps/DaftFlash.py            | 42
>> ++++++++++++++++++++++
>>  lib/python2.7/site-packages/autobuilder/config.py  |  1 +
>>  3 files changed, 44 insertions(+)
>>  create mode 100644 lib/python2.7/site-
>> packages/autobuilder/buildsteps/DaftFlash.py
>>
>> diff --git a/config/autobuilder.conf.example
>> b/config/autobuilder.conf.example
>> index e5ec16b..2e992cf 100644
>> --- a/config/autobuilder.conf.example
>> +++ b/config/autobuilder.conf.example
>> @@ -98,3 +98,4 @@ PERFORMANCE_MAIL_SIG = "Multiline\nSig\nLine"
>>
>>  [Daft]
>>  DAFT_WORKER_DEVICES_CFG = "/etc/daft/devices.cfg"
>> +DAFT_WORKER_WORKSPACE = "/home/ab/workspace"
>> diff --git a/lib/python2.7/site-
>> packages/autobuilder/buildsteps/DaftFlash.py b/lib/python2.7/site-
>> packages/autobuilder/buildsteps/DaftFlash.py
>> new file mode 100644
>> index 0000000..d5f6de8
>> --- /dev/null
>> +++ b/lib/python2.7/site-packages/autobuilder/buildsteps/DaftFlash.py
>> @@ -0,0 +1,42 @@
>> +import os
>> +
>> +from autobuilder.config import DAFT_WORKER_WORKSPACE
>> +from buildbot.steps.shell import ShellCommand
>> +
>> +class DaftFlash(ShellCommand):
>> +    haltOnFailure = True
>> +
>> +    name = "DaftFlash"
>> +
>> +    def __init__(self, factory, argdict=None, **kwargs):
>> +        self.tests = None
>> +        self.factory = factory
>> +        for k, v in argdict.iteritems():
>> +            setattr(self, k, v)
>> +        self.description = "DUT Flashing"
>> +        self.timeout = 100000
>> +        kwargs['timeout']=self.timeout
>> +        ShellCommand.__init__(self, **kwargs)
>> +
>> +    def start(self):
>> +        workspace_dir = DAFT_WORKER_WORKSPACE
>> +
>> +        dut_name = self.getProperty('dut_name')
>> +        workdir = self.getProperty('workdir')
>> +        image = self.getProperty('BUILDIMAGES')
>
> Remember to fix this when you rename the property.
>
>> +        machine = self.getProperty('MACHINE')
>> +
>> +        image_path = os.path.join(workdir, 'build', 'build', 'tmp',
>> 'deploy',
>> +                'images', machine, '%s-%s.hddimg' % (image,
>> machine))
>> +        image_name = os.path.basename(image_path)
>> +
>> +        # XXX: DAFT needs to have the image in a shared workspace
>> +        self.command = "cp %s %s; " % (image_path, workspace_dir)
>> +
>> +        self.command += "cd %s; " % (workspace_dir)
>> +
>> +        # XXX: DAFT needs to be executed by root user
>
> Why's that? Is that an intrinsic requirement of DAFT or is it because
> DAFT wants to use certain devices that are typically owned by root? Can
> we instead make the requirement that the user running daft has
> permissions and groups correctly configured to access these devices?

I don't have too much insight about this security issue, i think there
is because the deployment resources files are on /root directory, so i
agree that this needs to be changed but i don't know how much effort
will require.

DAFT is not run as root (by Refkit CI at least).

-- Mikko

[-- Attachment #2: Type: text/html, Size: 10077 bytes --]