[PATCH v2 0/3] remove ssh build controller

[PATCH v2 0/3] remove ssh build controller

[Ed Bartosh]

Hi,

This is a second attempt to get this through review. As ic branch has
been successfuly landed I hope this one can be upstreamed too.

sshbecontroller code and tests are broken for months. Its functionality(running
builds on remote machine) can be achieved by simply running manage.py runbuilds
on remote machine. There is not need to keep it in the codebase and confuse
users from my point of view.

This change would also make refactoring of buildcontroller code
[YOCTO #8806] easier.

Changes in v2: rebased on top of toaster-next.

The following changes since commit 31e7bb2c5e900c44aded447dcaf9b090dd2cb11d:

  toaster: change 'delete layer' to 'remove layer' (2016-02-25 14:26:41 +0000)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib ed/toaster/remove-ssh-controller
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/toaster/remove-ssh-controller

Ed Bartosh (3):
  toaster: raise NotImplementedError
  toaster: don't use sshbecontroller
  toaster: remove sshbecontroller module

 bitbake/lib/toaster/bldcontrol/bbcontroller.py     |  16 +-
 .../migrations/0002_auto_20160120_1250.py          |  19 +++
 bitbake/lib/toaster/bldcontrol/models.py           |  10 +-
 bitbake/lib/toaster/bldcontrol/sshbecontroller.py  | 169 ---------------------
 bitbake/lib/toaster/bldcontrol/tests.py            |  30 +---
 5 files changed, 32 insertions(+), 212 deletions(-)
 create mode 100644 bitbake/lib/toaster/bldcontrol/migrations/0002_auto_20160120_1250.py
 delete mode 100644 bitbake/lib/toaster/bldcontrol/sshbecontroller.py

--
Regards,
Ed

[PATCH v2 1/3] toaster: raise NotImplementedError

[Ed Bartosh]

Raised NotImplementedError instead of Exception to be
able to catch it.

This is a preparation for removing sshbecontroller module.
It has to be done as code in bldcontrol/tests.py imports custom
NotImplementedException from sshbecontroller.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/toaster/bldcontrol/bbcontroller.py | 10 +++++-----
 bitbake/lib/toaster/bldcontrol/models.py       |  8 ++++++--
 bitbake/lib/toaster/bldcontrol/tests.py        |  6 ++----
 3 files changed, 13 insertions(+), 11 deletions(-)

diff --git a/bitbake/lib/toaster/bldcontrol/bbcontroller.py b/bitbake/lib/toaster/bldcontrol/bbcontroller.py
index 1387bda..f228d37 100644
--- a/bitbake/lib/toaster/bldcontrol/bbcontroller.py
+++ b/bitbake/lib/toaster/bldcontrol/bbcontroller.py
@@ -138,7 +138,7 @@ class BuildEnvironmentController(object):
             After this method executes, self.be bbaddress/bbport MUST point to a running and free server,
             and the bbstate MUST be  updated to "started".
         """
-        raise Exception("FIXME: Must override in order to actually start the BB server")
+        raise NotImplementedError("FIXME: Must override in order to actually start the BB server")
 
 
     def setLayers(self, bitbake, ls):
@@ -149,7 +149,7 @@ class BuildEnvironmentController(object):
 
             a word of attention: by convention, the first layer for any build will be poky!
         """
-        raise Exception("FIXME: Must override setLayers")
+        raise NotImplementedError("FIXME: Must override setLayers")
 
 
     def getBBController(self):
@@ -176,16 +176,16 @@ class BuildEnvironmentController(object):
             up to the implementing BEC. The return MUST be a REST URL where a GET will actually return
             the content of the artifact, e.g. for use as a "download link" in a web UI.
         """
-        raise Exception("Must return the REST URL of the artifact")
+        raise NotImplementedError("Must return the REST URL of the artifact")
 
     def release(self):
         """ This stops the server and releases any resources. After this point, all resources
             are un-available for further reference
         """
-        raise Exception("Must override BE release")
+        raise NotImplementedError("Must override BE release")
 
     def triggerBuild(self, bitbake, layers, variables, targets):
-        raise Exception("Must override BE release")
+        raise NotImplementedError("Must override BE release")
 
 class ShellCmdException(Exception):
     pass
diff --git a/bitbake/lib/toaster/bldcontrol/models.py b/bitbake/lib/toaster/bldcontrol/models.py
index a3a49ce..bb613c6 100644
--- a/bitbake/lib/toaster/bldcontrol/models.py
+++ b/bitbake/lib/toaster/bldcontrol/models.py
@@ -42,13 +42,17 @@ class BuildEnvironment(models.Model):
     def get_artifact(self, path):
         if self.betype == BuildEnvironment.TYPE_LOCAL:
             return open(path, "r")
-        raise Exception("FIXME: artifact download not implemented for build environment type %s" % self.get_betype_display())
+        raise NotImplementedError("FIXME: artifact download not implemented "\
+                                  "for build environment type %s" % \
+                                  self.get_betype_display())
 
     def has_artifact(self, path):
         import os
         if self.betype == BuildEnvironment.TYPE_LOCAL:
             return os.path.exists(path)
-        raise Exception("FIXME: has artifact not implemented for build environment type %s" % self.get_betype_display())
+        raise NotImplementedError("FIXME: has artifact not implemented for "\
+                                  "build environment type %s" % \
+                                  self.get_betype_display())
 
 # a BuildRequest is a request that the scheduler will build using a BuildEnvironment
 # the build request queue is the table itself, ordered by state
diff --git a/bitbake/lib/toaster/bldcontrol/tests.py b/bitbake/lib/toaster/bldcontrol/tests.py
index 141b42a..e808991 100644
--- a/bitbake/lib/toaster/bldcontrol/tests.py
+++ b/bitbake/lib/toaster/bldcontrol/tests.py
@@ -48,13 +48,12 @@ class BEControllerTests(object):
         self.assertTrue(err == '', "bitbake server pid %s not stopped" % err)
 
     def test_serverStartAndStop(self):
-        from bldcontrol.sshbecontroller import NotImplementedException
         obe =  self._getBuildEnvironment()
         bc = self._getBEController(obe)
         try:
             # setting layers, skip any layer info
             bc.setLayers(BITBAKE_LAYER, POKY_LAYERS)
-        except NotImplementedException,  e:
+        except NotImplementedError:
             print "Test skipped due to command not implemented yet"
             return True
         # We are ok with the exception as we're handling the git already exists
@@ -74,14 +73,13 @@ class BEControllerTests(object):
         self._serverForceStop(bc)
 
     def test_getBBController(self):
-        from bldcontrol.sshbecontroller import NotImplementedException
         obe = self._getBuildEnvironment()
         bc = self._getBEController(obe)
         layerSet = False
         try:
             # setting layers, skip any layer info
             layerSet = bc.setLayers(BITBAKE_LAYER, POKY_LAYERS)
-        except NotImplementedException:
+        except NotImplementedError:
             print "Test skipped due to command not implemented yet"
             return True
         # We are ok with the exception as we're handling the git already exists
-- 
2.1.4

[PATCH v2 2/3] toaster: don't use sshbecontroller

[Ed Bartosh]

Removed usage of sshbecontroller from bbcontroller, models, tests
and database schema.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/toaster/bldcontrol/bbcontroller.py     |  6 ------
 .../migrations/0002_auto_20160120_1250.py          | 19 +++++++++++++++++
 bitbake/lib/toaster/bldcontrol/models.py           |  2 --
 bitbake/lib/toaster/bldcontrol/tests.py            | 24 ----------------------
 4 files changed, 19 insertions(+), 32 deletions(-)
 create mode 100644 bitbake/lib/toaster/bldcontrol/migrations/0002_auto_20160120_1250.py

diff --git a/bitbake/lib/toaster/bldcontrol/bbcontroller.py b/bitbake/lib/toaster/bldcontrol/bbcontroller.py
index f228d37..f40103c 100644
--- a/bitbake/lib/toaster/bldcontrol/bbcontroller.py
+++ b/bitbake/lib/toaster/bldcontrol/bbcontroller.py
@@ -76,13 +76,10 @@ def getBuildEnvironmentController(**kwargs):
     """
 
     from localhostbecontroller import LocalhostBEController
-    from sshbecontroller    import SSHBEController
 
     be = BuildEnvironment.objects.filter(Q(**kwargs))[0]
     if be.betype == BuildEnvironment.TYPE_LOCAL:
         return LocalhostBEController(be)
-    elif be.betype == BuildEnvironment.TYPE_SSH:
-        return SSHBEController(be)
     else:
         raise Exception("FIXME: Implement BEC for type %s" % str(be.betype))
 
@@ -105,9 +102,6 @@ class BuildEnvironmentController(object):
         on the local machine, with the "build/" directory under the "poky/" source checkout directory.
         Bash is expected to be available.
 
-            * SSH controller will run the Toaster BE on a remote machine, where the current user
-        can connect without raise Exception("FIXME: implement")word (set up with either ssh-agent or raise Exception("FIXME: implement")phrase-less key authentication)
-
     """
     def __init__(self, be):
         """ Takes a BuildEnvironment object as parameter that points to the settings of the BE.
diff --git a/bitbake/lib/toaster/bldcontrol/migrations/0002_auto_20160120_1250.py b/bitbake/lib/toaster/bldcontrol/migrations/0002_auto_20160120_1250.py
new file mode 100644
index 0000000..0c2475a
--- /dev/null
+++ b/bitbake/lib/toaster/bldcontrol/migrations/0002_auto_20160120_1250.py
@@ -0,0 +1,19 @@
+# -*- coding: utf-8 -*-
+from __future__ import unicode_literals
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('bldcontrol', '0001_initial'),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name='buildenvironment',
+            name='betype',
+            field=models.IntegerField(choices=[(0, b'local')]),
+        ),
+    ]
diff --git a/bitbake/lib/toaster/bldcontrol/models.py b/bitbake/lib/toaster/bldcontrol/models.py
index bb613c6..9244ed1 100644
--- a/bitbake/lib/toaster/bldcontrol/models.py
+++ b/bitbake/lib/toaster/bldcontrol/models.py
@@ -12,10 +12,8 @@ class BuildEnvironment(models.Model):
     )
 
     TYPE_LOCAL = 0
-    TYPE_SSH   = 1
     TYPE = (
         (TYPE_LOCAL, "local"),
-        (TYPE_SSH, "ssh"),
     )
 
     LOCK_FREE = 0
diff --git a/bitbake/lib/toaster/bldcontrol/tests.py b/bitbake/lib/toaster/bldcontrol/tests.py
index e808991..f20cc7d 100644
--- a/bitbake/lib/toaster/bldcontrol/tests.py
+++ b/bitbake/lib/toaster/bldcontrol/tests.py
@@ -9,7 +9,6 @@ from django.test import TestCase
 
 from bldcontrol.bbcontroller import BitbakeController, BuildSetupException
 from bldcontrol.localhostbecontroller import LocalhostBEController
-from bldcontrol.sshbecontroller import SSHBEController
 from bldcontrol.models import BuildEnvironment, BuildRequest
 from bldcontrol.management.commands.runbuilds import Command
 
@@ -110,29 +109,6 @@ class LocalhostBEControllerTests(TestCase, BEControllerTests):
     def _getBEController(self, obe):
         return LocalhostBEController(obe)
 
-class SSHBEControllerTests(TestCase, BEControllerTests):
-    def __init__(self, *args):
-        super(SSHBEControllerTests, self).__init__(*args)
-
-    def _getBuildEnvironment(self):
-        return BuildEnvironment.objects.create(
-                lock = BuildEnvironment.LOCK_FREE,
-                betype = BuildEnvironment.TYPE_SSH,
-                address = test_address,
-                sourcedir = test_sourcedir,
-                builddir = test_builddir )
-
-    def _getBEController(self, obe):
-        return SSHBEController(obe)
-
-    def test_pathExists(self):
-        obe = BuildEnvironment.objects.create(betype = BuildEnvironment.TYPE_SSH, address= test_address)
-        sbc = SSHBEController(obe)
-        self.assertTrue(sbc._pathexists("/"))
-        self.assertFalse(sbc._pathexists("/.deadbeef"))
-        self.assertTrue(sbc._pathexists(sbc._shellcmd("pwd")))
-
-
 class RunBuildsCommandTests(TestCase):
     def test_bec_select(self):
         """
-- 
2.1.4

[PATCH v2 3/3] toaster: remove sshbecontroller module

[Ed Bartosh]

The code of this module is broken for a long time.
The functionality of it can be easily achieved by running
'manage.py runbuilds.py' on remote machine.

[YOCTO #8806]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/toaster/bldcontrol/sshbecontroller.py | 169 ----------------------
 1 file changed, 169 deletions(-)
 delete mode 100644 bitbake/lib/toaster/bldcontrol/sshbecontroller.py

diff --git a/bitbake/lib/toaster/bldcontrol/sshbecontroller.py b/bitbake/lib/toaster/bldcontrol/sshbecontroller.py
deleted file mode 100644
index 17dd66c..0000000
--- a/bitbake/lib/toaster/bldcontrol/sshbecontroller.py
+++ /dev/null
@@ -1,169 +0,0 @@
-#
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
-#
-# BitBake Toaster Implementation
-#
-# Copyright (C) 2014        Intel Corporation
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-
-import sys
-import re
-from django.db import transaction
-from django.db.models import Q
-from bldcontrol.models import BuildEnvironment, BRLayer, BRVariable, BRTarget, BRBitbake
-import subprocess
-
-from toastermain import settings
-
-from bbcontroller import BuildEnvironmentController, ShellCmdException, BuildSetupException
-
-class NotImplementedException(Exception):
-    pass
-
-def DN(path):
-    return "/".join(path.split("/")[0:-1])
-
-class SSHBEController(BuildEnvironmentController):
-    """ Implementation of the BuildEnvironmentController for the localhost;
-        this controller manages the default build directory,
-        the server setup and system start and stop for the localhost-type build environment
-
-    """
-
-    def __init__(self, be):
-        super(SSHBEController, self).__init__(be)
-        self.dburl = settings.getDATABASE_URL()
-        self.pokydirname = None
-        self.islayerset = False
-
-    def _shellcmd(self, command, cwd = None):
-        if cwd is None:
-            cwd = self.be.sourcedir
-
-        p = subprocess.Popen("ssh %s 'cd %s && %s'" % (self.be.address, cwd, command), stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
-        (out,err) = p.communicate()
-        if p.returncode:
-            if len(err) == 0:
-                err = "command: %s \n%s" % (command, out)
-            else:
-                err = "command: %s \n%s" % (command, err)
-            raise ShellCmdException(err)
-        else:
-            return out.strip()
-
-    def _pathexists(self, path):
-        try:
-            self._shellcmd("test -e \"%s\"" % path)
-            return True
-        except ShellCmdException as e:
-            return False
-
-    def _pathcreate(self, path):
-        self._shellcmd("mkdir -p \"%s\"" % path)
-
-    def _setupBE(self):
-        assert self.pokydirname and self._pathexists(self.pokydirname)
-        self._pathcreate(self.be.builddir)
-        self._shellcmd("bash -c \"source %s/oe-init-build-env %s\"" % (self.pokydirname, self.be.builddir))
-
-    def startBBServer(self, brbe):
-        assert self.pokydirname and self._pathexists(self.pokydirname)
-        assert self.islayerset
-        cmd = self._shellcmd("bash -c \"source %s/oe-init-build-env %s && DATABASE_URL=%s source toaster start noweb brbe=%s\"" % (self.pokydirname, self.be.builddir, self.dburl, brbe))
-
-        port = "-1"
-        for i in cmd.split("\n"):
-            if i.startswith("Bitbake server address"):
-                port = i.split(" ")[-1]
-                print "Found bitbake server port ", port
-
-
-        assert self.be.sourcedir and self._pathexists(self.be.builddir)
-        self.be.bbaddress = self.be.address.split("@")[-1]
-        self.be.bbport = port
-        self.be.bbstate = BuildEnvironment.SERVER_STARTED
-        self.be.save()
-
-    def _copyFile(self, filepath1, filepath2):
-        p = subprocess.Popen("scp '%s' '%s'" % (filepath1, filepath2), stdout=subprocess.PIPE, stderr = subprocess.PIPE, shell=True)
-        (out, err) = p.communicate()
-        if p.returncode:
-            raise ShellCmdException(err)
-
-    def pullFile(self, local_filename, remote_filename):
-        _copyFile(local_filename, "%s:%s" % (self.be.address, remote_filename))
-
-    def pushFile(self, local_filename, remote_filename):
-        _copyFile("%s:%s" % (self.be.address, remote_filename), local_filename)
-
-    def setLayers(self, bitbakes, layers):
-        """ a word of attention: by convention, the first layer for any build will be poky! """
-
-        assert self.be.sourcedir is not None
-        assert len(bitbakes) == 1
-        # set layers in the layersource
-
-
-        raise NotImplementedException("Not implemented: SSH setLayers")
-        # 3. configure the build environment, so we have a conf/bblayers.conf
-        assert self.pokydirname is not None
-        self._setupBE()
-
-        # 4. update the bblayers.conf
-        bblayerconf = os.path.join(self.be.builddir, "conf/bblayers.conf")
-        if not self._pathexists(bblayerconf):
-            raise BuildSetupException("BE is not consistent: bblayers.conf file missing at %s" % bblayerconf)
-
-        import uuid
-        local_bblayerconf = "/tmp/" + uuid.uuid4() + "-bblayer.conf"
-
-        self.pullFile(bblayerconf, local_bblayerconf)
-
-        BuildEnvironmentController._updateBBLayers(local_bblayerconf, layerlist)
-        self.pushFile(local_bblayerconf, bblayerconf)
-
-        os.unlink(local_bblayerconf)
-
-        self.islayerset = True
-        return True
-
-    def release(self):
-        assert self.be.sourcedir and self._pathexists(self.be.builddir)
-        import shutil
-        shutil.rmtree(os.path.join(self.be.sourcedir, "build"))
-        assert not self._pathexists(self.be.builddir)
-
-    def triggerBuild(self, bitbake, layers, variables, targets):
-        # set up the buid environment with the needed layers
-        self.setLayers(bitbake, layers)
-        self.writeConfFile("conf/toaster-pre.conf", )
-        self.writeConfFile("conf/toaster.conf", raw = "INHERIT+=\"toaster buildhistory\"")
-
-        # get the bb server running with the build req id and build env id
-        bbctrl = self.getBBController()
-
-        # trigger the build command
-        task = reduce(lambda x, y: x if len(y)== 0 else y, map(lambda y: y.task, targets))
-        if len(task) == 0:
-            task = None
-
-        bbctrl.build(list(map(lambda x:x.target, targets)), task)
-
-        logger.debug("localhostbecontroller: Build launched, exiting. Follow build logs at %s/toaster_ui.log" % self.be.builddir)
-
-        # disconnect from the server
-        bbctrl.disconnect()
-- 
2.1.4

Re: [PATCH v2 0/3] remove ssh build controller

[Brian Avery]

upstreamed.
ty,
b

On Fri, Feb 26, 2016 at 12:06 AM, Ed Bartosh <ed.bartosh@linux.intel.com> wrote:
> Hi,
>
> This is a second attempt to get this through review. As ic branch has
> been successfuly landed I hope this one can be upstreamed too.
>
> sshbecontroller code and tests are broken for months. Its functionality(running
> builds on remote machine) can be achieved by simply running manage.py runbuilds
> on remote machine. There is not need to keep it in the codebase and confuse
> users from my point of view.
>
> This change would also make refactoring of buildcontroller code
> [YOCTO #8806] easier.
>
> Changes in v2: rebased on top of toaster-next.
>
> The following changes since commit 31e7bb2c5e900c44aded447dcaf9b090dd2cb11d:
>
>   toaster: change 'delete layer' to 'remove layer' (2016-02-25 14:26:41 +0000)
>
> are available in the git repository at:
>
>   git://git.yoctoproject.org/poky-contrib ed/toaster/remove-ssh-controller
>   http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/toaster/remove-ssh-controller
>
> Ed Bartosh (3):
>   toaster: raise NotImplementedError
>   toaster: don't use sshbecontroller
>   toaster: remove sshbecontroller module
>
>  bitbake/lib/toaster/bldcontrol/bbcontroller.py     |  16 +-
>  .../migrations/0002_auto_20160120_1250.py          |  19 +++
>  bitbake/lib/toaster/bldcontrol/models.py           |  10 +-
>  bitbake/lib/toaster/bldcontrol/sshbecontroller.py  | 169 ---------------------
>  bitbake/lib/toaster/bldcontrol/tests.py            |  30 +---
>  5 files changed, 32 insertions(+), 212 deletions(-)
>  create mode 100644 bitbake/lib/toaster/bldcontrol/migrations/0002_auto_20160120_1250.py
>  delete mode 100644 bitbake/lib/toaster/bldcontrol/sshbecontroller.py
>
> --
> Regards,
> Ed
> --
> _______________________________________________
> toaster mailing list
> toaster@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/toaster