* [PATCH v2 1/3] toaster: raise NotImplementedError
2016-02-26 8:06 [PATCH v2 0/3] remove ssh build controller Ed Bartosh
@ 2016-02-26 8:06 ` Ed Bartosh
2016-02-26 8:06 ` [PATCH v2 2/3] toaster: don't use sshbecontroller Ed Bartosh
` (2 subsequent siblings)
3 siblings, 0 replies; 5+ messages in thread
From: Ed Bartosh @ 2016-02-26 8:06 UTC (permalink / raw)
To: toaster
Raised NotImplementedError instead of Exception to be
able to catch it.
This is a preparation for removing sshbecontroller module.
It has to be done as code in bldcontrol/tests.py imports custom
NotImplementedException from sshbecontroller.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
bitbake/lib/toaster/bldcontrol/bbcontroller.py | 10 +++++-----
bitbake/lib/toaster/bldcontrol/models.py | 8 ++++++--
bitbake/lib/toaster/bldcontrol/tests.py | 6 ++----
3 files changed, 13 insertions(+), 11 deletions(-)
diff --git a/bitbake/lib/toaster/bldcontrol/bbcontroller.py b/bitbake/lib/toaster/bldcontrol/bbcontroller.py
index 1387bda..f228d37 100644
--- a/bitbake/lib/toaster/bldcontrol/bbcontroller.py
+++ b/bitbake/lib/toaster/bldcontrol/bbcontroller.py
@@ -138,7 +138,7 @@ class BuildEnvironmentController(object):
After this method executes, self.be bbaddress/bbport MUST point to a running and free server,
and the bbstate MUST be updated to "started".
"""
- raise Exception("FIXME: Must override in order to actually start the BB server")
+ raise NotImplementedError("FIXME: Must override in order to actually start the BB server")
def setLayers(self, bitbake, ls):
@@ -149,7 +149,7 @@ class BuildEnvironmentController(object):
a word of attention: by convention, the first layer for any build will be poky!
"""
- raise Exception("FIXME: Must override setLayers")
+ raise NotImplementedError("FIXME: Must override setLayers")
def getBBController(self):
@@ -176,16 +176,16 @@ class BuildEnvironmentController(object):
up to the implementing BEC. The return MUST be a REST URL where a GET will actually return
the content of the artifact, e.g. for use as a "download link" in a web UI.
"""
- raise Exception("Must return the REST URL of the artifact")
+ raise NotImplementedError("Must return the REST URL of the artifact")
def release(self):
""" This stops the server and releases any resources. After this point, all resources
are un-available for further reference
"""
- raise Exception("Must override BE release")
+ raise NotImplementedError("Must override BE release")
def triggerBuild(self, bitbake, layers, variables, targets):
- raise Exception("Must override BE release")
+ raise NotImplementedError("Must override BE release")
class ShellCmdException(Exception):
pass
diff --git a/bitbake/lib/toaster/bldcontrol/models.py b/bitbake/lib/toaster/bldcontrol/models.py
index a3a49ce..bb613c6 100644
--- a/bitbake/lib/toaster/bldcontrol/models.py
+++ b/bitbake/lib/toaster/bldcontrol/models.py
@@ -42,13 +42,17 @@ class BuildEnvironment(models.Model):
def get_artifact(self, path):
if self.betype == BuildEnvironment.TYPE_LOCAL:
return open(path, "r")
- raise Exception("FIXME: artifact download not implemented for build environment type %s" % self.get_betype_display())
+ raise NotImplementedError("FIXME: artifact download not implemented "\
+ "for build environment type %s" % \
+ self.get_betype_display())
def has_artifact(self, path):
import os
if self.betype == BuildEnvironment.TYPE_LOCAL:
return os.path.exists(path)
- raise Exception("FIXME: has artifact not implemented for build environment type %s" % self.get_betype_display())
+ raise NotImplementedError("FIXME: has artifact not implemented for "\
+ "build environment type %s" % \
+ self.get_betype_display())
# a BuildRequest is a request that the scheduler will build using a BuildEnvironment
# the build request queue is the table itself, ordered by state
diff --git a/bitbake/lib/toaster/bldcontrol/tests.py b/bitbake/lib/toaster/bldcontrol/tests.py
index 141b42a..e808991 100644
--- a/bitbake/lib/toaster/bldcontrol/tests.py
+++ b/bitbake/lib/toaster/bldcontrol/tests.py
@@ -48,13 +48,12 @@ class BEControllerTests(object):
self.assertTrue(err == '', "bitbake server pid %s not stopped" % err)
def test_serverStartAndStop(self):
- from bldcontrol.sshbecontroller import NotImplementedException
obe = self._getBuildEnvironment()
bc = self._getBEController(obe)
try:
# setting layers, skip any layer info
bc.setLayers(BITBAKE_LAYER, POKY_LAYERS)
- except NotImplementedException, e:
+ except NotImplementedError:
print "Test skipped due to command not implemented yet"
return True
# We are ok with the exception as we're handling the git already exists
@@ -74,14 +73,13 @@ class BEControllerTests(object):
self._serverForceStop(bc)
def test_getBBController(self):
- from bldcontrol.sshbecontroller import NotImplementedException
obe = self._getBuildEnvironment()
bc = self._getBEController(obe)
layerSet = False
try:
# setting layers, skip any layer info
layerSet = bc.setLayers(BITBAKE_LAYER, POKY_LAYERS)
- except NotImplementedException:
+ except NotImplementedError:
print "Test skipped due to command not implemented yet"
return True
# We are ok with the exception as we're handling the git already exists
--
2.1.4
^ permalink raw reply related [flat|nested] 5+ messages in thread* [PATCH v2 2/3] toaster: don't use sshbecontroller
2016-02-26 8:06 [PATCH v2 0/3] remove ssh build controller Ed Bartosh
2016-02-26 8:06 ` [PATCH v2 1/3] toaster: raise NotImplementedError Ed Bartosh
@ 2016-02-26 8:06 ` Ed Bartosh
2016-02-26 8:06 ` [PATCH v2 3/3] toaster: remove sshbecontroller module Ed Bartosh
2016-03-03 6:42 ` [PATCH v2 0/3] remove ssh build controller Brian Avery
3 siblings, 0 replies; 5+ messages in thread
From: Ed Bartosh @ 2016-02-26 8:06 UTC (permalink / raw)
To: toaster
Removed usage of sshbecontroller from bbcontroller, models, tests
and database schema.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
bitbake/lib/toaster/bldcontrol/bbcontroller.py | 6 ------
.../migrations/0002_auto_20160120_1250.py | 19 +++++++++++++++++
bitbake/lib/toaster/bldcontrol/models.py | 2 --
bitbake/lib/toaster/bldcontrol/tests.py | 24 ----------------------
4 files changed, 19 insertions(+), 32 deletions(-)
create mode 100644 bitbake/lib/toaster/bldcontrol/migrations/0002_auto_20160120_1250.py
diff --git a/bitbake/lib/toaster/bldcontrol/bbcontroller.py b/bitbake/lib/toaster/bldcontrol/bbcontroller.py
index f228d37..f40103c 100644
--- a/bitbake/lib/toaster/bldcontrol/bbcontroller.py
+++ b/bitbake/lib/toaster/bldcontrol/bbcontroller.py
@@ -76,13 +76,10 @@ def getBuildEnvironmentController(**kwargs):
"""
from localhostbecontroller import LocalhostBEController
- from sshbecontroller import SSHBEController
be = BuildEnvironment.objects.filter(Q(**kwargs))[0]
if be.betype == BuildEnvironment.TYPE_LOCAL:
return LocalhostBEController(be)
- elif be.betype == BuildEnvironment.TYPE_SSH:
- return SSHBEController(be)
else:
raise Exception("FIXME: Implement BEC for type %s" % str(be.betype))
@@ -105,9 +102,6 @@ class BuildEnvironmentController(object):
on the local machine, with the "build/" directory under the "poky/" source checkout directory.
Bash is expected to be available.
- * SSH controller will run the Toaster BE on a remote machine, where the current user
- can connect without raise Exception("FIXME: implement")word (set up with either ssh-agent or raise Exception("FIXME: implement")phrase-less key authentication)
-
"""
def __init__(self, be):
""" Takes a BuildEnvironment object as parameter that points to the settings of the BE.
diff --git a/bitbake/lib/toaster/bldcontrol/migrations/0002_auto_20160120_1250.py b/bitbake/lib/toaster/bldcontrol/migrations/0002_auto_20160120_1250.py
new file mode 100644
index 0000000..0c2475a
--- /dev/null
+++ b/bitbake/lib/toaster/bldcontrol/migrations/0002_auto_20160120_1250.py
@@ -0,0 +1,19 @@
+# -*- coding: utf-8 -*-
+from __future__ import unicode_literals
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+ dependencies = [
+ ('bldcontrol', '0001_initial'),
+ ]
+
+ operations = [
+ migrations.AlterField(
+ model_name='buildenvironment',
+ name='betype',
+ field=models.IntegerField(choices=[(0, b'local')]),
+ ),
+ ]
diff --git a/bitbake/lib/toaster/bldcontrol/models.py b/bitbake/lib/toaster/bldcontrol/models.py
index bb613c6..9244ed1 100644
--- a/bitbake/lib/toaster/bldcontrol/models.py
+++ b/bitbake/lib/toaster/bldcontrol/models.py
@@ -12,10 +12,8 @@ class BuildEnvironment(models.Model):
)
TYPE_LOCAL = 0
- TYPE_SSH = 1
TYPE = (
(TYPE_LOCAL, "local"),
- (TYPE_SSH, "ssh"),
)
LOCK_FREE = 0
diff --git a/bitbake/lib/toaster/bldcontrol/tests.py b/bitbake/lib/toaster/bldcontrol/tests.py
index e808991..f20cc7d 100644
--- a/bitbake/lib/toaster/bldcontrol/tests.py
+++ b/bitbake/lib/toaster/bldcontrol/tests.py
@@ -9,7 +9,6 @@ from django.test import TestCase
from bldcontrol.bbcontroller import BitbakeController, BuildSetupException
from bldcontrol.localhostbecontroller import LocalhostBEController
-from bldcontrol.sshbecontroller import SSHBEController
from bldcontrol.models import BuildEnvironment, BuildRequest
from bldcontrol.management.commands.runbuilds import Command
@@ -110,29 +109,6 @@ class LocalhostBEControllerTests(TestCase, BEControllerTests):
def _getBEController(self, obe):
return LocalhostBEController(obe)
-class SSHBEControllerTests(TestCase, BEControllerTests):
- def __init__(self, *args):
- super(SSHBEControllerTests, self).__init__(*args)
-
- def _getBuildEnvironment(self):
- return BuildEnvironment.objects.create(
- lock = BuildEnvironment.LOCK_FREE,
- betype = BuildEnvironment.TYPE_SSH,
- address = test_address,
- sourcedir = test_sourcedir,
- builddir = test_builddir )
-
- def _getBEController(self, obe):
- return SSHBEController(obe)
-
- def test_pathExists(self):
- obe = BuildEnvironment.objects.create(betype = BuildEnvironment.TYPE_SSH, address= test_address)
- sbc = SSHBEController(obe)
- self.assertTrue(sbc._pathexists("/"))
- self.assertFalse(sbc._pathexists("/.deadbeef"))
- self.assertTrue(sbc._pathexists(sbc._shellcmd("pwd")))
-
-
class RunBuildsCommandTests(TestCase):
def test_bec_select(self):
"""
--
2.1.4
^ permalink raw reply related [flat|nested] 5+ messages in thread* [PATCH v2 3/3] toaster: remove sshbecontroller module
2016-02-26 8:06 [PATCH v2 0/3] remove ssh build controller Ed Bartosh
2016-02-26 8:06 ` [PATCH v2 1/3] toaster: raise NotImplementedError Ed Bartosh
2016-02-26 8:06 ` [PATCH v2 2/3] toaster: don't use sshbecontroller Ed Bartosh
@ 2016-02-26 8:06 ` Ed Bartosh
2016-03-03 6:42 ` [PATCH v2 0/3] remove ssh build controller Brian Avery
3 siblings, 0 replies; 5+ messages in thread
From: Ed Bartosh @ 2016-02-26 8:06 UTC (permalink / raw)
To: toaster
The code of this module is broken for a long time.
The functionality of it can be easily achieved by running
'manage.py runbuilds.py' on remote machine.
[YOCTO #8806]
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
bitbake/lib/toaster/bldcontrol/sshbecontroller.py | 169 ----------------------
1 file changed, 169 deletions(-)
delete mode 100644 bitbake/lib/toaster/bldcontrol/sshbecontroller.py
diff --git a/bitbake/lib/toaster/bldcontrol/sshbecontroller.py b/bitbake/lib/toaster/bldcontrol/sshbecontroller.py
deleted file mode 100644
index 17dd66c..0000000
--- a/bitbake/lib/toaster/bldcontrol/sshbecontroller.py
+++ /dev/null
@@ -1,169 +0,0 @@
-#
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
-#
-# BitBake Toaster Implementation
-#
-# Copyright (C) 2014 Intel Corporation
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-
-import sys
-import re
-from django.db import transaction
-from django.db.models import Q
-from bldcontrol.models import BuildEnvironment, BRLayer, BRVariable, BRTarget, BRBitbake
-import subprocess
-
-from toastermain import settings
-
-from bbcontroller import BuildEnvironmentController, ShellCmdException, BuildSetupException
-
-class NotImplementedException(Exception):
- pass
-
-def DN(path):
- return "/".join(path.split("/")[0:-1])
-
-class SSHBEController(BuildEnvironmentController):
- """ Implementation of the BuildEnvironmentController for the localhost;
- this controller manages the default build directory,
- the server setup and system start and stop for the localhost-type build environment
-
- """
-
- def __init__(self, be):
- super(SSHBEController, self).__init__(be)
- self.dburl = settings.getDATABASE_URL()
- self.pokydirname = None
- self.islayerset = False
-
- def _shellcmd(self, command, cwd = None):
- if cwd is None:
- cwd = self.be.sourcedir
-
- p = subprocess.Popen("ssh %s 'cd %s && %s'" % (self.be.address, cwd, command), stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
- (out,err) = p.communicate()
- if p.returncode:
- if len(err) == 0:
- err = "command: %s \n%s" % (command, out)
- else:
- err = "command: %s \n%s" % (command, err)
- raise ShellCmdException(err)
- else:
- return out.strip()
-
- def _pathexists(self, path):
- try:
- self._shellcmd("test -e \"%s\"" % path)
- return True
- except ShellCmdException as e:
- return False
-
- def _pathcreate(self, path):
- self._shellcmd("mkdir -p \"%s\"" % path)
-
- def _setupBE(self):
- assert self.pokydirname and self._pathexists(self.pokydirname)
- self._pathcreate(self.be.builddir)
- self._shellcmd("bash -c \"source %s/oe-init-build-env %s\"" % (self.pokydirname, self.be.builddir))
-
- def startBBServer(self, brbe):
- assert self.pokydirname and self._pathexists(self.pokydirname)
- assert self.islayerset
- cmd = self._shellcmd("bash -c \"source %s/oe-init-build-env %s && DATABASE_URL=%s source toaster start noweb brbe=%s\"" % (self.pokydirname, self.be.builddir, self.dburl, brbe))
-
- port = "-1"
- for i in cmd.split("\n"):
- if i.startswith("Bitbake server address"):
- port = i.split(" ")[-1]
- print "Found bitbake server port ", port
-
-
- assert self.be.sourcedir and self._pathexists(self.be.builddir)
- self.be.bbaddress = self.be.address.split("@")[-1]
- self.be.bbport = port
- self.be.bbstate = BuildEnvironment.SERVER_STARTED
- self.be.save()
-
- def _copyFile(self, filepath1, filepath2):
- p = subprocess.Popen("scp '%s' '%s'" % (filepath1, filepath2), stdout=subprocess.PIPE, stderr = subprocess.PIPE, shell=True)
- (out, err) = p.communicate()
- if p.returncode:
- raise ShellCmdException(err)
-
- def pullFile(self, local_filename, remote_filename):
- _copyFile(local_filename, "%s:%s" % (self.be.address, remote_filename))
-
- def pushFile(self, local_filename, remote_filename):
- _copyFile("%s:%s" % (self.be.address, remote_filename), local_filename)
-
- def setLayers(self, bitbakes, layers):
- """ a word of attention: by convention, the first layer for any build will be poky! """
-
- assert self.be.sourcedir is not None
- assert len(bitbakes) == 1
- # set layers in the layersource
-
-
- raise NotImplementedException("Not implemented: SSH setLayers")
- # 3. configure the build environment, so we have a conf/bblayers.conf
- assert self.pokydirname is not None
- self._setupBE()
-
- # 4. update the bblayers.conf
- bblayerconf = os.path.join(self.be.builddir, "conf/bblayers.conf")
- if not self._pathexists(bblayerconf):
- raise BuildSetupException("BE is not consistent: bblayers.conf file missing at %s" % bblayerconf)
-
- import uuid
- local_bblayerconf = "/tmp/" + uuid.uuid4() + "-bblayer.conf"
-
- self.pullFile(bblayerconf, local_bblayerconf)
-
- BuildEnvironmentController._updateBBLayers(local_bblayerconf, layerlist)
- self.pushFile(local_bblayerconf, bblayerconf)
-
- os.unlink(local_bblayerconf)
-
- self.islayerset = True
- return True
-
- def release(self):
- assert self.be.sourcedir and self._pathexists(self.be.builddir)
- import shutil
- shutil.rmtree(os.path.join(self.be.sourcedir, "build"))
- assert not self._pathexists(self.be.builddir)
-
- def triggerBuild(self, bitbake, layers, variables, targets):
- # set up the buid environment with the needed layers
- self.setLayers(bitbake, layers)
- self.writeConfFile("conf/toaster-pre.conf", )
- self.writeConfFile("conf/toaster.conf", raw = "INHERIT+=\"toaster buildhistory\"")
-
- # get the bb server running with the build req id and build env id
- bbctrl = self.getBBController()
-
- # trigger the build command
- task = reduce(lambda x, y: x if len(y)== 0 else y, map(lambda y: y.task, targets))
- if len(task) == 0:
- task = None
-
- bbctrl.build(list(map(lambda x:x.target, targets)), task)
-
- logger.debug("localhostbecontroller: Build launched, exiting. Follow build logs at %s/toaster_ui.log" % self.be.builddir)
-
- # disconnect from the server
- bbctrl.disconnect()
--
2.1.4
^ permalink raw reply related [flat|nested] 5+ messages in thread* Re: [PATCH v2 0/3] remove ssh build controller
2016-02-26 8:06 [PATCH v2 0/3] remove ssh build controller Ed Bartosh
` (2 preceding siblings ...)
2016-02-26 8:06 ` [PATCH v2 3/3] toaster: remove sshbecontroller module Ed Bartosh
@ 2016-03-03 6:42 ` Brian Avery
3 siblings, 0 replies; 5+ messages in thread
From: Brian Avery @ 2016-03-03 6:42 UTC (permalink / raw)
To: Ed Bartosh; +Cc: toaster
upstreamed.
ty,
b
On Fri, Feb 26, 2016 at 12:06 AM, Ed Bartosh <ed.bartosh@linux.intel.com> wrote:
> Hi,
>
> This is a second attempt to get this through review. As ic branch has
> been successfuly landed I hope this one can be upstreamed too.
>
> sshbecontroller code and tests are broken for months. Its functionality(running
> builds on remote machine) can be achieved by simply running manage.py runbuilds
> on remote machine. There is not need to keep it in the codebase and confuse
> users from my point of view.
>
> This change would also make refactoring of buildcontroller code
> [YOCTO #8806] easier.
>
> Changes in v2: rebased on top of toaster-next.
>
> The following changes since commit 31e7bb2c5e900c44aded447dcaf9b090dd2cb11d:
>
> toaster: change 'delete layer' to 'remove layer' (2016-02-25 14:26:41 +0000)
>
> are available in the git repository at:
>
> git://git.yoctoproject.org/poky-contrib ed/toaster/remove-ssh-controller
> http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/toaster/remove-ssh-controller
>
> Ed Bartosh (3):
> toaster: raise NotImplementedError
> toaster: don't use sshbecontroller
> toaster: remove sshbecontroller module
>
> bitbake/lib/toaster/bldcontrol/bbcontroller.py | 16 +-
> .../migrations/0002_auto_20160120_1250.py | 19 +++
> bitbake/lib/toaster/bldcontrol/models.py | 10 +-
> bitbake/lib/toaster/bldcontrol/sshbecontroller.py | 169 ---------------------
> bitbake/lib/toaster/bldcontrol/tests.py | 30 +---
> 5 files changed, 32 insertions(+), 212 deletions(-)
> create mode 100644 bitbake/lib/toaster/bldcontrol/migrations/0002_auto_20160120_1250.py
> delete mode 100644 bitbake/lib/toaster/bldcontrol/sshbecontroller.py
>
> --
> Regards,
> Ed
> --
> _______________________________________________
> toaster mailing list
> toaster@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/toaster
^ permalink raw reply [flat|nested] 5+ messages in thread