[PATCH v2 00/15] per project build directory

[PATCH v2 00/15] per project build directory

[Ed Bartosh]

Hi,

This is a first draft of per project build directory implementation.
Default build directory is not going to be used for project builds anymore.
It can be used only for command line builds. Toaster will use build-toaster-<project>
directories for project builds.

This is how to test this:
 - run toaster
 - run command line build: build zlib
 - check if it's shown in Toaster UI
 - create project for Yocto master
 - trigger project build from UI
 - wait until it succeeds
 - create project for Jethro
 - trigger project build from UI
 - wait until it succeds
 - check that 2 build directories build-toaster-<project> exist and contain
   correct toaster.conf and bblayers.conf

Changes in v2: stop bitbake server after the build

The following changes since commit 3cee13dc65a1db5643126859ae9e5ce6bd282c81:

  toaster: rework task buildstats storage and display (2016-03-08 11:09:44 +0000)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib ed/toaster/project-build-dir-7880
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/toaster/project-build-dir-7880

Ed Bartosh (15):
  xmlrpc: fix bug in setting XMLRPCServer.single_use
  toaster: don't start bitbake server
  toaster: get rid of noui option
  toaster: set BITBAKE_UI environment variable
  toasterui: add brbe parameter to buildinfohelper
  uievent: improve BBUIEventQueue code
  buildinfohelper: improve handling of providermap
  toasterui: fix brbe reporting
  toaster: remove startBBServer API
  toaster: remove release API
  toaster: add brbe parameter to triggerBuild
  toaster: modified setLayers API
  toaster: reimplement triggerBuild
  toaster: add new parameter to _shellcmd
  toaster: stop bitbake server after the build

 bitbake/bin/toaster                                |  73 ++---------
 bitbake/lib/bb/server/xmlrpc.py                    |   3 +-
 bitbake/lib/bb/ui/buildinfohelper.py               |   7 +-
 bitbake/lib/bb/ui/toasterui.py                     |   9 +-
 bitbake/lib/bb/ui/uievent.py                       |  10 +-
 bitbake/lib/toaster/bldcontrol/bbcontroller.py     |  33 -----
 .../toaster/bldcontrol/localhostbecontroller.py    | 133 ++++++++-------------
 .../bldcontrol/management/commands/runbuilds.py    |   6 +-
 8 files changed, 80 insertions(+), 194 deletions(-)

--
Regards,
Ed

[PATCH v2 01/15] xmlrpc: fix bug in setting XMLRPCServer.single_use

[Ed Bartosh]

XMLRPCServer.single_use attribute was always set to False.
This caused xmlrpc server to keep running after build is done as
BitBakeServerCommands.removeClient only shuts down server if its
single_use attribute is set to True.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/bb/server/xmlrpc.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/bitbake/lib/bb/server/xmlrpc.py b/bitbake/lib/bb/server/xmlrpc.py
index 1ceca51..d627550 100644
--- a/bitbake/lib/bb/server/xmlrpc.py
+++ b/bitbake/lib/bb/server/xmlrpc.py
@@ -193,6 +193,8 @@ class XMLRPCServer(SimpleXMLRPCServer, BaseImplServer):
         BaseImplServer.__init__(self)
         if (interface[1] == 0):     # anonymous port, not getting reused
             self.single_use = True
+        else:
+            self.singe_use = False
         # Use auto port configuration
         if (interface[1] == -1):
             interface = (interface[0], 0)
@@ -205,7 +207,6 @@ class XMLRPCServer(SimpleXMLRPCServer, BaseImplServer):
         self.commands = BitBakeServerCommands(self)
         self.autoregister_all_functions(self.commands, "")
         self.interface = interface
-        self.single_use = False
 
     def addcooker(self, cooker):
         BaseImplServer.addcooker(self, cooker)
-- 
2.1.4

[PATCH v2 02/15] toaster: don't start bitbake server

[Ed Bartosh]

From now on toaster script will not run bitbake server.
It will be started by runbuilds and stopped after the build.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/bin/toaster | 67 +++++------------------------------------------------
 1 file changed, 6 insertions(+), 61 deletions(-)

diff --git a/bitbake/bin/toaster b/bitbake/bin/toaster
index d409554..6a85ae2 100755
--- a/bitbake/bin/toaster
+++ b/bitbake/bin/toaster
@@ -17,7 +17,7 @@
 # You should have received a copy of the GNU General Public License
 # along with this program. If not, see http://www.gnu.org/licenses/.
 
-# Usage: source toaster [start|stop|restart-bitbake]
+# Usage: source toaster [start|stop]
 #                       [webport=<port>] [noui] [noweb]
 
 # Helper function to kill a background toaster development server
@@ -95,7 +95,6 @@ stop_system()
         kill `cat ${BUILDDIR}/.toasterui.pid` 2>/dev/null
         rm ${BUILDDIR}/.toasterui.pid
     fi
-    stop_bitbake
     webserverKillAll
     # unset exported variables
     unset DATABASE_URL
@@ -106,31 +105,6 @@ stop_system()
     INSTOPSYSTEM=0
 }
 
-start_bitbake() {
-    unset BBSERVER
-    bitbake --read conf/toaster.conf --server-only -t xmlrpc -B 0.0.0.0:0
-    if [ $? -ne 0 ]; then
-        echo "Bitbake server start failed"
-        return 1
-    fi
-    export BBSERVER=0.0.0.0:-1
-    export DATABASE_URL=`$MANAGE get-dburl`
-    # we start the TOASTERUI only if not inhibited
-    if [ $NOTOASTERUI -eq 0 ]; then
-        bitbake --observe-only -u toasterui --remote-server=$BBSERVER -t xmlrpc \
-            >>${BUILDDIR}/toaster_ui.log 2>&1 \
-            & echo $! >${BUILDDIR}/.toasterui.pid
-    fi
-    return 0
-}
-
-stop_bitbake() {
-    BBSERVER=0.0.0.0:-1 bitbake -m
-    unset BBSERVER
-    # force stop any misbehaving bitbake server
-    lsof -t bitbake.lock | awk '{print $2}' | grep "[0-9]\+" | xargs -n1 -r kill
-}
-
 verify_prereq() {
     # Verify Django version
     reqfile=$(python -c "import os; print os.path.realpath('$BBBASEDIR/toaster-requirements.txt')")
@@ -229,15 +203,6 @@ if [ `basename \"$0\"` = `basename \"${TOASTER}\"` ]; then
     return 1
 fi
 
-if [ "$1" = 'restart-bitbake' ] ; then
-    stop_bitbake
-    sleep 1
-    start_bitbake
-    rc=$?
-    sleep 3
-    return $rc
-fi
-
 verify_prereq || return 1
 
 # We make sure we're running in the current shell and in a good environment
@@ -281,16 +246,6 @@ case $CMD in
              $MANAGE checksocket "0.0.0.0:$WEB_PORT" || return 1
         fi
 
-        # Make sure it's safe to start by checking bitbake lock
-        if [ -e $BUILDDIR/bitbake.lock ]; then
-            python -c "import fcntl; fcntl.flock(open(\"$BUILDDIR/bitbake.lock\"), fcntl.LOCK_EX|fcntl.LOCK_NB)" 2>/dev/null
-            if [ $? -ne 0 ] ; then
-                echo "Error: bitbake lock state error. File locks show that the system is on." 1>&2
-                echo "Please wait for the current build to finish, stop and then start the system again." 1>&2
-                return 3
-            fi
-        fi
-
         # kill Toaster web server if it's alive
         if [ -e $BUILDDIR/.toastermain.pid ] && kill -0 `cat $BUILDDIR/.toastermain.pid`; then
             echo "Warning: bitbake appears to be dead, but the Toaster web server is running." 1>&2
@@ -308,22 +263,12 @@ case $CMD in
             echo "Failed ${CMD}."
             return 4
         fi
-        start_bitbake
-        if [ $? -eq 0 ]; then
-            $MANAGE runbuilds & echo $! >${BUILDDIR}/.runbuilds.pid
-            # set fail safe stop system on terminal exit
-            trap stop_system SIGHUP
-            echo "Successful ${CMD}."
-            return 0
-        else
-            # failed start, do stop
-            stop_system
-            echo "Failed ${CMD}."
-            return 1
-        fi
-        # stop system on terminal exit
-        set -o monitor
+        export DATABASE_URL=`$MANAGE get-dburl`
+        $MANAGE runbuilds & echo $! >${BUILDDIR}/.runbuilds.pid
+        # set fail safe stop system on terminal exit
         trap stop_system SIGHUP
+        echo "Successful ${CMD}."
+        return 0
     ;;
     stop )
         stop_system
-- 
2.1.4

[PATCH v2 03/15] toaster: get rid of noui option

[Ed Bartosh]

noui command line option doesn't makes sense anymore as toaster doesn't
run bitbake. It should be safe to to remove it.

The purpose of this option was to skip running bitbake observer process.
This was never used before as it's not possible to run toaster build
without running observer.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/bin/toaster | 4 ----
 1 file changed, 4 deletions(-)

diff --git a/bitbake/bin/toaster b/bitbake/bin/toaster
index 6a85ae2..1e6e690 100755
--- a/bitbake/bin/toaster
+++ b/bitbake/bin/toaster
@@ -175,15 +175,11 @@ fi
 # make sure that the toaster.sqlite file doesn't default to `pwd` like it currently does.
 export TOASTER_DIR=`pwd`
 
-NOTOASTERUI=0
 WEBSERVER=1
 WEB_PORT="8000"
 unset CMD
 for param in $*; do
     case $param in
-    noui )
-            NOTOASTERUI=1
-    ;;
     noweb )
             WEBSERVER=0
     ;;
-- 
2.1.4

[PATCH v2 04/15] toaster: set BITBAKE_UI environment variable

[Ed Bartosh]

Set BITBAKE_UI variable to 'toastergui' for command line builds
to use toasterui as a default ui module for bitbake.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/bin/toaster | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/bitbake/bin/toaster b/bitbake/bin/toaster
index 1e6e690..5cc77ed 100755
--- a/bitbake/bin/toaster
+++ b/bitbake/bin/toaster
@@ -100,6 +100,7 @@ stop_system()
     unset DATABASE_URL
     unset TOASTER_CONF
     unset TOASTER_DIR
+    unset BITBAKE_UI
     trap - SIGHUP
     #trap - SIGCHLD
     INSTOPSYSTEM=0
@@ -259,6 +260,7 @@ case $CMD in
             echo "Failed ${CMD}."
             return 4
         fi
+        export BITBAKE_UI='toasterui'
         export DATABASE_URL=`$MANAGE get-dburl`
         $MANAGE runbuilds & echo $! >${BUILDDIR}/.runbuilds.pid
         # set fail safe stop system on terminal exit
-- 
2.1.4

[PATCH v2 05/15] toasterui: add brbe parameter to buildinfohelper

[Ed Bartosh]

In current toaster code BRBE(build request:build environment) value
is passed from toaster to buildinfohelper through the 'SetBRBE' event.

Passing it through environment variable is easier as it doesn't
involve rpc communication between toaster and bitbake server.
It also eliminates the need in running bitbake observer process.

Added parameter 'brbe' to BuildInfoHelper.__init__
Used environment variable TOASTER_BRBE to set brbe for
buildinfohelper object.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/bb/ui/buildinfohelper.py | 4 ++--
 bitbake/lib/bb/ui/toasterui.py       | 3 ++-
 2 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/bitbake/lib/bb/ui/buildinfohelper.py b/bitbake/lib/bb/ui/buildinfohelper.py
index 7fedb76..6f73c7c 100644
--- a/bitbake/lib/bb/ui/buildinfohelper.py
+++ b/bitbake/lib/bb/ui/buildinfohelper.py
@@ -840,7 +840,7 @@ class BuildInfoHelper(object):
     # pylint: disable=bad-continuation
     # we do not follow the python conventions for continuation indentation due to long lines here
 
-    def __init__(self, server, has_build_history = False):
+    def __init__(self, server, has_build_history = False, brbe = None):
         self.internal_state = {}
         self.internal_state['taskdata'] = {}
         self.internal_state['targets'] = []
@@ -856,7 +856,7 @@ class BuildInfoHelper(object):
 
         # this is set for Toaster-triggered builds by localhostbecontroller
         # via toasterui
-        self.brbe = None
+        self.brbe = brbe
 
         self.project = None
 
diff --git a/bitbake/lib/bb/ui/toasterui.py b/bitbake/lib/bb/ui/toasterui.py
index eee8d14..6ad27db 100644
--- a/bitbake/lib/bb/ui/toasterui.py
+++ b/bitbake/lib/bb/ui/toasterui.py
@@ -196,7 +196,8 @@ def main(server, eventHandler, params):
     taskfailures = []
     first = True
 
-    buildinfohelper = BuildInfoHelper(server, build_history_enabled)
+    buildinfohelper = BuildInfoHelper(server, build_history_enabled,
+                                      os.getenv('TOASTER_BRBE'))
 
     # write our own log files into bitbake's log directory;
     # we're only interested in the path to the parent directory of
-- 
2.1.4

[PATCH v2 06/15] uievent: improve BBUIEventQueue code

[Ed Bartosh]

Return value of self.BBServer.registerEventHandler differs between
jethro and master. To be able to build jethro toaster should be
able to communicate with jethro bitbake server i.e. it must work
with both old and new registerEventHandler call.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/bb/ui/uievent.py | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/bitbake/lib/bb/ui/uievent.py b/bitbake/lib/bb/ui/uievent.py
index 6b479bf..df093c5 100644
--- a/bitbake/lib/bb/ui/uievent.py
+++ b/bitbake/lib/bb/ui/uievent.py
@@ -24,7 +24,7 @@ server and queue them for the UI to process. This process must be used to avoid
 client/server deadlocks.
 """
 
-import socket, threading, pickle
+import socket, threading, pickle, collections
 from SimpleXMLRPCServer import SimpleXMLRPCServer, SimpleXMLRPCRequestHandler
 
 class BBUIEventQueue:
@@ -51,7 +51,13 @@ class BBUIEventQueue:
         # giving up
 
         for count_tries in range(5):
-            self.EventHandle, error = self.BBServer.registerEventHandler(self.host, self.port)
+            ret = self.BBServer.registerEventHandler(self.host, self.port)
+
+            if isinstance(ret, collections.Iterable):
+                self.EventHandle, error = ret
+            else:
+                self.EventHandle = ret
+                error = ""
 
             if self.EventHandle != None:
                 break
-- 
2.1.4

[PATCH v2 07/15] buildinfohelper: improve handling of providermap

[Ed Bartosh]

DepTreeGenerated event doesn't contain 'providermap' data in jethro.
Modified buildinfohelper to handle events without this data. This
should make it possible to handle jethro events coming from jethro
bitbake server by the latest buildinfohelper.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/bb/ui/buildinfohelper.py | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/bitbake/lib/bb/ui/buildinfohelper.py b/bitbake/lib/bb/ui/buildinfohelper.py
index 6f73c7c..cad2a21 100644
--- a/bitbake/lib/bb/ui/buildinfohelper.py
+++ b/bitbake/lib/bb/ui/buildinfohelper.py
@@ -1235,7 +1235,6 @@ class BuildInfoHelper(object):
         assert 'layer-priorities' in event._depgraph
         assert 'pn' in event._depgraph
         assert 'tdepends' in event._depgraph
-        assert 'providermap' in event._depgraph
 
         errormsg = ""
 
@@ -1318,7 +1317,7 @@ class BuildInfoHelper(object):
                 if dep in assume_provided:
                     continue
                 via = None
-                if dep in event._depgraph['providermap']:
+                if 'providermap' in event._depgraph and dep in event._depgraph['providermap']:
                     deprecipe = event._depgraph['providermap'][dep][0]
                     dependency = self.internal_state['recipes'][deprecipe]
                     via = Provides.objects.get_or_create(name=dep,
-- 
2.1.4

[PATCH v2 08/15] toasterui: fix brbe reporting

[Ed Bartosh]

buildinfohelper.brbe is lost when buildinfohelper is closed.
This causes incorrect report of brbe when build is done.

Saved brbe attribute before closing buildinfohelper and used
it to report correct brbe.

Got rid of useless and confusing 'ToasterUI build done 1'
log message.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/bb/ui/toasterui.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/bitbake/lib/bb/ui/toasterui.py b/bitbake/lib/bb/ui/toasterui.py
index 6ad27db..33607b5 100644
--- a/bitbake/lib/bb/ui/toasterui.py
+++ b/bitbake/lib/bb/ui/toasterui.py
@@ -373,9 +373,9 @@ def main(server, eventHandler, params):
 
                 # update the build info helper on BuildCompleted, not on CommandXXX
                 buildinfohelper.update_build_information(event, errors, warnings, taskfailures)
+
+                brbe = buildinfohelper.brbe
                 buildinfohelper.close(errorcode)
-                # mark the log output; controllers may kill the toasterUI after seeing this log
-                logger.info("ToasterUI build done 1, brbe: %s", buildinfohelper.brbe )
 
                 # we start a new build info
                 logger.debug("ToasterUI prepared for new build")
@@ -384,7 +384,7 @@ def main(server, eventHandler, params):
                 taskfailures = []
                 buildinfohelper = BuildInfoHelper(server, build_history_enabled)
 
-                logger.info("ToasterUI build done 2")
+                logger.info("ToasterUI build done, brbe: %s", brbe)
                 continue
 
             if isinstance(event, (bb.command.CommandCompleted,
-- 
2.1.4

[PATCH v2 09/15] toaster: remove startBBServer API

[Ed Bartosh]

We still will have to run bitbake server, but it will be done
different way and the code will be in triggerBuild function.

Removed startBBServer API from BuildEnvironmentController and
LocalhostBEController classes.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/toaster/bldcontrol/bbcontroller.py     |  9 ------
 .../toaster/bldcontrol/localhostbecontroller.py    | 32 ----------------------
 2 files changed, 41 deletions(-)

diff --git a/bitbake/lib/toaster/bldcontrol/bbcontroller.py b/bitbake/lib/toaster/bldcontrol/bbcontroller.py
index f40103c..535a398 100644
--- a/bitbake/lib/toaster/bldcontrol/bbcontroller.py
+++ b/bitbake/lib/toaster/bldcontrol/bbcontroller.py
@@ -127,14 +127,6 @@ class BuildEnvironmentController(object):
         bblayerconffile.write("# line added by toaster build control\nBBLAYERS = \"" + " ".join(layerlist) + "\"")
         bblayerconffile.close()
 
-    def startBBServer(self):
-        """ Starts a  BB server with Toaster toasterui set up to record the builds, an no controlling UI.
-            After this method executes, self.be bbaddress/bbport MUST point to a running and free server,
-            and the bbstate MUST be  updated to "started".
-        """
-        raise NotImplementedError("FIXME: Must override in order to actually start the BB server")
-
-
     def setLayers(self, bitbake, ls):
         """ Checks-out bitbake executor and layers from git repositories.
             Sets the layer variables in the config file, after validating local layer paths.
@@ -151,7 +143,6 @@ class BuildEnvironmentController(object):
             starts if needed; or reconnects to the server if we can
         """
         if not self.connection:
-            self.startBBServer()
             self.be.lock = BuildEnvironment.LOCK_RUNNING
             self.be.save()
 
diff --git a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
index efd82c3..9027291 100644
--- a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
+++ b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
@@ -70,38 +70,6 @@ class LocalhostBEController(BuildEnvironmentController):
             logger.debug("localhostbecontroller: shellcmd success")
             return out
 
-    def startBBServer(self):
-        assert self.pokydirname and os.path.exists(self.pokydirname)
-        assert self.islayerset
-
-        # find our own toasterui listener/bitbake
-        from toaster.bldcontrol.management.commands.loadconf import _reduce_canon_path
-
-        toaster = _reduce_canon_path(os.path.join(os.path.dirname(os.path.abspath(__file__)), "../../../bin/toaster"))
-        assert os.path.exists(toaster) and os.path.isfile(toaster)
-
-        # restart bitbake server and toastergui observer
-        self._shellcmd("bash -c 'source %s restart-bitbake'" % toaster, self.be.builddir)
-        logger.debug("localhostbecontroller: restarted bitbake server")
-
-        # read port number from bitbake.lock
-        self.be.bbport = ""
-        bblock = os.path.join(self.be.builddir, 'bitbake.lock')
-        if os.path.exists(bblock):
-            with open(bblock) as fplock:
-                for line in fplock:
-                    if ":" in line:
-                        self.be.bbport = line.split(":")[-1].strip()
-                        logger.debug("localhostbecontroller: bitbake port %s", self.be.bbport)
-                        break
-
-        if not self.be.bbport:
-            raise BuildSetupException("localhostbecontroller: can't read bitbake port from %s" % bblock)
-
-        self.be.bbaddress = "localhost"
-        self.be.bbstate = BuildEnvironment.SERVER_STARTED
-        self.be.save()
-
     def getGitCloneDirectory(self, url, branch):
         """Construct unique clone directory name out of url and branch."""
         if branch != "HEAD":
-- 
2.1.4

[PATCH v2 10/15] toaster: remove release API

[Ed Bartosh]

This API is quite dangerous as it removes build directory.
It's not used anywhere and most probably will not be used in future
as toaster is going to have one build directory per project.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/toaster/bldcontrol/bbcontroller.py          | 6 ------
 bitbake/lib/toaster/bldcontrol/localhostbecontroller.py | 6 ------
 2 files changed, 12 deletions(-)

diff --git a/bitbake/lib/toaster/bldcontrol/bbcontroller.py b/bitbake/lib/toaster/bldcontrol/bbcontroller.py
index 535a398..2ca2661 100644
--- a/bitbake/lib/toaster/bldcontrol/bbcontroller.py
+++ b/bitbake/lib/toaster/bldcontrol/bbcontroller.py
@@ -163,12 +163,6 @@ class BuildEnvironmentController(object):
         """
         raise NotImplementedError("Must return the REST URL of the artifact")
 
-    def release(self):
-        """ This stops the server and releases any resources. After this point, all resources
-            are un-available for further reference
-        """
-        raise NotImplementedError("Must override BE release")
-
     def triggerBuild(self, bitbake, layers, variables, targets):
         raise NotImplementedError("Must override BE release")
 
diff --git a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
index 9027291..007f3be 100644
--- a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
+++ b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
@@ -255,12 +255,6 @@ class LocalhostBEController(BuildEnvironmentController):
     def readServerLogFile(self):
         return open(os.path.join(self.be.builddir, "toaster_server.log"), "r").read()
 
-    def release(self):
-        assert self.be.sourcedir and os.path.exists(self.be.builddir)
-        import shutil
-        shutil.rmtree(os.path.join(self.be.sourcedir, "build"))
-        assert not os.path.exists(self.be.builddir)
-
 
     def triggerBuild(self, bitbake, layers, variables, targets):
         # set up the build environment with the needed layers
-- 
2.1.4

[PATCH v2 11/15] toaster: add brbe parameter to triggerBuild

[Ed Bartosh]

Called triggerBuild with brbe parameter instead of adding TOASTER_BRBE
variable to the database and fetching it in triggerBuild.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/toaster/bldcontrol/localhostbecontroller.py         | 2 +-
 bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py | 6 ++----
 2 files changed, 3 insertions(+), 5 deletions(-)

diff --git a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
index 007f3be..6f2ca33 100644
--- a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
+++ b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
@@ -256,7 +256,7 @@ class LocalhostBEController(BuildEnvironmentController):
         return open(os.path.join(self.be.builddir, "toaster_server.log"), "r").read()
 
 
-    def triggerBuild(self, bitbake, layers, variables, targets):
+    def triggerBuild(self, bitbake, layers, variables, targets, brbe):
         # set up the build environment with the needed layers
         self.setLayers(bitbake, layers, targets)
 
diff --git a/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py b/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py
index edf71a7..cbe0125 100644
--- a/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py
+++ b/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py
@@ -49,15 +49,13 @@ class Command(NoArgsCommand):
 
             logger.debug("runbuilds: starting build %s, environment %s" % (br, bec.be))
 
-            # write the build identification variable
-            BRVariable.objects.create(req = br, name="TOASTER_BRBE", value="%d:%d" % (br.pk, bec.be.pk))
-
             # let the build request know where it is being executed
             br.environment = bec.be
             br.save()
 
             # this triggers an async build
-            bec.triggerBuild(br.brbitbake, br.brlayer_set.all(), br.brvariable_set.all(), br.brtarget_set.all())
+            bec.triggerBuild(br.brbitbake, br.brlayer_set.all(), br.brvariable_set.all(),
+                             br.brtarget_set.all(), "%d:%d" % (br.pk, bec.be.pk))
 
         except Exception as e:
             logger.error("runbuilds: Error launching build %s" % e)
-- 
2.1.4

[PATCH v2 12/15] toaster: modified setLayers API

[Ed Bartosh]

Removed updating of bblayers.conf. It will be done in runBuild method.
Changed return value: return list of layers.
Removed _updateBBLayers method.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/toaster/bldcontrol/bbcontroller.py         | 18 ------------------
 .../lib/toaster/bldcontrol/localhostbecontroller.py    |  9 +--------
 2 files changed, 1 insertion(+), 26 deletions(-)

diff --git a/bitbake/lib/toaster/bldcontrol/bbcontroller.py b/bitbake/lib/toaster/bldcontrol/bbcontroller.py
index 2ca2661..058e490 100644
--- a/bitbake/lib/toaster/bldcontrol/bbcontroller.py
+++ b/bitbake/lib/toaster/bldcontrol/bbcontroller.py
@@ -109,24 +109,6 @@ class BuildEnvironmentController(object):
         self.be = be
         self.connection = None
 
-    @staticmethod
-    def _updateBBLayers(bblayerconf, layerlist):
-        conflines = open(bblayerconf, "r").readlines()
-
-        bblayerconffile = open(bblayerconf, "w")
-        skip = 0
-        for i in xrange(len(conflines)):
-            if skip > 0:
-                skip =- 1
-                continue
-            if conflines[i].startswith("# line added by toaster"):
-                skip = 1
-            else:
-                bblayerconffile.write(conflines[i])
-
-        bblayerconffile.write("# line added by toaster build control\nBBLAYERS = \"" + " ".join(layerlist) + "\"")
-        bblayerconffile.close()
-
     def setLayers(self, bitbake, ls):
         """ Checks-out bitbake executor and layers from git repositories.
             Sets the layer variables in the config file, after validating local layer paths.
diff --git a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
index 6f2ca33..833c685 100644
--- a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
+++ b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
@@ -179,11 +179,6 @@ class LocalhostBEController(BuildEnvironmentController):
 
         logger.debug("localhostbecontroller: current layer list %s " % pformat(layerlist))
 
-        # 4. update the bblayers.conf
-        bblayerconf = os.path.join(self.be.builddir, "conf/bblayers.conf")
-        if not os.path.exists(bblayerconf):
-            raise BuildSetupException("BE is not consistent: bblayers.conf file missing at %s" % bblayerconf)
-
         # 5. create custom layer and add custom recipes to it
         layerpath = os.path.join(self.be.sourcedir, "_meta-toaster-custom")
         if os.path.isdir(layerpath):
@@ -247,10 +242,8 @@ class LocalhostBEController(BuildEnvironmentController):
         if os.path.isdir(layerpath):
             layerlist.append(layerpath)
 
-        BuildEnvironmentController._updateBBLayers(bblayerconf, layerlist)
-
         self.islayerset = True
-        return True
+        return layerlist
 
     def readServerLogFile(self):
         return open(os.path.join(self.be.builddir, "toaster_server.log"), "r").read()
-- 
2.1.4

[PATCH v2 13/15] toaster: reimplement triggerBuild

[Ed Bartosh]

Reimplemented triggerBuild method to support one build directory
per project:
 - start bitbake server from the cloned repository
 - don't run observer
 - run bitbake build directly instead of triggering it
   through xmlrpc

[YOCTO #7880]
[YOCTO #9058]
[YOCTO #8958]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 .../toaster/bldcontrol/localhostbecontroller.py    | 80 +++++++++++++---------
 1 file changed, 46 insertions(+), 34 deletions(-)

diff --git a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
index 833c685..9923b13 100644
--- a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
+++ b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
@@ -250,44 +250,56 @@ class LocalhostBEController(BuildEnvironmentController):
 
 
     def triggerBuild(self, bitbake, layers, variables, targets, brbe):
-        # set up the build environment with the needed layers
-        self.setLayers(bitbake, layers, targets)
+        layers = self.setLayers(bitbake, layers, targets)
+
+        # init build environment from the clone
+        builddir = '%s-toaster-%s' % (self.be.builddir, bitbake.req.project.name)
+        oe_init = os.path.join(self.pokydirname, 'oe-init-build-env')
+        # init build environment
+        self._shellcmd('source %s %s' % (oe_init, builddir), self.be.sourcedir)
+
+        # update bblayers.conf
+        bblconfpath = os.path.join(builddir, "conf/bblayers.conf")
+        conflines = open(bblconfpath, "r").readlines()
+        skip = False
+        with open(bblconfpath, 'w') as bblayers:
+            for line in conflines:
+                if line.startswith("# line added by toaster"):
+                    skip = True
+                    continue
+                if skip:
+                    skip = False
+                else:
+                    bblayers.write(line)
+
+            bblayers.write('# line added by toaster build control\n'
+                           'BBLAYERS = "%s"' % ' '.join(layers))
 
         # write configuration file
-        filepath = os.path.join(self.be.builddir, "conf/toaster.conf")
-        with open(filepath, 'w') as conf:
+        confpath = os.path.join(builddir, 'conf/toaster.conf')
+        with open(confpath, 'w') as conf:
             for var in variables:
                 conf.write('%s="%s"\n' % (var.name, var.value))
             conf.write('INHERIT+="toaster buildhistory"')
 
-        # get the bb server running with the build req id and build env id
-        bbctrl = self.getBBController()
-
-        # set variables; TOASTER_BRBE is not set on the server, as this
-        # causes events from command-line builds to be attached to the last
-        # Toaster-triggered build; instead, TOASTER_BRBE is fired as an event so
-        # that toasterui can set it on the buildinfohelper;
-        # see https://bugzilla.yoctoproject.org/show_bug.cgi?id=9021
-        for var in variables:
-            if var.name == 'TOASTER_BRBE':
-                bbctrl.triggerEvent('bb.event.MetadataEvent("SetBRBE", "%s")' \
-                                     % var.value)
-            else:
-                bbctrl.setVariable(var.name, var.value)
-
-        # Add 'toaster' and 'buildhistory' to INHERIT variable
-        inherit = {item.strip() for item in bbctrl.getVariable('INHERIT').split()}
-        inherit = inherit.union(["toaster", "buildhistory"])
-        bbctrl.setVariable('INHERIT', ' '.join(inherit))
+        # run bitbake server from the clone
+        bitbake = os.path.join(self.pokydirname, 'bitbake', 'bin', 'bitbake')
+        self._shellcmd('source %s %s; BITBAKE_UI="" %s --read %s '
+                       '--server-only -t xmlrpc -B 0.0.0.0:0' % (oe_init, builddir,
+                       bitbake, confpath), self.be.sourcedir)
 
-        # trigger the build command
-        task = reduce(lambda x, y: x if len(y)== 0 else y, map(lambda y: y.task, targets))
-        if len(task) == 0:
-            task = None
-
-        bbctrl.build(list(map(lambda x:x.target, targets)), task)
-
-        logger.debug("localhostbecontroller: Build launched, exiting. Follow build logs at %s/toaster_ui.log" % self.be.builddir)
-
-        # disconnect from the server
-        bbctrl.disconnect()
+        bbtargets = ''
+        for target in targets:
+            task = target.task or 'do_build'
+            if not task.startswith('do_'):
+                task = 'do_' + task
+            bbtargets += '%s:%s ' % (target.target, task)
+
+        # run build with local bitbake
+        log = os.path.join(builddir, 'toaster_ui.log')
+        self._shellcmd('TOASTER_BRBE="%s" BBSERVER="0.0.0.0:-1" '
+                       '../bitbake/bin/bitbake %s -u toasterui '
+                       '>>%s 2>&1 &' % (brbe, bbtargets, log), builddir)
+
+        logger.debug('localhostbecontroller: Build launched, exiting. '
+                     'Follow build logs at %s' % log)
-- 
2.1.4

[PATCH v2 14/15] toaster: add new parameter to _shellcmd

[Ed Bartosh]

Added 'nowait' parameter to _shellcmd method to support
running chain of commands in a subshell.

This is going to be used to stop bitbake server after
the build.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/toaster/bldcontrol/localhostbecontroller.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
index 9923b13..bec9c8b 100644
--- a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
+++ b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
@@ -51,12 +51,14 @@ class LocalhostBEController(BuildEnvironmentController):
         self.pokydirname = None
         self.islayerset = False
 
-    def _shellcmd(self, command, cwd = None):
+    def _shellcmd(self, command, cwd=None, nowait=False):
         if cwd is None:
             cwd = self.be.sourcedir
 
         logger.debug("lbc_shellcmmd: (%s) %s" % (cwd, command))
         p = subprocess.Popen(command, cwd = cwd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+        if nowait:
+            return
         (out,err) = p.communicate()
         p.wait()
         if p.returncode:
-- 
2.1.4

[PATCH v2 15/15] toaster: stop bitbake server after the build

[Ed Bartosh]

Bitbake server is used only during the build. There is no need
in keeping server running after the build.

Running bitbake -m in the subshell after the build should stop
the server.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/toaster/bldcontrol/localhostbecontroller.py | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
index bec9c8b..b877ac0 100644
--- a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
+++ b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
@@ -297,11 +297,13 @@ class LocalhostBEController(BuildEnvironmentController):
                 task = 'do_' + task
             bbtargets += '%s:%s ' % (target.target, task)
 
-        # run build with local bitbake
+        # run build with local bitbake. stop the server after the build.
         log = os.path.join(builddir, 'toaster_ui.log')
-        self._shellcmd('TOASTER_BRBE="%s" BBSERVER="0.0.0.0:-1" '
-                       '../bitbake/bin/bitbake %s -u toasterui '
-                       '>>%s 2>&1 &' % (brbe, bbtargets, log), builddir)
+        self._shellcmd(['(TOASTER_BRBE="%s" BBSERVER="0.0.0.0:-1" '
+                        '../bitbake/bin/bitbake %s -u toasterui >>%s 2>&1;'
+                        'BITBAKE_UI="" BBSERVER=0.0.0.0:-1 %s -m)&' \
+                        % (brbe, bbtargets, log, bitbake)], builddir,
+                        nowait=True)
 
         logger.debug('localhostbecontroller: Build launched, exiting. '
                      'Follow build logs at %s' % log)
-- 
2.1.4

Re: [PATCH v2 00/15] per project build directory

[Barros Pena, Belen]

On 08/03/2016 13:53, "toaster-bounces@yoctoproject.org on behalf of Ed
Bartosh" <toaster-bounces@yoctoproject.org on behalf of
ed.bartosh@linux.intel.com> wrote:

>Hi,
>
>This is a first draft of per project build directory implementation.
>Default build directory is not going to be used for project builds
>anymore.
>It can be used only for command line builds. Toaster will use
>build-toaster-<project>
>directories for project builds.
>
>This is how to test this:
> - run toaster
> - run command line build: build zlib

This shows a warning about build history not being enabled. I know you
already know about this, but just for completeness ;)

> - check if it's shown in Toaster UI

Builds started from CLI do show in the web UI, but once they complete they
are missing layer and package information. I guess that's connected with
the build history issue above

> - create project for Yocto master
> - trigger project build from UI

The build I triggered from the Toaster web UI from a Yocto master project
(core-image-minimal) failed with this traceback:

/bin/sh: 1: source: not found

Traceback (most recent call last):
  File 
"/home/yocto/master/bitbake/lib/toaster/bldcontrol/management/commands/runb
uilds.py", line 58, in schedule
    br.brtarget_set.all(), "%d:%d" % (br.pk, bec.be.pk))
  File 
"/home/yocto/master/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
", line 261, in triggerBuild
    self._shellcmd('source %s %s' % (oe_init, builddir), self.be.sourcedir)
  File 
"/home/yocto/master/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
", line 70, in _shellcmd
    raise ShellCmdException(err)
ShellCmdException: command: source
/home/yocto/master/_toaster_clones/_git___git.yoctoproject.org_poky_master/
oe-init-build-env /home/yocto/master/build-toaster-master project
/bin/sh: 1: source: not found


> - wait until it succeeds
> - create project for Jethro
> - trigger project build from UI

Build fails with the same error as above

> - wait until it succeds
> - check that 2 build directories build-toaster-<project> exist and
>contain
>   correct toaster.conf and bblayers.conf

After all the above there is only one build directory created, the one
that was generated when running the build from CLI. Also:

* I can no longer find the toaster_ui.log file

* It looks like the builds are still using build/sstate-cache and
build/downloads as SSTATE_DIR and DL_DIR, but I think we agreed that
projects by default would share those 2 directories, which I guess means
they have to move outside the build directory, although I could be mistaken

* _toaster_clones is still being created in the root checkout directory,
which I guess makes sense

* A pending question is where the toaster-custom-images layer we generate
when building custom images should be. I had a brief chat with Michael
about this, and I think we concluded the layer should not be shared across
projects, which means we should create one per project and they should be
inside the project build dir. Right now the _meta-toaster-custom directory
is also in the root checkout directory

That's all, I think :)

Thanks!

Belén

>
>Changes in v2: stop bitbake server after the build
>
>The following changes since commit
>3cee13dc65a1db5643126859ae9e5ce6bd282c81:
>
>  toaster: rework task buildstats storage and display (2016-03-08
>11:09:44 +0000)
>
>are available in the git repository at:
>
>  git://git.yoctoproject.org/poky-contrib
>ed/toaster/project-build-dir-7880
>  
>http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/toaster/projec
>t-build-dir-7880
>
>Ed Bartosh (15):
>  xmlrpc: fix bug in setting XMLRPCServer.single_use
>  toaster: don't start bitbake server
>  toaster: get rid of noui option
>  toaster: set BITBAKE_UI environment variable
>  toasterui: add brbe parameter to buildinfohelper
>  uievent: improve BBUIEventQueue code
>  buildinfohelper: improve handling of providermap
>  toasterui: fix brbe reporting
>  toaster: remove startBBServer API
>  toaster: remove release API
>  toaster: add brbe parameter to triggerBuild
>  toaster: modified setLayers API
>  toaster: reimplement triggerBuild
>  toaster: add new parameter to _shellcmd
>  toaster: stop bitbake server after the build
>
> bitbake/bin/toaster                                |  73 ++---------
> bitbake/lib/bb/server/xmlrpc.py                    |   3 +-
> bitbake/lib/bb/ui/buildinfohelper.py               |   7 +-
> bitbake/lib/bb/ui/toasterui.py                     |   9 +-
> bitbake/lib/bb/ui/uievent.py                       |  10 +-
> bitbake/lib/toaster/bldcontrol/bbcontroller.py     |  33 -----
> .../toaster/bldcontrol/localhostbecontroller.py    | 133
>++++++++-------------
> .../bldcontrol/management/commands/runbuilds.py    |   6 +-
> 8 files changed, 80 insertions(+), 194 deletions(-)
>
>--
>Regards,
>Ed
>-- 
>_______________________________________________
>toaster mailing list
>toaster@yoctoproject.org
>https://lists.yoctoproject.org/listinfo/toaster

Re: [PATCH v2 00/15] per project build directory

[Ed Bartosh]

Hi Belen,

Thank you for the review. My comments are below.

On Wed, Mar 09, 2016 at 11:40:04AM +0000, Barros Pena, Belen wrote:
> >This is how to test this:
> > - run toaster
> > - run command line build: build zlib
> 
> This shows a warning about build history not being enabled. I know you
> already know about this, but just for completeness ;)
> 
Should be fixed in v3.

> > - check if it's shown in Toaster UI
> 
> Builds started from CLI do show in the web UI, but once they complete they
> are missing layer and package information. I guess that's connected with
> the build history issue above
> 
Most probably it is.

> > - create project for Yocto master
> > - trigger project build from UI
> 
> The build I triggered from the Toaster web UI from a Yocto master project
> (core-image-minimal) failed with this traceback:
> 
> /bin/sh: 1: source: not found
> 
> Traceback (most recent call last):
>   File 
> "/home/yocto/master/bitbake/lib/toaster/bldcontrol/management/commands/runb
> uilds.py", line 58, in schedule
>     br.brtarget_set.all(), "%d:%d" % (br.pk, bec.be.pk))
>   File 
> "/home/yocto/master/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
> ", line 261, in triggerBuild
>     self._shellcmd('source %s %s' % (oe_init, builddir), self.be.sourcedir)
>   File 
> "/home/yocto/master/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
> ", line 70, in _shellcmd
>     raise ShellCmdException(err)
> ShellCmdException: command: source
> /home/yocto/master/_toaster_clones/_git___git.yoctoproject.org_poky_master/
> oe-init-build-env /home/yocto/master/build-toaster-master project
> /bin/sh: 1: source: not found
> 
This is related to your shell (dash on Ubuntu I guess). Should be fixed
in v3.

> After all the above there is only one build directory created, the one
> that was generated when running the build from CLI. Also:
> 
> * I can no longer find the toaster_ui.log file
It will be in build directory. You didn't see it as toaster crashed and
didn't create build directory.

> * It looks like the builds are still using build/sstate-cache and
> build/downloads as SSTATE_DIR and DL_DIR, but I think we agreed that
> projects by default would share those 2 directories, which I guess means
> they have to move outside the build directory, although I could be mistaken
> 
This functionality is implemented in another patchset I think.
I'm going to make it working on top of my changes after I'm done with my
patchset. The same with build cancelation patchset - I'll make it
working after this one is accepted.

> * _toaster_clones is still being created in the root checkout directory,
> which I guess makes sense
> 
> * A pending question is where the toaster-custom-images layer we generate
> when building custom images should be. I had a brief chat with Michael
> about this, and I think we concluded the layer should not be shared across
> projects, which means we should create one per project and they should be
> inside the project build dir. Right now the _meta-toaster-custom directory
> is also in the root checkout directory
It makes sense to me. However, I'd rather not do it until this patchset
is tested and accepted. Otherwise amount of patches and new
functionality in this patchset will grow and it will be harder to
maintain it.

Thank you!
Ed