[PATCH 0/5] [dora] bitbake: fixes for dora

[PATCH 0/5] [dora] bitbake: fixes for dora

[Robert Yang]

The following changes since commit 53d2563ff13fcec74d4250bef5419e36169e55cc:

  bitbake: perforce: Fix path subdirectory issues (2013-12-03 17:51:31 +0000)

are available in the git repository at:

  git://git.pokylinux.org/poky-contrib robert/dora-next-bitbake
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=robert/dora-next-bitbake

Olof Johansson (1):
  bitbake: monitordisk: lower inode check warning to note

Peter Seebach (1):
  bitbake: build.py: add single-quotes around already-expanded
    directory name

Richard Purdie (3):
  bitbake: cooker/command: Add error state for the server and use for
    pre_serve errors
  bitbake: fetch2: Fix handling of SCM mirrors in MIRRORS
  bitbake: runqueue: Fix hole in setsceneverify skipped task logic

 bitbake/lib/bb/build.py           |    2 +-
 bitbake/lib/bb/command.py         |    2 ++
 bitbake/lib/bb/cooker.py          |    3 ++-
 bitbake/lib/bb/fetch2/__init__.py |    4 ++++
 bitbake/lib/bb/monitordisk.py     |    2 +-
 bitbake/lib/bb/runqueue.py        |   21 +++++++++++++++++++--
 6 files changed, 29 insertions(+), 5 deletions(-)

-- 
1.7.10.4

[PATCH 1/5] [dora] bitbake: cooker/command: Add error state for the server and use for pre_serve errors

[Robert Yang]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

Currently if errors occur when starting the PR service, there is a race that
occurs since the UI runs various commands including starting builds before
processing the CookerExit(). By adding the error state and refusing to run
async commands in this mode, builds are prevented from starting and the
UI reaches the exit code with the system shutting down cleanly.

(Bitbake master rev: 42fa34142ea685f91115a551e74416ca28ef1c91)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/lib/bb/command.py |    2 ++
 bitbake/lib/bb/cooker.py  |    3 ++-
 2 files changed, 4 insertions(+), 1 deletion(-)

diff --git a/bitbake/lib/bb/command.py b/bitbake/lib/bb/command.py
index 3ca27a6..a2795ce 100644
--- a/bitbake/lib/bb/command.py
+++ b/bitbake/lib/bb/command.py
@@ -86,6 +86,8 @@ class Command:
 
     def runAsyncCommand(self):
         try:
+            if self.cooker.state == bb.cooker.state.error:
+                return False
             if self.currentAsyncCommand is not None:
                 (command, options) = self.currentAsyncCommand
                 commandmethod = getattr(CommandsAsync, command)
diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index ad36b34..b504f45 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -61,7 +61,7 @@ class CollectionError(bb.BBHandledException):
     """
 
 class state:
-    initial, parsing, running, shutdown, forceshutdown, stopped = range(6)
+    initial, parsing, running, shutdown, forceshutdown, stopped, error = range(7)
 
 
 class SkippedPackage:
@@ -1321,6 +1321,7 @@ class BBCooker:
             self.prhost = prserv.serv.auto_start(self.data)
         except prserv.serv.PRServiceConfigError:
             bb.event.fire(CookerExit(), self.event_data)
+            self.state = state.error
         return
 
     def post_serve(self):
-- 
1.7.10.4

[PATCH 2/5] [dora] bitbake: monitordisk: lower inode check warning to note

[Robert Yang]

From: Olof Johansson <olof.johansson@axis.com>

Filesystems like btrfs and reiserfs sets the inode count to 0, since
they don't have an inode concept. This is expected, and having a warning
show up every time you run bitbake can cause undue concern.

(Bitbake master rev: f3ac2d3678f48c68a250a0a20c08cf8687322d38)

Signed-off-by: Olof Johansson <olof.johansson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/lib/bb/monitordisk.py |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/bitbake/lib/bb/monitordisk.py b/bitbake/lib/bb/monitordisk.py
index 3e6ecbd..fca43ee 100644
--- a/bitbake/lib/bb/monitordisk.py
+++ b/bitbake/lib/bb/monitordisk.py
@@ -243,7 +243,7 @@ class diskMonitor:
                     # zero, this is a feature of the fs, we disable the inode
                     # checking for such a fs.
                     if st.f_files == 0:
-                        logger.warn("Inode check for %s is unavaliable, will remove it from disk monitor" % path)
+                        logger.info("Inode check for %s is unavaliable, will remove it from disk monitor" % path)
                         self.devDict[k][2] = None
                         continue
                     # Always show warning, the self.checked would always be False if the action is WARN
-- 
1.7.10.4

[PATCH 3/5] [dora] bitbake: fetch2: Fix handling of SCM mirrors in MIRRORS

[Robert Yang]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

If an SCM mirror is in PREMIRRORS, the tarball is downloaded and then found
by the "upstream" check and handled correctly.

If an SCM mirror is in MIRRORS, the tarball is downloaded but not used
since there is no "upstream" run after MIRRORS completes. It therefore
sits there useless and unused. This code change forces the upstream to
run after a mirror tarball is found and fixes the usage of SCM mirrors
in MIRRORS.

(Bitbake master rev: a66ee0994645aa5658b2f5ea134ed17d89f8751a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/lib/bb/fetch2/__init__.py |    4 ++++
 1 file changed, 4 insertions(+)

diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py
index 451d104..b9f673c 100644
--- a/bitbake/lib/bb/fetch2/__init__.py
+++ b/bitbake/lib/bb/fetch2/__init__.py
@@ -805,6 +805,10 @@ def try_mirror_url(newuri, origud, ud, ld, check = False):
             dest = os.path.join(dldir, os.path.basename(ud.localpath))
             if not os.path.exists(dest):
                 os.symlink(ud.localpath, dest)
+            if not os.path.exists(origud.donestamp) or origud.method.need_update(origud.url, origud, ld):
+                origud.method.download(origud.url, origud, ld)
+                if hasattr(ud.method,"build_mirror_data"):
+                    origud.method.build_mirror_data(origud.url, origud, ld)
             return None
         # Otherwise the result is a local file:// and we symlink to it
         if not os.path.exists(origud.localpath):
-- 
1.7.10.4

[PATCH 4/5] [dora] bitbake: build.py: add single-quotes around already-expanded directory name

[Robert Yang]

From: Peter Seebach <peter.seebach@windriver.com>

If the computed name of a directory contains an undefined variable
reference, bitbake dutifully creates a directory with a name that has
${...} in it. However, the actual task script created then tries to cd
to that directory, and the cd command fails, because no such directory
exists -- because the shell has helpfully removed the ${...} which did
not match any actual variables.

Since we want the name to be used exactly-as-is, add single quotes around
the name so this doesn't cause strange failures running tasks, which
allows us to progress past such failures and get to a point where they
can be diagnosed.

(Bitbake master rev: 2809c2e6f2f35f9b08058950be896947ab5a0284)

Signed-off-by: Peter Seebach <peter.seebach@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/lib/bb/build.py |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/bitbake/lib/bb/build.py b/bitbake/lib/bb/build.py
index 2e49a09..edab556 100644
--- a/bitbake/lib/bb/build.py
+++ b/bitbake/lib/bb/build.py
@@ -285,7 +285,7 @@ set -e
         if bb.msg.loggerVerboseLogs:
             script.write("set -x\n")
         if cwd:
-            script.write("cd %s\n" % cwd)
+            script.write("cd '%s'\n" % cwd)
         script.write("%s\n" % func)
         script.write('''
 # cleanup
-- 
1.7.10.4

[PATCH 5/5] [dora] bitbake: runqueue: Fix hole in setsceneverify skipped task logic

[Robert Yang]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

We have do_bundle_initramfs which is a task inserted after compile and
before build. It is not covered by sstate.

If we run a build with a valid sstate cache present, the setsceneverify
function realises it will rerun the do_compile step (due to the
bundle_initramfs task) and hence marks do_populate_sysroot to rerun.
do_install, a dependency of do_populate_sysroot is left as marked as
covered by sstate.

What we need to do is traverse the dependency tree for any setsceneverify
invalided task and ensure any dependencies are also invalidated. We can
stop at any point we reach another setscene task though.

This means the do_populate_sysroot task has the data from do_install
available and doesn't crash.

(Bitbake master rev: f21910157d873c030b149c4cdc5b57c5062ab5a6)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/lib/bb/runqueue.py |   21 +++++++++++++++++++--
 1 file changed, 19 insertions(+), 2 deletions(-)

diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
index c09cfd4..72c0208 100644
--- a/bitbake/lib/bb/runqueue.py
+++ b/bitbake/lib/bb/runqueue.py
@@ -1204,6 +1204,8 @@ class RunQueueExecuteTasks(RunQueueExecute):
 
         self.stampcache = {}
 
+        initial_covered = self.rq.scenequeue_covered.copy()
+
         # Mark initial buildable tasks
         for task in xrange(self.stats.total):
             self.runq_running.append(0)
@@ -1257,13 +1259,28 @@ class RunQueueExecuteTasks(RunQueueExecute):
             except TypeError:
                 covered_remove = bb.utils.better_eval(call2, locs)
 
-        for task in covered_remove:
+        def removecoveredtask(task):
             fn = self.rqdata.taskData.fn_index[self.rqdata.runq_fnid[task]]
             taskname = self.rqdata.runq_task[task] + '_setscene'
             bb.build.del_stamp(taskname, self.rqdata.dataCache, fn)
-            logger.debug(1, 'Not skipping task %s due to setsceneverify', task)
             self.rq.scenequeue_covered.remove(task)
 
+        toremove = covered_remove
+        for task in toremove:
+            logger.debug(1, 'Not skipping task %s due to setsceneverify', task)
+        while toremove:
+            covered_remove = []
+            for task in toremove:
+                removecoveredtask(task)
+                for deptask in self.rqdata.runq_depends[task]:
+                    if deptask not in self.rq.scenequeue_covered:
+                        continue
+                    if deptask in toremove or deptask in covered_remove or deptask in initial_covered:
+                        continue
+                    logger.debug(1, 'Task %s depends on task %s so not skipping' % (task, deptask))
+                    covered_remove.append(deptask)
+            toremove = covered_remove
+
         logger.debug(1, 'Full skip list %s', self.rq.scenequeue_covered)
 
         event.fire(bb.event.StampUpdate(self.rqdata.target_pairs, self.rqdata.dataCache.stamp), self.cfgData)
-- 
1.7.10.4