[PATCH] bitbake: invalidate mtime cache if file doesn't exist

[PATCH] bitbake: invalidate mtime cache if file doesn't exist

[Ed Bartosh]

Mtime cache is desinged with assumption that files are not
removed. Unfortunately it's not always the case for memory-resident
bitbake as build/tmp can be removed to perform build from scratch.
In this case bitbake crashes when trying to create timestamps if
tmp/stamps/ hierarchy doesn't exist.

Simple check of file existance should solve this issue.

[YOCTO: #7562]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/bb/parse/__init__.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/bitbake/lib/bb/parse/__init__.py b/bitbake/lib/bb/parse/__init__.py
index 25effc2..e880732 100644
--- a/bitbake/lib/bb/parse/__init__.py
+++ b/bitbake/lib/bb/parse/__init__.py
@@ -62,6 +62,8 @@ def cached_mtime(f):
     return __mtime_cache[f]
 
 def cached_mtime_noerror(f):
+    if not os.path.exists(f):
+        return 0
     if f not in __mtime_cache:
         try:
             __mtime_cache[f] = os.stat(f)[stat.ST_MTIME]
-- 
2.1.4

Re: [PATCH] bitbake: invalidate mtime cache if file doesn't exist

[Richard Purdie]

On Tue, 2015-04-14 at 19:42 +0300, Ed Bartosh wrote:
> Mtime cache is desinged with assumption that files are not
> removed. Unfortunately it's not always the case for memory-resident
> bitbake as build/tmp can be removed to perform build from scratch.
> In this case bitbake crashes when trying to create timestamps if
> tmp/stamps/ hierarchy doesn't exist.
> 
> Simple check of file existance should solve this issue.
> 
> [YOCTO: #7562]
> 
> Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
> ---
>  bitbake/lib/bb/parse/__init__.py | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/bitbake/lib/bb/parse/__init__.py b/bitbake/lib/bb/parse/__init__.py
> index 25effc2..e880732 100644
> --- a/bitbake/lib/bb/parse/__init__.py
> +++ b/bitbake/lib/bb/parse/__init__.py
> @@ -62,6 +62,8 @@ def cached_mtime(f):
>      return __mtime_cache[f]
>  
>  def cached_mtime_noerror(f):
> +    if not os.path.exists(f):
> +        return 0
>      if f not in __mtime_cache:
>          try:
>              __mtime_cache[f] = os.stat(f)[stat.ST_MTIME]

I don't think this is right. The whole idea of this cache is so we don't
repeatedly hit stat() syscalls and by adding in the exists() check, the
cache is made worthless and I suspect we'd see a bad performance change.

Most of the use of this cache is from cooker/cache.py and there, we use
inotify to refresh the cache if something in the filesystem changes. We
probably need to teach the other users of this cache about the need to
deal with inotify (or simply remove any stamp data from the cache
entirely at the start of each bitbake invocation).

Cheers,

Richard

[PATCH] bitbake: reset mtime cache before the build

[Ed Bartosh]

Mtime cache is desinged with assumption that files are not
removed. Unfortunately it's not always the case for memory-resident
bitbake as build/tmp can be removed to perform build from scratch.
In this case bitbake crashes when trying to create timestamps if
tmp/stamps/ hierarchy doesn't exist.

Resetting mtime cache should solve this issue.

[YOCTO: #7562]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/bb/cooker.py         | 1 +
 bitbake/lib/bb/parse/__init__.py | 4 ++++
 2 files changed, 5 insertions(+)

diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index 9c101f2..383bd12 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -1343,6 +1343,7 @@ class BBCooker:
                 return True
             return retval
 
+        parse.reset_cache()
         self.buildSetVars()
 
         taskdata, runlist, fulltargetlist = self.buildTaskData(targets, task, self.configuration.abort)
diff --git a/bitbake/lib/bb/parse/__init__.py b/bitbake/lib/bb/parse/__init__.py
index 25effc2..c0867d4 100644
--- a/bitbake/lib/bb/parse/__init__.py
+++ b/bitbake/lib/bb/parse/__init__.py
@@ -78,6 +78,10 @@ def update_cache(f):
         logger.debug(1, "Updating mtime cache for %s" % f)
         update_mtime(f)
 
+def reset_cache():
+    global __mtime_cache
+    __mtime_cache = {}
+
 def mark_dependency(d, f):
     if f.startswith('./'):
         f = "%s/%s" % (os.getcwd(), f[2:])
-- 
2.1.4

[PATCH] bitbake: reset build mtime cache before the build

[Ed Bartosh]

Introduced build mtime cache structure. Reset it before the build
to prevent bitbake from crashing when build/tmp/stamps hierarchy
is removed.

[YOCTO: #7562]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/bb/build.py  | 17 ++++++++++++++++-
 bitbake/lib/bb/cooker.py |  3 ++-
 2 files changed, 18 insertions(+), 2 deletions(-)

diff --git a/bitbake/lib/bb/build.py b/bitbake/lib/bb/build.py
index 65cc851..0f6aa1a 100644
--- a/bitbake/lib/bb/build.py
+++ b/bitbake/lib/bb/build.py
@@ -31,6 +31,7 @@ import logging
 import shlex
 import glob
 import time
+import stat
 import bb
 import bb.msg
 import bb.process
@@ -42,6 +43,20 @@ logger = logging.getLogger('BitBake.Build')
 
 NULL = open(os.devnull, 'r+')
 
+__mtime_cache = {}
+
+def cached_mtime_noerror(f):
+    if f not in __mtime_cache:
+        try:
+            __mtime_cache[f] = os.stat(f)[stat.ST_MTIME]
+        except OSError:
+            return 0
+    return __mtime_cache[f]
+
+def reset_cache():
+    global __mtime_cache
+    __mtime_cache = {}
+
 # When we execute a Python function, we'd like certain things
 # in all namespaces, hence we add them to __builtins__.
 # If we do not do this and use the exec globals, they will
@@ -535,7 +550,7 @@ def stamp_internal(taskname, d, file_name, baseonly=False):
     stamp = bb.parse.siggen.stampfile(stamp, file_name, taskname, extrainfo)
 
     stampdir = os.path.dirname(stamp)
-    if bb.parse.cached_mtime_noerror(stampdir) == 0:
+    if cached_mtime_noerror(stampdir) == 0:
         bb.utils.mkdirhier(stampdir)
 
     return stamp
diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index 9c101f2..ddf5fed 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -35,7 +35,7 @@ from contextlib import closing
 from functools import wraps
 from collections import defaultdict
 import bb, bb.exceptions, bb.command
-from bb import utils, data, parse, event, cache, providers, taskdata, runqueue
+from bb import utils, data, parse, event, cache, providers, taskdata, runqueue, build
 import Queue
 import signal
 import prserv.serv
@@ -1343,6 +1343,7 @@ class BBCooker:
                 return True
             return retval
 
+        build.reset_cache()
         self.buildSetVars()
 
         taskdata, runlist, fulltargetlist = self.buildTaskData(targets, task, self.configuration.abort)
-- 
2.1.4