[PATCH 0/4] Progress fixes

[PATCH 0/4] Progress fixes

[Paul Eggleton]

Fixes for minor issues in the progress bar display, plus a patch for
some unclosed files that I noticed at the same time as working on these.


The following changes since commit 0a9b5d7d9655dbb09d458fc6e330e932f0f9dab6:

  toaster: buildinfohelper Add handling local layers (i.e. non-git) layers (2016-08-11 00:08:18 +0100)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib paule/bb-progress-fixes2
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=paule/bb-progress-fixes2

Paul Eggleton (4):
  runqueue: fix two minor issues with the initialising tasks progress
  knotty: fix task progress bar not starting at 0%
  knotty: don't show number of running tasks in quiet mode
  siggen: properly close files rather than opening them inline

 lib/bb/runqueue.py  |  5 +++--
 lib/bb/siggen.py    | 15 +++++++++------
 lib/bb/ui/knotty.py | 11 ++++++++---
 3 files changed, 20 insertions(+), 11 deletions(-)

-- 
2.5.5

[PATCH 1/4] runqueue: fix two minor issues with the initialising tasks progress

[Paul Eggleton]

A couple of fixes for the "Initialising tasks" progress bar behaviour:
* Properly finish the progress bar when using bitbake -S
* Finish the progress bar before calling BB_HASHCHECK_FUNCTION (so that
  in OE when that shows its own "Checking sstate mirror object
  availability"  progress bar it gets shown on the next line as it
  should).

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 lib/bb/runqueue.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index aa939d0..3a593b6 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -1148,6 +1148,7 @@ class RunQueue:
         if self.state is runQueueSceneInit:
             dump = self.cooker.configuration.dump_signatures
             if dump:
+                self.rqdata.init_progress_reporter.finish()
                 if 'printdiff' in dump:
                     invalidtasks = self.print_diffscenetasks()
                 self.dump_signatures(dump)
@@ -1967,6 +1968,8 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
             if len(self.sq_revdeps[tid]) == 0:
                 self.runq_buildable.add(tid)
 
+        self.rqdata.init_progress_reporter.finish()
+
         self.outrightfail = []
         if self.rq.hashvalidate:
             sq_hash = []
@@ -2018,8 +2021,6 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
                     logger.debug(2, 'No package found, so skipping setscene task %s', tid)
                     self.outrightfail.append(tid)
 
-        self.rqdata.init_progress_reporter.finish()
-
         logger.info('Executing SetScene Tasks')
 
         self.rq.state = runQueueSceneRun
-- 
2.5.5

[PATCH 2/4] knotty: fix task progress bar not starting at 0%

[Paul Eggleton]

If we have the task number here we need to subtract 1 to get the number
of tasks completed.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 lib/bb/ui/knotty.py | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/lib/bb/ui/knotty.py b/lib/bb/ui/knotty.py
index 1723a72..f89f7b3 100644
--- a/lib/bb/ui/knotty.py
+++ b/lib/bb/ui/knotty.py
@@ -260,13 +260,16 @@ class TerminalFilter(object):
                 content = "No currently running tasks (%s of %s)" % (self.helper.tasknumber_current, self.helper.tasknumber_total)
             else:
                 content = "Currently %2s running tasks (%s of %s)" % (len(activetasks), self.helper.tasknumber_current, self.helper.tasknumber_total)
-            maxtask = self.helper.tasknumber_total + 1
+            maxtask = self.helper.tasknumber_total
             if not self.main_progress or self.main_progress.maxval != maxtask:
                 widgets = [' ', progressbar.Percentage(), ' ', progressbar.Bar()]
                 self.main_progress = BBProgress("Running tasks", maxtask, widgets=widgets)
                 self.main_progress.start(False)
             self.main_progress.setmessage(content)
-            self.main_progress.update(self.helper.tasknumber_current)
+            progress = self.helper.tasknumber_current - 1
+            if progress < 0:
+                progress = 0
+            self.main_progress.update(progress)
             print('')
         lines = 1 + int(len(content) / (self.columns + 1))
         if not self.quiet:
-- 
2.5.5

[PATCH 3/4] knotty: don't show number of running tasks in quiet mode

[Paul Eggleton]

There's not a whole lot of point showing how many tasks are running when
we're in quiet mode, it just looks a bit strange particularly when it's
not running any tasks.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 lib/bb/ui/knotty.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/lib/bb/ui/knotty.py b/lib/bb/ui/knotty.py
index f89f7b3..b30135b 100644
--- a/lib/bb/ui/knotty.py
+++ b/lib/bb/ui/knotty.py
@@ -256,7 +256,9 @@ class TerminalFilter(object):
             content = "Waiting for %s running tasks to finish:" % len(activetasks)
             print(content)
         else:
-            if not len(activetasks):
+            if self.quiet:
+                content = "Running tasks (%s of %s)" % (self.helper.tasknumber_current, self.helper.tasknumber_total)
+            elif not len(activetasks):
                 content = "No currently running tasks (%s of %s)" % (self.helper.tasknumber_current, self.helper.tasknumber_total)
             else:
                 content = "Currently %2s running tasks (%s of %s)" % (len(activetasks), self.helper.tasknumber_current, self.helper.tasknumber_total)
-- 
2.5.5

[PATCH 4/4] siggen: properly close files rather than opening them inline

[Paul Eggleton]

If you don't do this, with Python 3 you get a warning on exit under some
circumstances.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 lib/bb/siggen.py | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)

diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py
index 5d8a253..9b2f658 100644
--- a/lib/bb/siggen.py
+++ b/lib/bb/siggen.py
@@ -363,10 +363,12 @@ def clean_basepaths_list(a):
 def compare_sigfiles(a, b, recursecb = None):
     output = []
 
-    p1 = pickle.Unpickler(open(a, "rb"))
-    a_data = p1.load()
-    p2 = pickle.Unpickler(open(b, "rb"))
-    b_data = p2.load()
+    with open(a, 'rb') as f:
+        p1 = pickle.Unpickler(ff)
+        a_data = p1.load()
+    with open(b, 'rb') as f:
+        p2 = pickle.Unpickler(f)
+        b_data = p2.load()
 
     def dict_diff(a, b, whitelist=set()):
         sa = set(a.keys())
@@ -563,8 +565,9 @@ def calc_taskhash(sigdata):
 def dump_sigfile(a):
     output = []
 
-    p1 = pickle.Unpickler(open(a, "rb"))
-    a_data = p1.load()
+    with open(a, 'rb') as f:
+        p1 = pickle.Unpickler(f)
+        a_data = p1.load()
 
     output.append("basewhitelist: %s" % (a_data['basewhitelist']))
 
-- 
2.5.5

Re: [PATCH 4/4] siggen: properly close files rather than opening them inline

[Peter Kjellerstedt]

> -----Original Message-----
> From: bitbake-devel-bounces@lists.openembedded.org [mailto:bitbake-
> devel-bounces@lists.openembedded.org] On Behalf Of Paul Eggleton
> Sent: den 11 augusti 2016 05:37
> To: bitbake-devel@lists.openembedded.org
> Subject: [bitbake-devel] [PATCH 4/4] siggen: properly close files
> rather than opening them inline
> 
> If you don't do this, with Python 3 you get a warning on exit under some
> circumstances.
> 
> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
> ---
>  lib/bb/siggen.py | 15 +++++++++------
>  1 file changed, 9 insertions(+), 6 deletions(-)
> 
> diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py
> index 5d8a253..9b2f658 100644
> --- a/lib/bb/siggen.py
> +++ b/lib/bb/siggen.py
> @@ -363,10 +363,12 @@ def clean_basepaths_list(a):
>  def compare_sigfiles(a, b, recursecb = None):
>      output = []
> 
> -    p1 = pickle.Unpickler(open(a, "rb"))
> -    a_data = p1.load()
> -    p2 = pickle.Unpickler(open(b, "rb"))
> -    b_data = p2.load()
> +    with open(a, 'rb') as f:
> +        p1 = pickle.Unpickler(ff)
                                 ^^
Shouldn't that be:

        p1 = pickle.Unpickler(f)

> +        a_data = p1.load()
> +    with open(b, 'rb') as f:
> +        p2 = pickle.Unpickler(f)
> +        b_data = p2.load()
> 
>      def dict_diff(a, b, whitelist=set()):
>          sa = set(a.keys())
> @@ -563,8 +565,9 @@ def calc_taskhash(sigdata):
>  def dump_sigfile(a):
>      output = []
> 
> -    p1 = pickle.Unpickler(open(a, "rb"))
> -    a_data = p1.load()
> +    with open(a, 'rb') as f:
> +        p1 = pickle.Unpickler(f)
> +        a_data = p1.load()
> 
>      output.append("basewhitelist: %s" % (a_data['basewhitelist']))
> 
> --
> 2.5.5

//Peter

Re: [PATCH 4/4] siggen: properly close files rather than opening them inline

[Paul Eggleton]

On Tue, 16 Aug 2016 21:16:17 Peter Kjellerstedt wrote:
> > -----Original Message-----
> > From: bitbake-devel-bounces@lists.openembedded.org [mailto:bitbake-
> > devel-bounces@lists.openembedded.org] On Behalf Of Paul Eggleton
> > Sent: den 11 augusti 2016 05:37
> > To: bitbake-devel@lists.openembedded.org
> > Subject: [bitbake-devel] [PATCH 4/4] siggen: properly close files
> > rather than opening them inline
> > 
> > If you don't do this, with Python 3 you get a warning on exit under some
> > circumstances.
> > 
> > Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
> > ---
> > 
> >  lib/bb/siggen.py | 15 +++++++++------
> >  1 file changed, 9 insertions(+), 6 deletions(-)
> > 
> > diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py
> > index 5d8a253..9b2f658 100644
> > --- a/lib/bb/siggen.py
> > +++ b/lib/bb/siggen.py
> > 
> > @@ -363,10 +363,12 @@ def clean_basepaths_list(a):
> >  def compare_sigfiles(a, b, recursecb = None):
> >      output = []
> > 
> > -    p1 = pickle.Unpickler(open(a, "rb"))
> > -    a_data = p1.load()
> > -    p2 = pickle.Unpickler(open(b, "rb"))
> > -    b_data = p2.load()
> > +    with open(a, 'rb') as f:
> > +        p1 = pickle.Unpickler(ff)
> 
>                                  ^^
> Shouldn't that be:
> 
>         p1 = pickle.Unpickler(f)
> 

Oops - yes it should be. I guess it just proves that even "simple, obvious" 
changes like this one do actually need to be tested. I'll send a v2.

Thanks,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre