* [RFC PATCH 0/6] Task progress reporting
@ 2016-05-10 23:24 Paul Eggleton
2016-05-10 23:24 ` [RFC PATCH 1/6] knotty: import latest python-progressbar Paul Eggleton
` (5 more replies)
0 siblings, 6 replies; 7+ messages in thread
From: Paul Eggleton @ 2016-05-10 23:24 UTC (permalink / raw)
To: openembedded-core
I've often thought it would be nice if we could print progress information
for individual tasks so that you could get some idea of whether the task
is doing anything and how much longer it might take, and so I've been working
on a set of patches to implement just that in the background for a while now
and this is the result. Unfortunately this isn't necessarily practical to do
for all tasks, but things like do_fetch are running processes that do output
progress information so it's not too hard to tap into that and present it in
the form of a progress bar next to the task, which is what this patchset
does. I was also able to enable progress for do_compile in recipes that use
cmake (since that outputs % complete as it runs) as well as do_rootfs where
it has a number of internal steps that can be counted and weighted.
Note that this patchset is on top of Poky because there are changes required
in both BitBake and OE-Core. However, for those that prefer BitBake + OE-core
separately I have pushed a paule/progress-bb branch to poky-contrib (on top
of bitbake master) and paule/progress-oecore to openembedded-core-contrib on
top of openembedded-core master containing the same changes - ensure you have
both if you go down this route.
I have contemplated and even prototyped going a step further and using the
count of lines in the output to track progress for *all* long-running tasks,
which does work reasonably well - the caveat is you actually need the
expected line count ahead of time for each task, so you have to be able to
get that from somewhere. I haven't worked out a practical means of
delivering that, suggestions would be very much welcome - my prototype stuff
is on poky-contrib paule/progress-wip and relies on someone profiling a
build containing all the tasks beforehand and that information somehow being
made available to everyone else.
Anyway, please let me know what you think.
Please review the following changes for suitability for inclusion. If you have
any objections or suggestions for improvement, please respond to the patches. If
you agree with the changes, please provide your Acked-by.
The following changes since commit 7ca60ec8bff7656b4e52f5a4d238913e851da089:
test-empty-image: Fix LIC_FILES_CHKSUM typo (2016-05-06 10:48:06 +0100)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib paule/progress
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=paule/progress
Paul Eggleton (6):
knotty: import latest python-progressbar
lib: implement basic task progress support
fetch2: implement progress support
bitbake.conf: whitelist progress varflag
classes/cmake: enable progress for do_compile
classes/image: implement progress support for do_rootfs
bitbake/lib/bb/build.py | 143 ++++++++
bitbake/lib/bb/fetch2/__init__.py | 4 +-
bitbake/lib/bb/fetch2/git.py | 47 ++-
bitbake/lib/bb/fetch2/wget.py | 26 +-
bitbake/lib/bb/ui/knotty.py | 76 ++++-
bitbake/lib/bb/ui/uihelper.py | 7 +-
bitbake/lib/progressbar.py | 384 ---------------------
bitbake/lib/progressbar/LICENSE.txt | 52 +++
bitbake/lib/progressbar/__init__.py | 49 +++
bitbake/lib/progressbar/compat.py | 44 +++
bitbake/lib/progressbar/progressbar.py | 315 +++++++++++++++++
bitbake/lib/progressbar/widgets.py | 391 ++++++++++++++++++++++
meta/classes/cmake.bbclass | 1 +
meta/classes/image.bbclass | 9 +-
meta/conf/bitbake.conf | 3 +-
meta/lib/oe/rootfs.py | 101 +++++-
meta/recipes-devtools/cmake/cmake-native_3.4.3.bb | 2 +
17 files changed, 1238 insertions(+), 416 deletions(-)
delete mode 100644 bitbake/lib/progressbar.py
create mode 100644 bitbake/lib/progressbar/LICENSE.txt
create mode 100644 bitbake/lib/progressbar/__init__.py
create mode 100644 bitbake/lib/progressbar/compat.py
create mode 100644 bitbake/lib/progressbar/progressbar.py
create mode 100644 bitbake/lib/progressbar/widgets.py
--
2.5.5
^ permalink raw reply [flat|nested] 7+ messages in thread
* [RFC PATCH 1/6] knotty: import latest python-progressbar
2016-05-10 23:24 [RFC PATCH 0/6] Task progress reporting Paul Eggleton
@ 2016-05-10 23:24 ` Paul Eggleton
2016-05-10 23:24 ` [RFC PATCH 2/6] lib: implement basic task progress support Paul Eggleton
` (4 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Paul Eggleton @ 2016-05-10 23:24 UTC (permalink / raw)
To: openembedded-core
Since we're going to make some minor extensions to it, it makes sense to
bring in the latest version of python-progressbar. Its structure has
changed a little but the API hasn't; however we do need to ensure our
overridden _needs_update() function's signature in BBProgress() matches
properly.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
bitbake/lib/bb/ui/knotty.py | 2 +-
bitbake/lib/progressbar.py | 384 ---------------------------------
bitbake/lib/progressbar/LICENSE.txt | 52 +++++
bitbake/lib/progressbar/__init__.py | 49 +++++
bitbake/lib/progressbar/compat.py | 44 ++++
bitbake/lib/progressbar/progressbar.py | 307 ++++++++++++++++++++++++++
bitbake/lib/progressbar/widgets.py | 355 ++++++++++++++++++++++++++++++
7 files changed, 808 insertions(+), 385 deletions(-)
delete mode 100644 bitbake/lib/progressbar.py
create mode 100644 bitbake/lib/progressbar/LICENSE.txt
create mode 100644 bitbake/lib/progressbar/__init__.py
create mode 100644 bitbake/lib/progressbar/compat.py
create mode 100644 bitbake/lib/progressbar/progressbar.py
create mode 100644 bitbake/lib/progressbar/widgets.py
diff --git a/bitbake/lib/bb/ui/knotty.py b/bitbake/lib/bb/ui/knotty.py
index 2685627..010b06d 100644
--- a/bitbake/lib/bb/ui/knotty.py
+++ b/bitbake/lib/bb/ui/knotty.py
@@ -51,7 +51,7 @@ class BBProgress(progressbar.ProgressBar):
self._resize_default = None
progressbar.ProgressBar.__init__(self, maxval, [self.msg + ": "] + widgets, fd=sys.stdout)
- def _handle_resize(self, signum, frame):
+ def _handle_resize(self, signum=None, frame=None):
progressbar.ProgressBar._handle_resize(self, signum, frame)
if self._resize_default:
self._resize_default(signum, frame)
diff --git a/bitbake/lib/progressbar.py b/bitbake/lib/progressbar.py
deleted file mode 100644
index 114cdc1..0000000
--- a/bitbake/lib/progressbar.py
+++ /dev/null
@@ -1,384 +0,0 @@
-#!/usr/bin/env python
-# -*- coding: iso-8859-1 -*-
-#
-# progressbar - Text progressbar library for python.
-# Copyright (c) 2005 Nilton Volpato
-#
-# This library is free software; you can redistribute it and/or
-# modify it under the terms of the GNU Lesser General Public
-# License as published by the Free Software Foundation; either
-# version 2.1 of the License, or (at your option) any later version.
-#
-# This library is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
-# Lesser General Public License for more details.
-#
-# You should have received a copy of the GNU Lesser General Public
-# License along with this library; if not, write to the Free Software
-# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
-
-
-"""Text progressbar library for python.
-
-This library provides a text mode progressbar. This is typically used
-to display the progress of a long running operation, providing a
-visual clue that processing is underway.
-
-The ProgressBar class manages the progress, and the format of the line
-is given by a number of widgets. A widget is an object that may
-display diferently depending on the state of the progress. There are
-three types of widget:
-- a string, which always shows itself;
-- a ProgressBarWidget, which may return a diferent value every time
-it's update method is called; and
-- a ProgressBarWidgetHFill, which is like ProgressBarWidget, except it
-expands to fill the remaining width of the line.
-
-The progressbar module is very easy to use, yet very powerful. And
-automatically supports features like auto-resizing when available.
-"""
-
-from __future__ import division
-
-__author__ = "Nilton Volpato"
-__author_email__ = "first-name dot last-name @ gmail.com"
-__date__ = "2006-05-07"
-__version__ = "2.3-dev"
-
-import sys, time, os
-from array import array
-try:
- from fcntl import ioctl
- import termios
-except ImportError:
- pass
-import signal
-try:
- basestring
-except NameError:
- basestring = (str,)
-
-class ProgressBarWidget(object):
- """This is an element of ProgressBar formatting.
-
- The ProgressBar object will call it's update value when an update
- is needed. It's size may change between call, but the results will
- not be good if the size changes drastically and repeatedly.
- """
- def update(self, pbar):
- """Returns the string representing the widget.
-
- The parameter pbar is a reference to the calling ProgressBar,
- where one can access attributes of the class for knowing how
- the update must be made.
-
- At least this function must be overriden."""
- pass
-
-class ProgressBarWidgetHFill(object):
- """This is a variable width element of ProgressBar formatting.
-
- The ProgressBar object will call it's update value, informing the
- width this object must the made. This is like TeX \\hfill, it will
- expand to fill the line. You can use more than one in the same
- line, and they will all have the same width, and together will
- fill the line.
- """
- def update(self, pbar, width):
- """Returns the string representing the widget.
-
- The parameter pbar is a reference to the calling ProgressBar,
- where one can access attributes of the class for knowing how
- the update must be made. The parameter width is the total
- horizontal width the widget must have.
-
- At least this function must be overriden."""
- pass
-
-
-class ETA(ProgressBarWidget):
- "Widget for the Estimated Time of Arrival"
- def format_time(self, seconds):
- return time.strftime('%H:%M:%S', time.gmtime(seconds))
- def update(self, pbar):
- if pbar.currval == 0:
- return 'ETA: --:--:--'
- elif pbar.finished:
- return 'Time: %s' % self.format_time(pbar.seconds_elapsed)
- else:
- elapsed = pbar.seconds_elapsed
- eta = elapsed * pbar.maxval / pbar.currval - elapsed
- return 'ETA: %s' % self.format_time(eta)
-
-class FileTransferSpeed(ProgressBarWidget):
- "Widget for showing the transfer speed (useful for file transfers)."
- def __init__(self, unit='B'):
- self.unit = unit
- self.fmt = '%6.2f %s'
- self.prefixes = ['', 'K', 'M', 'G', 'T', 'P']
- def update(self, pbar):
- if pbar.seconds_elapsed < 2e-6:#== 0:
- bps = 0.0
- else:
- bps = pbar.currval / pbar.seconds_elapsed
- spd = bps
- for u in self.prefixes:
- if spd < 1000:
- break
- spd /= 1000
- return self.fmt % (spd, u + self.unit + '/s')
-
-class RotatingMarker(ProgressBarWidget):
- "A rotating marker for filling the bar of progress."
- def __init__(self, markers='|/-\\'):
- self.markers = markers
- self.curmark = -1
- def update(self, pbar):
- if pbar.finished:
- return self.markers[0]
- self.curmark = (self.curmark + 1) % len(self.markers)
- return self.markers[self.curmark]
-
-class Percentage(ProgressBarWidget):
- "Just the percentage done."
- def update(self, pbar):
- return '%3d%%' % pbar.percentage()
-
-class SimpleProgress(ProgressBarWidget):
- "Returns what is already done and the total, e.g.: '5 of 47'"
- def __init__(self, sep=' of '):
- self.sep = sep
- def update(self, pbar):
- return '%d%s%d' % (pbar.currval, self.sep, pbar.maxval)
-
-class Bar(ProgressBarWidgetHFill):
- "The bar of progress. It will stretch to fill the line."
- def __init__(self, marker='#', left='|', right='|'):
- self.marker = marker
- self.left = left
- self.right = right
- def _format_marker(self, pbar):
- if isinstance(self.marker, basestring):
- return self.marker
- else:
- return self.marker.update(pbar)
- def update(self, pbar, width):
- percent = pbar.percentage()
- cwidth = width - len(self.left) - len(self.right)
- marked_width = int(percent * cwidth // 100)
- m = self._format_marker(pbar)
- bar = (self.left + (m * marked_width).ljust(cwidth) + self.right)
- return bar
-
-class ReverseBar(Bar):
- "The reverse bar of progress, or bar of regress. :)"
- def update(self, pbar, width):
- percent = pbar.percentage()
- cwidth = width - len(self.left) - len(self.right)
- marked_width = int(percent * cwidth // 100)
- m = self._format_marker(pbar)
- bar = (self.left + (m*marked_width).rjust(cwidth) + self.right)
- return bar
-
-default_widgets = [Percentage(), ' ', Bar()]
-class ProgressBar(object):
- """This is the ProgressBar class, it updates and prints the bar.
-
- A common way of using it is like:
- >>> pbar = ProgressBar().start()
- >>> for i in xrange(100):
- ... # do something
- ... pbar.update(i+1)
- ...
- >>> pbar.finish()
-
- You can also use a progressbar as an iterator:
- >>> progress = ProgressBar()
- >>> for i in progress(some_iterable):
- ... # do something
- ...
-
- But anything you want to do is possible (well, almost anything).
- You can supply different widgets of any type in any order. And you
- can even write your own widgets! There are many widgets already
- shipped and you should experiment with them.
-
- The term_width parameter must be an integer or None. In the latter case
- it will try to guess it, if it fails it will default to 80 columns.
-
- When implementing a widget update method you may access any
- attribute or function of the ProgressBar object calling the
- widget's update method. The most important attributes you would
- like to access are:
- - currval: current value of the progress, 0 <= currval <= maxval
- - maxval: maximum (and final) value of the progress
- - finished: True if the bar has finished (reached 100%), False o/w
- - start_time: the time when start() method of ProgressBar was called
- - seconds_elapsed: seconds elapsed since start_time
- - percentage(): percentage of the progress [0..100]. This is a method.
-
- The attributes above are unlikely to change between different versions,
- the other ones may change or cease to exist without notice, so try to rely
- only on the ones documented above if you are extending the progress bar.
- """
-
- __slots__ = ('currval', 'fd', 'finished', 'last_update_time', 'maxval',
- 'next_update', 'num_intervals', 'seconds_elapsed',
- 'signal_set', 'start_time', 'term_width', 'update_interval',
- 'widgets', '_iterable')
-
- _DEFAULT_MAXVAL = 100
-
- def __init__(self, maxval=None, widgets=default_widgets, term_width=None,
- fd=sys.stderr):
- self.maxval = maxval
- self.widgets = widgets
- self.fd = fd
- self.signal_set = False
- if term_width is not None:
- self.term_width = term_width
- else:
- try:
- self._handle_resize(None, None)
- signal.signal(signal.SIGWINCH, self._handle_resize)
- self.signal_set = True
- except (SystemExit, KeyboardInterrupt):
- raise
- except:
- self.term_width = int(os.environ.get('COLUMNS', 80)) - 1
-
- self.currval = 0
- self.finished = False
- self.start_time = None
- self.last_update_time = None
- self.seconds_elapsed = 0
- self._iterable = None
-
- def __call__(self, iterable):
- try:
- self.maxval = len(iterable)
- except TypeError:
- # If the iterable has no length, then rely on the value provided
- # by the user, otherwise fail.
- if not (isinstance(self.maxval, (int, long)) and self.maxval > 0):
- raise RuntimeError('Could not determine maxval from iterable. '
- 'You must explicitly provide a maxval.')
- self._iterable = iter(iterable)
- self.start()
- return self
-
- def __iter__(self):
- return self
-
- def next(self):
- try:
- next = self._iterable.next()
- self.update(self.currval + 1)
- return next
- except StopIteration:
- self.finish()
- raise
-
- def _handle_resize(self, signum, frame):
- h, w = array('h', ioctl(self.fd, termios.TIOCGWINSZ, '\0' * 8))[:2]
- self.term_width = w
-
- def percentage(self):
- "Returns the percentage of the progress."
- return self.currval * 100.0 / self.maxval
-
- def _format_widgets(self):
- r = []
- hfill_inds = []
- num_hfill = 0
- currwidth = 0
- for i, w in enumerate(self.widgets):
- if isinstance(w, ProgressBarWidgetHFill):
- r.append(w)
- hfill_inds.append(i)
- num_hfill += 1
- elif isinstance(w, basestring):
- r.append(w)
- currwidth += len(w)
- else:
- weval = w.update(self)
- currwidth += len(weval)
- r.append(weval)
- for iw in hfill_inds:
- widget_width = int((self.term_width - currwidth) // num_hfill)
- r[iw] = r[iw].update(self, widget_width)
- return r
-
- def _format_line(self):
- return ''.join(self._format_widgets()).ljust(self.term_width)
-
- def _next_update(self):
- return int((int(self.num_intervals *
- (self.currval / self.maxval)) + 1) *
- self.update_interval)
-
- def _need_update(self):
- """Returns true when the progressbar should print an updated line.
-
- You can override this method if you want finer grained control over
- updates.
-
- The current implementation is optimized to be as fast as possible and
- as economical as possible in the number of updates. However, depending
- on your usage you may want to do more updates. For instance, if your
- progressbar stays in the same percentage for a long time, and you want
- to update other widgets, like ETA, then you could return True after
- some time has passed with no updates.
-
- Ideally you could call self._format_line() and see if it's different
- from the previous _format_line() call, but calling _format_line() takes
- around 20 times more time than calling this implementation of
- _need_update().
- """
- return self.currval >= self.next_update
-
- def update(self, value):
- "Updates the progress bar to a new value."
- assert 0 <= value <= self.maxval, '0 <= %d <= %d' % (value, self.maxval)
- self.currval = value
- if not self._need_update():
- return
- if self.start_time is None:
- raise RuntimeError('You must call start() before calling update()')
- now = time.time()
- self.seconds_elapsed = now - self.start_time
- self.next_update = self._next_update()
- self.fd.write(self._format_line() + '\r')
- self.last_update_time = now
-
- def start(self):
- """Starts measuring time, and prints the bar at 0%.
-
- It returns self so you can use it like this:
- >>> pbar = ProgressBar().start()
- >>> for i in xrange(100):
- ... # do something
- ... pbar.update(i+1)
- ...
- >>> pbar.finish()
- """
- if self.maxval is None:
- self.maxval = self._DEFAULT_MAXVAL
- assert self.maxval > 0
-
- self.num_intervals = max(100, self.term_width)
- self.update_interval = self.maxval / self.num_intervals
- self.next_update = 0
-
- self.start_time = self.last_update_time = time.time()
- self.update(0)
- return self
-
- def finish(self):
- """Used to tell the progress is finished."""
- self.finished = True
- self.update(self.maxval)
- self.fd.write('\n')
- if self.signal_set:
- signal.signal(signal.SIGWINCH, signal.SIG_DFL)
diff --git a/bitbake/lib/progressbar/LICENSE.txt b/bitbake/lib/progressbar/LICENSE.txt
new file mode 100644
index 0000000..fc8ccdc
--- /dev/null
+++ b/bitbake/lib/progressbar/LICENSE.txt
@@ -0,0 +1,52 @@
+You can redistribute and/or modify this library under the terms of the
+GNU LGPL license or BSD license (or both).
+
+---
+
+progressbar - Text progress bar library for python.
+Copyright (C) 2005 Nilton Volpato
+
+This library is free software; you can redistribute it and/or
+modify it under the terms of the GNU Lesser General Public
+License as published by the Free Software Foundation; either
+version 2.1 of the License, or (at your option) any later version.
+
+This library is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+Lesser General Public License for more details.
+
+You should have received a copy of the GNU Lesser General Public
+License along with this library; if not, write to the Free Software
+Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+
+---
+
+progressbar - Text progress bar library for python
+Copyright (c) 2008 Nilton Volpato
+
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+ a. Redistributions of source code must retain the above copyright notice,
+ this list of conditions and the following disclaimer.
+ b. Redistributions in binary form must reproduce the above copyright
+ notice, this list of conditions and the following disclaimer in the
+ documentation and/or other materials provided with the distribution.
+ c. Neither the name of the author nor the names of its contributors
+ may be used to endorse or promote products derived from this software
+ without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE FOR
+ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
+DAMAGE.
diff --git a/bitbake/lib/progressbar/__init__.py b/bitbake/lib/progressbar/__init__.py
new file mode 100644
index 0000000..fbab744
--- /dev/null
+++ b/bitbake/lib/progressbar/__init__.py
@@ -0,0 +1,49 @@
+#!/usr/bin/python
+# -*- coding: utf-8 -*-
+#
+# progressbar - Text progress bar library for Python.
+# Copyright (c) 2005 Nilton Volpato
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2.1 of the License, or (at your option) any later version.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+
+"""Text progress bar library for Python.
+
+A text progress bar is typically used to display the progress of a long
+running operation, providing a visual cue that processing is underway.
+
+The ProgressBar class manages the current progress, and the format of the line
+is given by a number of widgets. A widget is an object that may display
+differently depending on the state of the progress bar. There are three types
+of widgets:
+ - a string, which always shows itself
+
+ - a ProgressBarWidget, which may return a different value every time its
+ update method is called
+
+ - a ProgressBarWidgetHFill, which is like ProgressBarWidget, except it
+ expands to fill the remaining width of the line.
+
+The progressbar module is very easy to use, yet very powerful. It will also
+automatically enable features like auto-resizing when the system supports it.
+"""
+
+__author__ = 'Nilton Volpato'
+__author_email__ = 'first-name dot last-name @ gmail.com'
+__date__ = '2011-05-14'
+__version__ = '2.3'
+
+from .compat import *
+from .widgets import *
+from .progressbar import *
diff --git a/bitbake/lib/progressbar/compat.py b/bitbake/lib/progressbar/compat.py
new file mode 100644
index 0000000..a39f4a1
--- /dev/null
+++ b/bitbake/lib/progressbar/compat.py
@@ -0,0 +1,44 @@
+# -*- coding: utf-8 -*-
+#
+# progressbar - Text progress bar library for Python.
+# Copyright (c) 2005 Nilton Volpato
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2.1 of the License, or (at your option) any later version.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+
+"""Compatibility methods and classes for the progressbar module."""
+
+
+# Python 3.x (and backports) use a modified iterator syntax
+# This will allow 2.x to behave with 3.x iterators
+try:
+ next
+except NameError:
+ def next(iter):
+ try:
+ # Try new style iterators
+ return iter.__next__()
+ except AttributeError:
+ # Fallback in case of a "native" iterator
+ return iter.next()
+
+
+# Python < 2.5 does not have "any"
+try:
+ any
+except NameError:
+ def any(iterator):
+ for item in iterator:
+ if item: return True
+ return False
diff --git a/bitbake/lib/progressbar/progressbar.py b/bitbake/lib/progressbar/progressbar.py
new file mode 100644
index 0000000..0b9dcf7
--- /dev/null
+++ b/bitbake/lib/progressbar/progressbar.py
@@ -0,0 +1,307 @@
+# -*- coding: utf-8 -*-
+#
+# progressbar - Text progress bar library for Python.
+# Copyright (c) 2005 Nilton Volpato
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2.1 of the License, or (at your option) any later version.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+
+"""Main ProgressBar class."""
+
+from __future__ import division
+
+import math
+import os
+import signal
+import sys
+import time
+
+try:
+ from fcntl import ioctl
+ from array import array
+ import termios
+except ImportError:
+ pass
+
+from .compat import * # for: any, next
+from . import widgets
+
+
+class UnknownLength: pass
+
+
+class ProgressBar(object):
+ """The ProgressBar class which updates and prints the bar.
+
+ A common way of using it is like:
+ >>> pbar = ProgressBar().start()
+ >>> for i in range(100):
+ ... # do something
+ ... pbar.update(i+1)
+ ...
+ >>> pbar.finish()
+
+ You can also use a ProgressBar as an iterator:
+ >>> progress = ProgressBar()
+ >>> for i in progress(some_iterable):
+ ... # do something
+ ...
+
+ Since the progress bar is incredibly customizable you can specify
+ different widgets of any type in any order. You can even write your own
+ widgets! However, since there are already a good number of widgets you
+ should probably play around with them before moving on to create your own
+ widgets.
+
+ The term_width parameter represents the current terminal width. If the
+ parameter is set to an integer then the progress bar will use that,
+ otherwise it will attempt to determine the terminal width falling back to
+ 80 columns if the width cannot be determined.
+
+ When implementing a widget's update method you are passed a reference to
+ the current progress bar. As a result, you have access to the
+ ProgressBar's methods and attributes. Although there is nothing preventing
+ you from changing the ProgressBar you should treat it as read only.
+
+ Useful methods and attributes include (Public API):
+ - currval: current progress (0 <= currval <= maxval)
+ - maxval: maximum (and final) value
+ - finished: True if the bar has finished (reached 100%)
+ - start_time: the time when start() method of ProgressBar was called
+ - seconds_elapsed: seconds elapsed since start_time and last call to
+ update
+ - percentage(): progress in percent [0..100]
+ """
+
+ __slots__ = ('currval', 'fd', 'finished', 'last_update_time',
+ 'left_justify', 'maxval', 'next_update', 'num_intervals',
+ 'poll', 'seconds_elapsed', 'signal_set', 'start_time',
+ 'term_width', 'update_interval', 'widgets', '_time_sensitive',
+ '__iterable')
+
+ _DEFAULT_MAXVAL = 100
+ _DEFAULT_TERMSIZE = 80
+ _DEFAULT_WIDGETS = [widgets.Percentage(), ' ', widgets.Bar()]
+
+ def __init__(self, maxval=None, widgets=None, term_width=None, poll=1,
+ left_justify=True, fd=sys.stderr):
+ """Initializes a progress bar with sane defaults."""
+
+ # Don't share a reference with any other progress bars
+ if widgets is None:
+ widgets = list(self._DEFAULT_WIDGETS)
+
+ self.maxval = maxval
+ self.widgets = widgets
+ self.fd = fd
+ self.left_justify = left_justify
+
+ self.signal_set = False
+ if term_width is not None:
+ self.term_width = term_width
+ else:
+ try:
+ self._handle_resize(None, None)
+ signal.signal(signal.SIGWINCH, self._handle_resize)
+ self.signal_set = True
+ except (SystemExit, KeyboardInterrupt): raise
+ except Exception as e:
+ print("DEBUG 5 %s" % e)
+ self.term_width = self._env_size()
+
+ self.__iterable = None
+ self._update_widgets()
+ self.currval = 0
+ self.finished = False
+ self.last_update_time = None
+ self.poll = poll
+ self.seconds_elapsed = 0
+ self.start_time = None
+ self.update_interval = 1
+ self.next_update = 0
+
+
+ def __call__(self, iterable):
+ """Use a ProgressBar to iterate through an iterable."""
+
+ try:
+ self.maxval = len(iterable)
+ except:
+ if self.maxval is None:
+ self.maxval = UnknownLength
+
+ self.__iterable = iter(iterable)
+ return self
+
+
+ def __iter__(self):
+ return self
+
+
+ def __next__(self):
+ try:
+ value = next(self.__iterable)
+ if self.start_time is None:
+ self.start()
+ else:
+ self.update(self.currval + 1)
+ return value
+ except StopIteration:
+ if self.start_time is None:
+ self.start()
+ self.finish()
+ raise
+
+
+ # Create an alias so that Python 2.x won't complain about not being
+ # an iterator.
+ next = __next__
+
+
+ def _env_size(self):
+ """Tries to find the term_width from the environment."""
+
+ return int(os.environ.get('COLUMNS', self._DEFAULT_TERMSIZE)) - 1
+
+
+ def _handle_resize(self, signum=None, frame=None):
+ """Tries to catch resize signals sent from the terminal."""
+
+ h, w = array('h', ioctl(self.fd, termios.TIOCGWINSZ, '\0' * 8))[:2]
+ self.term_width = w
+
+
+ def percentage(self):
+ """Returns the progress as a percentage."""
+ if self.currval >= self.maxval:
+ return 100.0
+ return (self.currval * 100.0 / self.maxval) if self.maxval else 100.00
+
+ percent = property(percentage)
+
+
+ def _format_widgets(self):
+ result = []
+ expanding = []
+ width = self.term_width
+
+ for index, widget in enumerate(self.widgets):
+ if isinstance(widget, widgets.WidgetHFill):
+ result.append(widget)
+ expanding.insert(0, index)
+ else:
+ widget = widgets.format_updatable(widget, self)
+ result.append(widget)
+ width -= len(widget)
+
+ count = len(expanding)
+ while count:
+ portion = max(int(math.ceil(width * 1. / count)), 0)
+ index = expanding.pop()
+ count -= 1
+
+ widget = result[index].update(self, portion)
+ width -= len(widget)
+ result[index] = widget
+
+ return result
+
+
+ def _format_line(self):
+ """Joins the widgets and justifies the line."""
+
+ widgets = ''.join(self._format_widgets())
+
+ if self.left_justify: return widgets.ljust(self.term_width)
+ else: return widgets.rjust(self.term_width)
+
+
+ def _need_update(self):
+ """Returns whether the ProgressBar should redraw the line."""
+ if self.currval >= self.next_update or self.finished: return True
+
+ delta = time.time() - self.last_update_time
+ return self._time_sensitive and delta > self.poll
+
+
+ def _update_widgets(self):
+ """Checks all widgets for the time sensitive bit."""
+
+ self._time_sensitive = any(getattr(w, 'TIME_SENSITIVE', False)
+ for w in self.widgets)
+
+
+ def update(self, value=None):
+ """Updates the ProgressBar to a new value."""
+
+ if value is not None and value is not UnknownLength:
+ if (self.maxval is not UnknownLength
+ and not 0 <= value <= self.maxval):
+
+ raise ValueError('Value out of range')
+
+ self.currval = value
+
+
+ if not self._need_update(): return
+ if self.start_time is None:
+ raise RuntimeError('You must call "start" before calling "update"')
+
+ now = time.time()
+ self.seconds_elapsed = now - self.start_time
+ self.next_update = self.currval + self.update_interval
+ self.fd.write(self._format_line() + '\r')
+ self.fd.flush()
+ self.last_update_time = now
+
+
+ def start(self):
+ """Starts measuring time, and prints the bar at 0%.
+
+ It returns self so you can use it like this:
+ >>> pbar = ProgressBar().start()
+ >>> for i in range(100):
+ ... # do something
+ ... pbar.update(i+1)
+ ...
+ >>> pbar.finish()
+ """
+
+ if self.maxval is None:
+ self.maxval = self._DEFAULT_MAXVAL
+
+ self.num_intervals = max(100, self.term_width)
+ self.next_update = 0
+
+ if self.maxval is not UnknownLength:
+ if self.maxval < 0: raise ValueError('Value out of range')
+ self.update_interval = self.maxval / self.num_intervals
+
+
+ self.start_time = self.last_update_time = time.time()
+ self.update(0)
+
+ return self
+
+
+ def finish(self):
+ """Puts the ProgressBar bar in the finished state."""
+
+ if self.finished:
+ return
+ self.finished = True
+ self.update(self.maxval)
+ self.fd.write('\n')
+ if self.signal_set:
+ signal.signal(signal.SIGWINCH, signal.SIG_DFL)
diff --git a/bitbake/lib/progressbar/widgets.py b/bitbake/lib/progressbar/widgets.py
new file mode 100644
index 0000000..6434ad5
--- /dev/null
+++ b/bitbake/lib/progressbar/widgets.py
@@ -0,0 +1,355 @@
+# -*- coding: utf-8 -*-
+#
+# progressbar - Text progress bar library for Python.
+# Copyright (c) 2005 Nilton Volpato
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2.1 of the License, or (at your option) any later version.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+
+"""Default ProgressBar widgets."""
+
+from __future__ import division
+
+import datetime
+import math
+
+try:
+ from abc import ABCMeta, abstractmethod
+except ImportError:
+ AbstractWidget = object
+ abstractmethod = lambda fn: fn
+else:
+ AbstractWidget = ABCMeta('AbstractWidget', (object,), {})
+
+
+def format_updatable(updatable, pbar):
+ if hasattr(updatable, 'update'): return updatable.update(pbar)
+ else: return updatable
+
+
+class Widget(AbstractWidget):
+ """The base class for all widgets.
+
+ The ProgressBar will call the widget's update value when the widget should
+ be updated. The widget's size may change between calls, but the widget may
+ display incorrectly if the size changes drastically and repeatedly.
+
+ The boolean TIME_SENSITIVE informs the ProgressBar that it should be
+ updated more often because it is time sensitive.
+ """
+
+ TIME_SENSITIVE = False
+ __slots__ = ()
+
+ @abstractmethod
+ def update(self, pbar):
+ """Updates the widget.
+
+ pbar - a reference to the calling ProgressBar
+ """
+
+
+class WidgetHFill(Widget):
+ """The base class for all variable width widgets.
+
+ This widget is much like the \\hfill command in TeX, it will expand to
+ fill the line. You can use more than one in the same line, and they will
+ all have the same width, and together will fill the line.
+ """
+
+ @abstractmethod
+ def update(self, pbar, width):
+ """Updates the widget providing the total width the widget must fill.
+
+ pbar - a reference to the calling ProgressBar
+ width - The total width the widget must fill
+ """
+
+
+class Timer(Widget):
+ """Widget which displays the elapsed seconds."""
+
+ __slots__ = ('format_string',)
+ TIME_SENSITIVE = True
+
+ def __init__(self, format='Elapsed Time: %s'):
+ self.format_string = format
+
+ @staticmethod
+ def format_time(seconds):
+ """Formats time as the string "HH:MM:SS"."""
+
+ return str(datetime.timedelta(seconds=int(seconds)))
+
+
+ def update(self, pbar):
+ """Updates the widget to show the elapsed time."""
+
+ return self.format_string % self.format_time(pbar.seconds_elapsed)
+
+
+class ETA(Timer):
+ """Widget which attempts to estimate the time of arrival."""
+
+ TIME_SENSITIVE = True
+
+ def update(self, pbar):
+ """Updates the widget to show the ETA or total time when finished."""
+
+ if pbar.currval == 0:
+ return 'ETA: --:--:--'
+ elif pbar.finished:
+ return 'Time: %s' % self.format_time(pbar.seconds_elapsed)
+ else:
+ elapsed = pbar.seconds_elapsed
+ eta = elapsed * pbar.maxval / pbar.currval - elapsed
+ return 'ETA: %s' % self.format_time(eta)
+
+
+class AdaptiveETA(Timer):
+ """Widget which attempts to estimate the time of arrival.
+
+ Uses a weighted average of two estimates:
+ 1) ETA based on the total progress and time elapsed so far
+ 2) ETA based on the progress as per the last 10 update reports
+
+ The weight depends on the current progress so that to begin with the
+ total progress is used and at the end only the most recent progress is
+ used.
+ """
+
+ TIME_SENSITIVE = True
+ NUM_SAMPLES = 10
+
+ def _update_samples(self, currval, elapsed):
+ sample = (currval, elapsed)
+ if not hasattr(self, 'samples'):
+ self.samples = [sample] * (self.NUM_SAMPLES + 1)
+ else:
+ self.samples.append(sample)
+ return self.samples.pop(0)
+
+ def _eta(self, maxval, currval, elapsed):
+ return elapsed * maxval / float(currval) - elapsed
+
+ def update(self, pbar):
+ """Updates the widget to show the ETA or total time when finished."""
+ if pbar.currval == 0:
+ return 'ETA: --:--:--'
+ elif pbar.finished:
+ return 'Time: %s' % self.format_time(pbar.seconds_elapsed)
+ else:
+ elapsed = pbar.seconds_elapsed
+ currval1, elapsed1 = self._update_samples(pbar.currval, elapsed)
+ eta = self._eta(pbar.maxval, pbar.currval, elapsed)
+ if pbar.currval > currval1:
+ etasamp = self._eta(pbar.maxval - currval1,
+ pbar.currval - currval1,
+ elapsed - elapsed1)
+ weight = (pbar.currval / float(pbar.maxval)) ** 0.5
+ eta = (1 - weight) * eta + weight * etasamp
+ return 'ETA: %s' % self.format_time(eta)
+
+
+class FileTransferSpeed(Widget):
+ """Widget for showing the transfer speed (useful for file transfers)."""
+
+ FORMAT = '%6.2f %s%s/s'
+ PREFIXES = ' kMGTPEZY'
+ __slots__ = ('unit',)
+
+ def __init__(self, unit='B'):
+ self.unit = unit
+
+ def update(self, pbar):
+ """Updates the widget with the current SI prefixed speed."""
+
+ if pbar.seconds_elapsed < 2e-6 or pbar.currval < 2e-6: # =~ 0
+ scaled = power = 0
+ else:
+ speed = pbar.currval / pbar.seconds_elapsed
+ power = int(math.log(speed, 1000))
+ scaled = speed / 1000.**power
+
+ return self.FORMAT % (scaled, self.PREFIXES[power], self.unit)
+
+
+class AnimatedMarker(Widget):
+ """An animated marker for the progress bar which defaults to appear as if
+ it were rotating.
+ """
+
+ __slots__ = ('markers', 'curmark')
+
+ def __init__(self, markers='|/-\\'):
+ self.markers = markers
+ self.curmark = -1
+
+ def update(self, pbar):
+ """Updates the widget to show the next marker or the first marker when
+ finished"""
+
+ if pbar.finished: return self.markers[0]
+
+ self.curmark = (self.curmark + 1) % len(self.markers)
+ return self.markers[self.curmark]
+
+# Alias for backwards compatibility
+RotatingMarker = AnimatedMarker
+
+
+class Counter(Widget):
+ """Displays the current count."""
+
+ __slots__ = ('format_string',)
+
+ def __init__(self, format='%d'):
+ self.format_string = format
+
+ def update(self, pbar):
+ return self.format_string % pbar.currval
+
+
+class Percentage(Widget):
+ """Displays the current percentage as a number with a percent sign."""
+
+ def update(self, pbar):
+ return '%3d%%' % pbar.percentage()
+
+
+class FormatLabel(Timer):
+ """Displays a formatted label."""
+
+ mapping = {
+ 'elapsed': ('seconds_elapsed', Timer.format_time),
+ 'finished': ('finished', None),
+ 'last_update': ('last_update_time', None),
+ 'max': ('maxval', None),
+ 'seconds': ('seconds_elapsed', None),
+ 'start': ('start_time', None),
+ 'value': ('currval', None)
+ }
+
+ __slots__ = ('format_string',)
+ def __init__(self, format):
+ self.format_string = format
+
+ def update(self, pbar):
+ context = {}
+ for name, (key, transform) in self.mapping.items():
+ try:
+ value = getattr(pbar, key)
+
+ if transform is None:
+ context[name] = value
+ else:
+ context[name] = transform(value)
+ except: pass
+
+ return self.format_string % context
+
+
+class SimpleProgress(Widget):
+ """Returns progress as a count of the total (e.g.: "5 of 47")."""
+
+ __slots__ = ('sep',)
+
+ def __init__(self, sep=' of '):
+ self.sep = sep
+
+ def update(self, pbar):
+ return '%d%s%d' % (pbar.currval, self.sep, pbar.maxval)
+
+
+class Bar(WidgetHFill):
+ """A progress bar which stretches to fill the line."""
+
+ __slots__ = ('marker', 'left', 'right', 'fill', 'fill_left')
+
+ def __init__(self, marker='#', left='|', right='|', fill=' ',
+ fill_left=True):
+ """Creates a customizable progress bar.
+
+ marker - string or updatable object to use as a marker
+ left - string or updatable object to use as a left border
+ right - string or updatable object to use as a right border
+ fill - character to use for the empty part of the progress bar
+ fill_left - whether to fill from the left or the right
+ """
+ self.marker = marker
+ self.left = left
+ self.right = right
+ self.fill = fill
+ self.fill_left = fill_left
+
+
+ def update(self, pbar, width):
+ """Updates the progress bar and its subcomponents."""
+
+ left, marked, right = (format_updatable(i, pbar) for i in
+ (self.left, self.marker, self.right))
+
+ width -= len(left) + len(right)
+ # Marked must *always* have length of 1
+ if pbar.maxval:
+ marked *= int(pbar.currval / pbar.maxval * width)
+ else:
+ marked = ''
+
+ if self.fill_left:
+ return '%s%s%s' % (left, marked.ljust(width, self.fill), right)
+ else:
+ return '%s%s%s' % (left, marked.rjust(width, self.fill), right)
+
+
+class ReverseBar(Bar):
+ """A bar which has a marker which bounces from side to side."""
+
+ def __init__(self, marker='#', left='|', right='|', fill=' ',
+ fill_left=False):
+ """Creates a customizable progress bar.
+
+ marker - string or updatable object to use as a marker
+ left - string or updatable object to use as a left border
+ right - string or updatable object to use as a right border
+ fill - character to use for the empty part of the progress bar
+ fill_left - whether to fill from the left or the right
+ """
+ self.marker = marker
+ self.left = left
+ self.right = right
+ self.fill = fill
+ self.fill_left = fill_left
+
+
+class BouncingBar(Bar):
+ def update(self, pbar, width):
+ """Updates the progress bar and its subcomponents."""
+
+ left, marker, right = (format_updatable(i, pbar) for i in
+ (self.left, self.marker, self.right))
+
+ width -= len(left) + len(right)
+
+ if pbar.finished: return '%s%s%s' % (left, width * marker, right)
+
+ position = int(pbar.currval % (width * 2 - 1))
+ if position > width: position = width * 2 - position
+ lpad = self.fill * (position - 1)
+ rpad = self.fill * (width - len(marker) - len(lpad))
+
+ # Swap if we want to bounce the other way
+ if not self.fill_left: rpad, lpad = lpad, rpad
+
+ return '%s%s%s%s%s' % (left, lpad, marker, rpad, right)
--
2.5.5
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [RFC PATCH 2/6] lib: implement basic task progress support
2016-05-10 23:24 [RFC PATCH 0/6] Task progress reporting Paul Eggleton
2016-05-10 23:24 ` [RFC PATCH 1/6] knotty: import latest python-progressbar Paul Eggleton
@ 2016-05-10 23:24 ` Paul Eggleton
2016-05-10 23:24 ` [RFC PATCH 3/6] fetch2: implement " Paul Eggleton
` (3 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Paul Eggleton @ 2016-05-10 23:24 UTC (permalink / raw)
To: openembedded-core
For long-running tasks where we have some output from the task that
gives us some idea of the progress of the task (such as a percentage
complete), provide the means to scrape the output for that progress
information and show it to the user in the default knotty terminal
output in the form of a progress bar. This is implemented using a new
TaskProgress event as well as some code we can insert to do output
scanning/filtering.
Any task can fire TaskProgress events; however, if you have a shell task
whose output you wish to scan for progress information, you just need to
set the "progress" varflag on the task. This can be set to:
* "percent" to just look for a number followed by a % sign
* "percent:<regex>" to specify your own regex matching a percentage
value (must have a single group which matches the percentage number)
* "outof:<regex>" to look for the specified regex matching x out of y
items completed (must have two groups - first group needs to be x,
second y).
We can potentially extend this in future but this should be a good
start.
Line changes such as the ones you get in git's output as it progresses
don't make it to the log files, you only get the final state of the line
so the logs aren't filled with progress information that's useless after
the fact.
Part of the implementation for [YOCTO #5383].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
bitbake/lib/bb/build.py | 112 +++++++++++++++++++++++++++++++++
bitbake/lib/bb/ui/knotty.py | 74 +++++++++++++++++++---
bitbake/lib/bb/ui/uihelper.py | 7 ++-
bitbake/lib/progressbar/progressbar.py | 16 +++--
bitbake/lib/progressbar/widgets.py | 36 +++++++++++
5 files changed, 231 insertions(+), 14 deletions(-)
diff --git a/bitbake/lib/bb/build.py b/bitbake/lib/bb/build.py
index a5b99ed..0b28492 100644
--- a/bitbake/lib/bb/build.py
+++ b/bitbake/lib/bb/build.py
@@ -35,6 +35,7 @@ import stat
import bb
import bb.msg
import bb.process
+import re
from contextlib import nested
from bb import event, utils
@@ -138,6 +139,13 @@ class TaskInvalid(TaskBase):
super(TaskInvalid, self).__init__(task, None, metadata)
self._message = "No such task '%s'" % task
+class TaskProgress(event.Event):
+ def __init__(self, task, progress, rate=None):
+ self.task = task
+ self.progress = progress
+ self.rate = rate
+ event.Event.__init__(self)
+
class LogTee(object):
def __init__(self, logger, outfile):
@@ -161,6 +169,96 @@ class LogTee(object):
def flush(self):
self.outfile.flush()
+
+class ProgressHandler(object):
+ """
+ Base class that can pretend to be a file object well enough to be
+ used to build objects to intercept console output and determine the
+ progress of some operation.
+ """
+ def __init__(self, d, outfile=None):
+ self._progress = 0
+ self._data = d
+ self._lastevent = 0
+ if outfile:
+ self._outfile = outfile
+ else:
+ self._outfile = sys.stdout
+
+ def write(self, string):
+ self._outfile.write(string)
+
+ def flush(self):
+ self._outfile.flush()
+
+ def update(self, progress, rate=None):
+ ts = time.time()
+ if progress > 100:
+ progress = 100
+ if progress != self._progress or self._lastevent + 1 < ts:
+ bb.event.fire(bb.build.TaskProgress(0, progress, rate), self._data)
+ self._lastevent = ts
+ self._progress = progress
+
+class BasicProgressHandler(ProgressHandler):
+ def __init__(self, d, regex=r'(\d+)%', outfile=None):
+ super(BasicProgressHandler, self).__init__(d, outfile)
+ self._regex = re.compile(regex)
+ # Send an initial progress event so the bar gets shown
+ bb.event.fire(bb.build.TaskProgress(0, 0), self._data)
+
+ def write(self, string):
+ percs = self._regex.findall(string)
+ if percs:
+ progress = int(percs[-1])
+ self.update(progress)
+ super(BasicProgressHandler, self).write(string)
+
+class OutOfProgressHandler(ProgressHandler):
+ def __init__(self, d, regex, outfile=None):
+ super(OutOfProgressHandler, self).__init__(d, outfile)
+ self._regex = re.compile(regex)
+ # Send an initial progress event so the bar gets shown
+ bb.event.fire(bb.build.TaskProgress(0, 0), self._data)
+
+ def write(self, string):
+ nums = self._regex.findall(string)
+ if nums:
+ progress = (float(nums[-1][0]) / float(nums[-1][1])) * 100
+ self.update(progress)
+ super(OutOfProgressHandler, self).write(string)
+
+class MultiStageProgressReporter(object):
+ """
+ Class which allows reporting progress without the caller
+ having to know where they are in the overall sequence. Useful
+ for tasks made up of python code spread across multiple
+ classes / functions.
+ """
+ def __init__(self, d, stage_weights):
+ self._data = d
+ self._stage_weights = stage_weights
+ self._stage = 0
+ self._base_progress = 0
+ # Send an initial progress event so the bar gets shown
+ bb.event.fire(bb.build.TaskProgress(0, 0), self._data)
+
+ def next_stage(self):
+ self._stage += 1
+ if self._stage < len(self._stage_weights):
+ self._base_progress = sum(self._stage_weights[:self._stage]) * 100
+ else:
+ bb.warn('ProgressReporter: current stage beyond declared number of stages')
+ self._base_progress = 100
+ bb.event.fire(bb.build.TaskProgress(0, self._base_progress), self._data)
+
+ def update(self, stageprogress):
+ progress = self._base_progress + (stageprogress * self._stage_weights[self._stage])
+ if progress > 100:
+ progress = 100
+ bb.event.fire(bb.build.TaskProgress(0, progress), self._data)
+
+
#
# pythonexception allows the python exceptions generated to be raised
# as the real exceptions (not FuncFailed) and without a backtrace at the
@@ -341,6 +439,20 @@ exit $ret
else:
logfile = sys.stdout
+ progress = d.getVarFlag(func, 'progress', True)
+ if progress:
+ if progress == 'percent':
+ # Use default regex
+ logfile = BasicProgressHandler(d, outfile=logfile)
+ elif progress.startswith('percent:'):
+ # Use specified regex
+ logfile = BasicProgressHandler(d, regex=progress.split(':', 1)[1], outfile=logfile)
+ elif progress.startswith('outof:'):
+ # Use specified regex
+ logfile = OutOfProgressHandler(d, regex=progress.split(':', 1)[1], outfile=logfile)
+ else:
+ bb.warn('%s: invalid task progress varflag value "%s", ignoring' % (func, progress))
+
def readfifo(data):
lines = data.split('\0')
for line in lines:
diff --git a/bitbake/lib/bb/ui/knotty.py b/bitbake/lib/bb/ui/knotty.py
index 010b06d..9d962b0 100644
--- a/bitbake/lib/bb/ui/knotty.py
+++ b/bitbake/lib/bb/ui/knotty.py
@@ -40,10 +40,13 @@ logger = logging.getLogger("BitBake")
interactive = sys.stdout.isatty()
class BBProgress(progressbar.ProgressBar):
- def __init__(self, msg, maxval):
+ def __init__(self, msg, maxval, widgets=None):
self.msg = msg
- widgets = [progressbar.Percentage(), ' ', progressbar.Bar(), ' ',
- progressbar.ETA()]
+ self.extrapos = -1
+ if not widgets:
+ widgets = [progressbar.Percentage(), ' ', progressbar.Bar(), ' ',
+ progressbar.ETA()]
+ self.extrapos = 4
try:
self._resize_default = signal.getsignal(signal.SIGWINCH)
@@ -55,11 +58,31 @@ class BBProgress(progressbar.ProgressBar):
progressbar.ProgressBar._handle_resize(self, signum, frame)
if self._resize_default:
self._resize_default(signum, frame)
+
def finish(self):
progressbar.ProgressBar.finish(self)
if self._resize_default:
signal.signal(signal.SIGWINCH, self._resize_default)
+ def setmessage(self, msg):
+ self.msg = msg
+ self.widgets[0] = msg
+
+ def setextra(self, extra):
+ if extra:
+ extrastr = str(extra)
+ if extrastr[0] != ' ':
+ extrastr = ' ' + extrastr
+ if extrastr[-1] != ' ':
+ extrastr += ' '
+ else:
+ extrastr = ' '
+ self.widgets[self.extrapos] = extrastr
+
+ def _need_update(self):
+ # We always want the bar to print when update() is called
+ return True
+
class NonInteractiveProgress(object):
fobj = sys.stdout
@@ -194,15 +217,31 @@ class TerminalFilter(object):
activetasks = self.helper.running_tasks
failedtasks = self.helper.failed_tasks
runningpids = self.helper.running_pids
- if self.footer_present and (self.lastcount == self.helper.tasknumber_current) and (self.lastpids == runningpids):
+ if self.footer_present and not self.helper.needUpdate:
return
+ self.helper.needUpdate = False
if self.footer_present:
self.clearFooter()
if (not self.helper.tasknumber_total or self.helper.tasknumber_current == self.helper.tasknumber_total) and not len(activetasks):
return
tasks = []
for t in runningpids:
- tasks.append("%s (pid %s)" % (activetasks[t]["title"], t))
+ progress = activetasks[t].get("progress", None)
+ if progress is not None:
+ pbar = activetasks[t].get("progressbar", None)
+ rate = activetasks[t].get("rate", None)
+ start_time = activetasks[t].get("starttime", None)
+ if not pbar or pbar.bouncing != (progress < 0):
+ if progress < 0:
+ pbar = BBProgress("0: %s (pid %s) " % (activetasks[t]["title"], t), 100, widgets=[progressbar.BouncingSlider()])
+ pbar.bouncing = True
+ else:
+ pbar = BBProgress("0: %s (pid %s) " % (activetasks[t]["title"], t), 100)
+ pbar.bouncing = False
+ activetasks[t]["progressbar"] = pbar
+ tasks.append((pbar, progress, rate, start_time))
+ else:
+ tasks.append("%s (pid %s)" % (activetasks[t]["title"], t))
if self.main.shutdown:
content = "Waiting for %s running tasks to finish:" % len(activetasks)
@@ -213,8 +252,23 @@ class TerminalFilter(object):
print(content)
lines = 1 + int(len(content) / (self.columns + 1))
for tasknum, task in enumerate(tasks[:(self.rows - 2)]):
- content = "%s: %s" % (tasknum, task)
- print(content)
+ if isinstance(task, tuple):
+ pbar, progress, rate, start_time = task
+ if not pbar.start_time:
+ pbar.start(False)
+ if start_time:
+ pbar.start_time = start_time
+ pbar.setmessage('%s:%s' % (tasknum, pbar.msg.split(':', 1)[1]))
+ if progress > -1:
+ pbar.setextra(rate)
+ output = pbar.update(progress)
+ else:
+ output = pbar.update(1)
+ if not output or (len(output) <= pbar.term_width):
+ print('')
+ else:
+ content = "%s: %s" % (tasknum, task)
+ print(content)
lines = lines + 1 + int(len(content) / (self.columns + 1))
self.footer_present = lines
self.lastpids = runningpids[:]
@@ -248,7 +302,8 @@ _evt_list = [ "bb.runqueue.runQueueExitWait", "bb.event.LogExecTTY", "logging.Lo
"bb.command.CommandExit", "bb.command.CommandCompleted", "bb.cooker.CookerExit",
"bb.event.MultipleProviders", "bb.event.NoProvider", "bb.runqueue.sceneQueueTaskStarted",
"bb.runqueue.runQueueTaskStarted", "bb.runqueue.runQueueTaskFailed", "bb.runqueue.sceneQueueTaskFailed",
- "bb.event.BuildBase", "bb.build.TaskStarted", "bb.build.TaskSucceeded", "bb.build.TaskFailedSilent"]
+ "bb.event.BuildBase", "bb.build.TaskStarted", "bb.build.TaskSucceeded", "bb.build.TaskFailedSilent",
+ "bb.build.TaskProgress"]
def main(server, eventHandler, params, tf = TerminalFilter):
@@ -527,7 +582,8 @@ def main(server, eventHandler, params, tf = TerminalFilter):
bb.event.OperationStarted,
bb.event.OperationCompleted,
bb.event.OperationProgress,
- bb.event.DiskFull)):
+ bb.event.DiskFull,
+ bb.build.TaskProgress)):
continue
logger.error("Unknown event: %s", event)
diff --git a/bitbake/lib/bb/ui/uihelper.py b/bitbake/lib/bb/ui/uihelper.py
index db70b76..1915e47 100644
--- a/bitbake/lib/bb/ui/uihelper.py
+++ b/bitbake/lib/bb/ui/uihelper.py
@@ -18,6 +18,7 @@
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import bb.build
+import time
class BBUIHelper:
def __init__(self):
@@ -31,7 +32,7 @@ class BBUIHelper:
def eventHandler(self, event):
if isinstance(event, bb.build.TaskStarted):
- self.running_tasks[event.pid] = { 'title' : "%s %s" % (event._package, event._task) }
+ self.running_tasks[event.pid] = { 'title' : "%s %s" % (event._package, event._task), 'starttime' : time.time() }
self.running_pids.append(event.pid)
self.needUpdate = True
if isinstance(event, bb.build.TaskSucceeded):
@@ -52,6 +53,10 @@ class BBUIHelper:
self.tasknumber_current = event.stats.completed + event.stats.active + event.stats.failed + 1
self.tasknumber_total = event.stats.total
self.needUpdate = True
+ if isinstance(event, bb.build.TaskProgress):
+ self.running_tasks[event.pid]['progress'] = event.progress
+ self.running_tasks[event.pid]['rate'] = event.rate
+ self.needUpdate = True
def getTasks(self):
self.needUpdate = False
diff --git a/bitbake/lib/progressbar/progressbar.py b/bitbake/lib/progressbar/progressbar.py
index 0b9dcf7..2873ad6 100644
--- a/bitbake/lib/progressbar/progressbar.py
+++ b/bitbake/lib/progressbar/progressbar.py
@@ -3,6 +3,8 @@
# progressbar - Text progress bar library for Python.
# Copyright (c) 2005 Nilton Volpato
#
+# (With some small changes after importing into BitBake)
+#
# This library is free software; you can redistribute it and/or
# modify it under the terms of the GNU Lesser General Public
# License as published by the Free Software Foundation; either
@@ -261,12 +263,14 @@ class ProgressBar(object):
now = time.time()
self.seconds_elapsed = now - self.start_time
self.next_update = self.currval + self.update_interval
- self.fd.write(self._format_line() + '\r')
+ output = self._format_line()
+ self.fd.write(output + '\r')
self.fd.flush()
self.last_update_time = now
+ return output
- def start(self):
+ def start(self, update=True):
"""Starts measuring time, and prints the bar at 0%.
It returns self so you can use it like this:
@@ -289,8 +293,12 @@ class ProgressBar(object):
self.update_interval = self.maxval / self.num_intervals
- self.start_time = self.last_update_time = time.time()
- self.update(0)
+ self.start_time = time.time()
+ if update:
+ self.last_update_time = self.start_time
+ self.update(0)
+ else:
+ self.last_update_time = 0
return self
diff --git a/bitbake/lib/progressbar/widgets.py b/bitbake/lib/progressbar/widgets.py
index 6434ad5..77285ca 100644
--- a/bitbake/lib/progressbar/widgets.py
+++ b/bitbake/lib/progressbar/widgets.py
@@ -353,3 +353,39 @@ class BouncingBar(Bar):
if not self.fill_left: rpad, lpad = lpad, rpad
return '%s%s%s%s%s' % (left, lpad, marker, rpad, right)
+
+
+class BouncingSlider(Bar):
+ """
+ A slider that bounces back and forth in response to update() calls
+ without reference to the actual value. Based on a combination of
+ BouncingBar from a newer version of this module and RotatingMarker.
+ """
+ def __init__(self, marker='<=>'):
+ self.curmark = -1
+ self.forward = True
+ Bar.__init__(self, marker=marker)
+ def update(self, pbar, width):
+ left, marker, right = (format_updatable(i, pbar) for i in
+ (self.left, self.marker, self.right))
+
+ width -= len(left) + len(right)
+ if width < 0:
+ return ''
+
+ if pbar.finished: return '%s%s%s' % (left, width * '=', right)
+
+ self.curmark = self.curmark + 1
+ position = int(self.curmark % (width * 2 - 1))
+ if position + len(marker) > width:
+ self.forward = not self.forward
+ self.curmark = 1
+ position = 1
+ lpad = ' ' * (position - 1)
+ rpad = ' ' * (width - len(marker) - len(lpad))
+
+ if not self.forward:
+ temp = lpad
+ lpad = rpad
+ rpad = temp
+ return '%s%s%s%s%s' % (left, lpad, marker, rpad, right)
--
2.5.5
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [RFC PATCH 3/6] fetch2: implement progress support
2016-05-10 23:24 [RFC PATCH 0/6] Task progress reporting Paul Eggleton
2016-05-10 23:24 ` [RFC PATCH 1/6] knotty: import latest python-progressbar Paul Eggleton
2016-05-10 23:24 ` [RFC PATCH 2/6] lib: implement basic task progress support Paul Eggleton
@ 2016-05-10 23:24 ` Paul Eggleton
2016-05-10 23:24 ` [RFC PATCH 4/6] bitbake.conf: whitelist progress varflag Paul Eggleton
` (2 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Paul Eggleton @ 2016-05-10 23:24 UTC (permalink / raw)
To: openembedded-core
Implement progress reporting support specifically for the fetchers. For
fetch tasks we don't necessarily know which fetcher will be used (we
might initially be fetching a git:// URI, but if we instead download a
mirror tarball we may fetch that over http using wget). These programs
also have different abilities as far as reporting progress goes (e.g.
wget gives us percentage complete and rate, git gives this some of the
time depending on what stage it's at). Additionally we filter out the
progress output before it makes it to the logs, in order to prevent the
logs filling up with junk.
At the moment this is only implemented for the wget and git fetchers
since they are the most commonly used (and svn doesn't seem to support
any kind of progress output, at least not without doing a relatively
expensive remote file listing first).
Part of the implementation for [YOCTO #5383].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
bitbake/lib/bb/build.py | 31 ++++++++++++++++++++++++++
bitbake/lib/bb/fetch2/__init__.py | 4 ++--
bitbake/lib/bb/fetch2/git.py | 47 +++++++++++++++++++++++++++++++++++++--
bitbake/lib/bb/fetch2/wget.py | 26 +++++++++++++++++++++-
4 files changed, 103 insertions(+), 5 deletions(-)
diff --git a/bitbake/lib/bb/build.py b/bitbake/lib/bb/build.py
index 0b28492..8cb80cf 100644
--- a/bitbake/lib/bb/build.py
+++ b/bitbake/lib/bb/build.py
@@ -200,6 +200,37 @@ class ProgressHandler(object):
self._lastevent = ts
self._progress = progress
+class LineFilterProgressHandler(ProgressHandler):
+ """
+ A ProgressHandler variant that provides the ability to filter out
+ the lines if they contain progress information. Additionally, it
+ filters out anything before the last line feed on a line. This can
+ be used to keep the logs clean of output that we've only enabled for
+ getting progress, assuming that that can be done on a per-line
+ basis.
+ """
+ def __init__(self, d, outfile=None):
+ self._linebuffer = ''
+ super(LineFilterProgressHandler, self).__init__(d, outfile)
+
+ def write(self, string):
+ self._linebuffer += string
+ while True:
+ breakpos = self._linebuffer.find('\n') + 1
+ if breakpos == 0:
+ break
+ line = self._linebuffer[:breakpos]
+ self._linebuffer = self._linebuffer[breakpos:]
+ # Drop any line feeds and anything that precedes them
+ lbreakpos = line.rfind('\r') + 1
+ if lbreakpos:
+ line = line[lbreakpos:]
+ if self.writeline(line):
+ super(LineFilterProgressHandler, self).write(line)
+
+ def writeline(self, line):
+ return True
+
class BasicProgressHandler(ProgressHandler):
def __init__(self, d, regex=r'(\d+)%', outfile=None):
super(BasicProgressHandler, self).__init__(d, outfile)
diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py
index 6ef0c6f..394c31c 100644
--- a/bitbake/lib/bb/fetch2/__init__.py
+++ b/bitbake/lib/bb/fetch2/__init__.py
@@ -784,7 +784,7 @@ def localpath(url, d):
fetcher = bb.fetch2.Fetch([url], d)
return fetcher.localpath(url)
-def runfetchcmd(cmd, d, quiet=False, cleanup=None):
+def runfetchcmd(cmd, d, quiet=False, cleanup=None, log=None):
"""
Run cmd returning the command output
Raise an error if interrupted or cmd fails
@@ -825,7 +825,7 @@ def runfetchcmd(cmd, d, quiet=False, cleanup=None):
error_message = ""
try:
- (output, errors) = bb.process.run(cmd, shell=True, stderr=subprocess.PIPE)
+ (output, errors) = bb.process.run(cmd, log=log, shell=True, stderr=subprocess.PIPE)
success = True
except bb.process.NotFoundError as e:
error_message = "Fetch command %s" % (e.command)
diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
index 526668b..c0160fd 100644
--- a/bitbake/lib/bb/fetch2/git.py
+++ b/bitbake/lib/bb/fetch2/git.py
@@ -71,11 +71,53 @@ import os
import re
import bb
import errno
+import bb.build
from bb import data
from bb.fetch2 import FetchMethod
from bb.fetch2 import runfetchcmd
from bb.fetch2 import logger
+
+class GitProgressHandler(bb.build.LineFilterProgressHandler):
+ """Extract progress information from git output"""
+ def __init__(self, d):
+ self._buffer = ''
+ self._count = 0
+ super(GitProgressHandler, self).__init__(d)
+ # Send an initial progress event so the bar gets shown
+ bb.event.fire(bb.build.TaskProgress(0, -1), self._data)
+
+ def write(self, string):
+ self._buffer += string
+ stages = ['Counting objects', 'Compressing objects', 'Receiving objects', 'Resolving deltas']
+ stage_weights = [0.2, 0.05, 0.5, 0.25]
+ stagenum = 0
+ for i, stage in reversed(list(enumerate(stages))):
+ if stage in self._buffer:
+ stagenum = i
+ self._buffer = ''
+ break
+ self._status = stages[stagenum]
+ percs = re.findall(r'(\d+)%', string)
+ if percs:
+ progress = int(round((int(percs[-1]) * stage_weights[stagenum]) + (sum(stage_weights[:stagenum]) * 100)))
+ rates = re.findall(r'([\d.]+ [a-zA-Z]*/s+)', string)
+ if rates:
+ rate = rates[-1]
+ else:
+ rate = None
+ self.update(progress, rate)
+ else:
+ if stagenum == 0:
+ percs = re.findall(r': (\d+)', string)
+ if percs:
+ count = int(percs[-1])
+ if count > self._count:
+ self._count = count
+ bb.event.fire(bb.build.TaskProgress(0, -count), self._data)
+ super(GitProgressHandler, self).write(string)
+
+
class Git(FetchMethod):
"""Class to fetch a module or modules from git repositories"""
def init(self, d):
@@ -196,10 +238,11 @@ class Git(FetchMethod):
# We do this since git will use a "-l" option automatically for local urls where possible
if repourl.startswith("file://"):
repourl = repourl[7:]
- clone_cmd = "%s clone --bare --mirror %s %s" % (ud.basecmd, repourl, ud.clonedir)
+ clone_cmd = "LANG=C %s clone --bare --mirror %s %s --progress" % (ud.basecmd, repourl, ud.clonedir)
if ud.proto.lower() != 'file':
bb.fetch2.check_network_access(d, clone_cmd)
- runfetchcmd(clone_cmd, d)
+ progresshandler = GitProgressHandler(d)
+ runfetchcmd(clone_cmd, d, log=progresshandler)
os.chdir(ud.clonedir)
# Update the checkout if needed
diff --git a/bitbake/lib/bb/fetch2/wget.py b/bitbake/lib/bb/fetch2/wget.py
index 8bc9e93..c07c3c8 100644
--- a/bitbake/lib/bb/fetch2/wget.py
+++ b/bitbake/lib/bb/fetch2/wget.py
@@ -31,6 +31,7 @@ import subprocess
import os
import logging
import bb
+import bb.build
import urllib
from bb import data
from bb.fetch2 import FetchMethod
@@ -41,6 +42,27 @@ from bb.utils import export_proxies
from bs4 import BeautifulSoup
from bs4 import SoupStrainer
+class WgetProgressHandler(bb.build.LineFilterProgressHandler):
+ """
+ Extract progress information from wget output.
+ Note: relies on --progress=dot --show-progress being specified on the
+ wget command line.
+ """
+ def __init__(self, d):
+ super(WgetProgressHandler, self).__init__(d)
+ # Send an initial progress event so the bar gets shown
+ bb.event.fire(bb.build.TaskProgress(0, 0), self._data)
+
+ def writeline(self, line):
+ percs = re.findall(r'(\d+)%\s+([\d.]+[A-Z])', line)
+ if percs:
+ progress = int(percs[-1][0])
+ rate = percs[-1][1] + '/s'
+ self.update(progress, rate)
+ return False
+ return True
+
+
class Wget(FetchMethod):
"""Class to fetch urls via 'wget'"""
def supports(self, ud, d):
@@ -70,9 +92,11 @@ class Wget(FetchMethod):
def _runwget(self, ud, d, command, quiet):
+ progresshandler = WgetProgressHandler(d)
+
logger.debug(2, "Fetching %s using command '%s'" % (ud.url, command))
bb.fetch2.check_network_access(d, command)
- runfetchcmd(command, d, quiet)
+ runfetchcmd(command + ' --progress=dot --show-progress', d, quiet, log=progresshandler)
def download(self, ud, d):
"""Fetch urls"""
--
2.5.5
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [RFC PATCH 4/6] bitbake.conf: whitelist progress varflag
2016-05-10 23:24 [RFC PATCH 0/6] Task progress reporting Paul Eggleton
` (2 preceding siblings ...)
2016-05-10 23:24 ` [RFC PATCH 3/6] fetch2: implement " Paul Eggleton
@ 2016-05-10 23:24 ` Paul Eggleton
2016-05-10 23:24 ` [RFC PATCH 5/6] classes/cmake: enable progress for do_compile Paul Eggleton
2016-05-10 23:24 ` [RFC PATCH 6/6] classes/image: implement progress support for do_rootfs Paul Eggleton
5 siblings, 0 replies; 7+ messages in thread
From: Paul Eggleton @ 2016-05-10 23:24 UTC (permalink / raw)
To: openembedded-core
We don't want the value of this varflag in any signatures since it's
only there for the purpose of aiding display of task execution.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/conf/bitbake.conf | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 1c4175f..14fee7f 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -814,7 +814,8 @@ BB_SIGNATURE_EXCLUDE_FLAGS ?= "doc deps depends \
lockfiles type vardepsexclude vardeps vardepvalue vardepvalueexclude \
file-checksums python func task export unexport noexec nostamp dirs cleandirs \
sstate-lockfile-shared prefuncs postfuncs export_func deptask rdeptask \
- recrdeptask nodeprrecs stamp-extra-info sstate-outputdirs filename lineno"
+ recrdeptask nodeprrecs stamp-extra-info sstate-outputdirs filename lineno \
+ progress"
MLPREFIX ??= ""
MULTILIB_VARIANTS ??= ""
--
2.5.5
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [RFC PATCH 5/6] classes/cmake: enable progress for do_compile
2016-05-10 23:24 [RFC PATCH 0/6] Task progress reporting Paul Eggleton
` (3 preceding siblings ...)
2016-05-10 23:24 ` [RFC PATCH 4/6] bitbake.conf: whitelist progress varflag Paul Eggleton
@ 2016-05-10 23:24 ` Paul Eggleton
2016-05-10 23:24 ` [RFC PATCH 6/6] classes/image: implement progress support for do_rootfs Paul Eggleton
5 siblings, 0 replies; 7+ messages in thread
From: Paul Eggleton @ 2016-05-10 23:24 UTC (permalink / raw)
To: openembedded-core
cmake outputs percentage complete as part of its compilation process, so
we can enable BitBake's new progress scanning for do_compile here.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/classes/cmake.bbclass | 1 +
meta/recipes-devtools/cmake/cmake-native_3.4.3.bb | 2 ++
2 files changed, 3 insertions(+)
diff --git a/meta/classes/cmake.bbclass b/meta/classes/cmake.bbclass
index 02f313a..f06600f 100644
--- a/meta/classes/cmake.bbclass
+++ b/meta/classes/cmake.bbclass
@@ -124,6 +124,7 @@ cmake_do_configure() {
-Wno-dev
}
+do_compile[progress] = "percent"
cmake_do_compile() {
cd ${B}
base_do_compile
diff --git a/meta/recipes-devtools/cmake/cmake-native_3.4.3.bb b/meta/recipes-devtools/cmake/cmake-native_3.4.3.bb
index cb4e749..33930fb 100644
--- a/meta/recipes-devtools/cmake/cmake-native_3.4.3.bb
+++ b/meta/recipes-devtools/cmake/cmake-native_3.4.3.bb
@@ -14,3 +14,5 @@ CMAKE_EXTRACONF = "\
-DENABLE_ACL=0 -DHAVE_ACL_LIBACL_H=0 \
-DHAVE_SYS_ACL_H=0 \
"
+
+do_compile[progress] = "percent"
--
2.5.5
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [RFC PATCH 6/6] classes/image: implement progress support for do_rootfs
2016-05-10 23:24 [RFC PATCH 0/6] Task progress reporting Paul Eggleton
` (4 preceding siblings ...)
2016-05-10 23:24 ` [RFC PATCH 5/6] classes/cmake: enable progress for do_compile Paul Eggleton
@ 2016-05-10 23:24 ` Paul Eggleton
5 siblings, 0 replies; 7+ messages in thread
From: Paul Eggleton @ 2016-05-10 23:24 UTC (permalink / raw)
To: openembedded-core
Use the new task progress functionality to report progress during
do_rootfs. I did some timings of the various parts of the do_rootfs code
and set some weightings accordingly. This is a little coarse and ideally
we would have some progress within the installation section, but it's
better than nothing.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/classes/image.bbclass | 9 +++-
meta/lib/oe/rootfs.py | 101 +++++++++++++++++++++++++++++++++++++++------
2 files changed, 95 insertions(+), 15 deletions(-)
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 4542e95..efa0629 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -198,6 +198,9 @@ fakeroot python do_rootfs () {
from oe.rootfs import create_rootfs
from oe.manifest import create_manifest
+ stage_weights = [0.01, 0.1, 0.18, 0.1, 0.03, 0.5, 0, 0.01, 0.02, 0.02, 0.02, 0.01, 0]
+ progress_reporter = bb.build.MultiStageProgressReporter(d, stage_weights)
+
# Handle package exclusions
excl_pkgs = d.getVar("PACKAGE_EXCLUDE", True).split()
inst_pkgs = d.getVar("PACKAGE_INSTALL", True).split()
@@ -230,8 +233,10 @@ fakeroot python do_rootfs () {
# Generate the initial manifest
create_manifest(d)
- # Generate rootfs
- create_rootfs(d)
+ progress_reporter.next_stage()
+
+ # generate rootfs
+ create_rootfs(d, progress_reporter=progress_reporter)
}
do_rootfs[dirs] = "${TOPDIR}"
do_rootfs[cleandirs] += "${S}"
diff --git a/meta/lib/oe/rootfs.py b/meta/lib/oe/rootfs.py
index 0546c1e..d143227 100644
--- a/meta/lib/oe/rootfs.py
+++ b/meta/lib/oe/rootfs.py
@@ -16,11 +16,12 @@ class Rootfs(object):
"""
__metaclass__ = ABCMeta
- def __init__(self, d):
+ def __init__(self, d, progress_reporter=None):
self.d = d
self.pm = None
self.image_rootfs = self.d.getVar('IMAGE_ROOTFS', True)
self.deploy_dir_image = self.d.getVar('DEPLOY_DIR_IMAGE', True)
+ self.progress_reporter = progress_reporter
self.install_order = Manifest.INSTALL_ORDER
@@ -195,6 +196,9 @@ class Rootfs(object):
execute_pre_post_process(self.d, pre_process_cmds)
+ if self.progress_reporter:
+ self.progress_reporter.next_stage()
+
# call the package manager dependent create method
self._create()
@@ -209,6 +213,9 @@ class Rootfs(object):
execute_pre_post_process(self.d, post_process_cmds)
+ if self.progress_reporter:
+ self.progress_reporter.next_stage()
+
if bb.utils.contains("IMAGE_FEATURES", "read-only-rootfs",
True, False, self.d):
delayed_postinsts = self._get_delayed_postinsts()
@@ -222,6 +229,9 @@ class Rootfs(object):
self._uninstall_unneeded()
+ if self.progress_reporter:
+ self.progress_reporter.next_stage()
+
self._insert_feed_uris()
self._run_ldconfig()
@@ -232,6 +242,10 @@ class Rootfs(object):
self._cleanup()
self._log_check()
+ if self.progress_reporter:
+ self.progress_reporter.next_stage()
+
+
def _uninstall_unneeded(self):
# Remove unneeded init script symlinks
delayed_postinsts = self._get_delayed_postinsts()
@@ -363,8 +377,8 @@ class Rootfs(object):
class RpmRootfs(Rootfs):
- def __init__(self, d, manifest_dir):
- super(RpmRootfs, self).__init__(d)
+ def __init__(self, d, manifest_dir, progress_reporter=None):
+ super(RpmRootfs, self).__init__(d, progress_reporter)
self.log_check_regex = '(unpacking of archive failed|Cannot find package'\
'|exit 1|ERROR: |Error: |Error |ERROR '\
'|Failed |Failed: |Failed$|Failed\(\d+\):)'
@@ -422,11 +436,17 @@ class RpmRootfs(Rootfs):
execute_pre_post_process(self.d, rpm_pre_process_cmds)
+ if self.progress_reporter:
+ self.progress_reporter.next_stage()
+
self.pm.dump_all_available_pkgs()
if self.inc_rpm_image_gen == "1":
self._create_incremental(pkgs_to_install)
+ if self.progress_reporter:
+ self.progress_reporter.next_stage()
+
self.pm.update()
pkgs = []
@@ -437,12 +457,24 @@ class RpmRootfs(Rootfs):
else:
pkgs += pkgs_to_install[pkg_type]
+ if self.progress_reporter:
+ self.progress_reporter.next_stage()
+
self.pm.install(pkgs)
+ if self.progress_reporter:
+ self.progress_reporter.next_stage()
+
self.pm.install(pkgs_attempt, True)
+ if self.progress_reporter:
+ self.progress_reporter.next_stage()
+
self.pm.install_complementary()
+ if self.progress_reporter:
+ self.progress_reporter.next_stage()
+
self._setup_dbg_rootfs(['/etc/rpm', '/var/lib/rpm', '/var/lib/smart'])
execute_pre_post_process(self.d, rpm_post_process_cmds)
@@ -454,6 +486,10 @@ class RpmRootfs(Rootfs):
self.pm.rpm_setup_smart_target_config()
+ if self.progress_reporter:
+ self.progress_reporter.next_stage()
+
+
@staticmethod
def _depends_list():
return ['DEPLOY_DIR_RPM', 'INC_RPM_IMAGE_GEN', 'RPM_PREPROCESS_COMMANDS',
@@ -524,8 +560,8 @@ class RpmRootfs(Rootfs):
bb.utils.remove(self.pm.install_dir_path, True)
class DpkgOpkgRootfs(Rootfs):
- def __init__(self, d):
- super(DpkgOpkgRootfs, self).__init__(d)
+ def __init__(self, d, progress_reporter=None):
+ super(DpkgOpkgRootfs, self).__init__(d, progress_reporter)
def _get_pkgs_postinsts(self, status_file):
def _get_pkg_depends_list(pkg_depends):
@@ -619,8 +655,8 @@ class DpkgOpkgRootfs(Rootfs):
num += 1
class DpkgRootfs(DpkgOpkgRootfs):
- def __init__(self, d, manifest_dir):
- super(DpkgRootfs, self).__init__(d)
+ def __init__(self, d, manifest_dir, progress_reporter=None):
+ super(DpkgRootfs, self).__init__(d, progress_reporter)
self.log_check_regex = '^E:'
self.log_check_expected_errors_regexes = \
[
@@ -648,15 +684,31 @@ class DpkgRootfs(DpkgOpkgRootfs):
execute_pre_post_process(self.d, deb_pre_process_cmds)
+ if self.progress_reporter:
+ self.progress_reporter.next_stage()
+ # Don't support incremental, so skip that
+ self.progress_reporter.next_stage()
+
self.pm.update()
+ if self.progress_reporter:
+ self.progress_reporter.next_stage()
+
for pkg_type in self.install_order:
if pkg_type in pkgs_to_install:
self.pm.install(pkgs_to_install[pkg_type],
[False, True][pkg_type == Manifest.PKG_TYPE_ATTEMPT_ONLY])
+ if self.progress_reporter:
+ # Don't support attemptonly, so skip that
+ self.progress_reporter.next_stage()
+ self.progress_reporter.next_stage()
+
self.pm.install_complementary()
+ if self.progress_reporter:
+ self.progress_reporter.next_stage()
+
self._setup_dbg_rootfs(['/var/lib/dpkg'])
self.pm.fix_broken_dependencies()
@@ -667,6 +719,9 @@ class DpkgRootfs(DpkgOpkgRootfs):
execute_pre_post_process(self.d, deb_post_process_cmds)
+ if self.progress_reporter:
+ self.progress_reporter.next_stage()
+
@staticmethod
def _depends_list():
return ['DEPLOY_DIR_DEB', 'DEB_SDK_ARCH', 'APTCONF_TARGET', 'APT_ARGS', 'DPKG_ARCH', 'DEB_PREPROCESS_COMMANDS', 'DEB_POSTPROCESS_COMMANDS']
@@ -692,8 +747,8 @@ class DpkgRootfs(DpkgOpkgRootfs):
class OpkgRootfs(DpkgOpkgRootfs):
- def __init__(self, d, manifest_dir):
- super(OpkgRootfs, self).__init__(d)
+ def __init__(self, d, manifest_dir, progress_reporter=None):
+ super(OpkgRootfs, self).__init__(d, progress_reporter)
self.log_check_regex = '(exit 1|Collected errors)'
self.manifest = OpkgManifest(d, manifest_dir)
@@ -887,13 +942,24 @@ class OpkgRootfs(DpkgOpkgRootfs):
execute_pre_post_process(self.d, opkg_pre_process_cmds)
+ if self.progress_reporter:
+ self.progress_reporter.next_stage()
+ # Steps are a bit different in order, skip next
+ self.progress_reporter.next_stage()
+
self.pm.update()
self.pm.handle_bad_recommendations()
+ if self.progress_reporter:
+ self.progress_reporter.next_stage()
+
if self.inc_opkg_image_gen == "1":
self._remove_extra_packages(pkgs_to_install)
+ if self.progress_reporter:
+ self.progress_reporter.next_stage()
+
for pkg_type in self.install_order:
if pkg_type in pkgs_to_install:
# For multilib, we perform a sanity test before final install
@@ -905,8 +971,14 @@ class OpkgRootfs(DpkgOpkgRootfs):
self.pm.install(pkgs_to_install[pkg_type],
[False, True][pkg_type == Manifest.PKG_TYPE_ATTEMPT_ONLY])
+ if self.progress_reporter:
+ self.progress_reporter.next_stage()
+
self.pm.install_complementary()
+ if self.progress_reporter:
+ self.progress_reporter.next_stage()
+
self._setup_dbg_rootfs(['/etc', '/var/lib/opkg'])
execute_pre_post_process(self.d, opkg_post_process_cmds)
@@ -914,6 +986,9 @@ class OpkgRootfs(DpkgOpkgRootfs):
if self.inc_opkg_image_gen == "1":
self.pm.backup_packaging_data()
+ if self.progress_reporter:
+ self.progress_reporter.next_stage()
+
@staticmethod
def _depends_list():
return ['IPKGCONF_SDK', 'IPK_FEED_URIS', 'DEPLOY_DIR_IPK', 'IPKGCONF_TARGET', 'INC_IPK_IMAGE_GEN', 'OPKG_ARGS', 'OPKGLIBDIR', 'OPKG_PREPROCESS_COMMANDS', 'OPKG_POSTPROCESS_COMMANDS', 'OPKGLIBDIR']
@@ -949,16 +1024,16 @@ def variable_depends(d, manifest_dir=None):
cls = get_class_for_type(img_type)
return cls._depends_list()
-def create_rootfs(d, manifest_dir=None):
+def create_rootfs(d, manifest_dir=None, progress_reporter=None):
env_bkp = os.environ.copy()
img_type = d.getVar('IMAGE_PKGTYPE', True)
if img_type == "rpm":
- RpmRootfs(d, manifest_dir).create()
+ RpmRootfs(d, manifest_dir, progress_reporter).create()
elif img_type == "ipk":
- OpkgRootfs(d, manifest_dir).create()
+ OpkgRootfs(d, manifest_dir, progress_reporter).create()
elif img_type == "deb":
- DpkgRootfs(d, manifest_dir).create()
+ DpkgRootfs(d, manifest_dir, progress_reporter).create()
os.environ.clear()
os.environ.update(env_bkp)
--
2.5.5
^ permalink raw reply related [flat|nested] 7+ messages in thread
end of thread, other threads:[~2016-05-10 23:25 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2016-05-10 23:24 [RFC PATCH 0/6] Task progress reporting Paul Eggleton
2016-05-10 23:24 ` [RFC PATCH 1/6] knotty: import latest python-progressbar Paul Eggleton
2016-05-10 23:24 ` [RFC PATCH 2/6] lib: implement basic task progress support Paul Eggleton
2016-05-10 23:24 ` [RFC PATCH 3/6] fetch2: implement " Paul Eggleton
2016-05-10 23:24 ` [RFC PATCH 4/6] bitbake.conf: whitelist progress varflag Paul Eggleton
2016-05-10 23:24 ` [RFC PATCH 5/6] classes/cmake: enable progress for do_compile Paul Eggleton
2016-05-10 23:24 ` [RFC PATCH 6/6] classes/image: implement progress support for do_rootfs Paul Eggleton
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox