Re: [PATCH] python-smartpm: Add an attempt install mode

[Mark Hatle <mark.hatle@windriver.com>]

Time measurements with:

MACHINE = "qemux86"
PACKAGE_CLASSES = "package_rpm"
EXTRA_IMAGE_FEATURES = "dev-pkgs staticdev-pkgs doc-pkgs dbg-pkgs ptest-pkgs"
BB_NUMBER_THREADS ?= "8"
PARALLEL_MAKE ?= "-j 8"

image: core-image-sato

Quad Core i7 workstation...

With all of the recipes built, and just executing the do_rootfs:

Before this change:
  real    26m2.541s
  user    21m8.458s
  sys     7m9.683s

After this change:
  real    14m43.436s
  user    15m57.525s
  sys     5m16.412s

This is a significant performance boost when using the 'complementary' installs 
and attemptonly installs.  The performance is by doing the complementary 
installs as a batch instead of one at a time, ignoring failures.

--Mark

On 9/10/13 8:59 PM, Mark Hatle wrote:
> [ YOCTO #3723 ]
>
> Add a mode to smart that will allow an installation to continue, instead of
> failure in the case that one or more items is uninstallable.
>
> Uninstallable packages are simply ignored, and no error is generated.
>
> Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
> ---
>   meta/classes/package_rpm.bbclass                   |  7 +--
>   .../python/python-smartpm/smart-attempt.patch      | 59 ++++++++++++++++++++++
>   .../python/python-smartpm_1.4.1.bb                 |  1 +
>   3 files changed, 62 insertions(+), 5 deletions(-)
>   create mode 100644 meta/recipes-devtools/python/python-smartpm/smart-attempt.patch
>
> diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass
> index 8ab42c4..699d0f2 100644
> --- a/meta/classes/package_rpm.bbclass
> +++ b/meta/classes/package_rpm.bbclass
> @@ -472,11 +472,8 @@ EOF
>   		echo "Attempting $pkgs_to_install"
>   		echo "Note: see `dirname ${BB_LOGFILE}`/log.do_${task}_attemptonly.${PID}"
>   		translate_oe_to_smart ${sdk_mode} --attemptonly $package_attemptonly
> -		for each_pkg in $pkgs_to_install ;  do
> -			# We need to try each package individually as a single dependency failure
> -			# will break the whole set otherwise.
> -			smart --data-dir=${target_rootfs}/var/lib/smart install -y $each_pkg >> "`dirname ${BB_LOGFILE}`/log.do_${task}_attemptonly.${PID}" 2>&1 || true
> -		done
> +		echo "Attempting $pkgs_to_install" >> "`dirname ${BB_LOGFILE}`/log.do_${task}_attemptonly.${PID}"
> +		smart --data-dir=${target_rootfs}/var/lib/smart install --attempt -y ${pkgs_to_install} >> "`dirname ${BB_LOGFILE}`/log.do_${task}_attemptonly.${PID}" 2>&1
>   	fi
>   }
>
> diff --git a/meta/recipes-devtools/python/python-smartpm/smart-attempt.patch b/meta/recipes-devtools/python/python-smartpm/smart-attempt.patch
> new file mode 100644
> index 0000000..0d603d3
> --- /dev/null
> +++ b/meta/recipes-devtools/python/python-smartpm/smart-attempt.patch
> @@ -0,0 +1,59 @@
> +Add a mechanism to attempt the install operation, w/o failing.
> +
> +For complementary and 'attemptonly' packages, we need a way to instruct smart to
> +try to install, but ignore any failures.
> +
> +This option only works for the install operation.
> +
> +Upstream-Status: Pending
> +
> +Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
> +
> +Index: smart-1.4.1/smart/commands/install.py
> +===================================================================
> +--- smart-1.4.1.orig/smart/commands/install.py
> ++++ smart-1.4.1/smart/commands/install.py
> +@@ -50,6 +50,8 @@ def option_parser():
> +     parser = OptionParser(usage=USAGE,
> +                           description=DESCRIPTION,
> +                           examples=EXAMPLES)
> ++    parser.add_option("--attempt", action="store_true",
> ++                      help=_("attempt to install packages, ignore failures"))
> +     parser.add_option("--stepped", action="store_true",
> +                       help=_("split operation in steps"))
> +     parser.add_option("--urls", action="store_true",
> +@@ -80,6 +82,9 @@ def main(ctrl, opts):
> +     if not opts.args:
> +         raise Error, _("no package(s) given")
> +
> ++    if opts.attempt:
> ++	sysconf.set("attempt-install", True, soft=True)
> ++
> +     if opts.explain:
> +         sysconf.set("explain-changesets", True, soft=True)
> +
> +Index: smart-1.4.1/smart/transaction.py
> +===================================================================
> +--- smart-1.4.1.orig/smart/transaction.py
> ++++ smart-1.4.1/smart/transaction.py
> +@@ -1216,9 +1216,17 @@ class Transaction(object):
> +                     else:
> +                         op = REMOVE
> +                 if op is INSTALL or op is REINSTALL:
> +-                    self._install(pkg, changeset, locked, pending)
> +-                    if pkg in changeset:
> +-                        changeset.setRequested(pkg, True)
> ++                    try:
> ++                        self._install(pkg, changeset, locked, pending)
> ++                        if pkg in changeset:
> ++                            changeset.setRequested(pkg, True)
> ++                    except Failed, e:
> ++                        if sysconf.has("attempt-install", soft=True):
> ++                            if pkg in changeset:
> ++                                del changeset[pkg]
> ++                            continue
> ++                        else:
> ++                            raise Failed, e
> +                 elif op is REMOVE:
> +                     self._remove(pkg, changeset, locked, pending)
> +                 elif op is UPGRADE:
> diff --git a/meta/recipes-devtools/python/python-smartpm_1.4.1.bb b/meta/recipes-devtools/python/python-smartpm_1.4.1.bb
> index 5d673b9..f406d69 100644
> --- a/meta/recipes-devtools/python/python-smartpm_1.4.1.bb
> +++ b/meta/recipes-devtools/python/python-smartpm_1.4.1.bb
> @@ -30,6 +30,7 @@ SRC_URI = "\
>             file://smart-flag-ignore-recommends.patch \
>             file://smart-flag-exclude-packages.patch \
>             file://smart-config-ignore-all-recommends.patch \
> +          file://smart-attempt.patch \
>             "
>
>   SRC_URI[md5sum] = "573ef32ba177a6b3c4bf7ef04873fcb6"
>