* [PATCH] python-smartpm: Add an attempt install mode
@ 2013-09-11 1:59 Mark Hatle
2013-09-11 2:04 ` Mark Hatle
0 siblings, 1 reply; 5+ messages in thread
From: Mark Hatle @ 2013-09-11 1:59 UTC (permalink / raw)
To: openembedded-core
[ YOCTO #3723 ]
Add a mode to smart that will allow an installation to continue, instead of
failure in the case that one or more items is uninstallable.
Uninstallable packages are simply ignored, and no error is generated.
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
---
meta/classes/package_rpm.bbclass | 7 +--
.../python/python-smartpm/smart-attempt.patch | 59 ++++++++++++++++++++++
.../python/python-smartpm_1.4.1.bb | 1 +
3 files changed, 62 insertions(+), 5 deletions(-)
create mode 100644 meta/recipes-devtools/python/python-smartpm/smart-attempt.patch
diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass
index 8ab42c4..699d0f2 100644
--- a/meta/classes/package_rpm.bbclass
+++ b/meta/classes/package_rpm.bbclass
@@ -472,11 +472,8 @@ EOF
echo "Attempting $pkgs_to_install"
echo "Note: see `dirname ${BB_LOGFILE}`/log.do_${task}_attemptonly.${PID}"
translate_oe_to_smart ${sdk_mode} --attemptonly $package_attemptonly
- for each_pkg in $pkgs_to_install ; do
- # We need to try each package individually as a single dependency failure
- # will break the whole set otherwise.
- smart --data-dir=${target_rootfs}/var/lib/smart install -y $each_pkg >> "`dirname ${BB_LOGFILE}`/log.do_${task}_attemptonly.${PID}" 2>&1 || true
- done
+ echo "Attempting $pkgs_to_install" >> "`dirname ${BB_LOGFILE}`/log.do_${task}_attemptonly.${PID}"
+ smart --data-dir=${target_rootfs}/var/lib/smart install --attempt -y ${pkgs_to_install} >> "`dirname ${BB_LOGFILE}`/log.do_${task}_attemptonly.${PID}" 2>&1
fi
}
diff --git a/meta/recipes-devtools/python/python-smartpm/smart-attempt.patch b/meta/recipes-devtools/python/python-smartpm/smart-attempt.patch
new file mode 100644
index 0000000..0d603d3
--- /dev/null
+++ b/meta/recipes-devtools/python/python-smartpm/smart-attempt.patch
@@ -0,0 +1,59 @@
+Add a mechanism to attempt the install operation, w/o failing.
+
+For complementary and 'attemptonly' packages, we need a way to instruct smart to
+try to install, but ignore any failures.
+
+This option only works for the install operation.
+
+Upstream-Status: Pending
+
+Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
+
+Index: smart-1.4.1/smart/commands/install.py
+===================================================================
+--- smart-1.4.1.orig/smart/commands/install.py
++++ smart-1.4.1/smart/commands/install.py
+@@ -50,6 +50,8 @@ def option_parser():
+ parser = OptionParser(usage=USAGE,
+ description=DESCRIPTION,
+ examples=EXAMPLES)
++ parser.add_option("--attempt", action="store_true",
++ help=_("attempt to install packages, ignore failures"))
+ parser.add_option("--stepped", action="store_true",
+ help=_("split operation in steps"))
+ parser.add_option("--urls", action="store_true",
+@@ -80,6 +82,9 @@ def main(ctrl, opts):
+ if not opts.args:
+ raise Error, _("no package(s) given")
+
++ if opts.attempt:
++ sysconf.set("attempt-install", True, soft=True)
++
+ if opts.explain:
+ sysconf.set("explain-changesets", True, soft=True)
+
+Index: smart-1.4.1/smart/transaction.py
+===================================================================
+--- smart-1.4.1.orig/smart/transaction.py
++++ smart-1.4.1/smart/transaction.py
+@@ -1216,9 +1216,17 @@ class Transaction(object):
+ else:
+ op = REMOVE
+ if op is INSTALL or op is REINSTALL:
+- self._install(pkg, changeset, locked, pending)
+- if pkg in changeset:
+- changeset.setRequested(pkg, True)
++ try:
++ self._install(pkg, changeset, locked, pending)
++ if pkg in changeset:
++ changeset.setRequested(pkg, True)
++ except Failed, e:
++ if sysconf.has("attempt-install", soft=True):
++ if pkg in changeset:
++ del changeset[pkg]
++ continue
++ else:
++ raise Failed, e
+ elif op is REMOVE:
+ self._remove(pkg, changeset, locked, pending)
+ elif op is UPGRADE:
diff --git a/meta/recipes-devtools/python/python-smartpm_1.4.1.bb b/meta/recipes-devtools/python/python-smartpm_1.4.1.bb
index 5d673b9..f406d69 100644
--- a/meta/recipes-devtools/python/python-smartpm_1.4.1.bb
+++ b/meta/recipes-devtools/python/python-smartpm_1.4.1.bb
@@ -30,6 +30,7 @@ SRC_URI = "\
file://smart-flag-ignore-recommends.patch \
file://smart-flag-exclude-packages.patch \
file://smart-config-ignore-all-recommends.patch \
+ file://smart-attempt.patch \
"
SRC_URI[md5sum] = "573ef32ba177a6b3c4bf7ef04873fcb6"
--
1.8.1.2.545.g2f19ada
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [PATCH] python-smartpm: Add an attempt install mode
2013-09-11 1:59 [PATCH] python-smartpm: Add an attempt install mode Mark Hatle
@ 2013-09-11 2:04 ` Mark Hatle
2013-09-11 10:04 ` Paul Eggleton
0 siblings, 1 reply; 5+ messages in thread
From: Mark Hatle @ 2013-09-11 2:04 UTC (permalink / raw)
To: openembedded-core
Time measurements with:
MACHINE = "qemux86"
PACKAGE_CLASSES = "package_rpm"
EXTRA_IMAGE_FEATURES = "dev-pkgs staticdev-pkgs doc-pkgs dbg-pkgs ptest-pkgs"
BB_NUMBER_THREADS ?= "8"
PARALLEL_MAKE ?= "-j 8"
image: core-image-sato
Quad Core i7 workstation...
With all of the recipes built, and just executing the do_rootfs:
Before this change:
real 26m2.541s
user 21m8.458s
sys 7m9.683s
After this change:
real 14m43.436s
user 15m57.525s
sys 5m16.412s
This is a significant performance boost when using the 'complementary' installs
and attemptonly installs. The performance is by doing the complementary
installs as a batch instead of one at a time, ignoring failures.
--Mark
On 9/10/13 8:59 PM, Mark Hatle wrote:
> [ YOCTO #3723 ]
>
> Add a mode to smart that will allow an installation to continue, instead of
> failure in the case that one or more items is uninstallable.
>
> Uninstallable packages are simply ignored, and no error is generated.
>
> Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
> ---
> meta/classes/package_rpm.bbclass | 7 +--
> .../python/python-smartpm/smart-attempt.patch | 59 ++++++++++++++++++++++
> .../python/python-smartpm_1.4.1.bb | 1 +
> 3 files changed, 62 insertions(+), 5 deletions(-)
> create mode 100644 meta/recipes-devtools/python/python-smartpm/smart-attempt.patch
>
> diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass
> index 8ab42c4..699d0f2 100644
> --- a/meta/classes/package_rpm.bbclass
> +++ b/meta/classes/package_rpm.bbclass
> @@ -472,11 +472,8 @@ EOF
> echo "Attempting $pkgs_to_install"
> echo "Note: see `dirname ${BB_LOGFILE}`/log.do_${task}_attemptonly.${PID}"
> translate_oe_to_smart ${sdk_mode} --attemptonly $package_attemptonly
> - for each_pkg in $pkgs_to_install ; do
> - # We need to try each package individually as a single dependency failure
> - # will break the whole set otherwise.
> - smart --data-dir=${target_rootfs}/var/lib/smart install -y $each_pkg >> "`dirname ${BB_LOGFILE}`/log.do_${task}_attemptonly.${PID}" 2>&1 || true
> - done
> + echo "Attempting $pkgs_to_install" >> "`dirname ${BB_LOGFILE}`/log.do_${task}_attemptonly.${PID}"
> + smart --data-dir=${target_rootfs}/var/lib/smart install --attempt -y ${pkgs_to_install} >> "`dirname ${BB_LOGFILE}`/log.do_${task}_attemptonly.${PID}" 2>&1
> fi
> }
>
> diff --git a/meta/recipes-devtools/python/python-smartpm/smart-attempt.patch b/meta/recipes-devtools/python/python-smartpm/smart-attempt.patch
> new file mode 100644
> index 0000000..0d603d3
> --- /dev/null
> +++ b/meta/recipes-devtools/python/python-smartpm/smart-attempt.patch
> @@ -0,0 +1,59 @@
> +Add a mechanism to attempt the install operation, w/o failing.
> +
> +For complementary and 'attemptonly' packages, we need a way to instruct smart to
> +try to install, but ignore any failures.
> +
> +This option only works for the install operation.
> +
> +Upstream-Status: Pending
> +
> +Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
> +
> +Index: smart-1.4.1/smart/commands/install.py
> +===================================================================
> +--- smart-1.4.1.orig/smart/commands/install.py
> ++++ smart-1.4.1/smart/commands/install.py
> +@@ -50,6 +50,8 @@ def option_parser():
> + parser = OptionParser(usage=USAGE,
> + description=DESCRIPTION,
> + examples=EXAMPLES)
> ++ parser.add_option("--attempt", action="store_true",
> ++ help=_("attempt to install packages, ignore failures"))
> + parser.add_option("--stepped", action="store_true",
> + help=_("split operation in steps"))
> + parser.add_option("--urls", action="store_true",
> +@@ -80,6 +82,9 @@ def main(ctrl, opts):
> + if not opts.args:
> + raise Error, _("no package(s) given")
> +
> ++ if opts.attempt:
> ++ sysconf.set("attempt-install", True, soft=True)
> ++
> + if opts.explain:
> + sysconf.set("explain-changesets", True, soft=True)
> +
> +Index: smart-1.4.1/smart/transaction.py
> +===================================================================
> +--- smart-1.4.1.orig/smart/transaction.py
> ++++ smart-1.4.1/smart/transaction.py
> +@@ -1216,9 +1216,17 @@ class Transaction(object):
> + else:
> + op = REMOVE
> + if op is INSTALL or op is REINSTALL:
> +- self._install(pkg, changeset, locked, pending)
> +- if pkg in changeset:
> +- changeset.setRequested(pkg, True)
> ++ try:
> ++ self._install(pkg, changeset, locked, pending)
> ++ if pkg in changeset:
> ++ changeset.setRequested(pkg, True)
> ++ except Failed, e:
> ++ if sysconf.has("attempt-install", soft=True):
> ++ if pkg in changeset:
> ++ del changeset[pkg]
> ++ continue
> ++ else:
> ++ raise Failed, e
> + elif op is REMOVE:
> + self._remove(pkg, changeset, locked, pending)
> + elif op is UPGRADE:
> diff --git a/meta/recipes-devtools/python/python-smartpm_1.4.1.bb b/meta/recipes-devtools/python/python-smartpm_1.4.1.bb
> index 5d673b9..f406d69 100644
> --- a/meta/recipes-devtools/python/python-smartpm_1.4.1.bb
> +++ b/meta/recipes-devtools/python/python-smartpm_1.4.1.bb
> @@ -30,6 +30,7 @@ SRC_URI = "\
> file://smart-flag-ignore-recommends.patch \
> file://smart-flag-exclude-packages.patch \
> file://smart-config-ignore-all-recommends.patch \
> + file://smart-attempt.patch \
> "
>
> SRC_URI[md5sum] = "573ef32ba177a6b3c4bf7ef04873fcb6"
>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH] python-smartpm: Add an attempt install mode
2013-09-11 2:04 ` Mark Hatle
@ 2013-09-11 10:04 ` Paul Eggleton
2013-09-11 13:47 ` Mark Hatle
0 siblings, 1 reply; 5+ messages in thread
From: Paul Eggleton @ 2013-09-11 10:04 UTC (permalink / raw)
To: Mark Hatle; +Cc: openembedded-core
Hi Mark,
On Tuesday 10 September 2013 21:04:29 Mark Hatle wrote:
> Time measurements with:
>
> MACHINE = "qemux86"
> PACKAGE_CLASSES = "package_rpm"
> EXTRA_IMAGE_FEATURES = "dev-pkgs staticdev-pkgs doc-pkgs dbg-pkgs
> ptest-pkgs" BB_NUMBER_THREADS ?= "8"
> PARALLEL_MAKE ?= "-j 8"
>
> image: core-image-sato
>
> Quad Core i7 workstation...
>
> With all of the recipes built, and just executing the do_rootfs:
>
> Before this change:
> real 26m2.541s
> user 21m8.458s
> sys 7m9.683s
>
> After this change:
> real 14m43.436s
> user 15m57.525s
> sys 5m16.412s
>
> This is a significant performance boost when using the 'complementary'
> installs and attemptonly installs. The performance is by doing the
> complementary installs as a batch instead of one at a time, ignoring
> failures.
This is a significant improvement, well done! My concern about not catching
failures stands though, which is why I hadn't implemented this yet. How do we
ensure that genuine failures are caught with this mode?
Cheers,
Paul
--
Paul Eggleton
Intel Open Source Technology Centre
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH] python-smartpm: Add an attempt install mode
2013-09-11 10:04 ` Paul Eggleton
@ 2013-09-11 13:47 ` Mark Hatle
2013-09-11 14:29 ` Paul Eggleton
0 siblings, 1 reply; 5+ messages in thread
From: Mark Hatle @ 2013-09-11 13:47 UTC (permalink / raw)
To: Paul Eggleton; +Cc: openembedded-core
On 9/11/13 5:04 AM, Paul Eggleton wrote:
> Hi Mark,
>
> On Tuesday 10 September 2013 21:04:29 Mark Hatle wrote:
>> Time measurements with:
>>
>> MACHINE = "qemux86"
>> PACKAGE_CLASSES = "package_rpm"
>> EXTRA_IMAGE_FEATURES = "dev-pkgs staticdev-pkgs doc-pkgs dbg-pkgs
>> ptest-pkgs" BB_NUMBER_THREADS ?= "8"
>> PARALLEL_MAKE ?= "-j 8"
>>
>> image: core-image-sato
>>
>> Quad Core i7 workstation...
>>
>> With all of the recipes built, and just executing the do_rootfs:
>>
>> Before this change:
>> real 26m2.541s
>> user 21m8.458s
>> sys 7m9.683s
>>
>> After this change:
>> real 14m43.436s
>> user 15m57.525s
>> sys 5m16.412s
>>
>> This is a significant performance boost when using the 'complementary'
>> installs and attemptonly installs. The performance is by doing the
>> complementary installs as a batch instead of one at a time, ignoring
>> failures.
>
> This is a significant improvement, well done! My concern about not catching
> failures stands though, which is why I hadn't implemented this yet. How do we
> ensure that genuine failures are caught with this mode?
The code only ignores failures in the resolution of the dependencies. As far as
I'm aware, this is the correct behavior in the attempt only mode(s). If the
dependencies can't be resolved for some reason, we simply throw away that item
and move on.
Any errors that occur during package installation -will- now cause a failure to
exist, and they will be logged to the attemptonly log-file.
In the previous implementation, any error when installing an attemptonly package
would be ignored, even if it turned out to be an installation problem and not a
dependency issue. So I think this is an improvement as well.
(For some of my testing I intentionally created dependency errors by adding
things to the PACKAGE_EXCLUDE, and the system did what it was supposed to.
Ignored the items that couldn't be installed.)
--Mark
> Cheers,
> Paul
>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH] python-smartpm: Add an attempt install mode
2013-09-11 13:47 ` Mark Hatle
@ 2013-09-11 14:29 ` Paul Eggleton
0 siblings, 0 replies; 5+ messages in thread
From: Paul Eggleton @ 2013-09-11 14:29 UTC (permalink / raw)
To: Mark Hatle; +Cc: openembedded-core
On Wednesday 11 September 2013 08:47:33 Mark Hatle wrote:
> On 9/11/13 5:04 AM, Paul Eggleton wrote:
> > On Tuesday 10 September 2013 21:04:29 Mark Hatle wrote:
> >> Time measurements with:
> >>
> >> MACHINE = "qemux86"
> >> PACKAGE_CLASSES = "package_rpm"
> >> EXTRA_IMAGE_FEATURES = "dev-pkgs staticdev-pkgs doc-pkgs dbg-pkgs
> >> ptest-pkgs" BB_NUMBER_THREADS ?= "8"
> >> PARALLEL_MAKE ?= "-j 8"
> >>
> >> image: core-image-sato
> >>
> >> Quad Core i7 workstation...
> >>
> >> With all of the recipes built, and just executing the do_rootfs:
> >>
> >> Before this change:
> >> real 26m2.541s
> >> user 21m8.458s
> >> sys 7m9.683s
> >>
> >> After this change:
> >> real 14m43.436s
> >> user 15m57.525s
> >> sys 5m16.412s
> >>
> >> This is a significant performance boost when using the 'complementary'
> >> installs and attemptonly installs. The performance is by doing the
> >> complementary installs as a batch instead of one at a time, ignoring
> >> failures.
> >
> > This is a significant improvement, well done! My concern about not
> > catching
> > failures stands though, which is why I hadn't implemented this yet. How do
> > we ensure that genuine failures are caught with this mode?
>
> The code only ignores failures in the resolution of the dependencies. As
> far as I'm aware, this is the correct behavior in the attempt only mode(s).
> If the dependencies can't be resolved for some reason, we simply throw
> away that item and move on.
>
> Any errors that occur during package installation -will- now cause a failure
> to exist, and they will be logged to the attemptonly log-file.
>
> In the previous implementation, any error when installing an attemptonly
> package would be ignored, even if it turned out to be an installation
> problem and not a dependency issue. So I think this is an improvement as
> well.
>
> (For some of my testing I intentionally created dependency errors by adding
> things to the PACKAGE_EXCLUDE, and the system did what it was supposed to.
> Ignored the items that couldn't be installed.)
Sounds like a great improvement. Thanks!
Cheers,
Paul
--
Paul Eggleton
Intel Open Source Technology Centre
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2013-09-11 14:29 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-09-11 1:59 [PATCH] python-smartpm: Add an attempt install mode Mark Hatle
2013-09-11 2:04 ` Mark Hatle
2013-09-11 10:04 ` Paul Eggleton
2013-09-11 13:47 ` Mark Hatle
2013-09-11 14:29 ` Paul Eggleton
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox