[PATCH V2 0/1] package_manager.py: fix rpm based do_rootfs failed while IMAGE_INSTALL_append = " python3"

[PATCH V2 0/1] package_manager.py: fix rpm based do_rootfs failed while IMAGE_INSTALL_append = " python3"

[Hongxu Jia]

Changed in V2: 
- Remove debug commented out line;
- Tweak the way of function invoking;

vim local.conf
...
IMAGE_INSTALL_append " python3"
...

bitbake core-image-minimal

//Hongxu

The following changes since commit dacc4ce59e48129a1a1e5316e10780f7358e29ef:

  nativesdk-cmake: Adjust toolchain paths dynamically (2014-10-24 21:59:46 +0100)

are available in the git repository at:

  git://git.pokylinux.org/poky-contrib hongxu/fix-rootfs
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=hongxu/fix-rootfs

Hongxu Jia (1):
  package_manager.py: fix rpm based do_rootfs failed while
    IMAGE_INSTALL_append = " python3"

 meta/lib/oe/package_manager.py | 22 ++++++++++++++++++++--
 1 file changed, 20 insertions(+), 2 deletions(-)

-- 
1.9.1

[PATCH 1/1] package_manager.py: fix rpm based do_rootfs failed while IMAGE_INSTALL_append = " python3"

[Hongxu Jia]

While a pkg name (such as python3) not existed in variable
PACKAGES, but provided by another pkg (such as python-core),
in this situation, rpm based do_rootfs could not search it in
the feeds.

The fix is to invoke 'smart query --provides' to do the search
if the feeds search failed. We don't drop the feeds search,
because the smart query search is too slow, and the feeds
search is fast and could work well with most pkgs.

[YOCTO #6918]

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 meta/lib/oe/package_manager.py | 22 ++++++++++++++++++++--
 1 file changed, 20 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
index ffb83b2..a0ecf3a 100644
--- a/meta/lib/oe/package_manager.py
+++ b/meta/lib/oe/package_manager.py
@@ -671,6 +671,22 @@ class RpmPM(PackageManager):
 
         return ""
 
+    def _search_pkg_provider_in_smart(self, pkg, feed_archs):
+        output = self._invoke_smart('query  --provides %s' % pkg) or ''
+        for provider in output.split('\n'):
+            # Filter out the line which is empty or start with space
+            if provider.strip() == '' or provider[0] == ' ':
+                continue
+
+            for arch in feed_archs:
+                arch = '@' + arch.replace('-', '_')
+                if arch in provider:
+                    # First found is best match
+                    return provider
+
+        return ''
+
+
     '''
     Translate the OE multilib format names to the RPM/Smart format names
     It searched the RPM/Smart format names in probable multilib feeds first,
@@ -691,7 +707,8 @@ class RpmPM(PackageManager):
                 # if the pkg in this multilib feed
                 if subst != pkg:
                     feed_archs = self.ml_prefix_list[mlib]
-                    new_pkg = self._search_pkg_name_in_feeds(subst, feed_archs)
+                    new_pkg = self._search_pkg_name_in_feeds(subst, feed_archs) or \
+                        self._search_pkg_provider_in_smart(subst, feed_archs)
                     if not new_pkg:
                         # Failed to translate, package not found!
                         err_msg = '%s not found in the %s feeds (%s).\n' % \
@@ -709,7 +726,8 @@ class RpmPM(PackageManager):
             if pkg == new_pkg:
                 # Search new_pkg in default archs
                 default_archs = self.ml_prefix_list['default']
-                new_pkg = self._search_pkg_name_in_feeds(pkg, default_archs)
+                new_pkg = self._search_pkg_name_in_feeds(pkg, default_archs) or \
+                    self._search_pkg_provider_in_smart(pkg, default_archs)
                 if not new_pkg:
                     err_msg = '%s not found in the base feeds (%s).\n' % \
                               (pkg, ' '.join(default_archs))
-- 
1.9.1

Re: [PATCH 1/1] package_manager.py: fix rpm based do_rootfs failed while IMAGE_INSTALL_append = " python3"

[Paul Eggleton]

Hi Hongxu,

On Friday 31 October 2014 17:46:24 Hongxu Jia wrote:
> While a pkg name (such as python3) not existed in variable
> PACKAGES, but provided by another pkg (such as python-core),
> in this situation, rpm based do_rootfs could not search it in
> the feeds.
> 
> The fix is to invoke 'smart query --provides' to do the search
> if the feeds search failed. We don't drop the feeds search,
> because the smart query search is too slow, and the feeds
> search is fast and could work well with most pkgs.
> 
> [YOCTO #6918]
> 
> Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
> ---
>  meta/lib/oe/package_manager.py | 22 ++++++++++++++++++++--
>  1 file changed, 20 insertions(+), 2 deletions(-)
> 
> diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
> index ffb83b2..a0ecf3a 100644
> --- a/meta/lib/oe/package_manager.py
> +++ b/meta/lib/oe/package_manager.py
> @@ -671,6 +671,22 @@ class RpmPM(PackageManager):
> 
>          return ""
> 
> +    def _search_pkg_provider_in_smart(self, pkg, feed_archs):
> +        output = self._invoke_smart('query  --provides %s' % pkg) or ''
> +        for provider in output.split('\n'):
> +            # Filter out the line which is empty or start with space
> +            if provider.strip() == '' or provider[0] == ' ':
> +                continue
> +
> +            for arch in feed_archs:
> +                arch = '@' + arch.replace('-', '_')
> +                if arch in provider:
> +                    # First found is best match
> +                    return provider
> +
> +        return ''
> +
> +
>      '''
>      Translate the OE multilib format names to the RPM/Smart format names
>      It searched the RPM/Smart format names in probable multilib feeds
> first, @@ -691,7 +707,8 @@ class RpmPM(PackageManager):
>                  # if the pkg in this multilib feed
>                  if subst != pkg:
>                      feed_archs = self.ml_prefix_list[mlib]
> -                    new_pkg = self._search_pkg_name_in_feeds(subst,
> feed_archs) +                    new_pkg =
> self._search_pkg_name_in_feeds(subst, feed_archs) or \ +                   
>     self._search_pkg_provider_in_smart(subst, feed_archs) if not new_pkg:
>                          # Failed to translate, package not found!
>                          err_msg = '%s not found in the %s feeds (%s).\n' %
> \ @@ -709,7 +726,8 @@ class RpmPM(PackageManager):
>              if pkg == new_pkg:
>                  # Search new_pkg in default archs
>                  default_archs = self.ml_prefix_list['default']
> -                new_pkg = self._search_pkg_name_in_feeds(pkg,
> default_archs) +                new_pkg =
> self._search_pkg_name_in_feeds(pkg, default_archs) or \ +                  
>  self._search_pkg_provider_in_smart(pkg, default_archs) if not new_pkg:
>                      err_msg = '%s not found in the base feeds (%s).\n' % \
>                                (pkg, ' '.join(default_archs))

It took a long time to reply to this, my apologies. This patch looks OK but 
Richard pointed out that it may have an impact on the speed of do_rootfs 
because of the extra calls out to smart. Do you have any performance impact 
figures?

Thanks,
Paul
-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 1/1] package_manager.py: fix rpm based do_rootfs failed while IMAGE_INSTALL_append = " python3"

[Hongxu Jia]

On 02/13/2015 12:46 AM, Paul Eggleton wrote:
>> default_archs) +                new_pkg =
>> >self._search_pkg_name_in_feeds(pkg, default_archs) or \ +
>> >  self._search_pkg_provider_in_smart(pkg, default_archs) if not new_pkg:
>> >                      err_msg = '%s not found in the base feeds (%s).\n' % \
>> >                                (pkg, ' '.join(default_archs))
> It took a long time to reply to this, my apologies. This patch looks OK but
> Richard pointed out that it may have an impact on the speed of do_rootfs
> because of the extra calls out to smart. Do you have any performance impact
> figures?

Hi Paul,

The extra 'smart' calling takes 3-5 seconds for one search,
but it will not do that for most packages, it goes into
self._search_pkg_name_in_feeds first, if not found and then in to
self._search_pkg_provider_in_smart, only if we have such as python3
in the build session, it will take extra 3-5 seconds.

//Hongxu



> Thanks,
> Paul
> -- 

Re: [PATCH 1/1] package_manager.py: fix rpm based do_rootfs failed while IMAGE_INSTALL_append = " python3"

[Richard Purdie]

On Fri, 2015-02-13 at 09:39 +0800, Hongxu Jia wrote:
> On 02/13/2015 12:46 AM, Paul Eggleton wrote:
> >> default_archs) +                new_pkg =
> >> >self._search_pkg_name_in_feeds(pkg, default_archs) or \ +
> >> >  self._search_pkg_provider_in_smart(pkg, default_archs) if not new_pkg:
> >> >                      err_msg = '%s not found in the base feeds (%s).\n' % \
> >> >                                (pkg, ' '.join(default_archs))
> > It took a long time to reply to this, my apologies. This patch looks OK but
> > Richard pointed out that it may have an impact on the speed of do_rootfs
> > because of the extra calls out to smart. Do you have any performance impact
> > figures?
> 
> Hi Paul,
> 
> The extra 'smart' calling takes 3-5 seconds for one search,
> but it will not do that for most packages, it goes into
> self._search_pkg_name_in_feeds first, if not found and then in to
> self._search_pkg_provider_in_smart, only if we have such as python3
> in the build session, it will take extra 3-5 seconds.

Could we test the time it takes to build core-image-sato-sdk before and
after this patch please?

Cheers,

Richard

Re: [PATCH V2 0/1] package_manager.py: fix rpm based do_rootfs failed while IMAGE_INSTALL_append = " python3"

[Hongxu Jia]

Ping

//Hongxu

On 10/31/2014 05:46 PM, Hongxu Jia wrote:
> Changed in V2:
> - Remove debug commented out line;
> - Tweak the way of function invoking;
>
> vim local.conf
> ...
> IMAGE_INSTALL_append " python3"
> ...
>
> bitbake core-image-minimal
>
> //Hongxu
>
> The following changes since commit dacc4ce59e48129a1a1e5316e10780f7358e29ef:
>
>    nativesdk-cmake: Adjust toolchain paths dynamically (2014-10-24 21:59:46 +0100)
>
> are available in the git repository at:
>
>    git://git.pokylinux.org/poky-contrib hongxu/fix-rootfs
>    http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=hongxu/fix-rootfs
>
> Hongxu Jia (1):
>    package_manager.py: fix rpm based do_rootfs failed while
>      IMAGE_INSTALL_append = " python3"
>
>   meta/lib/oe/package_manager.py | 22 ++++++++++++++++++++--
>   1 file changed, 20 insertions(+), 2 deletions(-)
>

Re: [PATCH V2 0/1] package_manager.py: fix rpm based do_rootfs failed while IMAGE_INSTALL_append = " python3"

[Hongxu Jia]

Ping

//Hongxu

On 10/31/2014 05:46 PM, Hongxu Jia wrote:
> Changed in V2:
> - Remove debug commented out line;
> - Tweak the way of function invoking;
>
> vim local.conf
> ...
> IMAGE_INSTALL_append " python3"
> ...
>
> bitbake core-image-minimal
>
> //Hongxu
>
> The following changes since commit dacc4ce59e48129a1a1e5316e10780f7358e29ef:
>
>    nativesdk-cmake: Adjust toolchain paths dynamically (2014-10-24 21:59:46 +0100)
>
> are available in the git repository at:
>
>    git://git.pokylinux.org/poky-contrib hongxu/fix-rootfs
>    http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=hongxu/fix-rootfs
>
> Hongxu Jia (1):
>    package_manager.py: fix rpm based do_rootfs failed while
>      IMAGE_INSTALL_append = " python3"
>
>   meta/lib/oe/package_manager.py | 22 ++++++++++++++++++++--
>   1 file changed, 20 insertions(+), 2 deletions(-)
>

Re: [PATCH V2 0/1] package_manager.py: fix rpm based do_rootfs failed while IMAGE_INSTALL_append = " python3"

[Hongxu Jia]

Ping

//Hongxu

On 10/31/2014 05:46 PM, Hongxu Jia wrote:
> Changed in V2:
> - Remove debug commented out line;
> - Tweak the way of function invoking;
>
> vim local.conf
> ...
> IMAGE_INSTALL_append " python3"
> ...
>
> bitbake core-image-minimal
>
> //Hongxu
>
> The following changes since commit dacc4ce59e48129a1a1e5316e10780f7358e29ef:
>
>    nativesdk-cmake: Adjust toolchain paths dynamically (2014-10-24 21:59:46 +0100)
>
> are available in the git repository at:
>
>    git://git.pokylinux.org/poky-contrib hongxu/fix-rootfs
>    http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=hongxu/fix-rootfs
>
> Hongxu Jia (1):
>    package_manager.py: fix rpm based do_rootfs failed while
>      IMAGE_INSTALL_append = " python3"
>
>   meta/lib/oe/package_manager.py | 22 ++++++++++++++++++++--
>   1 file changed, 20 insertions(+), 2 deletions(-)
>