[PATCH 0/1] package_manager.py: fix rpm based do_rootfs failed while IMAGE_INSTALL_append = " python3"

[PATCH 0/1] package_manager.py: fix rpm based do_rootfs failed while IMAGE_INSTALL_append = " python3"

[Hongxu Jia]

vim local.conf
...
IMAGE_INSTALL_append " python3"
...

bitbake core-image-minimal

//Hongxu

The following changes since commit dacc4ce59e48129a1a1e5316e10780f7358e29ef:

  nativesdk-cmake: Adjust toolchain paths dynamically (2014-10-24 21:59:46 +0100)

are available in the git repository at:

  git://git.pokylinux.org/poky-contrib hongxu/fix-rootfs
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=hongxu/fix-rootfs

Hongxu Jia (1):
  package_manager.py: fix rpm based do_rootfs failed while
    IMAGE_INSTALL_append = " python3"

 meta/lib/oe/package_manager.py | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

-- 
1.9.1

[PATCH 1/1] package_manager.py: fix rpm based do_rootfs failed while IMAGE_INSTALL_append = " python3"

[Hongxu Jia]

While a pkg name (such as python3) not existed in variable
PACKAGES, but provided by another pkg (such as python-core),
in this situation, rpm based do_rootfs could not search it in
the feeds.

The fix is to invoke 'smart query --provides' to do the search
if the feeds search failed. We don't drop the feeds search,
because the smart query search is too slow, and the feeds
search is fast and could work well with most pkgs.

[YOCTO #6918]

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 meta/lib/oe/package_manager.py | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
index ffb83b2..d5704e6 100644
--- a/meta/lib/oe/package_manager.py
+++ b/meta/lib/oe/package_manager.py
@@ -671,6 +671,23 @@ class RpmPM(PackageManager):
 
         return ""
 
+    def _search_pkg_provider_in_smart(self, pkg, feed_archs):
+        output = self._invoke_smart('query  --provides %s' % pkg) or ""
+        for provider in output.split('\n'):
+            # Filter out the line which is empty or start with space
+            if provider.strip() == '' or provider[0] == ' ':
+                continue
+
+            for arch in feed_archs:
+                arch = '@' + arch.replace('-', '_')
+                if arch in provider:
+                    # First found is best match
+                    # bb.note("%s -> %s" % (pkg, provider))
+                    return provider
+
+        return ''
+
+
     '''
     Translate the OE multilib format names to the RPM/Smart format names
     It searched the RPM/Smart format names in probable multilib feeds first,
@@ -693,6 +710,8 @@ class RpmPM(PackageManager):
                     feed_archs = self.ml_prefix_list[mlib]
                     new_pkg = self._search_pkg_name_in_feeds(subst, feed_archs)
                     if not new_pkg:
+                        new_pkg = self._search_pkg_provider_in_smart(subst, feed_archs)
+                    if not new_pkg:
                         # Failed to translate, package not found!
                         err_msg = '%s not found in the %s feeds (%s).\n' % \
                                   (pkg, mlib, " ".join(feed_archs))
@@ -711,6 +730,8 @@ class RpmPM(PackageManager):
                 default_archs = self.ml_prefix_list['default']
                 new_pkg = self._search_pkg_name_in_feeds(pkg, default_archs)
                 if not new_pkg:
+                    new_pkg = self._search_pkg_provider_in_smart(pkg, default_archs)
+                if not new_pkg:
                     err_msg = '%s not found in the base feeds (%s).\n' % \
                               (pkg, ' '.join(default_archs))
                     if not attempt_only:
-- 
1.9.1

Re: [PATCH 1/1] package_manager.py: fix rpm based do_rootfs failed while IMAGE_INSTALL_append = " python3"

[Laurentiu Palcu]

On Fri, Oct 31, 2014 at 03:05:58PM +0800, Hongxu Jia wrote:
> While a pkg name (such as python3) not existed in variable
> PACKAGES, but provided by another pkg (such as python-core),
> in this situation, rpm based do_rootfs could not search it in
> the feeds.
> 
> The fix is to invoke 'smart query --provides' to do the search
> if the feeds search failed. We don't drop the feeds search,
> because the smart query search is too slow, and the feeds
> search is fast and could work well with most pkgs.
> 
> [YOCTO #6918]
> 
> Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
> ---
>  meta/lib/oe/package_manager.py | 21 +++++++++++++++++++++
>  1 file changed, 21 insertions(+)
> 
> diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
> index ffb83b2..d5704e6 100644
> --- a/meta/lib/oe/package_manager.py
> +++ b/meta/lib/oe/package_manager.py
> @@ -671,6 +671,23 @@ class RpmPM(PackageManager):
>  
>          return ""
>  
> +    def _search_pkg_provider_in_smart(self, pkg, feed_archs):
> +        output = self._invoke_smart('query  --provides %s' % pkg) or ""
> +        for provider in output.split('\n'):
> +            # Filter out the line which is empty or start with space
> +            if provider.strip() == '' or provider[0] == ' ':
> +                continue
> +
> +            for arch in feed_archs:
> +                arch = '@' + arch.replace('-', '_')
> +                if arch in provider:
> +                    # First found is best match
> +                    # bb.note("%s -> %s" % (pkg, provider))
Why not remove the bb.note above since it's commented out?

> +                    return provider
> +
> +        return ''
> +
> +
>      '''
>      Translate the OE multilib format names to the RPM/Smart format names
>      It searched the RPM/Smart format names in probable multilib feeds first,
> @@ -693,6 +710,8 @@ class RpmPM(PackageManager):
>                      feed_archs = self.ml_prefix_list[mlib]
>                      new_pkg = self._search_pkg_name_in_feeds(subst, feed_archs)
>                      if not new_pkg:
> +                        new_pkg = self._search_pkg_provider_in_smart(subst, feed_archs)
> +                    if not new_pkg:
This entire construct:
...
new_pkg = self._search_pkg_name_in_feeds(subst, feed_archs)
if not new_pkg:
	new_pkg = self._search_pkg_provider_in_smart(subst, feed_archs)
if not new_pkg:
...

could be replaced with:
...
new_pkg = self._search_pkg_name_in_feeds(subst, feed_archs) or \
	self._search_pkg_provider_in_smart(subst, feed_archs)
if not new_pkg:
...

I'd say it looks cleaner but it's a matter of personal taste. :)

>                          # Failed to translate, package not found!
>                          err_msg = '%s not found in the %s feeds (%s).\n' % \
>                                    (pkg, mlib, " ".join(feed_archs))
> @@ -711,6 +730,8 @@ class RpmPM(PackageManager):
>                  default_archs = self.ml_prefix_list['default']
>                  new_pkg = self._search_pkg_name_in_feeds(pkg, default_archs)
>                  if not new_pkg:
> +                    new_pkg = self._search_pkg_provider_in_smart(pkg, default_archs)
> +                if not new_pkg:
ditto

laurentiu

Re: [PATCH 1/1] package_manager.py: fix rpm based do_rootfs failed while IMAGE_INSTALL_append = " python3"

[Hongxu Jia]

On 10/31/2014 04:50 PM, Laurentiu Palcu wrote:
> On Fri, Oct 31, 2014 at 03:05:58PM +0800, Hongxu Jia wrote:
>> While a pkg name (such as python3) not existed in variable
>> PACKAGES, but provided by another pkg (such as python-core),
>> in this situation, rpm based do_rootfs could not search it in
>> the feeds.
>>
>> The fix is to invoke 'smart query --provides' to do the search
>> if the feeds search failed. We don't drop the feeds search,
>> because the smart query search is too slow, and the feeds
>> search is fast and could work well with most pkgs.
>>
>> [YOCTO #6918]
>>
>> Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
>> ---
>>   meta/lib/oe/package_manager.py | 21 +++++++++++++++++++++
>>   1 file changed, 21 insertions(+)
>>
>> diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
>> index ffb83b2..d5704e6 100644
>> --- a/meta/lib/oe/package_manager.py
>> +++ b/meta/lib/oe/package_manager.py
>> @@ -671,6 +671,23 @@ class RpmPM(PackageManager):
>>   
>>           return ""
>>   
>> +    def _search_pkg_provider_in_smart(self, pkg, feed_archs):
>> +        output = self._invoke_smart('query  --provides %s' % pkg) or ""
>> +        for provider in output.split('\n'):
>> +            # Filter out the line which is empty or start with space
>> +            if provider.strip() == '' or provider[0] == ' ':
>> +                continue
>> +
>> +            for arch in feed_archs:
>> +                arch = '@' + arch.replace('-', '_')
>> +                if arch in provider:
>> +                    # First found is best match
>> +                    # bb.note("%s -> %s" % (pkg, provider))
> Why not remove the bb.note above since it's commented out?

The commented out line is left for debug purpose, it is helpful
to trace what changed when you debug.

If it caused confusion, I will remove it.

>
>> +                    return provider
>> +
>> +        return ''
>> +
>> +
>>       '''
>>       Translate the OE multilib format names to the RPM/Smart format names
>>       It searched the RPM/Smart format names in probable multilib feeds first,
>> @@ -693,6 +710,8 @@ class RpmPM(PackageManager):
>>                       feed_archs = self.ml_prefix_list[mlib]
>>                       new_pkg = self._search_pkg_name_in_feeds(subst, feed_archs)
>>                       if not new_pkg:
>> +                        new_pkg = self._search_pkg_provider_in_smart(subst, feed_archs)
>> +                    if not new_pkg:
> This entire construct:
> ...
> new_pkg = self._search_pkg_name_in_feeds(subst, feed_archs)
> if not new_pkg:
> 	new_pkg = self._search_pkg_provider_in_smart(subst, feed_archs)
> if not new_pkg:
> ...
>
> could be replaced with:
> ...
> new_pkg = self._search_pkg_name_in_feeds(subst, feed_archs) or \
> 	self._search_pkg_provider_in_smart(subst, feed_archs)
> if not new_pkg:
> ...
>
> I'd say it looks cleaner but it's a matter of personal taste. :)

Fine, I will improve it.

V2 incoming

//Hongxu



>
>>                           # Failed to translate, package not found!
>>                           err_msg = '%s not found in the %s feeds (%s).\n' % \
>>                                     (pkg, mlib, " ".join(feed_archs))
>> @@ -711,6 +730,8 @@ class RpmPM(PackageManager):
>>                   default_archs = self.ml_prefix_list['default']
>>                   new_pkg = self._search_pkg_name_in_feeds(pkg, default_archs)
>>                   if not new_pkg:
>> +                    new_pkg = self._search_pkg_provider_in_smart(pkg, default_archs)
>> +                if not new_pkg:
> ditto
>
> laurentiu