[PATCH] package_manager: Fix BAD_RECOMMENDATIONS for opkg

[PATCH] package_manager: Fix BAD_RECOMMENDATIONS for opkg

[Paul Barker]

In package_manager.py, when using opkg as the packager, the command 'opkg <args>
info <pkg>' is called to get information about each pkg in BAD_RECOMMENDATIONS
in a format that can be written to the status file. The 'Status: ...' line is
modified and all other lines are passed through. Changing the verbosity level
argument for this command will change what it written into the status file.
Crucially, with the default verbosity level, no blank lines are being printed by
the opkg command and so no blank lines are being written to the status file to
separate each package entry.

The package parsing code in opkg expects package entries in the status file to
be separated by at least one blank line. If no blank line is seen, the next
package entry is interpreted as a continuation of the last package entry, but
the new values overwrite the old values.

So with the default verbosity level, a blank line follows some package entries
and these are parsed. The others are dropped due to the lack of blank lines. As
the verbosity increases, more debugging messages add blank lines and more
packages are parsed.

The solution to ensure that this works correctly regardless of the verbosity
level is simply add a blank line after the output of 'opkg info' is written to
the status file, ensuring that the next package is separated from the current
package.

[YOCTO #6816]

Signed-off-by: Paul Barker <paul@paulbarker.me.uk>
Cc: Chris Carr <chris.carr@ge.com>
---
 meta/lib/oe/package_manager.py | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
index ffb83b2..85d7fd4 100644
--- a/meta/lib/oe/package_manager.py
+++ b/meta/lib/oe/package_manager.py
@@ -1408,6 +1408,10 @@ class OpkgPM(PackageManager):
                     else:
                         status.write(line + "\n")
 
+                # Append a blank line after each package entry to ensure that it
+                # is separated from the following entry
+                status.write("\n")
+
     '''
     The following function dummy installs pkgs and returns the log of output.
     '''
-- 
2.1.2

Re: [PATCH] package_manager: Fix BAD_RECOMMENDATIONS for opkg

[Paul Barker]

On 26 October 2014 19:36, Paul Barker <paul@paulbarker.me.uk> wrote:
> In package_manager.py, when using opkg as the packager, the command 'opkg <args>
> info <pkg>' is called to get information about each pkg in BAD_RECOMMENDATIONS
> in a format that can be written to the status file. The 'Status: ...' line is
> modified and all other lines are passed through. Changing the verbosity level
> argument for this command will change what it written into the status file.
> Crucially, with the default verbosity level, no blank lines are being printed by
> the opkg command and so no blank lines are being written to the status file to
> separate each package entry.
>
> The package parsing code in opkg expects package entries in the status file to
> be separated by at least one blank line. If no blank line is seen, the next
> package entry is interpreted as a continuation of the last package entry, but
> the new values overwrite the old values.
>
> So with the default verbosity level, a blank line follows some package entries
> and these are parsed. The others are dropped due to the lack of blank lines. As
> the verbosity increases, more debugging messages add blank lines and more
> packages are parsed.
>
> The solution to ensure that this works correctly regardless of the verbosity
> level is simply add a blank line after the output of 'opkg info' is written to
> the status file, ensuring that the next package is separated from the current
> package.
>
> [YOCTO #6816]
>
> Signed-off-by: Paul Barker <paul@paulbarker.me.uk>
> Cc: Chris Carr <chris.carr@ge.com>
> ---
>  meta/lib/oe/package_manager.py | 4 ++++
>  1 file changed, 4 insertions(+)
>
> diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
> index ffb83b2..85d7fd4 100644
> --- a/meta/lib/oe/package_manager.py
> +++ b/meta/lib/oe/package_manager.py
> @@ -1408,6 +1408,10 @@ class OpkgPM(PackageManager):
>                      else:
>                          status.write(line + "\n")
>
> +                # Append a blank line after each package entry to ensure that it
> +                # is separated from the following entry
> +                status.write("\n")
> +
>      '''
>      The following function dummy installs pkgs and returns the log of output.
>      '''
> --
> 2.1.2
>

Chris, could you give this a test and let us know if it fixes the issue.

Thanks,

-- 
Paul Barker

Email: paul@paulbarker.me.uk
http://www.paulbarker.me.uk

Re: [PATCH] package_manager: Fix BAD_RECOMMENDATIONS for opkg

[Johan Hovold]

On Sun, Oct 26, 2014 at 07:37:19PM +0000, Paul Barker wrote:
> On 26 October 2014 19:36, Paul Barker <paul@paulbarker.me.uk> wrote:
> > In package_manager.py, when using opkg as the packager, the command 'opkg <args>
> > info <pkg>' is called to get information about each pkg in BAD_RECOMMENDATIONS
> > in a format that can be written to the status file. The 'Status: ...' line is
> > modified and all other lines are passed through. Changing the verbosity level
> > argument for this command will change what it written into the status file.
> > Crucially, with the default verbosity level, no blank lines are being printed by
> > the opkg command and so no blank lines are being written to the status file to
> > separate each package entry.
> >
> > The package parsing code in opkg expects package entries in the status file to
> > be separated by at least one blank line. If no blank line is seen, the next
> > package entry is interpreted as a continuation of the last package entry, but
> > the new values overwrite the old values.
> >
> > So with the default verbosity level, a blank line follows some package entries
> > and these are parsed. The others are dropped due to the lack of blank lines. As
> > the verbosity increases, more debugging messages add blank lines and more
> > packages are parsed.

I was bitten by this after a recent update to Yocto 1.6.1. Only one of
two bad recommendations would be held back and the order of the entries
would affect which package was installed.

Adding two more entries to BAD_RECOMMENDATIONS still only prevented one
of the packages from being installed.

I had a chance to look into this briefly last Friday, and could conclude
that all four entries ended up in the status file, but that only one
remained after the language packages had been installed (first run of
opkg).

> > The solution to ensure that this works correctly regardless of the verbosity
> > level is simply add a blank line after the output of 'opkg info' is written to
> > the status file, ensuring that the next package is separated from the current
> > package.
> >
> > [YOCTO #6816]
> >
> > Signed-off-by: Paul Barker <paul@paulbarker.me.uk>
> > Cc: Chris Carr <chris.carr@ge.com>

I just verified that this patch fixes the problem, so feel free to add:

Tested-by: Johan Hovold <johan@kernel.org>

Thanks,
Johan

> > ---
> >  meta/lib/oe/package_manager.py | 4 ++++
> >  1 file changed, 4 insertions(+)
> >
> > diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
> > index ffb83b2..85d7fd4 100644
> > --- a/meta/lib/oe/package_manager.py
> > +++ b/meta/lib/oe/package_manager.py
> > @@ -1408,6 +1408,10 @@ class OpkgPM(PackageManager):
> >                      else:
> >                          status.write(line + "\n")
> >
> > +                # Append a blank line after each package entry to ensure that it
> > +                # is separated from the following entry
> > +                status.write("\n")
> > +
> >      '''
> >      The following function dummy installs pkgs and returns the log of output.
> >      '''
> > --
> > 2.1.2
> >
> 
> Chris, could you give this a test and let us know if it fixes the issue.
> 
> Thanks,

Re: [PATCH] package_manager: Fix BAD_RECOMMENDATIONS for opkg

[Carr, Chris (GE Intelligent Platforms)]

On 26/10/2014 19:37, Paul Barker wrote:
> On 26 October 2014 19:36, Paul Barker <paul@paulbarker.me.uk> wrote:
>> In package_manager.py, when using opkg as the packager, the command 'opkg <args>
>> info <pkg>' is called to get information about each pkg in BAD_RECOMMENDATIONS
>> in a format that can be written to the status file. The 'Status: ...' line is
>> modified and all other lines are passed through. Changing the verbosity level
>> argument for this command will change what it written into the status file.
>> Crucially, with the default verbosity level, no blank lines are being printed by
>> the opkg command and so no blank lines are being written to the status file to
>> separate each package entry.
>>
>> The package parsing code in opkg expects package entries in the status file to
>> be separated by at least one blank line. If no blank line is seen, the next
>> package entry is interpreted as a continuation of the last package entry, but
>> the new values overwrite the old values.
>>
>> So with the default verbosity level, a blank line follows some package entries
>> and these are parsed. The others are dropped due to the lack of blank lines. As
>> the verbosity increases, more debugging messages add blank lines and more
>> packages are parsed.
>>
>> The solution to ensure that this works correctly regardless of the verbosity
>> level is simply add a blank line after the output of 'opkg info' is written to
>> the status file, ensuring that the next package is separated from the current
>> package.
>>
>> [YOCTO #6816]
>>
>> Signed-off-by: Paul Barker <paul@paulbarker.me.uk>
>> Cc: Chris Carr <chris.carr@ge.com>
>> ---
>>  meta/lib/oe/package_manager.py | 4 ++++
>>  1 file changed, 4 insertions(+)
>>
>> diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
>> index ffb83b2..85d7fd4 100644
>> --- a/meta/lib/oe/package_manager.py
>> +++ b/meta/lib/oe/package_manager.py
>> @@ -1408,6 +1408,10 @@ class OpkgPM(PackageManager):
>>                      else:
>>                          status.write(line + "\n")
>>
>> +                # Append a blank line after each package entry to ensure that it
>> +                # is separated from the following entry
>> +                status.write("\n")
>> +
>>      '''
>>      The following function dummy installs pkgs and returns the log of output.
>>      '''
>> --
>> 2.1.2
>>
> 
> Chris, could you give this a test and let us know if it fixes the issue.
> 
> Thanks,
> 

Paul,

I've tried the patch with my build and all the packages in BAD_RECOMMENDATIONS are now being processed correctly and do not appear in the target rootfs.

[My subscription to the OE Core mailing list had quietly expired, so I've re-subscribed to ensure that this reply goes to the list as well]

Chris Carr