[PATCH v3] lib/oe/utils: add eol to format_pkg_list()

[PATCH v3] lib/oe/utils: add eol to format_pkg_list()

[grygorii tertychnyi]

Append '\n' to the non-empty formatted string before return. If you
write it to the (manifest) file, it will ensure file ends with a newline.

Many GNU utilities have problems processing the last line of a file
if it is not '\n' terminated. E.g. if the last line is not terminated
by a newline character, then "read" will read it but return false,
leaving the broken partial line in the read variable(s).
It can also break or adversely affect some text processing tools,
that operate on the file.

Signed-off-by: grygorii tertychnyi <gtertych@cisco.com>
---

Changes in v3:
 o write_image_manifest(): remove extra '\n'

PASS bitbake buildtools-tarball
PASS bitbake core-image-minimal                                                                                              
PASS bitbake core-image-sato -c do_populate_sdk_ext
PASS oe-selftest --run-tests manifest.VerifyManifest.test_image_manifest_entries

 meta/classes/rootfs-postcommands.bbclass | 1 -
 meta/lib/oe/utils.py                     | 8 +++++++-
 2 files changed, 7 insertions(+), 2 deletions(-)

diff --git a/meta/classes/rootfs-postcommands.bbclass b/meta/classes/rootfs-postcommands.bbclass
index e816824f28..bde58ad6cd 100644
--- a/meta/classes/rootfs-postcommands.bbclass
+++ b/meta/classes/rootfs-postcommands.bbclass
@@ -253,7 +253,6 @@ python write_image_manifest () {
     pkgs = image_list_installed_packages(d)
     with open(manifest_name, 'w+') as image_manifest:
         image_manifest.write(format_pkg_list(pkgs, "ver"))
-        image_manifest.write("\n")
 
     if os.path.exists(manifest_name):
         manifest_link = deploy_dir + "/" + link_name + ".manifest"
diff --git a/meta/lib/oe/utils.py b/meta/lib/oe/utils.py
index 93b0763b0a..d05f517a70 100644
--- a/meta/lib/oe/utils.py
+++ b/meta/lib/oe/utils.py
@@ -347,7 +347,13 @@ def format_pkg_list(pkg_dict, ret_format=None):
         for pkg in sorted(pkg_dict):
             output.append(pkg)
 
-    return '\n'.join(output)
+    output_str = '\n'.join(output)
+
+    if output_str:
+        # make sure last line is newline terminated
+        output_str += '\n'
+
+    return output_str
 
 def host_gcc_version(d, taskcontextonly=False):
     import re, subprocess
-- 
2.19.0

Re: [PATCH v3] lib/oe/utils: add eol to format_pkg_list()

[akuster808]

Grygorii,


On 10/10/2018 09:26 AM, grygorii tertychnyi via Openembedded-core wrote:
> Append '\n' to the non-empty formatted string before return. If you
> write it to the (manifest) file, it will ensure file ends with a newline.
>
> Many GNU utilities have problems processing the last line of a file
> if it is not '\n' terminated. E.g. if the last line is not terminated
> by a newline character, then "read" will read it but return false,
> leaving the broken partial line in the read variable(s).
> It can also break or adversely affect some text processing tools,
> that operate on the file.

is this needed for the cve patches?

- armin
> Signed-off-by: grygorii tertychnyi <gtertych@cisco.com>
> ---
>
> Changes in v3:
>  o write_image_manifest(): remove extra '\n'
>
> PASS bitbake buildtools-tarball
> PASS bitbake core-image-minimal                                                                                              
> PASS bitbake core-image-sato -c do_populate_sdk_ext
> PASS oe-selftest --run-tests manifest.VerifyManifest.test_image_manifest_entries
>
>  meta/classes/rootfs-postcommands.bbclass | 1 -
>  meta/lib/oe/utils.py                     | 8 +++++++-
>  2 files changed, 7 insertions(+), 2 deletions(-)
>
> diff --git a/meta/classes/rootfs-postcommands.bbclass b/meta/classes/rootfs-postcommands.bbclass
> index e816824f28..bde58ad6cd 100644
> --- a/meta/classes/rootfs-postcommands.bbclass
> +++ b/meta/classes/rootfs-postcommands.bbclass
> @@ -253,7 +253,6 @@ python write_image_manifest () {
>      pkgs = image_list_installed_packages(d)
>      with open(manifest_name, 'w+') as image_manifest:
>          image_manifest.write(format_pkg_list(pkgs, "ver"))
> -        image_manifest.write("\n")
>  
>      if os.path.exists(manifest_name):
>          manifest_link = deploy_dir + "/" + link_name + ".manifest"
> diff --git a/meta/lib/oe/utils.py b/meta/lib/oe/utils.py
> index 93b0763b0a..d05f517a70 100644
> --- a/meta/lib/oe/utils.py
> +++ b/meta/lib/oe/utils.py
> @@ -347,7 +347,13 @@ def format_pkg_list(pkg_dict, ret_format=None):
>          for pkg in sorted(pkg_dict):
>              output.append(pkg)
>  
> -    return '\n'.join(output)
> +    output_str = '\n'.join(output)
> +
> +    if output_str:
> +        # make sure last line is newline terminated
> +        output_str += '\n'
> +
> +    return output_str
>  
>  def host_gcc_version(d, taskcontextonly=False):
>      import re, subprocess

Re: [PATCH v3] lib/oe/utils: add eol to format_pkg_list()

[Grygorii Tertychnyi]

akuster808 writes:

> Grygorii,
>
>
> On 10/10/2018 09:26 AM, grygorii tertychnyi via Openembedded-core wrote:
>> Append '\n' to the non-empty formatted string before return. If you
>> write it to the (manifest) file, it will ensure file ends with a newline.
>>
>> Many GNU utilities have problems processing the last line of a file
>> if it is not '\n' terminated. E.g. if the last line is not terminated
>> by a newline character, then "read" will read it but return false,
>> leaving the broken partial line in the read variable(s).
>> It can also break or adversely affect some text processing tools,
>> that operate on the file.
>
> is this needed for the cve patches?

No

>
> - armin
>> Signed-off-by: grygorii tertychnyi <gtertych@cisco.com>
>> ---
>>
>> Changes in v3:
>>  o write_image_manifest(): remove extra '\n'
>>
>> PASS bitbake buildtools-tarball
>> PASS bitbake core-image-minimal                                                                                              
>> PASS bitbake core-image-sato -c do_populate_sdk_ext
>> PASS oe-selftest --run-tests manifest.VerifyManifest.test_image_manifest_entries
>>
>>  meta/classes/rootfs-postcommands.bbclass | 1 -
>>  meta/lib/oe/utils.py                     | 8 +++++++-
>>  2 files changed, 7 insertions(+), 2 deletions(-)
>>
>> diff --git a/meta/classes/rootfs-postcommands.bbclass b/meta/classes/rootfs-postcommands.bbclass
>> index e816824f28..bde58ad6cd 100644
>> --- a/meta/classes/rootfs-postcommands.bbclass
>> +++ b/meta/classes/rootfs-postcommands.bbclass
>> @@ -253,7 +253,6 @@ python write_image_manifest () {
>>      pkgs = image_list_installed_packages(d)
>>      with open(manifest_name, 'w+') as image_manifest:
>>          image_manifest.write(format_pkg_list(pkgs, "ver"))
>> -        image_manifest.write("\n")
>>  
>>      if os.path.exists(manifest_name):
>>          manifest_link = deploy_dir + "/" + link_name + ".manifest"
>> diff --git a/meta/lib/oe/utils.py b/meta/lib/oe/utils.py
>> index 93b0763b0a..d05f517a70 100644
>> --- a/meta/lib/oe/utils.py
>> +++ b/meta/lib/oe/utils.py
>> @@ -347,7 +347,13 @@ def format_pkg_list(pkg_dict, ret_format=None):
>>          for pkg in sorted(pkg_dict):
>>              output.append(pkg)
>>  
>> -    return '\n'.join(output)
>> +    output_str = '\n'.join(output)
>> +
>> +    if output_str:
>> +        # make sure last line is newline terminated
>> +        output_str += '\n'
>> +
>> +    return output_str
>>  
>>  def host_gcc_version(d, taskcontextonly=False):
>>      import re, subprocess