[PATCH 0/2] check and convert file depends on bash, perl and python

[PATCH 0/2] check and convert file depends on bash, perl and python

[Robert Yang]

The following changes since commit fa325e44f5b429b4038022b31285af9c94672943:

  insane: Ensure do_package_qa happens after do_packagedata (2014-07-11 23:19:18 +0100)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib rbt/filedep
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=rbt/filedep

Robert Yang (2):
  insane.bbclass: QAPATHTEST: check file depends on bash, perl and
    python
  package.bbclass: convert file depends into pkg depends

 meta/classes/insane.bbclass  |   32 ++++++++++++++++++++++++++++++--
 meta/classes/package.bbclass |   14 ++++++++++++++
 2 files changed, 44 insertions(+), 2 deletions(-)

-- 
1.7.9.5

[PATCH 1/2] insane.bbclass: QAPATHTEST: check file depends on bash, perl and python

[Robert Yang]

If it is a bash, perl or python script, and the RDEPENDS_pkg doesn't
have bash, python or perl, then warn.

This is mainly for ipk and deb, since rpm can add "/usr/bin/perl" to the
its "Requires" section if it is a perl script, but ipk or deb can't
depend on file such as "/usr/bin/perl", they can only depend on pkg, so
they would know nothing about perl, then there would be depend issues.

This check can help us figure out the depends, it nearly has no effect
on performance when enabled.

[YOCTO #1662]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta/classes/insane.bbclass |   32 ++++++++++++++++++++++++++++++--
 1 file changed, 30 insertions(+), 2 deletions(-)

diff --git a/meta/classes/insane.bbclass b/meta/classes/insane.bbclass
index 106ace7..a14e668 100644
--- a/meta/classes/insane.bbclass
+++ b/meta/classes/insane.bbclass
@@ -19,7 +19,7 @@
 
 # unsafe-references-in-binaries requires prelink-rtld from
 # prelink-native, but we don't want this DEPENDS for -native builds
-QADEPENDS = "prelink-native"
+QADEPENDS = "prelink-native file-native"
 QADEPENDS_class-native = ""
 QADEPENDS_class-nativesdk = ""
 QA_SANE = "True"
@@ -29,7 +29,7 @@ QA_SANE = "True"
 WARN_QA ?= "ldflags useless-rpaths rpaths staticdev libdir xorg-driver-abi \
             textrel already-stripped incompatible-license files-invalid \
             installed-vs-shipped compile-host-path install-host-path \
-            pn-overrides infodir \
+            pn-overrides infodir file-depends \
             "
 ERROR_QA ?= "dev-so debug-deps dev-deps debug-files arch pkgconfig la \
             perms dep-cmp pkgvarcheck perm-config perm-line perm-link \
@@ -605,6 +605,34 @@ def package_qa_check_symlink_to_sysroot(path, name, d, elf, messages):
                 trimmed = path.replace(os.path.join (d.getVar("PKGDEST", True), name), "")
                 messages.append("Symlink %s in %s points to TMPDIR" % (trimmed, name))
 
+QAPATHTEST[file-depends] = "package_qa_check_file_depends"
+def package_qa_check_file_depends(path, name, d, elf, messages):
+    """
+    Check that the package doesn't miss the RDEPENDS on perl, python and bash
+    """
+
+    if not os.access(path, os.X_OK) or os.path.islink(path) or elf:
+        return
+
+    import subprocess
+    import re
+
+    # The file command may return:
+    # - Perl or perl, Python or python.
+    # - "Bourne-Again shell" script or "bash" for bash script
+    prog_maps = {"[Pp]erl": "perl", "[Pp]ython": "python", "bash": "bash",
+                "Bourne-Again shell": "bash"}
+
+    file_output = bb.process.Popen(["file", "-b", path], stdout=subprocess.PIPE).stdout.read()
+    for p in prog_maps:
+        if re.search('%s .*script, ASCII text executable' % p, file_output):
+            rdepends = bb.utils.explode_deps(d.getVar('RDEPENDS_' + name, True) or "")
+            if prog_maps[p] not in rdepends:
+                if prog_maps[p] == "python" and "python-core" in rdepends:
+                    continue
+                messages.append("%s is a %s script, but %s is not in RDEPENDS_%s" % (path, prog_maps[p], prog_maps[p], name))
+                break
+
 def package_qa_check_license(workdir, d):
     """
     Check for changes in the license files 
-- 
1.7.9.5

Re: [PATCH 1/2] insane.bbclass: QAPATHTEST: check file depends on bash, perl and python

[Mark Hatle]

On 7/14/14, 4:41 AM, Robert Yang wrote:
> If it is a bash, perl or python script, and the RDEPENDS_pkg doesn't
> have bash, python or perl, then warn.
>
> This is mainly for ipk and deb, since rpm can add "/usr/bin/perl" to the
> its "Requires" section if it is a perl script, but ipk or deb can't
> depend on file such as "/usr/bin/perl", they can only depend on pkg, so
> they would know nothing about perl, then there would be depend issues.
>
> This check can help us figure out the depends, it nearly has no effect
> on performance when enabled.
>
> [YOCTO #1662]
>
> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> ---
>   meta/classes/insane.bbclass |   32 ++++++++++++++++++++++++++++++--
>   1 file changed, 30 insertions(+), 2 deletions(-)
>
> diff --git a/meta/classes/insane.bbclass b/meta/classes/insane.bbclass
> index 106ace7..a14e668 100644
> --- a/meta/classes/insane.bbclass
> +++ b/meta/classes/insane.bbclass
> @@ -19,7 +19,7 @@
>
>   # unsafe-references-in-binaries requires prelink-rtld from
>   # prelink-native, but we don't want this DEPENDS for -native builds
> -QADEPENDS = "prelink-native"
> +QADEPENDS = "prelink-native file-native"

If the perfile dependency processing is enabled, run in: package_do_filedeps

The system will already have gathered the information needed for the QA 
processing.  You can look at:

# Collect perfile run-time dependency metadata
# Output:
#  FILERPROVIDESFLIST_pkg - list of all files w/ deps
#  FILERPROVIDES_filepath_pkg - per file dep
#
#  FILERDEPENDSFLIST_pkg - list of all files w/ deps
#  FILERDEPENDS_filepath_pkg - per file dep

variables for the per-file dependencies and then warn if a dependency is not 
already declared in the manual RDEPENDS -- and/or promote them to real 
dependencies automatically.

This should be faster then iterating through and calling file directly.

--Mark

>   QADEPENDS_class-native = ""
>   QADEPENDS_class-nativesdk = ""
>   QA_SANE = "True"
> @@ -29,7 +29,7 @@ QA_SANE = "True"
>   WARN_QA ?= "ldflags useless-rpaths rpaths staticdev libdir xorg-driver-abi \
>               textrel already-stripped incompatible-license files-invalid \
>               installed-vs-shipped compile-host-path install-host-path \
> -            pn-overrides infodir \
> +            pn-overrides infodir file-depends \
>               "
>   ERROR_QA ?= "dev-so debug-deps dev-deps debug-files arch pkgconfig la \
>               perms dep-cmp pkgvarcheck perm-config perm-line perm-link \
> @@ -605,6 +605,34 @@ def package_qa_check_symlink_to_sysroot(path, name, d, elf, messages):
>                   trimmed = path.replace(os.path.join (d.getVar("PKGDEST", True), name), "")
>                   messages.append("Symlink %s in %s points to TMPDIR" % (trimmed, name))
>
> +QAPATHTEST[file-depends] = "package_qa_check_file_depends"
> +def package_qa_check_file_depends(path, name, d, elf, messages):
> +    """
> +    Check that the package doesn't miss the RDEPENDS on perl, python and bash
> +    """
> +
> +    if not os.access(path, os.X_OK) or os.path.islink(path) or elf:
> +        return
> +
> +    import subprocess
> +    import re
> +
> +    # The file command may return:
> +    # - Perl or perl, Python or python.
> +    # - "Bourne-Again shell" script or "bash" for bash script
> +    prog_maps = {"[Pp]erl": "perl", "[Pp]ython": "python", "bash": "bash",
> +                "Bourne-Again shell": "bash"}
> +
> +    file_output = bb.process.Popen(["file", "-b", path], stdout=subprocess.PIPE).stdout.read()
> +    for p in prog_maps:
> +        if re.search('%s .*script, ASCII text executable' % p, file_output):
> +            rdepends = bb.utils.explode_deps(d.getVar('RDEPENDS_' + name, True) or "")
> +            if prog_maps[p] not in rdepends:
> +                if prog_maps[p] == "python" and "python-core" in rdepends:
> +                    continue
> +                messages.append("%s is a %s script, but %s is not in RDEPENDS_%s" % (path, prog_maps[p], prog_maps[p], name))
> +                break
> +
>   def package_qa_check_license(workdir, d):
>       """
>       Check for changes in the license files
>

Re: [PATCH 1/2] insane.bbclass: QAPATHTEST: check file depends on bash, perl and python

[Robert Yang]

On 07/15/2014 04:02 AM, Mark Hatle wrote:
> On 7/14/14, 4:41 AM, Robert Yang wrote:
>> If it is a bash, perl or python script, and the RDEPENDS_pkg doesn't
>> have bash, python or perl, then warn.
>>
>> This is mainly for ipk and deb, since rpm can add "/usr/bin/perl" to the
>> its "Requires" section if it is a perl script, but ipk or deb can't
>> depend on file such as "/usr/bin/perl", they can only depend on pkg, so
>> they would know nothing about perl, then there would be depend issues.
>>
>> This check can help us figure out the depends, it nearly has no effect
>> on performance when enabled.
>>
>> [YOCTO #1662]
>>
>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>> ---
>>   meta/classes/insane.bbclass |   32 ++++++++++++++++++++++++++++++--
>>   1 file changed, 30 insertions(+), 2 deletions(-)
>>
>> diff --git a/meta/classes/insane.bbclass b/meta/classes/insane.bbclass
>> index 106ace7..a14e668 100644
>> --- a/meta/classes/insane.bbclass
>> +++ b/meta/classes/insane.bbclass
>> @@ -19,7 +19,7 @@
>>
>>   # unsafe-references-in-binaries requires prelink-rtld from
>>   # prelink-native, but we don't want this DEPENDS for -native builds
>> -QADEPENDS = "prelink-native"
>> +QADEPENDS = "prelink-native file-native"
>
> If the perfile dependency processing is enabled, run in: package_do_filedeps
>
> The system will already have gathered the information needed for the QA
> processing.  You can look at:
>
> # Collect perfile run-time dependency metadata
> # Output:
> #  FILERPROVIDESFLIST_pkg - list of all files w/ deps
> #  FILERPROVIDES_filepath_pkg - per file dep
> #
> #  FILERDEPENDSFLIST_pkg - list of all files w/ deps
> #  FILERDEPENDS_filepath_pkg - per file dep
>
> variables for the per-file dependencies and then warn if a dependency is not
> already declared in the manual RDEPENDS -- and/or promote them to real
> dependencies automatically.
>
> This should be faster then iterating through and calling file directly.
>

Thanks, I will only warn from package_do_filedeps and let the user add the
RDEPENDS manually, I will also fix a few of the RDEPENDS in V2.

Please feel free to let me know if others have any suggestions.

// Robert


> --Mark
>
>>   QADEPENDS_class-native = ""
>>   QADEPENDS_class-nativesdk = ""
>>   QA_SANE = "True"
>> @@ -29,7 +29,7 @@ QA_SANE = "True"
>>   WARN_QA ?= "ldflags useless-rpaths rpaths staticdev libdir xorg-driver-abi \
>>               textrel already-stripped incompatible-license files-invalid \
>>               installed-vs-shipped compile-host-path install-host-path \
>> -            pn-overrides infodir \
>> +            pn-overrides infodir file-depends \
>>               "
>>   ERROR_QA ?= "dev-so debug-deps dev-deps debug-files arch pkgconfig la \
>>               perms dep-cmp pkgvarcheck perm-config perm-line perm-link \
>> @@ -605,6 +605,34 @@ def package_qa_check_symlink_to_sysroot(path, name, d,
>> elf, messages):
>>                   trimmed = path.replace(os.path.join (d.getVar("PKGDEST",
>> True), name), "")
>>                   messages.append("Symlink %s in %s points to TMPDIR" %
>> (trimmed, name))
>>
>> +QAPATHTEST[file-depends] = "package_qa_check_file_depends"
>> +def package_qa_check_file_depends(path, name, d, elf, messages):
>> +    """
>> +    Check that the package doesn't miss the RDEPENDS on perl, python and bash
>> +    """
>> +
>> +    if not os.access(path, os.X_OK) or os.path.islink(path) or elf:
>> +        return
>> +
>> +    import subprocess
>> +    import re
>> +
>> +    # The file command may return:
>> +    # - Perl or perl, Python or python.
>> +    # - "Bourne-Again shell" script or "bash" for bash script
>> +    prog_maps = {"[Pp]erl": "perl", "[Pp]ython": "python", "bash": "bash",
>> +                "Bourne-Again shell": "bash"}
>> +
>> +    file_output = bb.process.Popen(["file", "-b", path],
>> stdout=subprocess.PIPE).stdout.read()
>> +    for p in prog_maps:
>> +        if re.search('%s .*script, ASCII text executable' % p, file_output):
>> +            rdepends = bb.utils.explode_deps(d.getVar('RDEPENDS_' + name,
>> True) or "")
>> +            if prog_maps[p] not in rdepends:
>> +                if prog_maps[p] == "python" and "python-core" in rdepends:
>> +                    continue
>> +                messages.append("%s is a %s script, but %s is not in
>> RDEPENDS_%s" % (path, prog_maps[p], prog_maps[p], name))
>> +                break
>> +
>>   def package_qa_check_license(workdir, d):
>>       """
>>       Check for changes in the license files
>>
>

[PATCH 2/2] package.bbclass: convert file depends into pkg depends

[Robert Yang]

The ipk or deb can't depend on file such as "/usr/bin/perl", so it would
know nothing about perl, and there would be dependencies problems when
we run "apt-get install" on the target. Convert the file depends into
pkg depends can fix the issue:

/bin/bash -> bash
/usr/bin/perl -> perl
/usr/bin/python -> python

NOTE, we still need the QA check package_qa_check_file_depends though we
have this fix, manually add the pkg to RDEPENDS is correct, anyway.

TODO:
- Add a global var for the paths, such as BASH_PATH = "/bin/bash" rather than
  directly use /bin/bash ?

[YOCTO #1662]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta/classes/package.bbclass |   14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index 26a20d1..419cdcf 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -1318,6 +1318,9 @@ python package_do_filedeps() {
     pool.close()
     pool.join()
 
+    # Used for converting file depends into pkg depends
+    prog_maps = {"/bin/bash": "bash", "/usr/bin/perl": "perl", "/usr/bin/python": "python"}
+
     provides_files = {}
     requires_files = {}
 
@@ -1334,10 +1337,21 @@ python package_do_filedeps() {
             key = "FILERPROVIDES_" + file + "_" + pkg
             d.setVar(key, " ".join(provides[file]))
 
+        req_file_pkgs = []
         for file in requires:
             requires_files[pkg].append(file)
             key = "FILERDEPENDS_" + file + "_" + pkg
             d.setVar(key, " ".join(requires[file]))
+            for req_file in requires[file]:
+                if req_file in prog_maps and prog_maps[req_file] not in req_file_pkgs:
+                    req_file_pkgs.append(prog_maps[req_file])
+
+        if req_file_pkgs:
+            rdepends = bb.utils.explode_deps(d.getVar('RDEPENDS_' + pkg, True) or "")
+            for p in req_file_pkgs:
+                if p not in rdepends:
+                    bb.note("Adding %s to RDEPENDS_%s" % (p, pkg))
+                    d.appendVar("RDEPENDS_" + pkg, ' ' + p)
 
     for pkg in requires_files:
         d.setVar("FILERDEPENDSFLIST_" + pkg, " ".join(requires_files[pkg]))
-- 
1.7.9.5

Re: [PATCH 2/2] package.bbclass: convert file depends into pkg depends

[Burton, Ross]

On 14 July 2014 10:41, Robert Yang <liezhi.yang@windriver.com> wrote:
> +    prog_maps = {"/bin/bash": "bash", "/usr/bin/perl": "perl", "/usr/bin/python": "python"}

Use something like d.expand("${bindir}/perl") instead of hard-coding paths.

Ross

Re: [PATCH 2/2] package.bbclass: convert file depends into pkg depends

[Robert Yang]

On 07/14/2014 06:29 PM, Burton, Ross wrote:
> On 14 July 2014 10:41, Robert Yang <liezhi.yang@windriver.com> wrote:
>> +    prog_maps = {"/bin/bash": "bash", "/usr/bin/perl": "perl", "/usr/bin/python": "python"}
>
> Use something like d.expand("${bindir}/perl") instead of hard-coding paths.

Good idea, thanks, will update it sooner.

// Robert

>
> Ross
>
>

Re: [PATCH 2/2] package.bbclass: convert file depends into pkg depends

[Robert Yang]

Hello,

Found that eglibc-utils/usr/bin/tzselect is a bash script, should we
add:

RDEPENDS_eglibc-utils += "bash" ?

The problem is that this would cause bash installed to meta-toolchain,
the dependencies chain is:

meta-toolchain -> eglibc-utils -> bash

Maybe we should only warn rather than add the RDEPENDS atm, please ?

// Robert

On 07/14/2014 06:29 PM, Burton, Ross wrote:
> On 14 July 2014 10:41, Robert Yang <liezhi.yang@windriver.com> wrote:
>> +    prog_maps = {"/bin/bash": "bash", "/usr/bin/perl": "perl", "/usr/bin/python": "python"}
>
> Use something like d.expand("${bindir}/perl") instead of hard-coding paths.
>
> Ross
>
>

Re: [PATCH 2/2] package.bbclass: convert file depends into pkg depends

[Mark Hatle]

On 7/14/14, 11:25 AM, Robert Yang wrote:
>
> Hello,
>
> Found that eglibc-utils/usr/bin/tzselect is a bash script, should we
> add:
>
> RDEPENDS_eglibc-utils += "bash" ?
>
> The problem is that this would cause bash installed to meta-toolchain,
> the dependencies chain is:
>
> meta-toolchain -> eglibc-utils -> bash
>
> Maybe we should only warn rather than add the RDEPENDS atm, please ?

Hmm at one point I thought someone has re-written tzselect as posix sh, to avoid 
the bash dependency.

I'd actually suggest this be a defect and we try to find someone who can fix it. 
  (I'd like to eliminate bash dependencies in anything core to a tiny system.)

--Mark

> // Robert
>
> On 07/14/2014 06:29 PM, Burton, Ross wrote:
>> On 14 July 2014 10:41, Robert Yang <liezhi.yang@windriver.com> wrote:
>>> +    prog_maps = {"/bin/bash": "bash", "/usr/bin/perl": "perl", "/usr/bin/python": "python"}
>>
>> Use something like d.expand("${bindir}/perl") instead of hard-coding paths.
>>
>> Ross
>>
>>