Re: [oe-commits] [openembedded-core] 01/02: package.bbclass: Restore functionality to detect RPM dependencies

* Re: [oe-commits] [openembedded-core] 01/02: package.bbclass: Restore functionality to detect RPM dependencies
       [not found] ` <20170816164450.1447E236566@git.openembedded.org>
@ 2017-08-21  9:00   ` Martin Jansa
  2017-08-21 20:56     ` [PATCH] libcheck: fix file-rdeps QA issue Martin Jansa
  2017-08-21 21:08     ` [oe-commits] [openembedded-core] 01/02: package.bbclass: Restore functionality to detect RPM dependencies Richard Purdie
  0 siblings, 2 replies; 3+ messages in thread
From: Martin Jansa @ 2017-08-21  9:00 UTC (permalink / raw)
  To: Peter Kjellerstedt,
	Patches and discussions about the oe-core layer
  Cc: openembedded-commits@lists.openembedded.org

[-- Attachment #1: Type: text/plain, Size: 8419 bytes --]

This seems to find a many more missed dependencies which is good thing.

There is +- 20 failures found in meta-oe recipes and there is still at
least one in oe-core:

ERROR: nativesdk-libcheck-0.10.0-r0 do_package_qa: QA Issue:
/usr/local/oecore-x86_64/sysroots/x86_64-oesdk-linux/usr/bin/checkmk
contained in package nativesdk-libcheck requires
/usr/local/oecore-x86_64/sysroots/x86_64-oesdk-linux/usr/bin/gawk, but no
providers found in RDEPENDS_nativesdk-libcheck? [file-rdeps]

I wonder why this one wasn't catched by oe-core autobuilder.

On Wed, Aug 16, 2017 at 6:44 PM, <git@git.openembedded.org> wrote:

> This is an automated email from the git hooks/post-receive script.
>
> rpurdie pushed a commit to branch master-next
> in repository openembedded-core.
>
> commit bf017930036f19b3d6df8e5b50d9979ee7045c5c
> Author: Peter Kjellerstedt <pkj@axis.com>
> AuthorDate: Tue Aug 15 16:41:56 2017 -0500
>
>     package.bbclass: Restore functionality to detect RPM dependencies
>
>     During the transition to dnf and rpm4, the functionality to
>     automatically make RPM determine dependencies was lost.
>
>     Before the transition, an OE specific tool called rpmdeps-oecore had
>     been added to the rpm suit. It was based on the rpmdeps tool that is
>     part of rpm. For each file specified on its command line, it would
>     output the provides and requires that RPM could determine.
>
>     During the transition to rpm4, rpmdeps-oecore was replaced with the
>     standard rpmdeps. However, what no one noticed was that unless rpmdeps
>     is given options, e.g., -P or -R, to tell it what it should output, it
>     will not output anything. Thus, it would do all the work to determine
>     the requirements, but would keep silent about it. And since no output
>     from rpmdeps is expected unless there are requirements, there were no
>     warnings indicating that everything was not working as expected.
>
>     Porting the old rpmdeps-oecore to work with rpm4 is not really
>     possible since it relied on being able to access internals of RPM that
>     are no longer available. However, it turned out that rpmdeps had a
>     debug option, --rpmfcdebug, that would output exactly the information
>     that we need, albeit in a different format and to stderr. To make this
>     usable, rpmdeps has now received a new option, --alldeps, which sends
>     the information we need to stdout.
>
>     (From OE-Core rev: 958501b3d9201aaabb81ec644c6049e0c9b737e7)
>
>     Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
>     Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
>     Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
>  meta/classes/package.bbclass |  5 ++--
>  meta/lib/oe/package.py       | 60 ++++++++++++++++++++++++++++++
> +-------------
>  2 files changed, 44 insertions(+), 21 deletions(-)
>
> diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
> index d2fa617..2fe30da 100644
> --- a/meta/classes/package.bbclass
> +++ b/meta/classes/package.bbclass
> @@ -1453,7 +1453,7 @@ if [ x"$D" = "x" ]; then
>  fi
>  }
>
> -RPMDEPS = "${STAGING_LIBDIR_NATIVE}/rpm/rpmdeps --rcfile
> ${STAGING_LIBDIR_NATIVE}/rpm/rpmrc  --macros ${STAGING_LIBDIR_NATIVE}/rpm/macros
> --define '_rpmconfigdir ${STAGING_LIBDIR_NATIVE}/rpm/'"
> +RPMDEPS = "${STAGING_LIBDIR_NATIVE}/rpm/rpmdeps --alldeps"
>
>  # Collect perfile run-time dependency metadata
>  # Output:
> @@ -1470,7 +1470,6 @@ python package_do_filedeps() {
>      pkgdest = d.getVar('PKGDEST')
>      packages = d.getVar('PACKAGES')
>      rpmdeps = d.getVar('RPMDEPS')
> -    magic = d.expand("${STAGING_DIR_NATIVE}${datadir_native}/misc/
> magic.mgc")
>
>      def chunks(files, n):
>          return [files[i:i+n] for i in range(0, len(files), n)]
> @@ -1482,7 +1481,7 @@ python package_do_filedeps() {
>          if pkg.endswith('-dbg') or pkg.endswith('-doc') or
> pkg.find('-locale-') != -1 or pkg.find('-localedata-') != -1 or
> pkg.find('-gconv-') != -1 or pkg.find('-charmap-') != -1 or
> pkg.startswith('kernel-module-') or pkg.endswith('-src'):
>              continue
>          for files in chunks(pkgfiles[pkg], 100):
> -            pkglist.append((pkg, files, rpmdeps, pkgdest, magic))
> +            pkglist.append((pkg, files, rpmdeps, pkgdest))
>
>      processed = oe.utils.multiprocess_exec( pkglist,
> oe.package.filedeprunner)
>
> diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py
> index 43748b2..a79c668 100644
> --- a/meta/lib/oe/package.py
> +++ b/meta/lib/oe/package.py
> @@ -162,44 +162,68 @@ def file_translate(file):
>  def filedeprunner(arg):
>      import re, subprocess, shlex
>
> -    (pkg, pkgfiles, rpmdeps, pkgdest, magic) = arg
> +    (pkg, pkgfiles, rpmdeps, pkgdest) = arg
>      provides = {}
>      requires = {}
>
> -    r = re.compile(r'[<>=]+ +[^ ]*')
> +    file_re = re.compile(r'\s+\d+\s(.*)')
> +    dep_re = re.compile(r'\s+(\S)\s+(.*)')
> +    r = re.compile(r'[<>=]+\s+\S*')
>
>      def process_deps(pipe, pkg, pkgdest, provides, requires):
> +        file = None
>          for line in pipe:
> -            f = line.decode("utf-8").split(" ", 1)[0].strip()
> -            line = line.decode("utf-8").split(" ", 1)[1].strip()
> +            line = line.decode("utf-8")
>
> -            if line.startswith("Requires:"):
> +            m = file_re.match(line)
> +            if m:
> +                file = m.group(1)
> +                file = file.replace(pkgdest + "/" + pkg, "")
> +                file = file_translate(file)
> +                continue
> +
> +            m = dep_re.match(line)
> +            if not m or not file:
> +                continue
> +
> +            type, dep = m.groups()
> +
> +            if type == 'R':
>                  i = requires
> -            elif line.startswith("Provides:"):
> +            elif type == 'P':
>                  i = provides
>              else:
> -                continue
> +               continue
>
> -            file = f.replace(pkgdest + "/" + pkg, "")
> -            file = file_translate(file)
> -            value = line.split(":", 1)[1].strip()
> -            value = r.sub(r'(\g<0>)', value)
> +            if dep.startswith("python("):
> +                continue
>
> -            if value.startswith("rpmlib("):
> +            # Ignore all perl(VMS::...) and perl(Mac::...) dependencies.
> These
> +            # are typically used conditionally from the Perl code, but are
> +            # generated as unconditional dependencies.
> +            if dep.startswith('perl(VMS::') or
> dep.startswith('perl(Mac::'):
>                  continue
> -            if value == "python":
> +
> +            # Ignore perl dependencies on .pl files.
> +            if dep.startswith('perl(') and dep.endswith('.pl)'):
>                  continue
> +
> +            # Remove perl versions and perl module versions since they
> typically
> +            # do not make sense when used as package versions.
> +            if dep.startswith('perl') and r.search(dep):
> +                dep = dep.split()[0]
> +
> +            # Put parentheses around any version specifications.
> +            dep = r.sub(r'(\g<0>)',dep)
> +
>              if file not in i:
>                  i[file] = []
> -            i[file].append(value)
> +            i[file].append(dep)
>
>          return provides, requires
>
> -    env = os.environ.copy()
> -    env["MAGIC"] = magic
> -
>      try:
> -        dep_popen = subprocess.Popen(shlex.split(rpmdeps) + pkgfiles,
> stdout=subprocess.PIPE, env=env)
> +        dep_popen = subprocess.Popen(shlex.split(rpmdeps) + pkgfiles,
> stdout=subprocess.PIPE)
>          provides, requires = process_deps(dep_popen.stdout, pkg, pkgdest,
> provides, requires)
>      except OSError as e:
>          bb.error("rpmdeps: '%s' command failed, '%s'" %
> (shlex.split(rpmdeps) + pkgfiles, e))
>
> --
> To stop receiving notification emails like this one, please contact
> the administrator of this repository.
> --
> _______________________________________________
> Openembedded-commits mailing list
> Openembedded-commits@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-commits
>

[-- Attachment #2: Type: text/html, Size: 10615 bytes --]

^ permalink raw reply	[flat|nested] 3+ messages in thread