* Re: [oe-commits] [openembedded-core] 01/02: package.bbclass: Restore functionality to detect RPM dependencies
[not found] ` <20170816164450.1447E236566@git.openembedded.org>
@ 2017-08-21 9:00 ` Martin Jansa
2017-08-21 20:56 ` [PATCH] libcheck: fix file-rdeps QA issue Martin Jansa
2017-08-21 21:08 ` [oe-commits] [openembedded-core] 01/02: package.bbclass: Restore functionality to detect RPM dependencies Richard Purdie
0 siblings, 2 replies; 3+ messages in thread
From: Martin Jansa @ 2017-08-21 9:00 UTC (permalink / raw)
To: Peter Kjellerstedt,
Patches and discussions about the oe-core layer
Cc: openembedded-commits@lists.openembedded.org
[-- Attachment #1: Type: text/plain, Size: 8419 bytes --]
This seems to find a many more missed dependencies which is good thing.
There is +- 20 failures found in meta-oe recipes and there is still at
least one in oe-core:
ERROR: nativesdk-libcheck-0.10.0-r0 do_package_qa: QA Issue:
/usr/local/oecore-x86_64/sysroots/x86_64-oesdk-linux/usr/bin/checkmk
contained in package nativesdk-libcheck requires
/usr/local/oecore-x86_64/sysroots/x86_64-oesdk-linux/usr/bin/gawk, but no
providers found in RDEPENDS_nativesdk-libcheck? [file-rdeps]
I wonder why this one wasn't catched by oe-core autobuilder.
On Wed, Aug 16, 2017 at 6:44 PM, <git@git.openembedded.org> wrote:
> This is an automated email from the git hooks/post-receive script.
>
> rpurdie pushed a commit to branch master-next
> in repository openembedded-core.
>
> commit bf017930036f19b3d6df8e5b50d9979ee7045c5c
> Author: Peter Kjellerstedt <pkj@axis.com>
> AuthorDate: Tue Aug 15 16:41:56 2017 -0500
>
> package.bbclass: Restore functionality to detect RPM dependencies
>
> During the transition to dnf and rpm4, the functionality to
> automatically make RPM determine dependencies was lost.
>
> Before the transition, an OE specific tool called rpmdeps-oecore had
> been added to the rpm suit. It was based on the rpmdeps tool that is
> part of rpm. For each file specified on its command line, it would
> output the provides and requires that RPM could determine.
>
> During the transition to rpm4, rpmdeps-oecore was replaced with the
> standard rpmdeps. However, what no one noticed was that unless rpmdeps
> is given options, e.g., -P or -R, to tell it what it should output, it
> will not output anything. Thus, it would do all the work to determine
> the requirements, but would keep silent about it. And since no output
> from rpmdeps is expected unless there are requirements, there were no
> warnings indicating that everything was not working as expected.
>
> Porting the old rpmdeps-oecore to work with rpm4 is not really
> possible since it relied on being able to access internals of RPM that
> are no longer available. However, it turned out that rpmdeps had a
> debug option, --rpmfcdebug, that would output exactly the information
> that we need, albeit in a different format and to stderr. To make this
> usable, rpmdeps has now received a new option, --alldeps, which sends
> the information we need to stdout.
>
> (From OE-Core rev: 958501b3d9201aaabb81ec644c6049e0c9b737e7)
>
> Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
> Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
> meta/classes/package.bbclass | 5 ++--
> meta/lib/oe/package.py | 60 ++++++++++++++++++++++++++++++
> +-------------
> 2 files changed, 44 insertions(+), 21 deletions(-)
>
> diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
> index d2fa617..2fe30da 100644
> --- a/meta/classes/package.bbclass
> +++ b/meta/classes/package.bbclass
> @@ -1453,7 +1453,7 @@ if [ x"$D" = "x" ]; then
> fi
> }
>
> -RPMDEPS = "${STAGING_LIBDIR_NATIVE}/rpm/rpmdeps --rcfile
> ${STAGING_LIBDIR_NATIVE}/rpm/rpmrc --macros ${STAGING_LIBDIR_NATIVE}/rpm/macros
> --define '_rpmconfigdir ${STAGING_LIBDIR_NATIVE}/rpm/'"
> +RPMDEPS = "${STAGING_LIBDIR_NATIVE}/rpm/rpmdeps --alldeps"
>
> # Collect perfile run-time dependency metadata
> # Output:
> @@ -1470,7 +1470,6 @@ python package_do_filedeps() {
> pkgdest = d.getVar('PKGDEST')
> packages = d.getVar('PACKAGES')
> rpmdeps = d.getVar('RPMDEPS')
> - magic = d.expand("${STAGING_DIR_NATIVE}${datadir_native}/misc/
> magic.mgc")
>
> def chunks(files, n):
> return [files[i:i+n] for i in range(0, len(files), n)]
> @@ -1482,7 +1481,7 @@ python package_do_filedeps() {
> if pkg.endswith('-dbg') or pkg.endswith('-doc') or
> pkg.find('-locale-') != -1 or pkg.find('-localedata-') != -1 or
> pkg.find('-gconv-') != -1 or pkg.find('-charmap-') != -1 or
> pkg.startswith('kernel-module-') or pkg.endswith('-src'):
> continue
> for files in chunks(pkgfiles[pkg], 100):
> - pkglist.append((pkg, files, rpmdeps, pkgdest, magic))
> + pkglist.append((pkg, files, rpmdeps, pkgdest))
>
> processed = oe.utils.multiprocess_exec( pkglist,
> oe.package.filedeprunner)
>
> diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py
> index 43748b2..a79c668 100644
> --- a/meta/lib/oe/package.py
> +++ b/meta/lib/oe/package.py
> @@ -162,44 +162,68 @@ def file_translate(file):
> def filedeprunner(arg):
> import re, subprocess, shlex
>
> - (pkg, pkgfiles, rpmdeps, pkgdest, magic) = arg
> + (pkg, pkgfiles, rpmdeps, pkgdest) = arg
> provides = {}
> requires = {}
>
> - r = re.compile(r'[<>=]+ +[^ ]*')
> + file_re = re.compile(r'\s+\d+\s(.*)')
> + dep_re = re.compile(r'\s+(\S)\s+(.*)')
> + r = re.compile(r'[<>=]+\s+\S*')
>
> def process_deps(pipe, pkg, pkgdest, provides, requires):
> + file = None
> for line in pipe:
> - f = line.decode("utf-8").split(" ", 1)[0].strip()
> - line = line.decode("utf-8").split(" ", 1)[1].strip()
> + line = line.decode("utf-8")
>
> - if line.startswith("Requires:"):
> + m = file_re.match(line)
> + if m:
> + file = m.group(1)
> + file = file.replace(pkgdest + "/" + pkg, "")
> + file = file_translate(file)
> + continue
> +
> + m = dep_re.match(line)
> + if not m or not file:
> + continue
> +
> + type, dep = m.groups()
> +
> + if type == 'R':
> i = requires
> - elif line.startswith("Provides:"):
> + elif type == 'P':
> i = provides
> else:
> - continue
> + continue
>
> - file = f.replace(pkgdest + "/" + pkg, "")
> - file = file_translate(file)
> - value = line.split(":", 1)[1].strip()
> - value = r.sub(r'(\g<0>)', value)
> + if dep.startswith("python("):
> + continue
>
> - if value.startswith("rpmlib("):
> + # Ignore all perl(VMS::...) and perl(Mac::...) dependencies.
> These
> + # are typically used conditionally from the Perl code, but are
> + # generated as unconditional dependencies.
> + if dep.startswith('perl(VMS::') or
> dep.startswith('perl(Mac::'):
> continue
> - if value == "python":
> +
> + # Ignore perl dependencies on .pl files.
> + if dep.startswith('perl(') and dep.endswith('.pl)'):
> continue
> +
> + # Remove perl versions and perl module versions since they
> typically
> + # do not make sense when used as package versions.
> + if dep.startswith('perl') and r.search(dep):
> + dep = dep.split()[0]
> +
> + # Put parentheses around any version specifications.
> + dep = r.sub(r'(\g<0>)',dep)
> +
> if file not in i:
> i[file] = []
> - i[file].append(value)
> + i[file].append(dep)
>
> return provides, requires
>
> - env = os.environ.copy()
> - env["MAGIC"] = magic
> -
> try:
> - dep_popen = subprocess.Popen(shlex.split(rpmdeps) + pkgfiles,
> stdout=subprocess.PIPE, env=env)
> + dep_popen = subprocess.Popen(shlex.split(rpmdeps) + pkgfiles,
> stdout=subprocess.PIPE)
> provides, requires = process_deps(dep_popen.stdout, pkg, pkgdest,
> provides, requires)
> except OSError as e:
> bb.error("rpmdeps: '%s' command failed, '%s'" %
> (shlex.split(rpmdeps) + pkgfiles, e))
>
> --
> To stop receiving notification emails like this one, please contact
> the administrator of this repository.
> --
> _______________________________________________
> Openembedded-commits mailing list
> Openembedded-commits@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-commits
>
[-- Attachment #2: Type: text/html, Size: 10615 bytes --]
^ permalink raw reply [flat|nested] 3+ messages in thread