[YOCTO 2.1] [PATCH 0/8] fixes for package_qa_walk() and buildpaths

[YOCTO 2.1] [PATCH 0/8] fixes for package_qa_walk() and buildpaths

[Robert Yang]

The following changes since commit 4038970f8ce27ac0d7a0afe2cdaa9a65108dfff5:

  os-release: fix do_compile() when RPM signing is enabled (2015-10-07 00:08:07 +0100)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib rbt/buildpaths
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=rbt/buildpaths

Robert Yang (8):
  insane.bbclass: do_package_qa depends on WARN_QA and ERROR_QA
  insane.bbclass: make package_qa_walk() can print all the messages
  insane.bbclass: buildpaths: ignore .pyc files
  insane.bbclass: skip buildpath qa check when set
    INHIBIT_PACKAGE_STRIP
  insane.bbclass: fix for ignoring debug files
  package.bbclass: strip archive files
  autotools.bbclass: use relative path to run configure script
  glibc-initial.inc: use relative path to run configure

 meta/classes/autotools.bbclass            |    4 +++-
 meta/classes/insane.bbclass               |   32 ++++++++++++++++++++++++-----
 meta/classes/package.bbclass              |   18 +++++++++++++++-
 meta/recipes-core/glibc/glibc-initial.inc |    2 +-
 4 files changed, 48 insertions(+), 8 deletions(-)

-- 
1.7.9.5

[PATCH 1/8] insane.bbclass: do_package_qa depends on WARN_QA and ERROR_QA

[Robert Yang]

do_package_qa() should rerun when WARN_QA or ERROR_QA changed

[YOCTO #8445]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta/classes/insane.bbclass |    1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/classes/insane.bbclass b/meta/classes/insane.bbclass
index 05e31a7..61db3e1 100644
--- a/meta/classes/insane.bbclass
+++ b/meta/classes/insane.bbclass
@@ -1113,6 +1113,7 @@ python do_package_qa () {
 }
 
 do_package_qa[rdeptask] = "do_packagedata"
+do_package_qa[vardeps] += "${WARN_QA} ${ERROR_QA}"
 addtask do_package_qa after do_packagedata do_package before do_build
 
 SSTATETASKS += "do_package_qa"
-- 
1.7.9.5

[PATCH 2/8] insane.bbclass: make package_qa_walk() can print all the messages

[Robert Yang]

* If more than one files have the same QA issue, package_qa_walk() can
  only print the last one, others are overrided, for example:
  messages["host-user-contaminated"] = "foo1"
  messages["host-user-contaminated"] = "foo2"
  Only foo2 will be printed, this patch fixes the issue.

* Remove unused parameter "path" from package_qa_walk()
  The path is a useless parameter in package_qa_walk() since it has been
  redined inside.

[YOCTO #8436]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta/classes/insane.bbclass |   20 +++++++++++++++++---
 1 file changed, 17 insertions(+), 3 deletions(-)

diff --git a/meta/classes/insane.bbclass b/meta/classes/insane.bbclass
index 61db3e1..e3a8cd7 100644
--- a/meta/classes/insane.bbclass
+++ b/meta/classes/insane.bbclass
@@ -749,8 +749,15 @@ def package_qa_check_staged(path,d):
 
     return sane
 
+# Merge src dict's content into dest
+def package_qa_merge_dict(dest, src):
+    if len(dest) == len(src) and dest != src:
+        for w in src:
+            if dest[w] != src[w]:
+                dest[w] = src[w] + '\n' + dest[w]
+
 # Walk over all files in a directory and call func
-def package_qa_walk(path, warnfuncs, errorfuncs, skip, package, d):
+def package_qa_walk(warnfuncs, errorfuncs, skip, package, d):
     import oe.qa
 
     #if this will throw an exception, then fix the dict above
@@ -766,9 +773,17 @@ def package_qa_walk(path, warnfuncs, errorfuncs, skip, package, d):
             except:
                 elf = None
             for func in warnfuncs:
+                warnings_orig = warnings.copy()
                 func(path, package, d, elf, warnings)
+                # warnings[foo] = "foo1" might be overrided by
+                # warnings [foo] = "foo2", check and merge.
+                package_qa_merge_dict(warnings, warnings_orig)
             for func in errorfuncs:
+                errors_orig = errors.copy()
                 func(path, package, d, elf, errors)
+                # errors[foo] = "foo1" might be overrided by
+                # errors [foo] = "foo2", check and merge.
+                package_qa_merge_dict(errors, errors_orig)
 
     for w in warnings:
         package_qa_handle_error(w, warnings[w], d)
@@ -1094,8 +1109,7 @@ python do_package_qa () {
             package_qa_handle_error("pkgname",
                     "%s doesn't match the [a-z0-9.+-]+ regex" % package, d)
 
-        path = "%s/%s" % (pkgdest, package)
-        if not package_qa_walk(path, warnchecks, errorchecks, skip, package, d):
+        if not package_qa_walk(warnchecks, errorchecks, skip, package, d):
             walk_sane  = False
         if not package_qa_check_rdepends(package, pkgdest, skip, taskdeps, packages, d):
             rdepends_sane = False
-- 
1.7.9.5

[PATCH 3/8] insane.bbclass: buildpaths: ignore .pyc files

[Robert Yang]

The .pyc files usually contain build path like:

i586-poky-linux-gcc  -m32 -march=i586 --sysroot=/buildarea/lyang1/test_p/tmp/sysroots/qemux86
And:
_PYTHON_PROJECT_BASE=/buildarea/lyang1/test_p/tmp/work/i586-poky-linux/python/2.7.9-r1/build

They are safe, so ignore them.

This can reduce 1065 QA issues for buildpaths in a world build.

[YOCTO #8446]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta/classes/insane.bbclass |    4 ++++
 1 file changed, 4 insertions(+)

diff --git a/meta/classes/insane.bbclass b/meta/classes/insane.bbclass
index e3a8cd7..664dfec 100644
--- a/meta/classes/insane.bbclass
+++ b/meta/classes/insane.bbclass
@@ -576,6 +576,10 @@ def package_qa_check_buildpaths(path, name, d, elf, messages):
     if path.find(".debug") != -1:
         return
 
+    # Ignore .pyc files, not interesting
+    if path.endswith(".pyc"):
+        return
+
     # Ignore symlinks
     if os.path.islink(path):
         return
-- 
1.7.9.5

[PATCH 4/8] insane.bbclass: skip buildpath qa check when set INHIBIT_PACKAGE_STRIP

[Robert Yang]

There are build paths in the file when set INHIBIT_PACKAGE_STRIP.

[YOCTO #8458]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta/classes/insane.bbclass |    7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/meta/classes/insane.bbclass b/meta/classes/insane.bbclass
index 664dfec..d6f7388 100644
--- a/meta/classes/insane.bbclass
+++ b/meta/classes/insane.bbclass
@@ -572,8 +572,11 @@ def package_qa_check_buildpaths(path, name, d, elf, messages):
     """
     Check for build paths inside target files and error if not found in the whitelist
     """
-    # Ignore .debug files, not interesting
-    if path.find(".debug") != -1:
+    # Ignore:
+    # - .debug files
+    # - which sets INHIBIT_PACKAGE_STRIP
+    if path.find(".debug") != -1 or \
+        d.getVar('INHIBIT_PACKAGE_STRIP', True) == "1":
         return
 
     # Ignore .pyc files, not interesting
-- 
1.7.9.5

[PATCH 5/8] insane.bbclass: fix for ignoring debug files

[Robert Yang]

Some recipes put debug files in "usr/src/debug/" rather than ".debug".

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta/classes/insane.bbclass |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/insane.bbclass b/meta/classes/insane.bbclass
index d6f7388..947d7ac 100644
--- a/meta/classes/insane.bbclass
+++ b/meta/classes/insane.bbclass
@@ -575,7 +575,7 @@ def package_qa_check_buildpaths(path, name, d, elf, messages):
     # Ignore:
     # - .debug files
     # - which sets INHIBIT_PACKAGE_STRIP
-    if path.find(".debug") != -1 or \
+    if path.find(".debug") != -1 or path.find("/src/debug/") != -1 or \
         d.getVar('INHIBIT_PACKAGE_STRIP', True) == "1":
         return
 
-- 
1.7.9.5

[PATCH 6/8] package.bbclass: strip archive files

[Robert Yang]

For saving space and removing build path in .a files.

[YOCTO #8457]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta/classes/package.bbclass |   18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)

diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index 09230b5..583848f 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -870,12 +870,25 @@ python split_and_strip_files () {
         return type
 
 
+    def isArchive(path):
+        ret, result = oe.utils.getstatusoutput("file \"%s\"" % path.replace("\"", "\\\""))
+        if ret:
+            msg = "split_and_strip_files: 'file %s' failed" % path
+            package_qa_handle_error("split-strip", msg, d)
+            return False
+
+        if "current ar archive" in result:
+            return True
+        else:
+            return False
+
     #
     # First lets figure out all of the files we may have to process ... do this only once!
     #
     elffiles = {}
     symlinks = {}
     kernmods = []
+    archives = []
     inodes = {}
     libdir = os.path.abspath(dvar + os.sep + d.getVar("libdir", True))
     baselibdir = os.path.abspath(dvar + os.sep + d.getVar("base_libdir", True))
@@ -886,6 +899,9 @@ python split_and_strip_files () {
                 if file.endswith(".ko") and file.find("/lib/modules/") != -1:
                     kernmods.append(file)
                     continue
+                if file.endswith('.a') and isArchive(file):
+                    archives.append(file)
+                    continue
 
                 # Skip debug files
                 if debugappend and file.endswith(debugappend):
@@ -1019,7 +1035,7 @@ python split_and_strip_files () {
             elf_file = int(elffiles[file])
             #bb.note("Strip %s" % file)
             sfiles.append((file, elf_file, strip))
-        for f in kernmods:
+        for f in kernmods + archives:
             sfiles.append((f, 16, strip))
 
         oe.utils.multiprocess_exec(sfiles, oe.package.runstrip)
-- 
1.7.9.5

[PATCH 7/8] autotools.bbclass: use relative path to run configure script

[Robert Yang]

The absolute path (/path/to/configure) caused VPATH in Makefile to be an
absolute path, and then it will be in elf files, use relative path to
run configure can fix the problem.

This will reduce 3127 buildpaths QA issues in a world buld (8957 ->
5830).

[YOCTO #8437]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta/classes/autotools.bbclass |    4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/meta/classes/autotools.bbclass b/meta/classes/autotools.bbclass
index 078f58f..c40e5f4 100644
--- a/meta/classes/autotools.bbclass
+++ b/meta/classes/autotools.bbclass
@@ -79,7 +79,9 @@ CONFIGUREOPT_DEPTRACK ?= "--disable-dependency-tracking"
 
 
 oe_runconf () {
-	cfgscript="${S}/configure"
+	# Use relative path for S to avoid buildpaths in files
+	relative_s=`python -c "import os; print os.path.relpath('${S}', '.')"`
+	cfgscript="$relative_s/configure"
 	if [ -x "$cfgscript" ] ; then
 		bbnote "Running $cfgscript ${CONFIGUREOPTS} ${EXTRA_OECONF} $@"
 		set +e
-- 
1.7.9.5

[PATCH 8/8] glibc-initial.inc: use relative path to run configure

[Robert Yang]

Avoid potential build path in output files.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta/recipes-core/glibc/glibc-initial.inc |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-core/glibc/glibc-initial.inc b/meta/recipes-core/glibc/glibc-initial.inc
index 3500848..a09e8a9 100644
--- a/meta/recipes-core/glibc/glibc-initial.inc
+++ b/meta/recipes-core/glibc/glibc-initial.inc
@@ -11,7 +11,7 @@ TOOLCHAIN_OPTIONS = " --sysroot=${STAGING_DIR_TCBOOTSTRAP}"
 do_configure () {
 	(cd ${S} && gnu-configize) || die "failure in running gnu-configize"
 	find ${S} -name "configure" | xargs touch
-	${S}/configure --host=${TARGET_SYS} --build=${BUILD_SYS} \
+	../git/configure --host=${TARGET_SYS} --build=${BUILD_SYS} \
 		--prefix=/usr \
 		--without-cvs --disable-sanity-checks \
 		--with-headers=${STAGING_DIR_TARGET}${includedir} \
-- 
1.7.9.5

Re: [PATCH 7/8] autotools.bbclass: use relative path to run configure script

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 620 bytes --]

On Thu, Oct 8, 2015 at 6:16 AM, Robert Yang <liezhi.yang@windriver.com>
wrote:

>  oe_runconf () {
> -       cfgscript="${S}/configure"
> +       # Use relative path for S to avoid buildpaths in files
> +       relative_s=`python -c "import os; print os.path.relpath('${S}',
> '.')"`
> +       cfgscript="$relative_s/configure"
>

Why not just use inline python here, rather than calling out to an external
python process at task execution time?
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 1081 bytes --]

Re: [PATCH 7/8] autotools.bbclass: use relative path to run configure script

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 801 bytes --]

On Thu, Oct 8, 2015 at 2:00 PM, Christopher Larson <clarson@kergoth.com>
wrote:

> On Thu, Oct 8, 2015 at 6:16 AM, Robert Yang <liezhi.yang@windriver.com>
> wrote:
>
>>  oe_runconf () {
>> -       cfgscript="${S}/configure"
>> +       # Use relative path for S to avoid buildpaths in files
>> +       relative_s=`python -c "import os; print os.path.relpath('${S}',
>> '.')"`
>> +       cfgscript="$relative_s/configure"
>>
>
> Why not just use inline python here, rather than calling out to an
> external python process at task execution time?


Oh, nevermind, of course oe_runconf can be run from any number of paths :)
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 1474 bytes --]

Re: [PATCH 7/8] autotools.bbclass: use relative path to run configure script

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 693 bytes --]

On 8 October 2015 at 22:00, Christopher Larson <clarson@kergoth.com> wrote:

> +       # Use relative path for S to avoid buildpaths in files
>>> +       relative_s=`python -c "import os; print os.path.relpath('${S}',
>>> '.')"`
>>> +       cfgscript="$relative_s/configure"
>>>
>>
>> Why not just use inline python here, rather than calling out to an
>> external python process at task execution time?
>
>
> Oh, nevermind, of course oe_runconf can be run from any number of paths :)


Why not use realpath?

ross@flashheart ~/Yocto/poky/meta/recipes-devtools/qemu (mut)
$ realpath --relative-to=. ~/Yocto/poky/meta/recipes-support/mpfr/
../../recipes-support/mpfr

Ross

[-- Attachment #2: Type: text/html, Size: 1727 bytes --]

Re: [PATCH 7/8] autotools.bbclass: use relative path to run configure script

[Robert Yang]

Hi Chris and Ross,

On 10/09/2015 05:06 AM, Burton, Ross wrote:
>
> On 8 October 2015 at 22:00, Christopher Larson <clarson@kergoth.com
> <mailto:clarson@kergoth.com>> wrote:
>
>             +       # Use relative path for S to avoid buildpaths in files
>             +       relative_s=`python -c "import os; print
>             os.path.relpath('${S}', '.')"`
>             +       cfgscript="$relative_s/configure"
>
>
>         Why not just use inline python here, rather than calling out to an
>         external python process at task execution time?
>
>
>     Oh, nevermind, of course oe_runconf can be run from any number of paths :)

Yes, "inline python" is executed at parsing time, but we need it be
a building time path.

>
>
> Why not use realpath?
>
> ross@flashheart ~/Yocto/poky/meta/recipes-devtools/qemu (mut)
> $ realpath --relative-to=. ~/Yocto/poky/meta/recipes-support/mpfr/
> ../../recipes-support/mpfr

I'm afraid that realpath is not generally existed on the build host ?

// Robert

>
> Ross

Re: [PATCH 7/8] autotools.bbclass: use relative path to run configure script

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 316 bytes --]

On 9 October 2015 at 02:28, Robert Yang <liezhi.yang@windriver.com> wrote:

> I'm afraid that realpath is not generally existed on the build host ?
>

It's part of coreutils on my Debian stable box.  I guess the real question
is "does it exist on Centos 6?".  [looks].  Hmm, it's not present on that.

Ross

[-- Attachment #2: Type: text/html, Size: 752 bytes --]

Re: [PATCH 7/8] autotools.bbclass: use relative path to run configure script

[Robert Yang]

On 10/09/2015 06:51 PM, Burton, Ross wrote:
>
> On 9 October 2015 at 02:28, Robert Yang <liezhi.yang@windriver.com
> <mailto:liezhi.yang@windriver.com>> wrote:
>
>     I'm afraid that realpath is not generally existed on the build host ?
>
>
> It's part of coreutils on my Debian stable box.  I guess the real question is
> "does it exist on Centos 6?".  [looks].  Hmm, it's not present on that.

Ubuntu 14.03 desktop doesn't have it, either.

// Robert

>
> Ross

Re: [PATCH 7/8] autotools.bbclass: use relative path to run configure script

[Paul Eggleton]

On Friday 09 October 2015 19:30:07 Robert Yang wrote:
> On 10/09/2015 06:51 PM, Burton, Ross wrote:
> > On 9 October 2015 at 02:28, Robert Yang <liezhi.yang@windriver.com
> > 
> > <mailto:liezhi.yang@windriver.com>> wrote:
> >     I'm afraid that realpath is not generally existed on the build host ?
> > 
> > It's part of coreutils on my Debian stable box.  I guess the real question
> > is "does it exist on Centos 6?".  [looks].  Hmm, it's not present on
> > that.
> Ubuntu 14.03 desktop doesn't have it, either.

Does readlink -f suffice ?

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 7/8] autotools.bbclass: use relative path to run configure script

[Robert Yang]

On 10/13/2015 03:08 PM, Paul Eggleton wrote:
> On Friday 09 October 2015 19:30:07 Robert Yang wrote:
>> On 10/09/2015 06:51 PM, Burton, Ross wrote:
>>> On 9 October 2015 at 02:28, Robert Yang <liezhi.yang@windriver.com
>>>
>>> <mailto:liezhi.yang@windriver.com>> wrote:
>>>      I'm afraid that realpath is not generally existed on the build host ?
>>>
>>> It's part of coreutils on my Debian stable box.  I guess the real question
>>> is "does it exist on Centos 6?".  [looks].  Hmm, it's not present on
>>> that.
>> Ubuntu 14.03 desktop doesn't have it, either.
>
> Does readlink -f suffice ?

Hi Paul,

We deed change an absolute path to relative path here, readlink -f does the
opposite ?

// Robert

>
> Cheers,
> Paul
>

Re: [PATCH 7/8] autotools.bbclass: use relative path to run configure script

[Paul Eggleton]

On Tuesday 13 October 2015 15:17:45 Robert Yang wrote:
> On 10/13/2015 03:08 PM, Paul Eggleton wrote:
> > On Friday 09 October 2015 19:30:07 Robert Yang wrote:
> >> On 10/09/2015 06:51 PM, Burton, Ross wrote:
> >>> On 9 October 2015 at 02:28, Robert Yang <liezhi.yang@windriver.com
> >>> 
> >>> <mailto:liezhi.yang@windriver.com>> wrote:
> >>>      I'm afraid that realpath is not generally existed on the build host
> >>>      ?
> >>> 
> >>> It's part of coreutils on my Debian stable box.  I guess the real
> >>> question
> >>> is "does it exist on Centos 6?".  [looks].  Hmm, it's not present on
> >>> that.
> >> 
> >> Ubuntu 14.03 desktop doesn't have it, either.
> > 
> > Does readlink -f suffice ?
> 
> Hi Paul,
> 
> We deed change an absolute path to relative path here, readlink -f does the
> opposite ?

Ah, my misunderstanding - sorry for the noise.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre