[PATCH 0/4] Implement multilib SDK support

[PATCH 0/4] Implement multilib SDK support

[Mark Hatle]

The attached patch set implements multilib SDK support.  This means that 
when generating an SDK (either by a traditional SDK recipe or the
do_populate_sdk task), that support infrastructure for each multilib
will be included in the SDK.

Note, this code does -not- change the toolchain.  If the generated toolchain
does not work properly with the multilib configuration, that will need to
be a seperate fix!

This was tested with both the oe-core and our custom Wind River toolchain.
Generation passed on the oe-core SDK, but the functionality of the toolchain
was not tested.  The Wind River toolchain did get properly generated and
functioned as expected.

Note:  The 0003 patch that affects processing the environment file was
agreed to with the other author of the function.

The following changes since commit 06d27cf0fbcc4004e6f456880eca49893c9290bf:

  xorg: remove XF86 BigFont extension (2012-10-22 14:57:26 +0100)

are available in the git repository at:
  git://git.yoctoproject.org/poky-contrib mhatle/sdk
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=mhatle/sdk

Mark Hatle (4):
  multilib: Add support for cross-canadian multilib packages
  multilib - crosssdk: Stop building multilib for crosssdk packages
  populate_sdk_base: Update extraction script for multilibs
  populate_sdk_base: Ensure that the multilib cross-canadian tools are
    used

 meta/classes/base.bbclass              |   43 +++++++++++++++++++++++++++++---
 meta/classes/multilib.bbclass          |   20 +++++++++++---
 meta/classes/populate_sdk_base.bbclass |    7 +++--
 meta/lib/oe/classextend.py             |    4 +-
 4 files changed, 60 insertions(+), 14 deletions(-)

-- 
1.7.3.4

[PATCH 1/4] multilib: Add support for cross-canadian multilib packages

[Mark Hatle]

Add support for the generation of cross-canadian packages.

Each cross-canadian package has:
 PN = "pkg-cross-canadian-${TRANSLATED_TARGET_ARCH}"

in order for that to be evaluated properly with multilibs enabled, it was
necessary to detect both the presence of the cross-canadian packages and
then update the vars using the OVERRIDE for the multilib.  Additional checks
were made to ensure that any dependency that sais "cross-canadian" did not
get prefixed with the MLPREFIX.

Also, make sure that even when building multilib cross-canadian packages,
we only use the single SDK PACKAGE_ARCH, we don't want or need variants.

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
---
 meta/classes/base.bbclass     |   39 +++++++++++++++++++++++++++++++++++++--
 meta/classes/multilib.bbclass |   18 ++++++++++++++----
 meta/lib/oe/classextend.py    |    4 ++--
 3 files changed, 53 insertions(+), 8 deletions(-)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index f254306..12a30c9 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -183,6 +183,8 @@ def preferred_ml_updates(d):
             providers.append(v)
 
     for pkg, reason in blacklists.items():
+        if pkg.endswith("-native") or pkg.startswith("nativesdk-") or 'cross-canadian' in pkg:
+            continue
         for p in prefixes:
             newpkg = p + "-" + pkg
             if not d.getVarFlag('PNBLACKLIST', newpkg, True):
@@ -193,6 +195,17 @@ def preferred_ml_updates(d):
         pkg = v.replace("PREFERRED_VERSION_", "")
         if pkg.endswith("-native") or pkg.startswith("nativesdk-"):
             continue
+        if 'cross-canadian' in pkg:
+            for p in prefixes:
+                localdata = bb.data.createCopy(d)
+                override = ":virtclass-multilib-" + p
+                localdata.setVar("OVERRIDES", localdata.getVar("OVERRIDES", False) + override)
+                bb.data.update_data(localdata)
+                newname = localdata.expand(v)
+                if newname != v:
+                    newval = localdata.getVar(v, True)
+                    d.setVar(newname, newval)
+            continue
         for p in prefixes:
             newname = "PREFERRED_VERSION_" + p + "-" + pkg
             if not d.getVar(newname, False):
@@ -203,14 +216,36 @@ def preferred_ml_updates(d):
         pkg = prov.replace("PREFERRED_PROVIDER_", "")
         if pkg.endswith("-native") or pkg.startswith("nativesdk-"):
             continue
+        if 'cross-canadian' in pkg:
+            for p in prefixes:
+                localdata = bb.data.createCopy(d)
+                override = ":virtclass-multilib-" + p
+                localdata.setVar("OVERRIDES", localdata.getVar("OVERRIDES", False) + override)
+                bb.data.update_data(localdata)
+                newname = localdata.expand(prov)
+                if newname != prov:
+                    newval = localdata.expand(val)
+                    d.setVar(newname, newval)
+            continue
         virt = ""
         if pkg.startswith("virtual/"):
             pkg = pkg.replace("virtual/", "")
             virt = "virtual/"
         for p in prefixes:
-            newname = "PREFERRED_PROVIDER_" + virt + p + "-" + pkg
             if pkg != "kernel":
                 val = p + "-" + val
+
+            # implement variable keys
+            localdata = bb.data.createCopy(d)
+            override = ":virtclass-multilib-" + p
+            localdata.setVar("OVERRIDES", localdata.getVar("OVERRIDES", False) + override)
+            bb.data.update_data(localdata)
+            newname = localdata.expand(prov)
+            if newname != prov:
+                d.setVar(newname, localdata.expand(val))
+
+            # implement alternative multilib name
+            newname = localdata.expand("PREFERRED_PROVIDER_" + virt + p + "-" + pkg)
             if not d.getVar(newname, False):
                 d.setVar(newname, val)
 
@@ -218,7 +253,7 @@ def preferred_ml_updates(d):
     mp = (d.getVar("MULTI_PROVIDER_WHITELIST", True) or "").split()
     extramp = []
     for p in mp:
-        if p.endswith("-native") or p.startswith("nativesdk-"):
+        if p.endswith("-native") or p.startswith("nativesdk-") or 'cross-canadian' in p:
             continue
         virt = ""
         if p.startswith("virtual/"):
diff --git a/meta/classes/multilib.bbclass b/meta/classes/multilib.bbclass
index 3824f3b..de6e59e 100644
--- a/meta/classes/multilib.bbclass
+++ b/meta/classes/multilib.bbclass
@@ -18,6 +18,13 @@ python multilib_virtclass_handler () {
         e.data.setVar("PN", variant + "-" + e.data.getVar("PN", False))
         return
 
+    if bb.data.inherits_class('cross-canadian', e.data):
+        e.data.setVar("MLPREFIX", variant + "-")
+        override = ":virtclass-multilib-" + variant
+        e.data.setVar("OVERRIDES", e.data.getVar("OVERRIDES", False) + override)
+        bb.data.update_data(e.data)
+        return
+
     if bb.data.inherits_class('native', e.data):
         raise bb.parse.SkipPackage("We can't extend native recipes")
 
@@ -69,12 +76,16 @@ python __anonymous () {
     if bb.data.inherits_class('image', d) or bb.data.inherits_class('populate_sdk_base', d):
         return
 
+    clsextend.map_depends_variable("DEPENDS")
+    clsextend.map_variable("PROVIDES")
+
+    if bb.data.inherits_class('cross-canadian', d):
+        return
+
     clsextend.rename_packages()
     clsextend.rename_package_variables((d.getVar("PACKAGEVARS", True) or "").split())
 
-    clsextend.map_depends_variable("DEPENDS")
     clsextend.map_packagevars()
-    clsextend.map_variable("PROVIDES")
     clsextend.map_regexp_variable("PACKAGES_DYNAMIC")
     clsextend.map_variable("PACKAGE_INSTALL")
     clsextend.map_variable("INITSCRIPT_PACKAGES")
@@ -90,7 +101,7 @@ python do_package_qa_multilib() {
         for i in values:
             if i.startswith('virtual/'):
                 i = i[len('virtual/'):]
-            if (not i.startswith('kernel-module')) and (not i.startswith(mlprefix)):
+            if (not i.startswith('kernel-module')) and (not i.startswith(mlprefix)) and (not 'cross-canadian' in i):
                 candidates.append(i)
         if len(candidates) > 0:
             bb.warn("Multilib QA Issue: %s package %s - suspicious values '%s' in %s" 
@@ -109,4 +120,3 @@ python do_package_qa_multilib() {
         check_mlprefix(pkg, 'RREPLACES', ml)
         check_mlprefix(pkg, 'RCONFLICTS', ml)
 }
-
diff --git a/meta/lib/oe/classextend.py b/meta/lib/oe/classextend.py
index 857a6c9..c982fb7 100644
--- a/meta/lib/oe/classextend.py
+++ b/meta/lib/oe/classextend.py
@@ -50,7 +50,7 @@ class ClassExtender(object):
         return newdata
 
     def map_depends(self, dep):
-        if dep.endswith(("-native", "-native-runtime")):
+        if dep.endswith(("-native", "-native-runtime")) or ('nativesdk-' in dep) or ('cross-canadian' in dep):
             return dep
         else:
             return self.extend_name(dep)
@@ -93,7 +93,7 @@ class ClassExtender(object):
 
 class NativesdkClassExtender(ClassExtender):
     def map_depends(self, dep):
-        if dep.endswith(("-native", "-native-runtime", "-cross")):
+        if dep.endswith(("-native", "-native-runtime", "-cross")) or ('nativesdk-' in dep):
             return dep
         elif dep.endswith(("-gcc-intermediate", "-gcc-initial", "-gcc", "-g++")):
             return dep + "-crosssdk"
-- 
1.7.3.4

[PATCH 2/4] multilib - crosssdk: Stop building multilib for crosssdk packages

[Mark Hatle]

Crosssdk packages are not actually multilib packages, so treat them
the same as other nativesdk packages in the multilib, base, and
classextend components.

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
---
 meta/classes/base.bbclass     |    8 ++++----
 meta/classes/multilib.bbclass |    4 ++--
 meta/lib/oe/classextend.py    |    4 ++--
 3 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 12a30c9..783b64d 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -183,7 +183,7 @@ def preferred_ml_updates(d):
             providers.append(v)
 
     for pkg, reason in blacklists.items():
-        if pkg.endswith("-native") or pkg.startswith("nativesdk-") or 'cross-canadian' in pkg:
+        if pkg.endswith(("-native", "-crosssdk")) or pkg.startswith("nativesdk-") or 'cross-canadian' in pkg:
             continue
         for p in prefixes:
             newpkg = p + "-" + pkg
@@ -193,7 +193,7 @@ def preferred_ml_updates(d):
     for v in versions:
         val = d.getVar(v, False)
         pkg = v.replace("PREFERRED_VERSION_", "")
-        if pkg.endswith("-native") or pkg.startswith("nativesdk-"):
+        if pkg.endswith(("-native", "-crosssdk")) or pkg.startswith("nativesdk-"):
             continue
         if 'cross-canadian' in pkg:
             for p in prefixes:
@@ -214,7 +214,7 @@ def preferred_ml_updates(d):
     for prov in providers:
         val = d.getVar(prov, False)
         pkg = prov.replace("PREFERRED_PROVIDER_", "")
-        if pkg.endswith("-native") or pkg.startswith("nativesdk-"):
+        if pkg.endswith(("-native", "-crosssdk")) or pkg.startswith("nativesdk-"):
             continue
         if 'cross-canadian' in pkg:
             for p in prefixes:
@@ -253,7 +253,7 @@ def preferred_ml_updates(d):
     mp = (d.getVar("MULTI_PROVIDER_WHITELIST", True) or "").split()
     extramp = []
     for p in mp:
-        if p.endswith("-native") or p.startswith("nativesdk-") or 'cross-canadian' in p:
+        if p.endswith(("-native", "-crosssdk")) or p.startswith("nativesdk-") or 'cross-canadian' in p:
             continue
         virt = ""
         if p.startswith("virtual/"):
diff --git a/meta/classes/multilib.bbclass b/meta/classes/multilib.bbclass
index de6e59e..c249925 100644
--- a/meta/classes/multilib.bbclass
+++ b/meta/classes/multilib.bbclass
@@ -28,7 +28,7 @@ python multilib_virtclass_handler () {
     if bb.data.inherits_class('native', e.data):
         raise bb.parse.SkipPackage("We can't extend native recipes")
 
-    if bb.data.inherits_class('nativesdk', e.data):
+    if bb.data.inherits_class('nativesdk', e.data) or bb.data.inherits_class('crosssdk', e.data):
         raise bb.parse.SkipPackage("We can't extend nativesdk recipes")
 
     save_var_name=e.data.getVar("MULTILIB_SAVE_VARNAME", True) or ""
@@ -101,7 +101,7 @@ python do_package_qa_multilib() {
         for i in values:
             if i.startswith('virtual/'):
                 i = i[len('virtual/'):]
-            if (not i.startswith('kernel-module')) and (not i.startswith(mlprefix)) and (not 'cross-canadian' in i):
+            if (not i.startswith('kernel-module')) and (not i.startswith(mlprefix)) and (not 'cross-canadian' in i) and (not i.startswith("nativesdk-")):
                 candidates.append(i)
         if len(candidates) > 0:
             bb.warn("Multilib QA Issue: %s package %s - suspicious values '%s' in %s" 
diff --git a/meta/lib/oe/classextend.py b/meta/lib/oe/classextend.py
index c982fb7..c46332c 100644
--- a/meta/lib/oe/classextend.py
+++ b/meta/lib/oe/classextend.py
@@ -50,7 +50,7 @@ class ClassExtender(object):
         return newdata
 
     def map_depends(self, dep):
-        if dep.endswith(("-native", "-native-runtime")) or ('nativesdk-' in dep) or ('cross-canadian' in dep):
+        if dep.endswith(("-native", "-native-runtime", "-crosssdk")) or ('nativesdk-' in dep) or ('cross-canadian' in dep):
             return dep
         else:
             return self.extend_name(dep)
@@ -93,7 +93,7 @@ class ClassExtender(object):
 
 class NativesdkClassExtender(ClassExtender):
     def map_depends(self, dep):
-        if dep.endswith(("-native", "-native-runtime", "-cross")) or ('nativesdk-' in dep):
+        if dep.endswith(("-native", "-native-runtime", "-cross", "-crosssdk")) or ('nativesdk-' in dep):
             return dep
         elif dep.endswith(("-gcc-intermediate", "-gcc-initial", "-gcc", "-g++")):
             return dep + "-crosssdk"
-- 
1.7.3.4

[PATCH 3/4] populate_sdk_base: Update extraction script for multilibs

[Mark Hatle]

When multilibs are enabled, there will be more then one environment
file created.  We need to be sure to process each environment file.
The next function can simply use the last environment file processed
to get the magic value(s) that it requires.

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
---
 meta/classes/populate_sdk_base.bbclass |    5 +++--
 1 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/meta/classes/populate_sdk_base.bbclass b/meta/classes/populate_sdk_base.bbclass
index a914a2c..8258224 100644
--- a/meta/classes/populate_sdk_base.bbclass
+++ b/meta/classes/populate_sdk_base.bbclass
@@ -160,8 +160,9 @@ echo "done"
 
 printf "Setting it up..."
 # fix environment paths
-env_setup_script=$(find $target_sdk_dir/ -name "environment-setup-${REAL_MULTIMACH_TARGET_SYS}")
-sed -e "s:$DEFAULT_INSTALL_DIR:$target_sdk_dir:g" -i $env_setup_script
+for env_setup_script in `ls $target_sdk_dir/environment-setup-*`; do
+  sed -e "s:$DEFAULT_INSTALL_DIR:$target_sdk_dir:g" -i $env_setup_script
+done
 
 # fix dynamic loader paths in all ELF SDK binaries
 native_sysroot=$(cat $env_setup_script |grep OECORE_NATIVE_SYSROOT|cut -d'=' -f2|tr -d '"')
-- 
1.7.3.4

[PATCH 4/4] populate_sdk_base: Ensure that the multilib cross-canadian tools are used

[Mark Hatle]

Update the host toolchain list, for cross-canadian toolchains, to ensure
that all of the supported multilibs are built and installed.  This
dynamically generates the dependnecy set based on the current multilib
configuration.

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
---
 meta/classes/populate_sdk_base.bbclass |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/meta/classes/populate_sdk_base.bbclass b/meta/classes/populate_sdk_base.bbclass
index 8258224..2fe7f15 100644
--- a/meta/classes/populate_sdk_base.bbclass
+++ b/meta/classes/populate_sdk_base.bbclass
@@ -7,7 +7,7 @@ SDK_DEPLOY = "${TMPDIR}/deploy/sdk"
 
 SDKTARGETSYSROOT = "${SDKPATH}/sysroots/${MULTIMACH_TARGET_SYS}"
 
-TOOLCHAIN_HOST_TASK ?= "nativesdk-packagegroup-sdk-host packagegroup-cross-canadian-${TRANSLATED_TARGET_ARCH}"
+TOOLCHAIN_HOST_TASK ?= "nativesdk-packagegroup-sdk-host packagegroup-cross-canadian-${@' packagegroup-cross-canadian-'.join(all_multilib_tune_values(d, 'TRANSLATED_TARGET_ARCH').split())}"
 TOOLCHAIN_HOST_TASK_ATTEMPTONLY ?= ""
 TOOLCHAIN_TARGET_TASK ?= "packagegroup-core-standalone-sdk-target packagegroup-core-standalone-sdk-target-dbg"
 TOOLCHAIN_TARGET_TASK_ATTEMPTONLY ?= ""
-- 
1.7.3.4

Re: [PATCH 0/4] Implement multilib SDK support

[Saul Wold]

On 10/22/2012 08:20 AM, Mark Hatle wrote:
> The attached patch set implements multilib SDK support.  This means that
> when generating an SDK (either by a traditional SDK recipe or the
> do_populate_sdk task), that support infrastructure for each multilib
> will be included in the SDK.
>
> Note, this code does -not- change the toolchain.  If the generated toolchain
> does not work properly with the multilib configuration, that will need to
> be a seperate fix!
>
> This was tested with both the oe-core and our custom Wind River toolchain.
> Generation passed on the oe-core SDK, but the functionality of the toolchain
> was not tested.  The Wind River toolchain did get properly generated and
> functioned as expected.
>
> Note:  The 0003 patch that affects processing the environment file was
> agreed to with the other author of the function.
>
> The following changes since commit 06d27cf0fbcc4004e6f456880eca49893c9290bf:
>
>    xorg: remove XF86 BigFont extension (2012-10-22 14:57:26 +0100)
>
> are available in the git repository at:
>    git://git.yoctoproject.org/poky-contrib mhatle/sdk
>    http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=mhatle/sdk
>
> Mark Hatle (4):
>    multilib: Add support for cross-canadian multilib packages
>    multilib - crosssdk: Stop building multilib for crosssdk packages
>    populate_sdk_base: Update extraction script for multilibs
>    populate_sdk_base: Ensure that the multilib cross-canadian tools are
>      used
>
>   meta/classes/base.bbclass              |   43 +++++++++++++++++++++++++++++---
>   meta/classes/multilib.bbclass          |   20 +++++++++++---
>   meta/classes/populate_sdk_base.bbclass |    7 +++--
>   meta/lib/oe/classextend.py             |    4 +-
>   4 files changed, 60 insertions(+), 14 deletions(-)
>

Merged into OE-Core

Thanks
	Sau!