[PATCH] base: Improve handling of switching virtual/x providers

[PATCH] base: Improve handling of switching virtual/x providers

[Richard Purdie]

If you build virtual/kernel, then change PREFERRED_PROVIDER_virtual/kernel from say
"linux-yocto" to "linux-yocto-dev", you see errors from the sysroot about overlapping
files. The automatic uninstall logic doesn't trigger since the other recipes is
still technically parsed/buildable.

What we can do is look at the value of PREFERRED_PROVIDER_virtual/X and raise SkipRecipe
(skip parsing) if it provides this thing and its not selected. We skip cases no preferred
provider is set, or the value is in MULTI_PROVIDER_WHITELIST.We also inform the user
if they try to build something which conflicts with the configuration:

$ bitbake linux-yocto-tiny
ERROR: Nothing PROVIDES 'linux-yocto-tiny'
ERROR: linux-yocto-tiny was skipped: PREFERRED_PROVIDER_virtual/kernel set to linux-yocto, not linux-yocto-tiny

[YOCTO #4102]

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 44ca781..b8f2aea 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -204,7 +204,7 @@ def buildcfg_neededvars(d):
         bb.fatal('The following variable(s) were not set: %s\nPlease set them directly, or choose a MACHINE or DISTRO that sets them.' % ', '.join(pesteruser))
 
 addhandler base_eventhandler
-base_eventhandler[eventmask] = "bb.event.ConfigParsed bb.event.BuildStarted bb.event.RecipePreFinalise bb.runqueue.sceneQueueComplete"
+base_eventhandler[eventmask] = "bb.event.ConfigParsed bb.event.BuildStarted bb.event.RecipePreFinalise bb.runqueue.sceneQueueComplete bb.event.RecipeParsed"
 python base_eventhandler() {
     import bb.runqueue
 
@@ -257,6 +257,25 @@ python base_eventhandler() {
             bb.debug(1, "Executing SceneQueue Completion commands: %s" % "\n".join(cmds))
             bb.build.exec_func("completion_function", e.data)
             os.remove(completions)
+
+    if isinstance(e, bb.event.RecipeParsed):
+        #
+        # If we have multiple providers of virtual/X and a PREFERRED_PROVIDER_virtual/X is set
+        # skip parsing for all the other providers which will mean they get uninstalled from the
+        # sysroot since they're now "unreachable". This makes switching virtual/kernel work in 
+        # particular.
+        #
+        pn = d.getVar('PN', True)
+        source_mirror_fetch = d.getVar('SOURCE_MIRROR_FETCH', False)
+        if not source_mirror_fetch:
+            provs = (d.getVar("PROVIDES", True) or "").split()
+            multiwhitelist = (d.getVar("MULTI_PROVIDER_WHITELIST", True) or "").split()
+            for p in provs:
+                if p.startswith("virtual/") and p not in multiwhitelist:
+                    profprov = d.getVar("PREFERRED_PROVIDER_" + p, True)
+                    if profprov and pn != profprov:
+                        bb.warn("PREFERRED_PROVIDER_%s set to %s, not %s" % (p, profprov, pn))
+                        raise bb.parse.SkipPackage("PREFERRED_PROVIDER_%s set to %s, not %s" % (p, profprov, pn))
 }
 
 CONFIGURESTAMPFILE = "${WORKDIR}/configure.sstate"

Re: [PATCH] base: Improve handling of switching virtual/x providers

[Richard Purdie]

On Fri, 2015-11-06 at 12:37 +0000, Richard Purdie wrote:
> If you build virtual/kernel, then change PREFERRED_PROVIDER_virtual/kernel from say
> "linux-yocto" to "linux-yocto-dev", you see errors from the sysroot about overlapping
> files. The automatic uninstall logic doesn't trigger since the other recipes is
> still technically parsed/buildable.
> 
> What we can do is look at the value of PREFERRED_PROVIDER_virtual/X and raise SkipRecipe
> (skip parsing) if it provides this thing and its not selected. We skip cases no preferred
> provider is set, or the value is in MULTI_PROVIDER_WHITELIST.We also inform the user
> if they try to build something which conflicts with the configuration:
> 
> $ bitbake linux-yocto-tiny
> ERROR: Nothing PROVIDES 'linux-yocto-tiny'
> ERROR: linux-yocto-tiny was skipped: PREFERRED_PROVIDER_virtual/kernel set to linux-yocto, not linux-yocto-tiny
> 
> [YOCTO #4102]
> 
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> 
> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
> index 44ca781..b8f2aea 100644
> --- a/meta/classes/base.bbclass
> +++ b/meta/classes/base.bbclass
> @@ -204,7 +204,7 @@ def buildcfg_neededvars(d):
>          bb.fatal('The following variable(s) were not set: %s\nPlease set them directly, or choose a MACHINE or DISTRO that sets them.' % ', '.join(pesteruser))
>  
>  addhandler base_eventhandler
> -base_eventhandler[eventmask] = "bb.event.ConfigParsed bb.event.BuildStarted bb.event.RecipePreFinalise bb.runqueue.sceneQueueComplete"
> +base_eventhandler[eventmask] = "bb.event.ConfigParsed bb.event.BuildStarted bb.event.RecipePreFinalise bb.runqueue.sceneQueueComplete bb.event.RecipeParsed"
>  python base_eventhandler() {
>      import bb.runqueue
>  
> @@ -257,6 +257,25 @@ python base_eventhandler() {
>              bb.debug(1, "Executing SceneQueue Completion commands: %s" % "\n".join(cmds))
>              bb.build.exec_func("completion_function", e.data)
>              os.remove(completions)
> +
> +    if isinstance(e, bb.event.RecipeParsed):
> +        #
> +        # If we have multiple providers of virtual/X and a PREFERRED_PROVIDER_virtual/X is set
> +        # skip parsing for all the other providers which will mean they get uninstalled from the
> +        # sysroot since they're now "unreachable". This makes switching virtual/kernel work in 
> +        # particular.
> +        #
> +        pn = d.getVar('PN', True)
> +        source_mirror_fetch = d.getVar('SOURCE_MIRROR_FETCH', False)
> +        if not source_mirror_fetch:
> +            provs = (d.getVar("PROVIDES", True) or "").split()
> +            multiwhitelist = (d.getVar("MULTI_PROVIDER_WHITELIST", True) or "").split()
> +            for p in provs:
> +                if p.startswith("virtual/") and p not in multiwhitelist:
> +                    profprov = d.getVar("PREFERRED_PROVIDER_" + p, True)
> +                    if profprov and pn != profprov:
> +                        bb.warn("PREFERRED_PROVIDER_%s set to %s, not %s" % (p, profprov, pn))

Obviously minus the above line. I'll remove that in any version I add to
-next.

Cheers,

Richard

> +                        raise bb.parse.SkipPackage("PREFERRED_PROVIDER_%s set to %s, not %s" % (p, profprov, pn))
>  }
>  
>  CONFIGURESTAMPFILE = "${WORKDIR}/configure.sstate"
> 
> 

Re: [PATCH] base: Improve handling of switching virtual/x providers

[Andre McCurdy]

On Fri, Nov 6, 2015 at 4:37 AM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> If you build virtual/kernel, then change PREFERRED_PROVIDER_virtual/kernel from say
> "linux-yocto" to "linux-yocto-dev", you see errors from the sysroot about overlapping
> files. The automatic uninstall logic doesn't trigger since the other recipes is
> still technically parsed/buildable.
>
> What we can do is look at the value of PREFERRED_PROVIDER_virtual/X and raise SkipRecipe
> (skip parsing) if it provides this thing and its not selected. We skip cases no preferred
> provider is set, or the value is in MULTI_PROVIDER_WHITELIST.We also inform the user
> if they try to build something which conflicts with the configuration:
>
> $ bitbake linux-yocto-tiny
> ERROR: Nothing PROVIDES 'linux-yocto-tiny'
> ERROR: linux-yocto-tiny was skipped: PREFERRED_PROVIDER_virtual/kernel set to linux-yocto, not linux-yocto-tiny

This change seems to cause problems for musl libc builds (I'm using
today's oe-core master and Khem's meta-musl layer).

  ERROR: Nothing RPROVIDES 'nativesdk-glibc' (but
/home/andre/rdk/rdk-master-musl/openembedded-core/meta/recipes-core/meta/uninative-tarball.bb
RDEPENDS on or otherwise requires it)
  ERROR: nativesdk-glibc was skipped:
PREFERRED_PROVIDER_virtual/nativesdk-x86_64-rdksdk-linux-libc-for-gcc
set to nativesdk-musl, not nativesdk-glibc
  ERROR: Required build target 'rdk-image-wpe' has no buildable providers.
  Missing or unbuildable dependency chain was: ['rdk-image-wpe',
'uninative-tarball', 'nativesdk-glibc']

Reverting this one commit (oe-core 9a3b992) gets the builds working again.


> [YOCTO #4102]
>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
>
> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
> index 44ca781..b8f2aea 100644
> --- a/meta/classes/base.bbclass
> +++ b/meta/classes/base.bbclass
> @@ -204,7 +204,7 @@ def buildcfg_neededvars(d):
>          bb.fatal('The following variable(s) were not set: %s\nPlease set them directly, or choose a MACHINE or DISTRO that sets them.' % ', '.join(pesteruser))
>
>  addhandler base_eventhandler
> -base_eventhandler[eventmask] = "bb.event.ConfigParsed bb.event.BuildStarted bb.event.RecipePreFinalise bb.runqueue.sceneQueueComplete"
> +base_eventhandler[eventmask] = "bb.event.ConfigParsed bb.event.BuildStarted bb.event.RecipePreFinalise bb.runqueue.sceneQueueComplete bb.event.RecipeParsed"
>  python base_eventhandler() {
>      import bb.runqueue
>
> @@ -257,6 +257,25 @@ python base_eventhandler() {
>              bb.debug(1, "Executing SceneQueue Completion commands: %s" % "\n".join(cmds))
>              bb.build.exec_func("completion_function", e.data)
>              os.remove(completions)
> +
> +    if isinstance(e, bb.event.RecipeParsed):
> +        #
> +        # If we have multiple providers of virtual/X and a PREFERRED_PROVIDER_virtual/X is set
> +        # skip parsing for all the other providers which will mean they get uninstalled from the
> +        # sysroot since they're now "unreachable". This makes switching virtual/kernel work in
> +        # particular.
> +        #
> +        pn = d.getVar('PN', True)
> +        source_mirror_fetch = d.getVar('SOURCE_MIRROR_FETCH', False)
> +        if not source_mirror_fetch:
> +            provs = (d.getVar("PROVIDES", True) or "").split()
> +            multiwhitelist = (d.getVar("MULTI_PROVIDER_WHITELIST", True) or "").split()
> +            for p in provs:
> +                if p.startswith("virtual/") and p not in multiwhitelist:
> +                    profprov = d.getVar("PREFERRED_PROVIDER_" + p, True)
> +                    if profprov and pn != profprov:
> +                        bb.warn("PREFERRED_PROVIDER_%s set to %s, not %s" % (p, profprov, pn))
> +                        raise bb.parse.SkipPackage("PREFERRED_PROVIDER_%s set to %s, not %s" % (p, profprov, pn))
>  }
>
>  CONFIGURESTAMPFILE = "${WORKDIR}/configure.sstate"
>
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [PATCH] base: Improve handling of switching virtual/x providers

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 1519 bytes --]

On 8 December 2015 at 18:58, Andre McCurdy <armccurdy@gmail.com> wrote:

> This change seems to cause problems for musl libc builds (I'm using
> today's oe-core master and Khem's meta-musl layer).
>
>   ERROR: Nothing RPROVIDES 'nativesdk-glibc' (but
> /home/andre/rdk/rdk-master-musl/openembedded-core/meta/recipes-core/meta/
> uninative-tarball.bb
> RDEPENDS on or otherwise requires it)
>   ERROR: nativesdk-glibc was skipped:
> PREFERRED_PROVIDER_virtual/nativesdk-x86_64-rdksdk-linux-libc-for-gcc
> set to nativesdk-musl, not nativesdk-glibc
>   ERROR: Required build target 'rdk-image-wpe' has no buildable providers.
>   Missing or unbuildable dependency chain was: ['rdk-image-wpe',
> 'uninative-tarball', 'nativesdk-glibc']
>
> Reverting this one commit (oe-core 9a3b992) gets the builds working again.
>

It's breaking universe fetch too:

ERROR: Nothing PROVIDES 'linux-yocto-rt' (but
/home/pokybuild/yocto-autobuilder/yocto-worker/build-appliance/build/meta/recipes-rt/images/
core-image-rt-sdk.bb,
/home/pokybuild/yocto-autobuilder/yocto-worker/build-appliance/build/meta/recipes-rt/images/
core-image-rt.bb DEPENDS on or otherwise requires it)
ERROR: linux-yocto-rt was skipped: PREFERRED_PROVIDER_virtual/kernel set to
linux-yocto, not linux-yocto-rt
ERROR: linux-yocto-rt was skipped: PREFERRED_PROVIDER_virtual/kernel set to
linux-yocto, not linux-yocto-rt
ERROR: Nothing RPROVIDES 'core-image-rt-sdk'
ERROR: No eligible RPROVIDERs exist for 'core-image-rt-sdk'

Ross

[-- Attachment #2: Type: text/html, Size: 2273 bytes --]