[PATCHv3 0/5] Add dummy tools to help identify needed dependencies

[PATCHv3 0/5] Add dummy tools to help identify needed dependencies

[Peter Kjellerstedt]

After the introduction of RSS, I still found it hard to get
dependencies on some common tools that are typically installed on the
build host correct. Using the wrong version of tools like pkg-config,
gdbus-codegen and dbus-binding-tool can cause build failures.

To circumvent this, I created dummy versions of the tools that always
fail and placed them in the scripts directory. Thus, if the real tool
has not been installed in the RSS, the dummy version is used and the
build fails. For good measures I even output a message that says what
needs to be corrected in the recipe.

Included is also an update to linux-libc-headers as it apparently
needs to inherit pkgconfig, which the introduction of the dummy
version of pkg-config showed.

PATCHv2: Also inherit pkgconfig in module.bbclass since pkg-config
is needed for the make_scripts task.

PATCHv3: Add a missing Signed-off-by footer.

//Peter

The following changes since commit 8f1ed1758748e4c00eaa6b4b3528e79f36d62c19:

  bitbake: fetch2: add initial Amazon AWS S3 fetcher (2017-03-03 00:19:37 +0000)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib pkj/dummy_tools
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=pkj/dummy_tools

Peter Kjellerstedt (5):
  module.bbclass: Add inherit of pkgconfig
  linux-libc-headers: Add inherit of pkgconfig
  scripts/dbus-binding-tool: Add a dummy version that always fails
  scripts/gdbus-codegen: Add a dummy version that always fails
  scripts/pkg-config: Add a dummy version that always fails

 meta/classes/module.bbclass                              |  2 +-
 .../linux-libc-headers/linux-libc-headers.inc            |  2 +-
 scripts/dbus-binding-tool                                | 11 +++++++++++
 scripts/gdbus-codegen                                    | 11 +++++++++++
 scripts/pkg-config                                       | 16 ++++++++++++++++
 5 files changed, 40 insertions(+), 2 deletions(-)
 create mode 100755 scripts/dbus-binding-tool
 create mode 100755 scripts/gdbus-codegen
 create mode 100755 scripts/pkg-config

-- 
2.9.0

[PATCHv3 1/5] module.bbclass: Add inherit of pkgconfig

[Peter Kjellerstedt]

This is needed for the make_scripts task.

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
---
 meta/classes/module.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/module.bbclass b/meta/classes/module.bbclass
index a588873..99b7ebc 100644
--- a/meta/classes/module.bbclass
+++ b/meta/classes/module.bbclass
@@ -1,4 +1,4 @@
-inherit module-base kernel-module-split
+inherit module-base kernel-module-split pkgconfig
 
 addtask make_scripts after do_prepare_recipe_sysroot before do_compile
 do_make_scripts[lockfiles] = "${TMPDIR}/kernel-scripts.lock"
-- 
2.9.0

[PATCHv3 2/5] linux-libc-headers: Add inherit of pkgconfig

[Peter Kjellerstedt]

pkg-config is used by the kernel build system when creating the
configuration tools.

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
---
 meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc b/meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc
index 30d2243..653a470 100644
--- a/meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc
+++ b/meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc
@@ -42,7 +42,7 @@ python __anonymous () {
         d.setVar("HEADER_FETCH_VER", "2.6")
 }
 
-inherit kernel-arch
+inherit kernel-arch pkgconfig
 
 KORG_ARCHIVE_COMPRESSION ?= "xz"
 
-- 
2.9.0

[PATCHv3 3/5] scripts/dbus-binding-tool: Add a dummy version that always fails

[Peter Kjellerstedt]

This is intended to catch missing dependencies on the real version.

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
---
 scripts/dbus-binding-tool | 11 +++++++++++
 1 file changed, 11 insertions(+)
 create mode 100755 scripts/dbus-binding-tool

diff --git a/scripts/dbus-binding-tool b/scripts/dbus-binding-tool
new file mode 100755
index 0000000..2e85987
--- /dev/null
+++ b/scripts/dbus-binding-tool
@@ -0,0 +1,11 @@
+#!/bin/sh
+
+# This is a dummy version that always fails. It is intended to catch
+# missing dependencies on the real version.
+
+myname=${0##*/}
+
+echo "The correct version of $myname is not being used!" >&2
+echo "Make sure the recipe DEPENDS on 'dbus-glib-native'." >&2
+
+exit 1
-- 
2.9.0

[PATCHv3 4/5] scripts/gdbus-codegen: Add a dummy version that always fails

[Peter Kjellerstedt]

This is intended to catch missing dependencies on the real version.

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
---
 scripts/gdbus-codegen | 11 +++++++++++
 1 file changed, 11 insertions(+)
 create mode 100755 scripts/gdbus-codegen

diff --git a/scripts/gdbus-codegen b/scripts/gdbus-codegen
new file mode 100755
index 0000000..d91386b
--- /dev/null
+++ b/scripts/gdbus-codegen
@@ -0,0 +1,11 @@
+#!/bin/sh
+
+# This is a dummy version that always fails. It is intended to catch
+# missing dependencies on the real version.
+
+myname=${0##*/}
+
+echo "The correct version of $myname is not being used!" >&2
+echo "Make sure the recipe DEPENDS on 'glib-2.0-native'." >&2
+
+exit 1
-- 
2.9.0

[PATCHv3 5/5] scripts/pkg-config: Add a dummy version that always fails

[Peter Kjellerstedt]

This is intended to catch missing dependencies on the real version.

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
---
 scripts/pkg-config | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
 create mode 100755 scripts/pkg-config

diff --git a/scripts/pkg-config b/scripts/pkg-config
new file mode 100755
index 0000000..c6376ec
--- /dev/null
+++ b/scripts/pkg-config
@@ -0,0 +1,16 @@
+#!/bin/sh
+
+# This is a dummy version that always fails. It is intended to catch
+# missing dependencies on the real version.
+
+myname=${0##*/}
+
+echo "The correct version of $myname is not being used!" >&2
+echo "Make sure the recipe inherits 'pkgconfig'." >&2
+
+# Return something that will make the compiler barf. This is for when
+# pkg-config is used directly in a Makefile in a construct like:
+# CFLAGS += $(shell pkg-config --cflags glib-2.0)
+echo "The-correct-version-of-$myname-is-not-being-used"
+
+exit 1
-- 
2.9.0

Re: [PATCHv3 0/5] Add dummy tools to help identify needed dependencies

[Max Krummenacher]

Hi

2017-03-03 12:17 GMT+01:00 Peter Kjellerstedt <peter.kjellerstedt@axis.com>:

> To circumvent this, I created dummy versions of the tools that always
> fail and placed them in the scripts directory. Thus, if the real tool
> has not been installed in the RSS, the dummy version is used and the
> build fails. For good measures I even output a message that says what
> needs to be corrected in the recipe.
>
> Included is also an update to linux-libc-headers as it apparently
> needs to inherit pkgconfig, which the introduction of the dummy
> version of pkg-config showed.
>

I tried the whole series, i.e. applied on top of openembedded-core,
20c6e87 and bitbake 6fe07ed.

The resulting build failed in linux-libc-headers because it picked up
the dummy script instead of the real pkg-config binary which actually
is installed in the RSS.

run.do_configure has a PATH setting with first ...meta/scripts and
only then the paths to the different RSS directories.

Changing that order in meta/conf/bitbake.conf line 450 fixed this for
me, however I'm unsure of posible sideeffects of that reordering.

Max

Re: [PATCHv3 0/5] Add dummy tools to help identify needed dependencies

[Peter Kjellerstedt]

> -----Original Message-----
> From: Max Krummenacher [mailto:max.oss.09@gmail.com]
> Sent: den 3 mars 2017 17:24
> To: Peter Kjellerstedt
> Cc: OE-core
> Subject: Re: [OE-core] [PATCHv3 0/5] Add dummy tools to help identify
> needed dependencies
> 
> Hi
> 
> 2017-03-03 12:17 GMT+01:00 Peter Kjellerstedt
> <peter.kjellerstedt@axis.com>:
> 
> > To circumvent this, I created dummy versions of the tools that always
> > fail and placed them in the scripts directory. Thus, if the real tool
> > has not been installed in the RSS, the dummy version is used and the
> > build fails. For good measures I even output a message that says what
> > needs to be corrected in the recipe.
> >
> > Included is also an update to linux-libc-headers as it apparently
> > needs to inherit pkgconfig, which the introduction of the dummy
> > version of pkg-config showed.
> >
> 
> I tried the whole series, i.e. applied on top of openembedded-core,
> 20c6e87 and bitbake 6fe07ed.
> 
> The resulting build failed in linux-libc-headers because it picked up
> the dummy script instead of the real pkg-config binary which actually
> is installed in the RSS.
> 
> run.do_configure has a PATH setting with first ...meta/scripts and
> only then the paths to the different RSS directories.
> 
> Changing that order in meta/conf/bitbake.conf line 450 fixed this for
> me, however I'm unsure of posible sideeffects of that reordering.
> 
> Max

Bah, you are correct. That is what I get for not testing exactly 
what I sent for integration. Shame on me.

Anyway, I see a couple of possible solutions:

* Put the scripts in bitbake/bin instead. This should work as it 
  is only added to $PATH by oe-buildenv-internal (contrary to 
  ${COREBASE}/scripts which is added to $PATH both by 
  oe-buildenv-internal and bitbake.conf). However, I do not like 
  this idea as it feels wrong to put them there.
* Put the scripts in a subdirectory of ${COREBASE}/scripts, e.g., 
  ${COREBASE}/scripts/blacklisted, and add that path to $PATH in 
  bitbake.conf, but at the other end of it so that the RSS paths 
  appear before it. This sounds a lot more appealing to me.
* Implement the whitelist solution Patrick Ohly mentioned instead.
  Although this is probably the best solution, prioritizing it 
  for 2.3 does not sound likely to happen.

I have implemented solution two above (properly tested this time), 
and will send that along in a little while.

//Peter

Re: [PATCHv3 0/5] Add dummy tools to help identify needed dependencies

[Peter Kjellerstedt]

> -----Original Message-----
> From: openembedded-core-bounces@lists.openembedded.org
> [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf Of
> Peter Kjellerstedt
> Sent: den 3 mars 2017 20:53
> To: Max Krummenacher
> Cc: OE-core
> Subject: Re: [OE-core] [PATCHv3 0/5] Add dummy tools to help identify
> needed dependencies
> 
> > -----Original Message-----
> > From: Max Krummenacher [mailto:max.oss.09@gmail.com]
> > Sent: den 3 mars 2017 17:24
> > To: Peter Kjellerstedt
> > Cc: OE-core
> > Subject: Re: [OE-core] [PATCHv3 0/5] Add dummy tools to help identify
> > needed dependencies
> >
> > Hi
> >
> > 2017-03-03 12:17 GMT+01:00 Peter Kjellerstedt
> > <peter.kjellerstedt@axis.com>:
> >
> > > To circumvent this, I created dummy versions of the tools that
> always
> > > fail and placed them in the scripts directory. Thus, if the real
> tool
> > > has not been installed in the RSS, the dummy version is used and
> the
> > > build fails. For good measures I even output a message that says
> what
> > > needs to be corrected in the recipe.
> > >
> > > Included is also an update to linux-libc-headers as it apparently
> > > needs to inherit pkgconfig, which the introduction of the dummy
> > > version of pkg-config showed.
> > >
> >
> > I tried the whole series, i.e. applied on top of openembedded-core,
> > 20c6e87 and bitbake 6fe07ed.
> >
> > The resulting build failed in linux-libc-headers because it picked up
> > the dummy script instead of the real pkg-config binary which actually
> > is installed in the RSS.
> >
> > run.do_configure has a PATH setting with first ...meta/scripts and
> > only then the paths to the different RSS directories.
> >
> > Changing that order in meta/conf/bitbake.conf line 450 fixed this for
> > me, however I'm unsure of posible sideeffects of that reordering.
> >
> > Max
> 
> Bah, you are correct. That is what I get for not testing exactly
> what I sent for integration. Shame on me.
> 
> Anyway, I see a couple of possible solutions:
> 
> * Put the scripts in bitbake/bin instead. This should work as it
>   is only added to $PATH by oe-buildenv-internal (contrary to
>   ${COREBASE}/scripts which is added to $PATH both by
>   oe-buildenv-internal and bitbake.conf). However, I do not like
>   this idea as it feels wrong to put them there.
> * Put the scripts in a subdirectory of ${COREBASE}/scripts, e.g.,
>   ${COREBASE}/scripts/blacklisted, and add that path to $PATH in
>   bitbake.conf, but at the other end of it so that the RSS paths
>   appear before it. This sounds a lot more appealing to me.
> * Implement the whitelist solution Patrick Ohly mentioned instead.
>   Although this is probably the best solution, prioritizing it
>   for 2.3 does not sound likely to happen.
> 
> I have implemented solution two above (properly tested this time),
> and will send that along in a little while.
> 
> //Peter

Richard,

since I see that you have integrated/staged the two patches that add 
inherits of pkgconfig, but not the patches that add the dummy commands, 
I assume you have some reservations to these patches. What are your 
take on the subject of blacklisting vs whitelisting the commands from 
the build host? I know that having these dummy commands in place helped 
me a great deal when updating our recipes to build correctly with RSS.

//Peter

Re: [PATCHv3 0/5] Add dummy tools to help identify needed dependencies

[Khem Raj]

On 17-03-08 09:43:58, Peter Kjellerstedt wrote:
> > -----Original Message-----
> > From: openembedded-core-bounces@lists.openembedded.org
> > [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf Of
> > Peter Kjellerstedt
> > Sent: den 3 mars 2017 20:53
> > To: Max Krummenacher
> > Cc: OE-core
> > Subject: Re: [OE-core] [PATCHv3 0/5] Add dummy tools to help identify
> > needed dependencies
> > 
> > > -----Original Message-----
> > > From: Max Krummenacher [mailto:max.oss.09@gmail.com]
> > > Sent: den 3 mars 2017 17:24
> > > To: Peter Kjellerstedt
> > > Cc: OE-core
> > > Subject: Re: [OE-core] [PATCHv3 0/5] Add dummy tools to help identify
> > > needed dependencies
> > >
> > > Hi
> > >
> > > 2017-03-03 12:17 GMT+01:00 Peter Kjellerstedt
> > > <peter.kjellerstedt@axis.com>:
> > >
> > > > To circumvent this, I created dummy versions of the tools that
> > always
> > > > fail and placed them in the scripts directory. Thus, if the real
> > tool
> > > > has not been installed in the RSS, the dummy version is used and
> > the
> > > > build fails. For good measures I even output a message that says
> > what
> > > > needs to be corrected in the recipe.
> > > >
> > > > Included is also an update to linux-libc-headers as it apparently
> > > > needs to inherit pkgconfig, which the introduction of the dummy
> > > > version of pkg-config showed.
> > > >
> > >
> > > I tried the whole series, i.e. applied on top of openembedded-core,
> > > 20c6e87 and bitbake 6fe07ed.
> > >
> > > The resulting build failed in linux-libc-headers because it picked up
> > > the dummy script instead of the real pkg-config binary which actually
> > > is installed in the RSS.
> > >
> > > run.do_configure has a PATH setting with first ...meta/scripts and
> > > only then the paths to the different RSS directories.
> > >
> > > Changing that order in meta/conf/bitbake.conf line 450 fixed this for
> > > me, however I'm unsure of posible sideeffects of that reordering.
> > >
> > > Max
> > 
> > Bah, you are correct. That is what I get for not testing exactly
> > what I sent for integration. Shame on me.
> > 
> > Anyway, I see a couple of possible solutions:
> > 
> > * Put the scripts in bitbake/bin instead. This should work as it
> >   is only added to $PATH by oe-buildenv-internal (contrary to
> >   ${COREBASE}/scripts which is added to $PATH both by
> >   oe-buildenv-internal and bitbake.conf). However, I do not like
> >   this idea as it feels wrong to put them there.
> > * Put the scripts in a subdirectory of ${COREBASE}/scripts, e.g.,
> >   ${COREBASE}/scripts/blacklisted, and add that path to $PATH in
> >   bitbake.conf, but at the other end of it so that the RSS paths
> >   appear before it. This sounds a lot more appealing to me.
> > * Implement the whitelist solution Patrick Ohly mentioned instead.
> >   Although this is probably the best solution, prioritizing it
> >   for 2.3 does not sound likely to happen.
> > 
> > I have implemented solution two above (properly tested this time),
> > and will send that along in a little while.
> > 
> > //Peter
> 
> Richard,
> 
> since I see that you have integrated/staged the two patches that add 
> inherits of pkgconfig, but not the patches that add the dummy commands, 
> I assume you have some reservations to these patches. What are your 
> take on the subject of blacklisting vs whitelisting the commands from 
> the build host? I know that having these dummy commands in place helped 
> me a great deal when updating our recipes to build correctly with RSS.

perhaps if we have a way to containerize the binaries/scripts used from
buildhost would be comprehensive but hard to tool in may be blacklisting
is contained in that we can weed out the known problems but if we can
sanitize and create a comprehensive list of needed stuff from buildhost
would be cool, it will probably help the reproducability cause as well.
> 
> //Peter
> 
> -- 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [PATCHv3 0/5] Add dummy tools to help identify needed dependencies

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 835 bytes --]

On 8 March 2017 at 09:43, Peter Kjellerstedt <peter.kjellerstedt@axis.com>
wrote:

> since I see that you have integrated/staged the two patches that add
> inherits of pkgconfig, but not the patches that add the dummy commands,
> I assume you have some reservations to these patches. What are your
> take on the subject of blacklisting vs whitelisting the commands from
> the build host? I know that having these dummy commands in place helped
> me a great deal when updating our recipes to build correctly with RSS.
>

FWIW I wasn't entirely keen on how the blacklisting had to be very careful
about how PATH was manipulated, which is why I merged to my staging branch
the fixes but not the dummy commands.

If the majority of people agree that this should be merged I don't have a
strong opinion either way.

Ross

[-- Attachment #2: Type: text/html, Size: 1413 bytes --]

Re: [PATCHv3 0/5] Add dummy tools to help identify needed dependencies

[Mark Hatle]

On 3/8/17 11:21 AM, Burton, Ross wrote:
> 
> On 8 March 2017 at 09:43, Peter Kjellerstedt <peter.kjellerstedt@axis.com
> <mailto:peter.kjellerstedt@axis.com>> wrote:
> 
>     since I see that you have integrated/staged the two patches that add
>     inherits of pkgconfig, but not the patches that add the dummy commands,
>     I assume you have some reservations to these patches. What are your
>     take on the subject of blacklisting vs whitelisting the commands from
>     the build host? I know that having these dummy commands in place helped
>     me a great deal when updating our recipes to build correctly with RSS.
> 
> 
> FWIW I wasn't entirely keen on how the blacklisting had to be very careful about
> how PATH was manipulated, which is why I merged to my staging branch the fixes
> but not the dummy commands.
> 
> If the majority of people agree that this should be merged I don't have a strong
> opinion either way.

The scripts would be very useful for people to avoid host-contamination issues.

If we could build up a set of these types of tools (and associated 'please do
the following' behaviors) I think it would be very useful.  (Starting with these
three items will probably catch the majority of things.)

--Mark

> Ross
> 
> 

Re: [PATCHv3 0/5] Add dummy tools to help identify needed dependencies

[Richard Purdie]

On Wed, 2017-03-08 at 17:21 +0000, Burton, Ross wrote:
> 
> On 8 March 2017 at 09:43, Peter Kjellerstedt <peter.kjellerstedt@axis
> .com> wrote:
> > since I see that you have integrated/staged the two patches that
> > add
> > inherits of pkgconfig, but not the patches that add the dummy
> > commands,
> > I assume you have some reservations to these patches. What are your
> > take on the subject of blacklisting vs whitelisting the commands
> > from
> > the build host? I know that having these dummy commands in place
> > helped
> > me a great deal when updating our recipes to build correctly with
> > RSS.
> > 
> FWIW I wasn't entirely keen on how the blacklisting had to be very
> careful about how PATH was manipulated, which is why I merged to my
> staging branch the fixes but not the dummy commands.
> 
> If the majority of people agree that this should be merged I don't
> have a strong opinion either way.

Ultimately I think there is a better solution than this but I haven't
had time to look at that as yet.

I do worry that it makes assumptions about PATH that may or may not be
valid everywhere, equally I do see it could be useful.

So I also have mixed feelings...

Cheers,

Richard

Re: [PATCHv3 0/5] Add dummy tools to help identify needed dependencies

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 2048 bytes --]

On Wed, Mar 08, 2017 at 06:00:30PM +0000, Richard Purdie wrote:
> On Wed, 2017-03-08 at 17:21 +0000, Burton, Ross wrote:
> > 
> > On 8 March 2017 at 09:43, Peter Kjellerstedt <peter.kjellerstedt@axis
> > .com> wrote:
> > > since I see that you have integrated/staged the two patches that
> > > add
> > > inherits of pkgconfig, but not the patches that add the dummy
> > > commands,
> > > I assume you have some reservations to these patches. What are your
> > > take on the subject of blacklisting vs whitelisting the commands
> > > from
> > > the build host? I know that having these dummy commands in place
> > > helped
> > > me a great deal when updating our recipes to build correctly with
> > > RSS.
> > > 
> > FWIW I wasn't entirely keen on how the blacklisting had to be very
> > careful about how PATH was manipulated, which is why I merged to my
> > staging branch the fixes but not the dummy commands.
> > 
> > If the majority of people agree that this should be merged I don't
> > have a strong opinion either way.
> 
> Ultimately I think there is a better solution than this but I haven't
> had time to look at that as yet.
> 
> I do worry that it makes assumptions about PATH that may or may not be
> valid everywhere, equally I do see it could be useful.
> 
> So I also have mixed feelings...

I got many complains about my PNBLACKLISTs saying that the issues weren't
reproducible elsewhere and in the end it was in 90% one of these
commonly installed tools like pkgconfig being used from the host.

So I agree that PATH changes are tricky and that whitelisting would be
even better than this, but still I would prefer to get this merged as it
will help to identify and fix most common issues and changing this to
whitelist would be easier in future.

My jenkins buildhost is quite minimal (that's why it detected so many
issues with RSS, but I'll include the rest of this series in next run to
see if it will find even more).

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 201 bytes --]

Re: [PATCHv3 0/5] Add dummy tools to help identify needed dependencies

[Richard Purdie]

On Wed, 2017-03-08 at 19:41 +0100, Martin Jansa wrote:
> On Wed, Mar 08, 2017 at 06:00:30PM +0000, Richard Purdie wrote:
> > 
> > On Wed, 2017-03-08 at 17:21 +0000, Burton, Ross wrote:
> > > 
> > > 
> > > On 8 March 2017 at 09:43, Peter Kjellerstedt <peter.kjellerstedt@
> > > axis
> > > .com> wrote:
> > > > 
> > > > since I see that you have integrated/staged the two patches
> > > > that
> > > > add
> > > > inherits of pkgconfig, but not the patches that add the dummy
> > > > commands,
> > > > I assume you have some reservations to these patches. What are
> > > > your
> > > > take on the subject of blacklisting vs whitelisting the
> > > > commands
> > > > from
> > > > the build host? I know that having these dummy commands in
> > > > place
> > > > helped
> > > > me a great deal when updating our recipes to build correctly
> > > > with
> > > > RSS.
> > > > 
> > > FWIW I wasn't entirely keen on how the blacklisting had to be
> > > very
> > > careful about how PATH was manipulated, which is why I merged to
> > > my
> > > staging branch the fixes but not the dummy commands.
> > > 
> > > If the majority of people agree that this should be merged I
> > > don't
> > > have a strong opinion either way.
> > Ultimately I think there is a better solution than this but I
> > haven't
> > had time to look at that as yet.
> > 
> > I do worry that it makes assumptions about PATH that may or may not
> > be
> > valid everywhere, equally I do see it could be useful.
> > 
> > So I also have mixed feelings...
> I got many complains about my PNBLACKLISTs saying that the issues
> weren't reproducible elsewhere and in the end it was in 90% one of
> these commonly installed tools like pkgconfig being used from the
> host.
> 
> So I agree that PATH changes are tricky and that whitelisting would
> be even better than this, but still I would prefer to get this merged
> as it will help to identify and fix most common issues and changing
> this to whitelist would be easier in future.

I decided to see how hard adding a whitelist would be. I came up with
the patch below. Admittedly it changes both bitbake and OE-Core,
injects itself into OE-Core's layer.conf and is generally pretty
horrible in some ways but equally it does let me start to run builds.
It crashes and burns somewhere after libtool-native and the list of
tools clearly needs work but it does show what we could do.

I'm therefore torn on Peter's blacklist, or whether its worth a push to
the whitelist now...

How badly do people dislike the patch below?

From: Richard Purdie <richard.purdie@linuxfoundation.org>
Subject: bitbake/oe-core: Filter contents of PATH

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py
index d6bcfa3..dbb74dd 100644
--- a/bitbake/lib/bb/utils.py
+++ b/bitbake/lib/bb/utils.py
@@ -1526,3 +1526,13 @@ class LogCatcher(logging.Handler):
         self.messages.append(bb.build.logformatter.format(record))
     def contains(self, message):
         return (message in self.messages)
+
+def setup_native_bindir(dest, toolsvar, d):
+    tools = d.getVar(toolsvar).split()
+    path = os.environ.get("PATH")
+    mkdirhier(dest)
+    for tool in tools:
+        desttool = os.path.join(dest, tool)
+        if not os.path.exists(desttool):
+            srctool = which(path, tool)
+            os.symlink(srctool, desttool)
diff --git a/meta/conf/layer.conf b/meta/conf/layer.conf
index 87c235f..21265ed 100644
--- a/meta/conf/layer.conf
+++ b/meta/conf/layer.conf
@@ -59,3 +59,14 @@ SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS += " \
   oprofile->virtual/kernel \
 "
 
+NATIVETOOLS = " \
+    bash sh cut sed gcc ld git rm install which find xargs cat true mktemp \
+    grep tar gzip touch cp mv basename dirname tr getopt sort awk head tail \
+    mkdir patch uniq perl python chmod python3 ar strip expr ls make as \
+    ranlib egrep echo chown cpio tee wc wget bzip2 stat date rmdir od diff \
+    md5sum unlzma dd chrpath file pod2man gunzip python2.7 ln g++ [ \
+    taskset \
+"
+
+DUMMY := "${@bb.utils.setup_native_bindir('${TOPDIR}/nativetools', 'NATIVETOOLS', d)}"
+PATH = "${TOPDIR}/nativetools"

Re: [PATCHv3 0/5] Add dummy tools to help identify needed dependencies

[Richard Purdie]

On Thu, 2017-03-09 at 00:18 +0000, Richard Purdie wrote:
> How badly do people dislike the patch below?
> 
> From: Richard Purdie <richard.purdie@linuxfoundation.org>
> Subject: bitbake/oe-core: Filter contents of PATH
> 
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> 
> diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py
> index d6bcfa3..dbb74dd 100644
> --- a/bitbake/lib/bb/utils.py
> +++ b/bitbake/lib/bb/utils.py
> @@ -1526,3 +1526,13 @@ class LogCatcher(logging.Handler):
>          self.messages.append(bb.build.logformatter.format(record))
>      def contains(self, message):
>          return (message in self.messages)
> +
> +def setup_native_bindir(dest, toolsvar, d):
> +    tools = d.getVar(toolsvar).split()
> +    path = os.environ.get("PATH")
> +    mkdirhier(dest)
> +    for tool in tools:
> +        desttool = os.path.join(dest, tool)
> +        if not os.path.exists(desttool):
> +            srctool = which(path, tool)
> +            os.symlink(srctool, desttool)
> diff --git a/meta/conf/layer.conf b/meta/conf/layer.conf
> index 87c235f..21265ed 100644
> --- a/meta/conf/layer.conf
> +++ b/meta/conf/layer.conf
> @@ -59,3 +59,14 @@ SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS += " \
>    oprofile->virtual/kernel \
>  "
>  
> +NATIVETOOLS = " \
> +    bash sh cut sed gcc ld git rm install which find xargs cat true
> mktemp \
> +    grep tar gzip touch cp mv basename dirname tr getopt sort awk
> head tail \
> +    mkdir patch uniq perl python chmod python3 ar strip expr ls make
> as \
> +    ranlib egrep echo chown cpio tee wc wget bzip2 stat date rmdir
> od diff \
> +    md5sum unlzma dd chrpath file pod2man gunzip python2.7 ln g++ [
> \
> +    taskset \

Adding "false true uname test hostname nm objdump objcopy cmp printf
env" gets this going much further. taskset is only above as I have
local patches in my build which use it for other experiments and I
added it just to get things working.

Cheers,

Richard