ASSUME_PROVIDED versus SANITY_REQUIRED_UTILITIES versus "The Build Host Packages"

ASSUME_PROVIDED versus SANITY_REQUIRED_UTILITIES versus "The Build Host Packages"

[Robert P. J. Day]

  a bit confused by what i'm seeing in a recent
qemuppc/core-image-minimal build on my fedora system regarding which
native packages are built, despite what's in bitbake.conf (using
current poky layer checkout).

  first, here's a snippet from bitbake.conf:

  ASSUME_PROVIDED = "\
    bzip2-native \          <---
    chrpath-native \
    file-native \           <---
    findutils-native \
    git-native \
    grep-native \
    diffstat-native \
    ... snip ...

suggesting that (among other things) bzip2-native and file-native
shouldn't be built -- it's the developer's responsibility to install
them, yes? but if i peek under tmp/work/x86_64-linux, i can see:

  ...
  byacc-native/
  bzip2-native/             <---
  createrepo-native/
  cross-localedef-native/
  cryptodev-linux-native/
  db-native/
  dtc-native/
  e2fsprogs-native/
  elfutils-native/
  expat-native/
  file-native/              <---
  flex-native/
  ...

so that's ... confusing.

  add to that this bit in the YP Quick Start Guide at
http://www.yoctoproject.org/docs/2.1/yocto-project-qs/yocto-project-qs.html:

  "Build Host Packages
  ...
  Fedora

     $ sudo dnf install gawk make wget tar bzip2 gzip python unzip perl patch \
     diffutils diffstat git cpp gcc gcc-c++ glibc-devel texinfo chrpath \
     ccache perl-Data-Dumper perl-Text-ParseWords perl-Thread-Queue perl-bignum socat \
     findutils which SDL-devel xterm

my understanding was that that list represented what the developer was
required to install on his build host *manually* before going any
further, so you can see "bzip2" in that list, but you don't see
"file". given that the "file" command comes in its own package, should
that list of allegedly mandatory Build Host Packages not include
*everything* listed in ASSUME_PROVIDED?

  finally, in sanity.bbclass, one reads:

  SANITY_REQUIRED_UTILITIES ?= "patch diffstat makeinfo git bzip2 tar \
      gzip gawk chrpath wget cpio perl file"

which oddly *does* list the "file" command, but says nothing about
"sed", which *is* mentioned in ASSUME_PROVIDED, but is also not
mentioned in the "Build Host Packages" in the YP Quick Start Guide.

  is it just me, or does some of this seem out of sync?

rday

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

Re: ASSUME_PROVIDED versus SANITY_REQUIRED_UTILITIES versus "The Build Host Packages"

[Richard Purdie]

On Mon, 2016-11-28 at 06:20 -0500, Robert P. J. Day wrote:
>   a bit confused by what i'm seeing in a recent
> qemuppc/core-image-minimal build on my fedora system regarding which
> native packages are built, despite what's in bitbake.conf (using
> current poky layer checkout).
> 
>   first, here's a snippet from bitbake.conf:
> 
>   ASSUME_PROVIDED = "\
>     bzip2-native \          <---
>     chrpath-native \
>     file-native \           <---
>     findutils-native \
>     git-native \
>     grep-native \
>     diffstat-native \
>     ... snip ...
> 
> suggesting that (among other things) bzip2-native and file-native
> shouldn't be built -- it's the developer's responsibility to install
> them, yes? but if i peek under tmp/work/x86_64-linux, i can see:

There are two ways "file-native" can be used in a build. It can be used
as the host provided "file" command and it is also needed by the file
recipe to build file for the target. We need to build file-native in
order for the build for the target file command to build/work. This is
why it gets built.

With bzip2, we need libbz2 (native) during the build process and we
don't ASSUME_PROVIDED libbz2-devel, hence it builds bzip2-native only
for that piece. Confusing perhaps but necessary unfortunately.

Cheers,

Richard

Re: ASSUME_PROVIDED versus SANITY_REQUIRED_UTILITIES versus "The Build Host Packages"

[Patrick Ohly]

On Mon, 2016-11-28 at 12:04 +0000, Richard Purdie wrote:
> On Mon, 2016-11-28 at 06:20 -0500, Robert P. J. Day wrote:
> >   a bit confused by what i'm seeing in a recent
> > qemuppc/core-image-minimal build on my fedora system regarding which
> > native packages are built, despite what's in bitbake.conf (using
> > current poky layer checkout).
> > 
> >   first, here's a snippet from bitbake.conf:
> > 
> >   ASSUME_PROVIDED = "\
> >     bzip2-native \          <---
> >     chrpath-native \
> >     file-native \           <---
> >     findutils-native \
> >     git-native \
> >     grep-native \
> >     diffstat-native \
> >     ... snip ...
> > 
> > suggesting that (among other things) bzip2-native and file-native
> > shouldn't be built -- it's the developer's responsibility to install
> > them, yes? but if i peek under tmp/work/x86_64-linux, i can see:
> 
> There are two ways "file-native" can be used in a build. It can be used
> as the host provided "file" command and it is also needed by the file
> recipe to build file for the target.

I recently ran into a third usage of "file-native": swupd-server links
against libmagic from file and therefore has a DEPENDS = "file". But
building swupd-server-native didn't actually build file because of
ASSUME_PROVIDED and because I hadn't installed libmagic-dev on my build
host, the build was failing.

Is there a way to declare that ASSUME_PROVIDED does not apply to this
case? It sounds like there is a way (based on your comments about
building file-native when building file and the libbz2-devel example),
but it did not become clear to me how that works in practice.

Or is it just a case of educating the developer that libmagic-dev needs
to be installed on the build host in addition to the file command?

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: ASSUME_PROVIDED versus SANITY_REQUIRED_UTILITIES versus "The Build Host Packages"

[Richard Purdie]

On Mon, 2016-11-28 at 14:17 +0100, Patrick Ohly wrote:
> I recently ran into a third usage of "file-native": swupd-server
> links against libmagic from file and therefore has a DEPENDS =
> "file". But building swupd-server-native didn't actually build file
> because of ASSUME_PROVIDED and because I hadn't installed libmagic-
> dev on my build host, the build was failing.
> 
> Is there a way to declare that ASSUME_PROVIDED does not apply to this
> case? It sounds like there is a way (based on your comments about
> building file-native when building file and the libbz2-devel
> example), but it did not become clear to me how that works in
> practice.

If you look at the file recipe, you'll see:

DEPENDS = "zlib file-replacement-native"


> Or is it just a case of educating the developer that libmagic-dev
> needs to be installed on the build host in addition to the file
> command?

No, DEPENDS on file-replacement-native.

Cheers,

Richard

Re: ASSUME_PROVIDED versus SANITY_REQUIRED_UTILITIES versus "The Build Host Packages"

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 499 bytes --]

On 28 November 2016 at 13:17, Patrick Ohly <patrick.ohly@intel.com> wrote:

> I recently ran into a third usage of "file-native": swupd-server links
> against libmagic from file and therefore has a DEPENDS = "file". But
> building swupd-server-native didn't actually build file because of
> ASSUME_PROVIDED and because I hadn't installed libmagic-dev on my build
> host, the build was failing.
>

For this case you can use file-replacement-native to get a full file-native
built.

Ross

[-- Attachment #2: Type: text/html, Size: 943 bytes --]

Re: ASSUME_PROVIDED versus SANITY_REQUIRED_UTILITIES versus "The Build Host Packages"

[Robert P. J. Day]

[-- Attachment #1: Type: text/plain, Size: 2502 bytes --]

On Mon, 28 Nov 2016, Richard Purdie wrote:

> On Mon, 2016-11-28 at 06:20 -0500, Robert P. J. Day wrote:
> >   a bit confused by what i'm seeing in a recent
> > qemuppc/core-image-minimal build on my fedora system regarding which
> > native packages are built, despite what's in bitbake.conf (using
> > current poky layer checkout).
> >
> >   first, here's a snippet from bitbake.conf:
> >
> >   ASSUME_PROVIDED = "\
> >     bzip2-native \          <---
> >     chrpath-native \
> >     file-native \           <---
> >     findutils-native \
> >     git-native \
> >     grep-native \
> >     diffstat-native \
> >     ... snip ...
> >
> > suggesting that (among other things) bzip2-native and file-native
> > shouldn't be built -- it's the developer's responsibility to install
> > them, yes? but if i peek under tmp/work/x86_64-linux, i can see:
>
> There are two ways "file-native" can be used in a build. It can be
> used as the host provided "file" command and it is also needed by
> the file recipe to build file for the target. We need to build
> file-native in order for the build for the target file command to
> build/work. This is why it gets built.
>
> With bzip2, we need libbz2 (native) during the build process and we
> don't ASSUME_PROVIDED libbz2-devel, hence it builds bzip2-native
> only for that piece. Confusing perhaps but necessary unfortunately.

  ah, gotcha ... so in cases like that, even if a recipe needs to be
*partially* built for libraries, that in no way affects the
ASSUME_PROVIDED setting, correct? as in, once I ASSUME_PROVIDED
"bzip2-native", then that *better* exist on my build host, correct?

rday

P.S. there is still some lack of sync amongst:

  * ASSUME_PROVIDED
  * sanity.bbclass
  * YP quick start guide

one example i mentioned is that the YP guide does not list "sed" as a
build host requirement, but "sed-native" is in ASSUME_PROVIDED;
however, it's not listed in SANITY_REQUIRED_UTILITIES. that seems
inconsistent.

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

Re: ASSUME_PROVIDED versus SANITY_REQUIRED_UTILITIES versus "The Build Host Packages"

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 677 bytes --]

On 29 November 2016 at 10:47, Robert P. J. Day <rpjday@crashcourse.ca>
wrote:

> one example i mentioned is that the YP guide does not list "sed" as a
> build host requirement, but "sed-native" is in ASSUME_PROVIDED;
> however, it's not listed in SANITY_REQUIRED_UTILITIES. that seems
> inconsistent.
>

It would be somewhat useful if there where a way to annotate in recipes
what binary to check for on the host, so just doing
ASSUME_PROVIDED=sed-native would look up in the sed recipe to check that
"sed" exists on the host, so that SANITY_REQUIRED_UTILITIES doesn't need to
be maintained separately.

Very low hanging fruit though.  Patches welcome!

Ross

[-- Attachment #2: Type: text/html, Size: 1215 bytes --]

Re: ASSUME_PROVIDED versus SANITY_REQUIRED_UTILITIES versus "The Build Host Packages"

[Paul Eggleton]

On Tue, 29 Nov 2016 11:31:44 Burton, Ross wrote:
> On 29 November 2016 at 10:47, Robert P. J. Day <rpjday@crashcourse.ca>
> wrote:
> > one example i mentioned is that the YP guide does not list "sed" as a
> > build host requirement, but "sed-native" is in ASSUME_PROVIDED;
> > however, it's not listed in SANITY_REQUIRED_UTILITIES. that seems
> > inconsistent.
> 
> It would be somewhat useful if there where a way to annotate in recipes
> what binary to check for on the host, so just doing
> ASSUME_PROVIDED=sed-native would look up in the sed recipe to check that
> "sed" exists on the host, so that SANITY_REQUIRED_UTILITIES doesn't need to
> be maintained separately.
> 
> Very low hanging fruit though.  Patches welcome!

How would you get access to that recipe-space information within 
sanity.bbclass though?

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: ASSUME_PROVIDED versus SANITY_REQUIRED_UTILITIES versus "The Build Host Packages"

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 242 bytes --]

On 29 November 2016 at 17:24, Paul Eggleton <paul.eggleton@linux.intel.com>
wrote:

> How would you get access to that recipe-space information within
> sanity.bbclass though?
>

Yeah good point, that might be "a bit" tricky.

Ross

[-- Attachment #2: Type: text/html, Size: 652 bytes --]