Extending images

Extending images

[Gary Thomas]

I've always used 'IMAGE_INSTALL += " xyz"' in my local.conf
to add new packages to a build.  That said, I had a working
build for qemuarm (probably doesn't matter) and I added:
   IMAGE_INSTALL += " strace"
This produced a completely broken image which barely came
up to a shell, lots of missing programs, etc.

I then tried
   CORE_IMAGE_EXTRA_INSTALL += " strace"
which produced a perfectly working build (just as previous)
with the new package added.

I can see that the images produced are vastly different:

* Original working build
-rw-r--r-- 1 gthomas gthomas     11719 Jul 18 09:34 core-image-sato-qemuarm-20140718124453.rootfs.manifest
-rw-r--r-- 1 gthomas gthomas  87611203 Jul 18 09:34 core-image-sato-qemuarm-20140718124453.rootfs.tar.bz2

* After 'IMAGE_INSTALL += '
-rw-r--r-- 1 gthomas gthomas      9986 Jul 18 09:55 core-image-sato-qemuarm-20140718155134.rootfs.manifest
-rw-r--r-- 1 gthomas gthomas  37859884 Jul 18 09:56 core-image-sato-qemuarm-20140718155134.rootfs.tar.bz2

* After 'CORE_IMAGE_EXTRA_INSTALL += '
-rw-r--r-- 1 gthomas gthomas     11738 Jul 18 10:05 core-image-sato-qemuarm-20140718160108.rootfs.manifest
-rw-r--r-- 1 gthomas gthomas  87720106 Jul 18 10:05 core-image-sato-qemuarm-20140718160108.rootfs.tar.bz2

What's the difference and why did the first attempt fail?

Thanks

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: Extending images

[Robert P. J. Day]

On Fri, 18 Jul 2014, Gary Thomas wrote:

> I've always used 'IMAGE_INSTALL += " xyz"' in my local.conf
> to add new packages to a build.  That said, I had a working
> build for qemuarm (probably doesn't matter) and I added:
>   IMAGE_INSTALL += " strace"
> This produced a completely broken image which barely came
> up to a shell, lots of missing programs, etc.
>
> I then tried
>   CORE_IMAGE_EXTRA_INSTALL += " strace"
> which produced a perfectly working build (just as previous)
> with the new package added.
>
> I can see that the images produced are vastly different:
>
> * Original working build
> -rw-r--r-- 1 gthomas gthomas     11719 Jul 18 09:34
> core-image-sato-qemuarm-20140718124453.rootfs.manifest
> -rw-r--r-- 1 gthomas gthomas  87611203 Jul 18 09:34
> core-image-sato-qemuarm-20140718124453.rootfs.tar.bz2
>
> * After 'IMAGE_INSTALL += '
> -rw-r--r-- 1 gthomas gthomas      9986 Jul 18 09:55
> core-image-sato-qemuarm-20140718155134.rootfs.manifest
> -rw-r--r-- 1 gthomas gthomas  37859884 Jul 18 09:56
> core-image-sato-qemuarm-20140718155134.rootfs.tar.bz2
>
> * After 'CORE_IMAGE_EXTRA_INSTALL += '
> -rw-r--r-- 1 gthomas gthomas     11738 Jul 18 10:05
> core-image-sato-qemuarm-20140718160108.rootfs.manifest
> -rw-r--r-- 1 gthomas gthomas  87720106 Jul 18 10:05
> core-image-sato-qemuarm-20140718160108.rootfs.tar.bz2
>
> What's the difference and why did the first attempt fail?

  i'm going to guess this is related to my recent whining about how
weirdly core-image-* recipes are defined.  if i use the "bb" command
to see what will be installed on a qemuarm in a core-image-sato image,
i see:

$ bb show IMAGE_INSTALL -r core-image-sato
Parsing recipes..done.
# IMAGE_INSTALL=${CORE_IMAGE_BASE_INSTALL} packagegroup-core-x11-sato-games
IMAGE_INSTALL="packagegroup-core-boot packagegroup-base-extended \
  packagegroup-core-x11-sato-games"
$

  which looks good. but the instant i add the line:

IMAGE_INSTALL += "strace" to my local.conf ... whoops:

$ bb show IMAGE_INSTALL -r core-image-sato
Parsing recipes..done.
IMAGE_INSTALL=" strace packagegroup-core-x11-sato-games"
$

  note the loss of the essential core-boot and base-extended
packagegroups, which is why your image is broken, which is because of
the way IMAGE_INSTALL is assigned in core-image.bbclass.

  lesson: do not mess with IMAGE_INSTALL directly unless you truly
know what you're doing.

rday

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

Re: Extending images

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 2109 bytes --]

On Fri, Jul 18, 2014 at 9:26 AM, Gary Thomas <gary@mlbassoc.com> wrote:

> 've always used 'IMAGE_INSTALL += " xyz"' in my local.conf
> to add new packages to a build.  That said, I had a working
> build for qemuarm (probably doesn't matter) and I added:
>   IMAGE_INSTALL += " strace"
> This produced a completely broken image which barely came
> up to a shell, lots of missing programs, etc.
>
> I then tried
>   CORE_IMAGE_EXTRA_INSTALL += " strace"
> which produced a perfectly working build (just as previous)
> with the new package added.
>
> I can see that the images produced are vastly different:
>
> * Original working build
> -rw-r--r-- 1 gthomas gthomas     11719 Jul 18 09:34
> core-image-sato-qemuarm-20140718124453.rootfs.manifest
> -rw-r--r-- 1 gthomas gthomas  87611203 Jul 18 09:34
> core-image-sato-qemuarm-20140718124453.rootfs.tar.bz2
>
> * After 'IMAGE_INSTALL += '
> -rw-r--r-- 1 gthomas gthomas      9986 Jul 18 09:55
> core-image-sato-qemuarm-20140718155134.rootfs.manifest
> -rw-r--r-- 1 gthomas gthomas  37859884 Jul 18 09:56
> core-image-sato-qemuarm-20140718155134.rootfs.tar.bz2
>
> * After 'CORE_IMAGE_EXTRA_INSTALL += '
> -rw-r--r-- 1 gthomas gthomas     11738 Jul 18 10:05
> core-image-sato-qemuarm-20140718160108.rootfs.manifest
> -rw-r--r-- 1 gthomas gthomas  87720106 Jul 18 10:05
> core-image-sato-qemuarm-20140718160108.rootfs.tar.bz2
>
> What's the difference and why did the first attempt fail?
>

Most likely the image defined its own IMAGE_INSTALL using ?=, so defining
it yourself in the configuration data overrode its default definition,
since ?= is "set only if unset". If the recipe didn't use ?=, then your
IMAGE_INSTALL += would have had no effect at all. To append to
IMAGE_INSTALL directly without using CORE_IMAGE_EXTRA_INSTALL you could
have used IMAGE_INSTALL_append, since that's a postponed operation that
happens at the end of the recipe parsing.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 2928 bytes --]

Re: Extending images

[Gary Thomas]

On 2014-07-18 10:49, Christopher Larson wrote:
>
> On Fri, Jul 18, 2014 at 9:26 AM, Gary Thomas <gary@mlbassoc.com <mailto:gary@mlbassoc.com>> wrote:
>
>     've always used 'IMAGE_INSTALL += " xyz"' in my local.conf
>     to add new packages to a build.  That said, I had a working
>     build for qemuarm (probably doesn't matter) and I added:
>        IMAGE_INSTALL += " strace"
>     This produced a completely broken image which barely came
>     up to a shell, lots of missing programs, etc.
>
>     I then tried
>        CORE_IMAGE_EXTRA_INSTALL += " strace"
>     which produced a perfectly working build (just as previous)
>     with the new package added.
>
>     I can see that the images produced are vastly different:
>
>     * Original working build
>     -rw-r--r-- 1 gthomas gthomas     11719 Jul 18 09:34 core-image-sato-qemuarm-__20140718124453.rootfs.manifest
>     -rw-r--r-- 1 gthomas gthomas  87611203 Jul 18 09:34 core-image-sato-qemuarm-__20140718124453.rootfs.tar.bz2
>
>     * After 'IMAGE_INSTALL += '
>     -rw-r--r-- 1 gthomas gthomas      9986 Jul 18 09:55 core-image-sato-qemuarm-__20140718155134.rootfs.manifest
>     -rw-r--r-- 1 gthomas gthomas  37859884 Jul 18 09:56 core-image-sato-qemuarm-__20140718155134.rootfs.tar.bz2
>
>     * After 'CORE_IMAGE_EXTRA_INSTALL += '
>     -rw-r--r-- 1 gthomas gthomas     11738 Jul 18 10:05 core-image-sato-qemuarm-__20140718160108.rootfs.manifest
>     -rw-r--r-- 1 gthomas gthomas  87720106 Jul 18 10:05 core-image-sato-qemuarm-__20140718160108.rootfs.tar.bz2
>
>     What's the difference and why did the first attempt fail?
>
>
> Most likely the image defined its own IMAGE_INSTALL using ?=, so defining it yourself in the configuration data overrode its default definition, since ?= is "set only if unset". If
> the recipe didn't use ?=, then your IMAGE_INSTALL += would have had no effect at all. To append to IMAGE_INSTALL directly without using CORE_IMAGE_EXTRA_INSTALL you could have used
> IMAGE_INSTALL_append, since that's a postponed operation that happens at the end of the recipe parsing.

That makes sense, thanks.  I'll stick to using CORE_IMAGE_EXTRA_INSTALL
from now on!

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: Extending images

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 1400 bytes --]

On Fri, Jul 18, 2014 at 9:54 AM, Gary Thomas <gary@mlbassoc.com> wrote:

> Most likely the image defined its own IMAGE_INSTALL using ?=, so defining
>> it yourself in the configuration data overrode its default definition,
>> since ?= is "set only if unset". If
>> the recipe didn't use ?=, then your IMAGE_INSTALL += would have had no
>> effect at all. To append to IMAGE_INSTALL directly without using
>> CORE_IMAGE_EXTRA_INSTALL you could have used
>> IMAGE_INSTALL_append, since that's a postponed operation that happens at
>> the end of the recipe parsing.
>>
>
> That makes sense, thanks.  I'll stick to using CORE_IMAGE_EXTRA_INSTALL
> from now on!


That's best, indeed. This is a case where there's a real difference between
variables intended for the user to override from configuration, and those
which are not. IMAGE_INSTALL is a detail of how images are constructed,
CORE_IMAGE_EXTRA_INSTALL is an explicit hook provided to the user to
non-destructively add packages to an image, so we don't have to try to
guess at what the recipe will do with it -- it has explicit semantics, so
we know this. There might be value in differentiating between cases like
this in the documentation, if we don't already.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 1893 bytes --]

Re: Extending images

[Robert P. J. Day]

On Fri, 18 Jul 2014, Gary Thomas wrote:

> On 2014-07-18 10:49, Christopher Larson wrote:
> >
> > On Fri, Jul 18, 2014 at 9:26 AM, Gary Thomas <gary@mlbassoc.com
> > <mailto:gary@mlbassoc.com>> wrote:
> >
> >     've always used 'IMAGE_INSTALL += " xyz"' in my local.conf
> >     to add new packages to a build.  That said, I had a working
> >     build for qemuarm (probably doesn't matter) and I added:
> >        IMAGE_INSTALL += " strace"
> >     This produced a completely broken image which barely came
> >     up to a shell, lots of missing programs, etc.
> >
> >     I then tried
> >        CORE_IMAGE_EXTRA_INSTALL += " strace"
> >     which produced a perfectly working build (just as previous)
> >     with the new package added.
> >
> >     I can see that the images produced are vastly different:
> >
> >     * Original working build
> >     -rw-r--r-- 1 gthomas gthomas     11719 Jul 18 09:34
> >     core-image-sato-qemuarm-__20140718124453.rootfs.manifest
> >     -rw-r--r-- 1 gthomas gthomas  87611203 Jul 18 09:34
> >     core-image-sato-qemuarm-__20140718124453.rootfs.tar.bz2
> >
> >     * After 'IMAGE_INSTALL += '
> >     -rw-r--r-- 1 gthomas gthomas      9986 Jul 18 09:55
> >     core-image-sato-qemuarm-__20140718155134.rootfs.manifest
> >     -rw-r--r-- 1 gthomas gthomas  37859884 Jul 18 09:56
> >     core-image-sato-qemuarm-__20140718155134.rootfs.tar.bz2
> >
> >     * After 'CORE_IMAGE_EXTRA_INSTALL += '
> >     -rw-r--r-- 1 gthomas gthomas     11738 Jul 18 10:05
> >     core-image-sato-qemuarm-__20140718160108.rootfs.manifest
> >     -rw-r--r-- 1 gthomas gthomas  87720106 Jul 18 10:05
> >     core-image-sato-qemuarm-__20140718160108.rootfs.tar.bz2
> >
> >     What's the difference and why did the first attempt fail?
> >
> >
> > Most likely the image defined its own IMAGE_INSTALL using ?=, so
> > defining it yourself in the configuration data overrode its
> > default definition, since ?= is "set only if unset". If the recipe
> > didn't use ?=, then your IMAGE_INSTALL += would have had no effect
> > at all. To append to IMAGE_INSTALL directly without using
> > CORE_IMAGE_EXTRA_INSTALL you could have used IMAGE_INSTALL_append,
> > since that's a postponed operation that happens at the end of the
> > recipe parsing.
>
> That makes sense, thanks.  I'll stick to using
> CORE_IMAGE_EXTRA_INSTALL from now on!

  not to sound repetitively repetitive, but this is related to what i
thought was the weird way core images are defined based off of
core-image.bbclass, where core-image.bbclass does *exactly* what you
describe above:

CORE_IMAGE_BASE_INSTALL = '\
    packagegroup-core-boot \
    packagegroup-base-extended \
    \
    ${CORE_IMAGE_EXTRA_INSTALL} \
    '

CORE_IMAGE_EXTRA_INSTALL ?= ""

IMAGE_INSTALL ?= "${CORE_IMAGE_BASE_INSTALL}"

  the danger here is that if someone has been defining/building images
for a while based off of the lower-level image.bbclass, then (i think)
using "IMAGE_INSTALL +=" will work just fine, and the poor developer
gets lulled into a false sense of security and carries that habit over
to her first core-image and ... boom.

  i know there's a standard to define variables prefixed with "EXTRA_"
meant for users to add features, and i think that's a great idea. i
think the docs should strongly emphasize that users should look for
the EXTRA_ variable before messing with anything else.

rday

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

Re: Extending images

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 3803 bytes --]

On Fri, Jul 18, 2014 at 11:45 AM, Robert P. J. Day <rpjday@crashcourse.ca>
wrote:

>
> > On 2014-07-18 10:49, Christopher Larson wrote:
> > >
> > > On Fri, Jul 18, 2014 at 9:26 AM, Gary Thomas <gary@mlbassoc.com
> > > <mailto:gary@mlbassoc.com>> wrote:
> > >
> > >     've always used 'IMAGE_INSTALL += " xyz"' in my local.conf
> > >     to add new packages to a build.  That said, I had a working
> > >     build for qemuarm (probably doesn't matter) and I added:
> > >        IMAGE_INSTALL += " strace"
> > >     This produced a completely broken image which barely came
> > >     up to a shell, lots of missing programs, etc.
> > >
> > >     I then tried
> > >        CORE_IMAGE_EXTRA_INSTALL += " strace"
> > >     which produced a perfectly working build (just as previous)
> > >     with the new package added.
> > >
> > >     I can see that the images produced are vastly different:
> > >
> > >     * Original working build
> > >     -rw-r--r-- 1 gthomas gthomas     11719 Jul 18 09:34
> > >     core-image-sato-qemuarm-__20140718124453.rootfs.manifest
> > >     -rw-r--r-- 1 gthomas gthomas  87611203 Jul 18 09:34
> > >     core-image-sato-qemuarm-__20140718124453.rootfs.tar.bz2
> > >
> > >     * After 'IMAGE_INSTALL += '
> > >     -rw-r--r-- 1 gthomas gthomas      9986 Jul 18 09:55
> > >     core-image-sato-qemuarm-__20140718155134.rootfs.manifest
> > >     -rw-r--r-- 1 gthomas gthomas  37859884 Jul 18 09:56
> > >     core-image-sato-qemuarm-__20140718155134.rootfs.tar.bz2
> > >
> > >     * After 'CORE_IMAGE_EXTRA_INSTALL += '
> > >     -rw-r--r-- 1 gthomas gthomas     11738 Jul 18 10:05
> > >     core-image-sato-qemuarm-__20140718160108.rootfs.manifest
> > >     -rw-r--r-- 1 gthomas gthomas  87720106 Jul 18 10:05
> > >     core-image-sato-qemuarm-__20140718160108.rootfs.tar.bz2
> > >
> > >     What's the difference and why did the first attempt fail?
> > >
> > >
> > > Most likely the image defined its own IMAGE_INSTALL using ?=, so
> > > defining it yourself in the configuration data overrode its
> > > default definition, since ?= is "set only if unset". If the recipe
> > > didn't use ?=, then your IMAGE_INSTALL += would have had no effect
> > > at all. To append to IMAGE_INSTALL directly without using
> > > CORE_IMAGE_EXTRA_INSTALL you could have used IMAGE_INSTALL_append,
> > > since that's a postponed operation that happens at the end of the
> > > recipe parsing.
> >
> > That makes sense, thanks.  I'll stick to using
> > CORE_IMAGE_EXTRA_INSTALL from now on!
>
>   not to sound repetitively repetitive, but this is related to what i
> thought was the weird way core images are defined based off of
> core-image.bbclass, where core-image.bbclass does *exactly* what you
> describe above:
>
> CORE_IMAGE_BASE_INSTALL = '\
>     packagegroup-core-boot \
>     packagegroup-base-extended \
>     \
>     ${CORE_IMAGE_EXTRA_INSTALL} \
>     '
>
> CORE_IMAGE_EXTRA_INSTALL ?= ""
>
> IMAGE_INSTALL ?= "${CORE_IMAGE_BASE_INSTALL}"
>
>   the danger here is that if someone has been defining/building images
> for a while based off of the lower-level image.bbclass, then (i think)
> using "IMAGE_INSTALL +=" will work just fine, and the poor developer
> gets lulled into a false sense of security and carries that habit over
> to her first core-image and ... boom.
>

I really don't think this was ever safe, as it completely depended on how
the image recipe added its bits to IMAGE_INSTALL, which could easily vary
from image to image. I've *always* told folks to use IMAGE_INSTALL_append
if anything, even back in oe classic :)
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 5000 bytes --]

Re: Extending images

[Robert P. J. Day]

On Fri, 18 Jul 2014, Christopher Larson wrote:

> I really don't think this was ever safe, as it completely depended
> on how the image recipe added its bits to IMAGE_INSTALL, which could
> easily vary from image to image. I've *always* told folks to use
> IMAGE_INSTALL_append if anything, even back in oe classic :) --

  a quick grep in openembedded-core/meta shows:

$ grep -r "IMAGE_INSTALL +=" *
recipes-core/images/core-image-minimal-mtdutils.bb:IMAGE_INSTALL += "mtd-utils"
recipes-extended/images/core-image-testmaster.bb:IMAGE_INSTALL += "\
recipes-extended/images/core-image-lsb-sdk.bb:IMAGE_INSTALL += "kernel-dev"
recipes-graphics/images/core-image-directfb.bb:IMAGE_INSTALL += "\
recipes-qt/images/qt4e-demo-image.bb:IMAGE_INSTALL += "\
recipes-rt/images/core-image-rt-sdk.bb:IMAGE_INSTALL += "rt-tests hwlatdetect kernel-dev"
recipes-rt/images/core-image-rt.bb:IMAGE_INSTALL += "rt-tests hwlatdetect"
recipes-sato/images/core-image-sato-sdk.bb:IMAGE_INSTALL += "kernel-dev"
recipes-sato/images/core-image-sato.bb:IMAGE_INSTALL += "packagegroup-core-x11-sato-games"
$

  now i'm scared to look. :-)

rday

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

Re: Extending images

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 2642 bytes --]

On Fri, Jul 18, 2014 at 11:55 AM, Robert P. J. Day <rpjday@crashcourse.ca>
wrote:

> On Fri, 18 Jul 2014, Christopher Larson wrote:
>
> > I really don't think this was ever safe, as it completely depended
> > on how the image recipe added its bits to IMAGE_INSTALL, which could
> > easily vary from image to image. I've *always* told folks to use
> > IMAGE_INSTALL_append if anything, even back in oe classic :) --
>
>   a quick grep in openembedded-core/meta shows:
>
> $ grep -r "IMAGE_INSTALL +=" *
> recipes-core/images/core-image-minimal-mtdutils.bb:IMAGE_INSTALL +=
> "mtd-utils"
> recipes-extended/images/core-image-testmaster.bb:IMAGE_INSTALL += "\
> recipes-extended/images/core-image-lsb-sdk.bb:IMAGE_INSTALL +=
> "kernel-dev"
> recipes-graphics/images/core-image-directfb.bb:IMAGE_INSTALL += "\
> recipes-qt/images/qt4e-demo-image.bb:IMAGE_INSTALL += "\
> recipes-rt/images/core-image-rt-sdk.bb:IMAGE_INSTALL += "rt-tests
> hwlatdetect kernel-dev"
> recipes-rt/images/core-image-rt.bb:IMAGE_INSTALL += "rt-tests hwlatdetect"
> recipes-sato/images/core-image-sato-sdk.bb:IMAGE_INSTALL += "kernel-dev"
> recipes-sato/images/core-image-sato.bb:IMAGE_INSTALL +=
> "packagegroup-core-x11-sato-games"
> $
>
>   now i'm scared to look. :-)


This is what they should be doing in most cases. The only time you need
_append is when you're defining it from local.conf, because the behavior of
IMAGE_INSTALL += in local.conf depends on what the recipe is doing (does it
IMAGE_INSTALL = (overrides the user's value), ?= (only uses the user's
value, not its own), or += (adds to an already set value)). This is where
the value lies in providing the user explicit variables with known
semantics, so the user doesn't have to know these peculiarities and can
just set the variable already :)

That said, this is definitely another case of many you've pointed out where
our recipes just aren't as consistent as would be ideal. IMO when a recipe
takes a list variable like this and does just an '=' it's basically saying
"I must have only these things, throw away anything already set", so the
only time it should be set that way is when there's such an explicit need.
Whereas most of the time what they really wanted to say was "I need these
things", in which case += is more appropriate (and I'm not just talking
about IMAGE_INSTALL here, but also variables like RDEPENDS and DEPENDS) :)
But that's just my personal view on the subject.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 3206 bytes --]