SDK question: does "-c populate_sdk" build SDK based on entire image?

SDK question: does "-c populate_sdk" build SDK based on entire image?

[Robert P. J. Day]

  almost certainly a silly question as i'm still poring over the
mechanics of standard SDK creation, but if i define a perfectly normal
image, then build the corresponding SDK with:

  $ bitbake -c populate_sdk my_image

is the resulting SDK populated based on the entire contents of the
target image? that is, if i subsequently add a new package to the
target and rebuild the SDK, will the new SDK now contain the
corresponding content from that newly-added package? (i'm just about
to test this with a build but that's going to take over an hour on
this server. *sigh* ...)

  now, i realize that if i just want to add content to the SDK without
adding to target, i can use one or both of:

  TOOLCHAIN_HOST_TASK
  TOOLCHAIN_TARGET_TASK

but if someone asks for new content both for the target and the SDK,
is it sufficient to just add to target?

  and one more question that inspired all of this. i was asked to add
a particular executable to an SDK, call it /usr/bin/fubar. fair
enough, i thought, it was short work to determine that that binary
came from the recipe fubar_1.0.bb. but i checked and that package was,
in fact, being installed in the target, so i was puzzled as to why it
wasn't in the SDK as well.

  i checked the recipe for fubar and, to my chagrin, it contained the
snippet:

  do_install_append() {
    rm -rf ${D}/usr/bin/fubar
  }

well, that explained it, so as a first test, i'm just going to dump
that snippet from the recipe, which will install the executable on the
target, and will(?) also install it in the SDK?

rday

Re: [OE-core] SDK question: does "-c populate_sdk" build SDK based on entire image?

[Quentin Schulz]

Hi Robert,

No SDK expert here, so as usual, to be taken with a grain of salt.

On Fri, May 07, 2021 at 09:11:37AM -0400, Robert P. J. Day wrote:
> 
>   almost certainly a silly question as i'm still poring over the
> mechanics of standard SDK creation, but if i define a perfectly normal
> image, then build the corresponding SDK with:
> 
>   $ bitbake -c populate_sdk my_image
> 
> is the resulting SDK populated based on the entire contents of the
> target image? that is, if i subsequently add a new package to the
> target and rebuild the SDK, will the new SDK now contain the
> corresponding content from that newly-added package? (i'm just about
> to test this with a build but that's going to take over an hour on
> this server. *sigh* ...)
> 

I think you're mixing SDK and eSDK. AFAIU so far, you're looking for an
eSDK. An SDK just gives you the minimal toolchain and associated tools,
to be able to compile something. I would go as far as saying it's
basically what a PC distro package for cross-compile with gcc is doing.

Most of headers, shlib and binaries from an image won't make it to the
SDK. A much bigger portion of those will be installed in an SDK, though
not all of them. I have never used an eSDK so can't comment on what's
missing or how to add packages to the eSDK.

>   now, i realize that if i just want to add content to the SDK without
> adding to target, i can use one or both of:
> 
>   TOOLCHAIN_HOST_TASK
>   TOOLCHAIN_TARGET_TASK
> 
> but if someone asks for new content both for the target and the SDK,
> is it sufficient to just add to target?
> 

I'd assume that TOOLCHAIN_HOST_TASK is for native packages and
TOOLCHAIN_TARGET_TASK for target packages, for an SDK,
TOOLCHAIN_HOST_TASK would contain arm-linux-gcc, make, cmake, whatever
common build tools needed and TOOLCHAIN_TARGET_TASK headers/shlib for
compiling binaries to be run on your target, compiled via tools from
TOOLCHAIN_HOST_TASK.

Not sure this is remotely correct, just some thoughts.

Hope this can help/guide a bit?

Cheers,
Quentin

Re: [OE-core] SDK question: does "-c populate_sdk" build SDK based on entire image?

[Robert P. J. Day]

On Fri, 7 May 2021, Quentin Schulz wrote:

> Hi Robert,
>
> No SDK expert here, so as usual, to be taken with a grain of salt.
>
> On Fri, May 07, 2021 at 09:11:37AM -0400, Robert P. J. Day wrote:
> >
> >   almost certainly a silly question as i'm still poring over the
> > mechanics of standard SDK creation, but if i define a perfectly normal
> > image, then build the corresponding SDK with:
> >
> >   $ bitbake -c populate_sdk my_image
> >
> > is the resulting SDK populated based on the entire contents of the
> > target image? that is, if i subsequently add a new package to the
> > target and rebuild the SDK, will the new SDK now contain the
> > corresponding content from that newly-added package? (i'm just about
> > to test this with a build but that's going to take over an hour on
> > this server. *sigh* ...)
> >
>
> I think you're mixing SDK and eSDK. AFAIU so far, you're looking for
> an eSDK. An SDK just gives you the minimal toolchain and associated
> tools, to be able to compile something. I would go as far as saying
> it's basically what a PC distro package for cross-compile with gcc
> is doing.

  ... snip ...

i suspect you're right, i've been reading both sections in the SDK
manual and have probably just started mixing them up. i swear, once i
figure out how it all works, i'll rewrite the manual. :-)

rday

Re: [OE-core] SDK question: does "-c populate_sdk" build SDK based on entire image?

[Robert P. J. Day]

On Fri, 7 May 2021, Quentin Schulz wrote:

> Hi Robert,
>
> No SDK expert here, so as usual, to be taken with a grain of salt.
>
> On Fri, May 07, 2021 at 09:11:37AM -0400, Robert P. J. Day wrote:
> >
> >   almost certainly a silly question as i'm still poring over the
> > mechanics of standard SDK creation, but if i define a perfectly normal
> > image, then build the corresponding SDK with:
> >
> >   $ bitbake -c populate_sdk my_image
> >
> > is the resulting SDK populated based on the entire contents of the
> > target image? that is, if i subsequently add a new package to the
> > target and rebuild the SDK, will the new SDK now contain the
> > corresponding content from that newly-added package? (i'm just about
> > to test this with a build but that's going to take over an hour on
> > this server. *sigh* ...)
> >
>
> I think you're mixing SDK and eSDK. AFAIU so far, you're looking for
> an eSDK. An SDK just gives you the minimal toolchain and associated
> tools, to be able to compile something. I would go as far as saying
> it's basically what a PC distro package for cross-compile with gcc
> is doing.

   ... snip ...

it's slowly coming back to me, as i once examined the recipe
meta-toolchain.bb for building a toolchain, whose contents are
effectively:

  inherit populate_sdk

and when you run "bitbake meta-toolchain", you're obviously not
supplying a particular image as an argument, so whatever is being done
has to be based on fundamental information like MACHINE and so on,
with no reference to a particular image so, yes, you'd get a "minimal"
toolchain as you describe above independent of any image.

  however, as populate_sdk.bbclass inherits populate_sdk_base.bbclass,
and that latter class file contains:

TOOLCHAIN_HOST_TASK ?= "nativesdk-packagegroup-sdk-host packagegroup-cross-canadian-${MACHINE}"
TOOLCHAIN_HOST_TASK_ATTEMPTONLY ?= ""
TOOLCHAIN_TARGET_TASK ?= "${@multilib_pkg_extend(d, 'packagegroup-core-standalone-sdk-target')} target-sdk-provides-dummy"
TOOLCHAIN_TARGET_TASK_ATTEMPTONLY ?= ""
TOOLCHAIN_OUTPUTNAME ?= "${SDK_NAME}-toolchain-${SDK_VERSION}"

it seems clear that you can augment that minimal toolchain with
exactly the two variables i mentioned earlier. ok, i think i've
refreshed my memory on this. now off to figure out the eSDK.

rday

Re: [OE-core] SDK question: does "-c populate_sdk" build SDK based on entire image?

[Andre McCurdy]

On Fri, May 7, 2021 at 6:47 AM Robert P. J. Day <rpjday@crashcourse.ca> wrote:
>
> On Fri, 7 May 2021, Quentin Schulz wrote:
>
> > Hi Robert,
> >
> > No SDK expert here, so as usual, to be taken with a grain of salt.
> >
> > On Fri, May 07, 2021 at 09:11:37AM -0400, Robert P. J. Day wrote:
> > >
> > >   almost certainly a silly question as i'm still poring over the
> > > mechanics of standard SDK creation, but if i define a perfectly normal
> > > image, then build the corresponding SDK with:
> > >
> > >   $ bitbake -c populate_sdk my_image
> > >
> > > is the resulting SDK populated based on the entire contents of the
> > > target image? that is, if i subsequently add a new package to the
> > > target and rebuild the SDK, will the new SDK now contain the
> > > corresponding content from that newly-added package? (i'm just about
> > > to test this with a build but that's going to take over an hour on
> > > this server. *sigh* ...)
> > >
> >
> > I think you're mixing SDK and eSDK. AFAIU so far, you're looking for
> > an eSDK. An SDK just gives you the minimal toolchain and associated
> > tools, to be able to compile something. I would go as far as saying
> > it's basically what a PC distro package for cross-compile with gcc
> > is doing.
>
>    ... snip ...
>
> it's slowly coming back to me, as i once examined the recipe
> meta-toolchain.bb for building a toolchain, whose contents are
> effectively:
>
>   inherit populate_sdk
>
> and when you run "bitbake meta-toolchain", you're obviously not
> supplying a particular image as an argument, so whatever is being done
> has to be based on fundamental information like MACHINE and so on,
> with no reference to a particular image so, yes, you'd get a "minimal"
> toolchain as you describe above independent of any image.
>
>   however, as populate_sdk.bbclass inherits populate_sdk_base.bbclass,
> and that latter class file contains:
>
> TOOLCHAIN_HOST_TASK ?= "nativesdk-packagegroup-sdk-host packagegroup-cross-canadian-${MACHINE}"
> TOOLCHAIN_HOST_TASK_ATTEMPTONLY ?= ""
> TOOLCHAIN_TARGET_TASK ?= "${@multilib_pkg_extend(d, 'packagegroup-core-standalone-sdk-target')} target-sdk-provides-dummy"
> TOOLCHAIN_TARGET_TASK_ATTEMPTONLY ?= ""
> TOOLCHAIN_OUTPUTNAME ?= "${SDK_NAME}-toolchain-${SDK_VERSION}"
>
> it seems clear that you can augment that minimal toolchain with
> exactly the two variables i mentioned earlier. ok, i think i've
> refreshed my memory on this. now off to figure out the eSDK.

Didn't we discuss exactly this (ie the difference between a pure SDK
recipe such as meta-toolchain and the populate_sdk task of an image
recipe) a couple of weeks ago?

Both are valid ways to create an SDK and there are no rules about one
or the other being specific to creating a "minimal" toolchain.
Basically a pure SDK recipe gives you full control while the
populare_sdk task of an image recipe is easier to set up (no new
recipe to write) at the expense of the final SDK containing extra junk
such as busybox init scripts, etc which are needed in the image but
not an SDK.

Re: [OE-core] SDK question: does "-c populate_sdk" build SDK based on entire image?

[Robert P. J. Day]

On Fri, 7 May 2021, Andre McCurdy wrote:

> Didn't we discuss exactly this (ie the difference between a pure SDK
> recipe such as meta-toolchain and the populate_sdk task of an image
> recipe) a couple of weeks ago?

  yes, i just got confused as i was digging into eSDK and mixed the
two. it's been a long week.

rday

Re: [OE-core] SDK question: does "-c populate_sdk" build SDK based on entire image?

[Denys Dmytriyenko]

On Fri, May 07, 2021 at 09:49:32AM -0700, Andre McCurdy wrote:
> On Fri, May 7, 2021 at 6:47 AM Robert P. J. Day <rpjday@crashcourse.ca> wrote:
> >
> > On Fri, 7 May 2021, Quentin Schulz wrote:
> >
> > > Hi Robert,
> > >
> > > No SDK expert here, so as usual, to be taken with a grain of salt.
> > >
> > > On Fri, May 07, 2021 at 09:11:37AM -0400, Robert P. J. Day wrote:
> > > >
> > > >   almost certainly a silly question as i'm still poring over the
> > > > mechanics of standard SDK creation, but if i define a perfectly normal
> > > > image, then build the corresponding SDK with:
> > > >
> > > >   $ bitbake -c populate_sdk my_image
> > > >
> > > > is the resulting SDK populated based on the entire contents of the
> > > > target image? that is, if i subsequently add a new package to the
> > > > target and rebuild the SDK, will the new SDK now contain the
> > > > corresponding content from that newly-added package? (i'm just about
> > > > to test this with a build but that's going to take over an hour on
> > > > this server. *sigh* ...)
> > > >
> > >
> > > I think you're mixing SDK and eSDK. AFAIU so far, you're looking for
> > > an eSDK. An SDK just gives you the minimal toolchain and associated
> > > tools, to be able to compile something. I would go as far as saying
> > > it's basically what a PC distro package for cross-compile with gcc
> > > is doing.
> >
> >    ... snip ...
> >
> > it's slowly coming back to me, as i once examined the recipe
> > meta-toolchain.bb for building a toolchain, whose contents are
> > effectively:
> >
> >   inherit populate_sdk
> >
> > and when you run "bitbake meta-toolchain", you're obviously not
> > supplying a particular image as an argument, so whatever is being done
> > has to be based on fundamental information like MACHINE and so on,
> > with no reference to a particular image so, yes, you'd get a "minimal"
> > toolchain as you describe above independent of any image.
> >
> >   however, as populate_sdk.bbclass inherits populate_sdk_base.bbclass,
> > and that latter class file contains:
> >
> > TOOLCHAIN_HOST_TASK ?= "nativesdk-packagegroup-sdk-host packagegroup-cross-canadian-${MACHINE}"
> > TOOLCHAIN_HOST_TASK_ATTEMPTONLY ?= ""
> > TOOLCHAIN_TARGET_TASK ?= "${@multilib_pkg_extend(d, 'packagegroup-core-standalone-sdk-target')} target-sdk-provides-dummy"
> > TOOLCHAIN_TARGET_TASK_ATTEMPTONLY ?= ""
> > TOOLCHAIN_OUTPUTNAME ?= "${SDK_NAME}-toolchain-${SDK_VERSION}"
> >
> > it seems clear that you can augment that minimal toolchain with
> > exactly the two variables i mentioned earlier. ok, i think i've
> > refreshed my memory on this. now off to figure out the eSDK.
> 
> Didn't we discuss exactly this (ie the difference between a pure SDK
> recipe such as meta-toolchain and the populate_sdk task of an image
> recipe) a couple of weeks ago?
> 
> Both are valid ways to create an SDK and there are no rules about one
> or the other being specific to creating a "minimal" toolchain.
> Basically a pure SDK recipe gives you full control while the
> populare_sdk task of an image recipe is easier to set up (no new
> recipe to write) at the expense of the final SDK containing extra junk
> such as busybox init scripts, etc which are needed in the image but
> not an SDK.

Exactly! Thanks for this excellent summary!

I've heard so many times before that my meta-toolchain recipe is not the 
"Yocto" way of doing things and I should instead use populate_sdk. I got 
tired arguing with people and explaining that meta-toolchain is quite valid 
and has its merits...

-- 
Regards,
Denys Dmytriyenko <denis@denix.org>
PGP: 0x420902729A92C964 - https://denix.org/0x420902729A92C964
Fingerprint: 25FC E4A5 8A72 2F69 1186  6D76 4209 0272 9A92 C964

Re: [OE-core] SDK question: does "-c populate_sdk" build SDK based on entire image?

[Diego Santa Cruz]

> -----Original Message-----
> From: openembedded-core@lists.openembedded.org <openembedded-
> core@lists.openembedded.org> On Behalf Of Robert P. J. Day via
> lists.openembedded.org
> Sent: 07 May 2021 15:12
> To: OE Core mailing list <openembedded-core@lists.openembedded.org>
> Subject: [OE-core] SDK question: does "-c populate_sdk" build SDK based on
> entire image?
> 
> 
>   almost certainly a silly question as i'm still poring over the
> mechanics of standard SDK creation, but if i define a perfectly normal
> image, then build the corresponding SDK with:
> 
>   $ bitbake -c populate_sdk my_image
> 
> is the resulting SDK populated based on the entire contents of the
> target image? that is, if i subsequently add a new package to the
> target and rebuild the SDK, will the new SDK now contain the
> corresponding content from that newly-added package? (i'm just about
> to test this with a build but that's going to take over an hour on
> this server. *sigh* ...)

We use the SDK a lot to give application developers an easy to use a dev environment which has all the support to build against the libraries included in target images. The SDK of an image recipe will contain all the *-dev packages matching the packages installed in the target for the same image (i.e. if foobar is installed in the target image then foobar-dev is installed in the SDK image, if it exists).

> 
>   now, i realize that if i just want to add content to the SDK without
> adding to target, i can use one or both of:
> 
>   TOOLCHAIN_HOST_TASK
>   TOOLCHAIN_TARGET_TASK
> 
> but if someone asks for new content both for the target and the SDK,
> is it sufficient to just add to target?

Yes, with the exception of recipes for which only the headers are used when compiling a recipe that is included in the target image. As no library is being linked for it then no package is installed into the target, so no corresponding *-dev package is included in the SDK. A typical example of this is boost when the header-only boost libs are used (and I suspect the same happens for static libs). In such a case you can add the recipes to TOOLCHAIN_TARGET_TASK (we actually have an sdk packagegroup that we add to TOOLCHAIN_TARGET_TASK that pulls in all target SDK things that we want in the SDK but not in the target image).

Other things that you may need to add manually to TOOLCHAIN_HOST_TASK are supporting tools to run on the build host which are not included in the SDK by default, like nativesdk-glib-2.0-codegen to get the gdbus-codegen utility.

> 
>   and one more question that inspired all of this. i was asked to add
> a particular executable to an SDK, call it /usr/bin/fubar. fair
> enough, i thought, it was short work to determine that that binary
> came from the recipe fubar_1.0.bb. but i checked and that package was,
> in fact, being installed in the target, so i was puzzled as to why it
> wasn't in the SDK as well.
> 
>   i checked the recipe for fubar and, to my chagrin, it contained the
> snippet:
> 
>   do_install_append() {
>     rm -rf ${D}/usr/bin/fubar
>   }
> 
> well, that explained it, so as a first test, i'm just going to dump
> that snippet from the recipe, which will install the executable on the
> target, and will(?) also install it in the SDK?
> 
> rday
-- 
Diego Santa Cruz, PhD
Technology Architect
spinetix.com