Bitbake for native x86 project?

Bitbake for native x86 project?

[Chris Morgan]

Hello.

I can't be sure that I'm asking the question correctly so please
forgive me if I've gotten something wrong here.

I'm working on a project that generates several libraries and
applications. Today we build and run exclusively on x86 Linux desktop
machines and use cmake for the build system. In the future we plan to
add cross-compiling of these same libraries and applications for an
embedded arm platform, Yocto based. We would like to maintain both
types of build, for x86 and for arm.

I've seen guides on how to use Yocto (or other bitbake based
distributions) to generate x86 images but these appear to involve
building the entire x86 toolchain, kernel, all libraries, creating a
virtual root and building a root filesystem image that one would run
under an emulator.

In my case I'd like to use bitbake to build libraries and applications
and create a virtual root, but not create a kernel, toolchain and
other system provided libraries. Then, when we decide to transition
over to Yocto I'd like to leverage the recipes by plugging them into
the Yocto system somehow (I'm still starting to understand the
process).

I haven't seen any guides or information on how to use bitbake for
native desktop applications. I feel like its certainly possible but it
appears to be quite a bit of work just to proof of concept things.

Is there anything inherently wrong with what I'm proposing here with
bitbake? Does it make sense that it would save time to develop the
recipes once and use bitbake for the desktop like this?

Chris

Re: Bitbake for native x86 project?

[Trevor Woerner]

Hi Chris,

I don't think there's anything "wrong" with what you're proposing, but
your usage is so far outside what's intended that you'd be pretty much
on your own. I don't know that there are many people using bitbake
standalone (outside of OE/Yocto), but it would certainly be
interesting to hear of such projects.

Having OE/Yocto perform your complete build from the ground up, even
when compiling on an x86 machine for an x86 target, makes a huge
amount of sense in any scenario.

Even if there is only one developer: if your one and only build
machine dies (or needs to be replaced) and you are using OE/Yocto for
your entire build, you can easily switch to a new machine, start a new
build, and be right back where you were on the last machine. In other
words you don't have to "setup" your build machine in any way to make
sure it looks exactly like the old machine. If you are using OE/Yocto
for your entire build, it takes care of "setting up your build
environment" so your build performs the same on multiple machines.

But if you do have multiple developers, if you use OE/Yocto for the
entire build, all the developers can create the same image without
having to worry about native contamination. For example, if you didn't
want to use OE/Yocto for the entire build (and wanted everyone to use
their host environment as part of their build) then they all have to
be running the exact same version of whatever Linux distribution
you're using (so they all have the same compiler, linker, headers,
etc) and they all have to have the exact same packages installed (for
configuration). This becomes rather hard to enforce.

As another example, perhaps your x86 build and target machines are
currently 32-bit, but maybe you replace your build machine with a
64-bit machine in the future; in this case you'll need your build to
be completely self-contained (and not contaminated by the host's
environment). You will no longer be able to use your host's tools for
your build, you'll need to have something setup so that OE/Yocto
performs the entire build.

As another example. Maybe you release your product today but
development continues for the next 2 years. In this case you'll need
to make sure you "set aside" your build environment (maybe you're
lucky and this is easier to do because it's all been setup in a VM).
But if you're not using a VM and you're using a physical machine for
your build, you'll need to set aside your build machine and never
update it since you'd risk building something in the future which
links to a (for example) newer version of the C library and trying to
deploy that on your device which uses an older library version. If
you're using OE/Yocto now for your entire build then all you need to
do is continue using the current release in the future and you'll
still be able to update your build machine (new features, security
fixes, etc).

When you do move to your ARM platform you will need to compile
everything from the ground up to create your image. What OE/Yocto does
is to allow you to create your distribution. Ideally switching from
your x86 target to your ARM one would involve nothing more than
changing your MACHINE definition in your local.conf and you'll end up
with the exact same image in either case, the only difference being
one runs on x86 and the other runs on ARM. But if your x86 build isn't
using OE/Yocto completely then you'll never be quite sure what's in
that one.

Hope this helps,
    Trevor

Re: Bitbake for native x86 project?

[Richard Purdie]

On Fri, 2013-10-11 at 09:22 -0400, Chris Morgan wrote:
> I'm working on a project that generates several libraries and
> applications. Today we build and run exclusively on x86 Linux desktop
> machines and use cmake for the build system. In the future we plan to
> add cross-compiling of these same libraries and applications for an
> embedded arm platform, Yocto based. We would like to maintain both
> types of build, for x86 and for arm.
> 
> I've seen guides on how to use Yocto (or other bitbake based
> distributions) to generate x86 images but these appear to involve
> building the entire x86 toolchain, kernel, all libraries, creating a
> virtual root and building a root filesystem image that one would run
> under an emulator.

This isn't entirely accurate. You can choose just to build a given
library for example (bitbake libfoo). All its default dependencies are
also built though, which means in most cases, a cross toolchain would be
triggered. There is no need to build a kernel or a complete rootfs.

> In my case I'd like to use bitbake to build libraries and applications
> and create a virtual root, but not create a kernel, toolchain and
> other system provided libraries. Then, when we decide to transition
> over to Yocto I'd like to leverage the recipes by plugging them into
> the Yocto system somehow (I'm still starting to understand the
> process).

To try and clarify, we have roughly three types of target recipes:

a) target system

These are for the target system defined by MACHINE and are cross
compiled by a cross toolchain against a cross compiled libc. target can
be different to the build architecture. They are packaged and managed by
a package manager.

b) nativesdk

These are standalone binaries designed to run on SDKMACHINE. They are
cross compiled and link against their own libc which is designed to run
in some location defined at install time. We use these to generate
toolchain binaries and tools that can run on pretty much an Linux
system, regardless of its kernel, libc or system layout. They are
packaged and managed by a package manager.

c) native

These are compiled to run on the build system. They work very much like
you describe except they are not packaged at all.

(I'm ignoring -cross and -crosssdk since these are basically toolchain
bootstrap details).

> I haven't seen any guides or information on how to use bitbake for
> native desktop applications. I feel like its certainly possible but it
> appears to be quite a bit of work just to proof of concept things.
> 
> Is there anything inherently wrong with what I'm proposing here with
> bitbake? Does it make sense that it would save time to develop the
> recipes once and use bitbake for the desktop like this?

Its certainly possible, its just that nobody has done it. Your options
would be to either reconfigure the native recipes to be packaged, or to
add in a configuration which shortcircuits the cross compiler and adds
in some native magic to specific machine targets.

Whilst adding packaging to native sounds attractive, the reality is that
its not easy and doesn't really make sense technically. Adding in those
short-circuits I mention by comparison shouldn't be hard and if I were
doing this, its the way I'd go.

Cheers,

Richard

Re: Bitbake for native x86 project?

[Chris Morgan]

On Sun, Oct 13, 2013 at 12:53 PM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> On Fri, 2013-10-11 at 09:22 -0400, Chris Morgan wrote:
>> I'm working on a project that generates several libraries and
>> applications. Today we build and run exclusively on x86 Linux desktop
>> machines and use cmake for the build system. In the future we plan to
>> add cross-compiling of these same libraries and applications for an
>> embedded arm platform, Yocto based. We would like to maintain both
>> types of build, for x86 and for arm.
>>
>> I've seen guides on how to use Yocto (or other bitbake based
>> distributions) to generate x86 images but these appear to involve
>> building the entire x86 toolchain, kernel, all libraries, creating a
>> virtual root and building a root filesystem image that one would run
>> under an emulator.
>
> This isn't entirely accurate. You can choose just to build a given
> library for example (bitbake libfoo). All its default dependencies are
> also built though, which means in most cases, a cross toolchain would be
> triggered. There is no need to build a kernel or a complete rootfs.
>
>> In my case I'd like to use bitbake to build libraries and applications
>> and create a virtual root, but not create a kernel, toolchain and
>> other system provided libraries. Then, when we decide to transition
>> over to Yocto I'd like to leverage the recipes by plugging them into
>> the Yocto system somehow (I'm still starting to understand the
>> process).
>
> To try and clarify, we have roughly three types of target recipes:
>
> a) target system
>
> These are for the target system defined by MACHINE and are cross
> compiled by a cross toolchain against a cross compiled libc. target can
> be different to the build architecture. They are packaged and managed by
> a package manager.
>
> b) nativesdk
>
> These are standalone binaries designed to run on SDKMACHINE. They are
> cross compiled and link against their own libc which is designed to run
> in some location defined at install time. We use these to generate
> toolchain binaries and tools that can run on pretty much an Linux
> system, regardless of its kernel, libc or system layout. They are
> packaged and managed by a package manager.
>
> c) native
>
> These are compiled to run on the build system. They work very much like
> you describe except they are not packaged at all.
>
> (I'm ignoring -cross and -crosssdk since these are basically toolchain
> bootstrap details).
>
>> I haven't seen any guides or information on how to use bitbake for
>> native desktop applications. I feel like its certainly possible but it
>> appears to be quite a bit of work just to proof of concept things.
>>
>> Is there anything inherently wrong with what I'm proposing here with
>> bitbake? Does it make sense that it would save time to develop the
>> recipes once and use bitbake for the desktop like this?
>
> Its certainly possible, its just that nobody has done it. Your options
> would be to either reconfigure the native recipes to be packaged, or to
> add in a configuration which shortcircuits the cross compiler and adds
> in some native magic to specific machine targets.
>
> Whilst adding packaging to native sounds attractive, the reality is that
> its not easy and doesn't really make sense technically. Adding in those
> short-circuits I mention by comparison shouldn't be hard and if I were
> doing this, its the way I'd go.
>
> Cheers,
>
> Richard
>


Ahh interesting. I was thinking that this kind of a short-circuit
approach might make sense. Its good to hear that the concept isn't so
crazy although I've literally found no information about anyone doing
anything of the sort on the web.

I considered using the full blown x86/x86-64 targets but our build
time right now using cmake and native tools is ~30 seconds with only a
handful of MB of data to download.

Would using a sstate cache + a custom image target be able to get down
to that kind of time? Today each developer is building on their
desktop and from my own testing with angstrom it looks like the builds
take nearly an hour and generate 30+GB of data. This is almost
entirely why I've been hesitant to try to use the built-in x86/x86-64
support. I love the idea of ensuring that everything matches exactly
on each developer machine but the overhead is pretty crazy.

Chris

Re: Bitbake for native x86 project?

[Richard Purdie]

On Sun, 2013-10-13 at 17:53 +0100, Richard Purdie wrote:

> Its certainly possible, its just that nobody has done it. Your options
> would be to either reconfigure the native recipes to be packaged, or to
> add in a configuration which shortcircuits the cross compiler and adds
> in some native magic to specific machine targets.
> 
> Whilst adding packaging to native sounds attractive, the reality is that
> its not easy and doesn't really make sense technically. Adding in those
> short-circuits I mention by comparison shouldn't be hard and if I were
> doing this, its the way I'd go.

I had a kind of morbid curiosity about this. I came up with the patches
here:

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t2&id=c1c02d166f22179fb18e430aeae2806bb5d51a3f

where I've basically tried to wire in the host gcc yet compile
everything else for the target as usual including the libc. Its a bit of
an arbitrary choice but should in theory be easier than providing a
whole libc from the native system.

It works to a point (e.g. gettext compiled) but ncurses did not due to
symbol errors with __poll_chk. Why? Basically the compiler/linker do not
respect the sysroot, go searching for libc and find the system one
rather than the one we built. It then fails to link as there is version
mismatch. There isn't any compiler commandline option you can tweak to
stop it doing this :(.

Some questions this raises:

a) Do you use a sysroot of "/" or STAGING_DIR_TARGET?

I did the latter, it wasn't particularly successful as the tools don't
all respect it properly. In theory you can link the pieces inside the
STAGING_DIR to outside the sysroot if you want the compiler to see them,
the reality is some search paths are hardcoded into gcc and they leak
in. When we compile our compiler, we configure it not to do that.

To use "/" you'd have to do what native.bbclass does, set
STAGING_DIR_TARGET to be "/" and then set prefix to be inside the
STAGING_DIR but this means that binaries would expect to always run with
such a prefix.

I guess a hybrid would be possible, add -L and -I for the sysroot and
accept it linking with the system lib/include directories, ditching the
sysroot option a bit like the system did before gcc has sysroot support.

b) Can you use the system compiler without the libc?

Answer appears to be no, you can't. This means you would have to add in
some kind of tweaking to the output packages to add in dependencies on
libc and anything else that was system provided. You can therefore not
use this approach to build complete systems. Packaging up the system
provided libc would also seem to be ruled out.

Cheers,

Richard

Re: Bitbake for native x86 project?

[Richard Purdie]

On Mon, 2013-10-14 at 09:28 -0400, Chris Morgan wrote:
> Ahh interesting. I was thinking that this kind of a short-circuit
> approach might make sense. Its good to hear that the concept isn't so
> crazy although I've literally found no information about anyone doing
> anything of the sort on the web.

I just sent out my notes/patch attempt at just short circuiting in gcc.
Its a bit messy and I found some issues I didn't expect. It would be
possible to do it with both gcc/libc I'd guess but its not something I'm
going to have any more time to try any time soon...

> I considered using the full blown x86/x86-64 targets but our build
> time right now using cmake and native tools is ~30 seconds with only a
> handful of MB of data to download.
> 
> Would using a sstate cache + a custom image target be able to get down
> to that kind of time? Today each developer is building on their
> desktop and from my own testing with angstrom it looks like the builds
> take nearly an hour and generate 30+GB of data. This is almost
> entirely why I've been hesitant to try to use the built-in x86/x86-64
> support. I love the idea of ensuring that everything matches exactly
> on each developer machine but the overhead is pretty crazy.

You can build a complete rootfs from sstate in about 4 minutes so I'd
suggest trying it and see. If you've just building a cmake app, it will
need to download and install a toolchain but it will just do that once
for any given build directory and I suspect you could make it work. 

Cheers,

Richard

Re: Bitbake for native x86 project?

[Chris Morgan]

On Mon, Oct 14, 2013 at 10:40 AM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> On Mon, 2013-10-14 at 09:28 -0400, Chris Morgan wrote:
>> Ahh interesting. I was thinking that this kind of a short-circuit
>> approach might make sense. Its good to hear that the concept isn't so
>> crazy although I've literally found no information about anyone doing
>> anything of the sort on the web.
>
> I just sent out my notes/patch attempt at just short circuiting in gcc.
> Its a bit messy and I found some issues I didn't expect. It would be
> possible to do it with both gcc/libc I'd guess but its not something I'm
> going to have any more time to try any time soon...
>
>> I considered using the full blown x86/x86-64 targets but our build
>> time right now using cmake and native tools is ~30 seconds with only a
>> handful of MB of data to download.
>>
>> Would using a sstate cache + a custom image target be able to get down
>> to that kind of time? Today each developer is building on their
>> desktop and from my own testing with angstrom it looks like the builds
>> take nearly an hour and generate 30+GB of data. This is almost
>> entirely why I've been hesitant to try to use the built-in x86/x86-64
>> support. I love the idea of ensuring that everything matches exactly
>> on each developer machine but the overhead is pretty crazy.
>
> You can build a complete rootfs from sstate in about 4 minutes so I'd
> suggest trying it and see. If you've just building a cmake app, it will
> need to download and install a toolchain but it will just do that once
> for any given build directory and I suspect you could make it work.
>
> Cheers,
>
> Richard
>
>

Ahh. I see.

Thank you for those examples, I'll take a look at it in more detail
and see what I can learn. I appreciate the help.

Today I'm going to try to setup a sstate on a remote machine that has
a few hundred GB of free space, do a build and then try locally to do
the same. One question I had was whether developers, when using the
sstate, would receive tens of gigabytes of files downloaded to their
machines. I think that would be ok but not idea but oh well.


Would you be interested in doing some consulting/contracting around
the issue? or know if some one that might be? It would probably be 10x
faster than me doing it and we could avoid the various pitfalls that
we might hit during experimentation. If you or anyone is interested
you can contact me at chmorgan@gmail.com. We are in the Boston area
but telecommute works too.

Chris

Re: Bitbake for native x86 project?

[Trevor Woerner]

Chris,

Are all of your developers working on creating the target's kernel,
rootfs, and/or bootloader? If most of the developers are only doing
target, user-space, application development then all they need is an
appropriate SDK (which only needs to be generated whenever any target
packages change). This means a huge build only needs to be done once,
and then everyone can use the results. In this scenario a
network-shared sstate can still be employed for further build-time
savings.

Re: Bitbake for native x86 project?

[Richard Purdie]

On Mon, 2013-10-14 at 11:02 -0400, Chris Morgan wrote:
> Ahh. I see.
> 
> Thank you for those examples, I'll take a look at it in more detail
> and see what I can learn. I appreciate the help.
> 
> Today I'm going to try to setup a sstate on a remote machine that has
> a few hundred GB of free space, do a build and then try locally to do
> the same. One question I had was whether developers, when using the
> sstate, would receive tens of gigabytes of files downloaded to their
> machines. I think that would be ok but not idea but oh well.

They shouldn't see tens of gigabytes. I'd say we're talking a few
hundred megabytes.

As Trevor also mentions, there is the SDK/ADT standalone toolchain for
app development work which is worth looking at too. That could move the
need for the core builds to a small number of people and/or central
server.

> Would you be interested in doing some consulting/contracting around
> the issue? or know if some one that might be? It would probably be 10x
> faster than me doing it and we could avoid the various pitfalls that
> we might hit during experimentation. If you or anyone is interested
> you can contact me at chmorgan@gmail.com. We are in the Boston area
> but telecommute works too.

For me personally this isn't possible however there are a number of
community members offering consultancy around the project who may be
able to help and may get in touch :).

Cheers,

Richard

Re: Bitbake for native x86 project?

[Chris Morgan]

>Chris,
>
>Are all of your developers working on creating the target's kernel,
>rootfs, and/or bootloader? If most of the developers are only doing
>target, user-space, application development then all they need is an
>appropriate SDK (which only needs to be generated whenever any target
>packages change). This means a huge build only needs to be done once,
>and then everyone can use the results. In this scenario a
>network-shared sstate can still be employed for further build-time
>savings.

I suspect that we could get away with the appropriate sdk. I can see
how this would work for the cross-compilation process, with an arm
sdk, but if we had an x86-64 sdk for running things natively I'm not
sure the libraries would be in sync enough for the applications to run
on the developer's Linux machines. I'm thinking a mismatch in any
library would prevent the executables from being run without going
through the qemu approach.

I can certainly see that there is a path forward though and thank you
for brining up the sdk idea. I had read about that but haven't gotten
to trying it or trying to figure out where it best fits yet.

Chris

Re: Bitbake for native x86 project?

[Darren Hart]

On Tue, 2013-10-15 at 08:17 -0400, Chris Morgan wrote:
> >Chris,
> >
> >Are all of your developers working on creating the target's kernel,
> >rootfs, and/or bootloader? If most of the developers are only doing
> >target, user-space, application development then all they need is an
> >appropriate SDK (which only needs to be generated whenever any target
> >packages change). This means a huge build only needs to be done once,
> >and then everyone can use the results. In this scenario a
> >network-shared sstate can still be employed for further build-time
> >savings.
> 
> I suspect that we could get away with the appropriate sdk. I can see
> how this would work for the cross-compilation process, with an arm
> sdk, but if we had an x86-64 sdk for running things natively I'm not
> sure the libraries would be in sync enough for the applications to run
> on the developer's Linux machines. I'm thinking a mismatch in any
> library would prevent the executables from being run without going
> through the qemu approach.
> 
> I can certainly see that there is a path forward though and thank you
> for brining up the sdk idea. I had read about that but haven't gotten
> to trying it or trying to figure out where it best fits yet.
> 

I was also going to suggest the SDK for the app developers. In most
product groups I've met with using the Yocto Project, the developers
maintain their existing workflow through use of the SDK (which ensures
they are using the same compiler, libraries, etc) and the bitbake tasks
are owned by a small build team of 1 or 2 engineers.

-- 
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel