Working with OEMs

Working with OEMs

[Jeremy Grant]

   I am currently working on project that we are planning on using
   openembedded.  We will be using 6 ereader devices initially from 2
   different manufactures.  One of the manufactures already uses open
   embedded the other is not.  We are needing to explain to the other OEM
   why they should be uses open embedded to build the images used on the
   devices so that all devices are consistent and can pull from one feed
   source for updates.
   Here are a few of the requirements I have right now:
   Must use qt-embedded 4.6.x
   Would be nice if it can uses arora-e
   I have a few question on what I should be doing.
   Should I create my own distribution based on angstrom or just create
   and image with preferred-versions for the ereaders and used angstrom as
   the distro?
   What suggestions does anyone have on working with the OEM that does not
   uses OE already to help them see why using OE would be beneficial to
   them?  I find information on why developers and distribution
   maintainers should uses OE but not much information on how an OEM could
   create machine.conf to be used with OE or use OE.  I may just have not
   found the correct presentation or blog that has the information I am
   looking for.
   Thanks for any help you can give.
   Jeremy Grant
   VML | Lead Systems Administrator
   2020 Baltimore, Kansas City, MO 64108
   office: +1.816.218.1956
   fax: +1.816.472.5234
   mobile: +1.913.636.1112
   email: [1]jgrant@vml.com

   Before you print, please think about the environment.
   Confidentiality Statement This transmission is confidential and
   intended solely for the party to whom it is addressed. If the reader of
   this email is not the intended recipient, you are hereby notified that
   it may contain privileged, confidential and trade secret information,
   and that any dissemination, distribution, copying or use of the
   information in this transmission is strictly prohibited. If you have
   received this transmission in error, please immediately notify the
   sender at the email address above, and delete all copies of it from
   your computer.

References

   1. file://localhost/tmp/jgrant@vml.com

Re: Working with OEMs

[Holger Hans Peter Freyther]

On Friday 05 February 2010 20:28:59 Jeremy Grant wrote:

Hi Jeremy,

in general we encourage people to use one of the existing distributions 
(angstrom, micro, minimal). In some recipes these distro's have done 
configuration or other policy. This is done with the "OVERRIDE" system of 
bitbake which is working by name. So just copying angstrom.conf to a new name 
will most likely not work right a way.

For the next part. I would suggest creating a "task-oem-kind" and a "oem-kind-
image" which would install Qt Embedded 4.6, the required plugins and 
additional applications like the arora browser.

My general suggestion is to use OE only for the tasks it is good. E.g. if you 
force a developer to use it for application development it can be pretty 
painful on both sides.

At the OEM I would encourage someone to write the machine config, makes sure 
the init files are working. As the output of the OEM they should have the 
machine file, the sourcecode, a bootable image, and the result of something 
like "meta-toolchain-qte" or "meta-toolchain" to allow easy application 
development for the target hardware.



holger

Re: Working with OEMs

[C Michael Sundius]

Jeremy,

I brought OE in as my employers build system to replace what was a miserable
set of make files and tarballs. we had several groups placed all over the
world and each location had their own build system and then a final build
system that was used to patch lots of tarballs together. For me (a kernel
engineer) to build the drivers and application stack was nearly impossible.

for us the reasons to use OE were these:
1) keep track of software runtime and build time dependencies.
(staging area is great at forcing developers to only publish the bare
minimum apis and libraries)
2) a consistent way of building and configuring ALL of the software used in
the distrobution
3) decouple the software source and the congiration meta data. helps in
developing portable or reusable code.

We build software for a pretty well known set top box. Since we were already
established with a large distro of software and configuration, we decided to
create a new distribution. we ended up creating 2 directories, one our local
recipes and classes, and one for the standard opensource recipes. (local and
openembedded). In the openembedded recipes, we used any recipes that existed
for versions of software that we were using, but created new recipes (i.e.
cut/paste) for unsupported packages or versions. We also (as Holger
mentioned) used overrides where we could to specify our own configuration or
patches that have to be applied. The intent is push this back out once our
development stablizes.

note that we trimmed down the openembedded directory to only those recipes
that we use so that it does not take so long to parse all the recipes which
takes too much time (though I've been thinking there's got to be a better
way of doing that.)

in our local recipes (those for our own code) we've made a very simple class
that creates a link instead of copying the source code into the work
directory (bypassing the fetch all together). This is a great help for
developers since we are not using a supported source control software and
even so, I don't think the way that the fetchers work wrt to source control
is very developer friendly.

I also recently made a hack to our bitbake to allow dependent recipes to
rebuild when a dependency was rebuilt.

> Holger writes:
> My general suggestion is to use OE only for the tasks it is good. E.g. if
you
> force a developer to use it for application development it can be pretty
> painful on both sides.

- I find it a bit dishartening that Holger does not recommend you to use OE
for software development. One of the main reasons we were in search for a
comprehensive build system was so that we could offer a consistent means of
buliding our entire distribution to all developers and Configurtion Mgmt,
Testing, Release teams alike with out everyone having to be an expert with
all parts of the system (and all different build tools that had cropped up
in the different locations we have engineers - india, US east coast, west
coast). It *is* difficult to make it work for devlopers and build teams
alike, but having a consistent means building software is a huge step
creating a really great build system. Further if people are discouraged from
using OE as a tool for developers, then it will never get any better and
this hole of support in OE will not be filled.

Holger, will all due respect to you and the rest of the list (they have been
the only way I would have been able to get our team switched over to OE -
thank you thank you to everyone on this list), I hope I am misunderstanding
that comment. Further, I wonder if there are others out there who also feel
this way? My understanding is that Monta Vista is using OE; are there voices
on this list from MV or others that use OE for creating a consistent means
of building for developers and Config/Release Managers? If so, what are your
tricks and complaints with respect to this issue.

On Fri, Feb 5, 2010 at 11:28 AM, Jeremy Grant <jgrant@vml.com> wrote:

>   I am currently working on project that we are planning on using
>   openembedded.  We will be using 6 ereader devices initially from 2
>   different manufactures.  One of the manufactures already uses open
>   embedded the other is not.  We are needing to explain to the other OEM
>   why they should be uses open embedded to build the images used on the
>   devices so that all devices are consistent and can pull from one feed
>   source for updates.
>   Here are a few of the requirements I have right now:
>   Must use qt-embedded 4.6.x
>   Would be nice if it can uses arora-e
>   I have a few question on what I should be doing.
>   Should I create my own distribution based on angstrom or just create
>   and image with preferred-versions for the ereaders and used angstrom as
>   the distro?
>   What suggestions does anyone have on working with the OEM that does not
>   uses OE already to help them see why using OE would be beneficial to
>   them?  I find information on why developers and distribution
>   maintainers should uses OE but not much information on how an OEM could
>   create machine.conf to be used with OE or use OE.  I may just have not
>   found the correct presentation or blog that has the information I am
>   looking for.
>   Thanks for any help you can give.
>   Jeremy Grant
>   VML | Lead Systems Administrator
>   2020 Baltimore, Kansas City, MO 64108
>   office: +1.816.218.1956
>   fax: +1.816.472.5234
>   mobile: +1.913.636.1112
>   email: [1]jgrant@vml.com
>
>   Before you print, please think about the environment.
>   Confidentiality Statement This transmission is confidential and
>   intended solely for the party to whom it is addressed. If the reader of
>   this email is not the intended recipient, you are hereby notified that
>   it may contain privileged, confidential and trade secret information,
>   and that any dissemination, distribution, copying or use of the
>   information in this transmission is strictly prohibited. If you have
>   received this transmission in error, please immediately notify the
>   sender at the email address above, and delete all copies of it from
>   your computer.
>
> References
>
>   1. file://localhost/tmp/jgrant@vml.com
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
>

Re: Working with OEMs

[Philip Balister]

On 02/08/2010 11:48 AM, C Michael Sundius wrote:
> - I find it a bit dishartening that Holger does not recommend you to use OE
> for software development. One of the main reasons we were in search for a
> comprehensive build system was so that we could offer a consistent means of
> buliding our entire distribution to all developers and Configurtion Mgmt,
> Testing, Release teams alike with out everyone having to be an expert with
> all parts of the system (and all different build tools that had cropped up
> in the different locations we have engineers - india, US east coast, west
> coast). It *is* difficult to make it work for devlopers and build teams
> alike, but having a consistent means building software is a huge step
> creating a really great build system. Further if people are discouraged from
> using OE as a tool for developers, then it will never get any better and
> this hole of support in OE will not be filled.

Another way of looking at Holder's response is that we feel we could do 
a better job of supporting people using OE to do software development. 
There are certainly ways to use OE to do SW development.

I would really like to see us look at improving the experience for 
people doing sw dev for OE based distributions.

Philip

Re: Working with OEMs

[C Michael Sundius]

>
> Another way of looking at Holder's response is that we feel we could do a
> better job of supporting people using OE to do software development. There
> are certainly ways to use OE to do SW development.
>
> agreed. I really wanted to be cautious regarding my comment since I've
gotten such great help from this list, further, I do want to encourage
others organizations to use and promote OE as a tool to help make the task
of building a uniform process for developers and CM alike.

>
>
>

> I would really like to see us look at improving the experience for people
> doing sw dev for OE based distributions.
>
>
that said I'd still love to hear from other developers about problems that
they have encountered and how they have overcome or didn't. I think it would
be an interesting discussion.

mike

Re: Working with OEMs

[Michael Smith]

C Michael Sundius wrote:
> Further, I wonder if there are others out there who also feel
> this way? My understanding is that Monta Vista is using OE; are there voices
> on this list from MV or others that use OE for creating a consistent means
> of building for developers and Config/Release Managers? If so, what are your
> tricks and complaints with respect to this issue.

Hi Mike,

I'm using COLLECTIONS, the srctree and gitver classes, plus amend.inc. I 
think all of the above come from Chris at MV. It's about three billion 
times better than the custom build system we had before.

We have four trees:

- public: playground for new recipes before submitting to OE; also, 
amend.inc's to tweak packaging behaviour or configure flags.
- private: recipes to build closed-source drivers from our suppliers;
- oe: full org.openembedded.dev;
- proj: git repos for internal projects get cloned here, and added to 
COLLECTIONS with wildcards

Building the internal projects with OE means they barely even need to be 
cross-compile aware. All they need is a simple Makefile, and a git tag 
to set the package version. It's way easier than writing a one-off build 
system for each project.

The only problem is that bitbake takes forever to start up: COLLECTIONS 
causes a re-exec of Python, I think, and there are about 8,000 recipes 
in the tree. But we get around it by passing in a specific recipe file 
with -b.

For release management, I have a class that creates a list of the .deb 
packages that would have gone into an image, and a script that checks 
them into the right place in CVS. If you use Angstrom I think you get 
the same list for free every time you build an image. The class just 
saves me from waiting for rootfs to run.

Mike

Re: Working with OEMs

[Peter Chubb]

We are also attempting to use BitBake +OE for development.  We have a
several packagaes (a hypervisor, a kernel patched to use that
hypervisor, some libraries that act as glue between the two, and some
bits for bootloading) that are under active development, and build
several customised root images baseed on micro to run on top.

Problems we've faced are that bitbake doesn't keep track of
interpackage dependencies except by PR, so the normal `test think edit
build test ...' cycle is complicated by having to remember to remove
the packages we want rebuilt from the various caches that oe
maintains.


Setting BB_STAMP_POLICY="full" helps for this.
--
Dr Peter Chubb        www.nicta.com.au      peter DOT chubb AT nicta.com.au
http://www.ertos.nicta.com.au           ERTOS within National ICT Australia
From Imagination to Impact                       Imagining the (ICT) Future

Re: Working with OEMs

[C Michael Sundius]

On Tue, Feb 9, 2010 at 2:33 PM, Peter Chubb <peter.chubb@nicta.com.au>wrote:

> We are also attempting to use BitBake +OE for development.  We have a
> several packagaes (a hypervisor, a kernel patched to use that
> hypervisor, some libraries that act as glue between the two, and some
> bits for bootloading) that are under active development, and build
> several customised root images baseed on micro to run on top.
>
> Problems we've faced are that bitbake doesn't keep track of
> interpackage dependencies except by PR, so the normal `test think edit
> build test ...' cycle is complicated by having to remember to remove
> the packages we want rebuilt from the various caches that oe
> maintains.
>
>
> Setting BB_STAMP_POLICY="full" helps for this.
>

I was never able to make that trick work.I always ran into a problem where
it gave me a circular dependency with some of the native tools... (quilt if
I remember)

anyhow, I created a patch to bitbake just last week I posted it. I created a
variable BB_STAMP_BLACKLIST which creates a list of recipes which should be
checked for time stamp dependencies between recipes... It seems to work
well, and is maybe a little less checking (faster?) if you know that there
are a few apis you can just list those recieps...

mike

Re: Working with OEMs

[C Michael Sundius]

On Tue, Feb 9, 2010 at 1:47 PM, Michael Smith <msmith@cbnco.com> wrote:

> C Michael Sundius wrote:
>
>> Further, I wonder if there are others out there who also feel
>> this way? My understanding is that Monta Vista is using OE; are there
>> voices
>> on this list from MV or others that use OE for creating a consistent means
>> of building for developers and Config/Release Managers? If so, what are
>> your
>> tricks and complaints with respect to this issue.
>>
>
> Hi Mike,
>
> I'm using COLLECTIONS, the srctree and gitver classes, plus amend.inc. I
> think all of the above come from Chris at MV. It's about three billion times
> better than the custom build system we had before.
>
> We have four trees:
>
> - public: playground for new recipes before submitting to OE; also,
> amend.inc's to tweak packaging behaviour or configure flags.
> - private: recipes to build closed-source drivers from our suppliers;
> - oe: full org.openembedded.dev;
> - proj: git repos for internal projects get cloned here, and added to
> COLLECTIONS with wildcards
>

we used just two collections:
openembedded:  a trimmed down openembedded tree with just the recipes that
we use
local: just like your "private"


> Building the internal projects with OE means they barely even need to be
> cross-compile aware. All they need is a simple Makefile, and a git tag to
> set the package version. It's way easier than writing a one-off build system
> for each project.
>
> The only problem is that bitbake takes forever to start up: COLLECTIONS
> causes a re-exec of Python, I think, and there are about 8,000 recipes in
> the tree. But we get around it by passing in a specific recipe file with -b.
>
> this is why we only use a trimmed down version of the OE tree. our
developers would completely revolt if we had to parse all 20k recipes... I
think its really 20k...

this is easier for building, but makes it harder to sync with the main tree.


 but I'm thinking that maybe I'll just create a file with a list of the
recipes (or just the recipe dirs) that I can used to build my BBFILES list
with. That way I'll be able to keep the whole oe tree..

We do not use git here :[ we used something call Synergy CCM (I had never
heard of it before either, but I guess its a company owned by IBM... its not
very friendly) ,  anyhow there is no fetcher currently supported. I see that
we will likely need to write our own fetcher so that we can enforce
versioning better.

we've, however, really actually been very successful by creating a class
that adds a task before the patch task.. it just sets up a symlink to the
checked out version of the code... its simple and I'm sure we could do
something fancier, but it wroks really well... it just means that we need to
pair the baseline of the OE repo w/ the source code repo.

-----------------

I have not used the srctree class, though I noticed it a while back and have
not had time to check out what it is or how it works.. (Now I guess I have
to go and educate myself...). Also not aware of the amend.inc file

-----------------

we've had long discussions about how to reconsile our software versions (for
proprietery code) and the recipe versions since they don't really follow the
version - revision way of doing things that OE uses. This has been difficult
one, I'd like to push them to be better with versioning, but infact we need
to have a real fetcher for our source control software before we can enforce
which version is built by each recipe.

---------

Finally, our drivers and applications teams just condensed their nicly
modularized recipes into just two recipes because intra recipe dependency
checking was not working (i.e. when you rebuild one recipe its dependant
recipes should rerun, but don't). I finally provided them a solution but
they had already created monster recipes for all of their source code :}



> For release management, I have a class that creates a list of the .deb
> packages that would have gone into an image, and a script that checks them
> into the right place in CVS. If you use Angstrom I think you get the same
> list for free every time you build an image. The class just saves me from
> waiting for rootfs to run.
>

we use lots of magic that puts the image in a form that can be installed on
the box. this is done in the image-postprocessing function (theres a hook
for this in the image class, forget what its called). we don't really track
the individual packages.. just the image since we never allow installing
individual packages the box.

 thanks for your comments.