How to handle meta-intel/openembedded repos with multiple developers

How to handle meta-intel/openembedded repos with multiple developers

[Olsson Rikard (RBSN/ESW1)]

[-- Attachment #1: Type: text/plain, Size: 2090 bytes --]

Hello Yocto Members,

My second posting, first one got great answers so hoping for two for two...

I come from a git /repo background where:

* repo init -u <URL> [<OPTIONS>]

               * Will get the manifest.xml file with the trees

* repo sync

Will download the forest to my local HDD and I am good to go.

I am having some problem understanding how to use Yocto/Bitbake in a development team and using source control/CM similar to what we had before. For example all out team members will use:
poky
meta-intel
meta-openembedded
meta-oracle-java
...
....
plus a few other ones. Is the idea that each developer should download these necessary git trees for example meta-intel/openembedded manually using git clone then update build/config/bblayers.conf manually? With this approach I feel that each developed needs to do multiple steps to setup their build environment just to get started.

I have tried to add my own BB file and I am able to download the repo's by running bitbake XXXX-meta-intel-lr
-rw-rw-r--. 1 XXX XXX 555 Mar  3 09:08 XXXX-meta-intel-lr_1.0.bb
-rw-rw-r--. 1 XXX XXX 564 Mar  3 10:47 XXXX-meta-openembedded-lr_1.0.bb
-rw-rw-r--. 1 XXX XXX 607 Mar  3 10:25 XXXX-meta-oracle-java-lr_1.0.b
And my code end up in tmp/work/i586-poky-linux/XXX-meta-oracle-java-lr/1.0-r0/git/ and I am then able to update build/config/bblayers.conf ==> But this cant be the correct way to handle this, path might change and tmp is what the name implies, a temporary directory........

For me I would like to have the necessary meta-XXXX downloaded by yocto in the same way repo init/sync handles it, this would allow developers to:
1) Clone poky from local repository
2) execute a yocto file/script (which was part of the poky clone) to download (no need to build anything) all the needed meta-XXXX.
3) Update build/config/bblayers.conf (if needed) with correct paths to meta-XXXX
4) Build
Is this possible in Yocto or do we need to wrap our development environment in something like repo int/sync to be able to solve this?

Regards
Rikard




[-- Attachment #2: Type: text/html, Size: 10934 bytes --]

Re: How to handle meta-intel/openembedded repos with multiple developers

[Mark Hatle]

On 3/3/16 7:15 AM, Olsson Rikard (RBSN/ESW1) wrote:
> Hello Yocto Members,
> 
>  
> 
> My second posting, first one got great answers so hoping for two for two…
> 

I'm going to ignore the semantics of git, and just focus on general usage.

OpenEmbedded/Yocto Project is based around a concept of layers.  When you do
team work, one strategy I recommend is:

Everything use the same 'bitbake, oe-core, meta-yocto*, and other non-company
layers'.  You will have to devise a scheme to do this.  In my company, our SCM
servers (which are git) are setup as mirrors and we mirror the data regularly.
The developers are in the habit of updating to the latest version of the code
when it's convenient for their development, and always before submitting code
for review.  These non-company assets are read-only.  Local users never modify them.

Think of this as something like:

git.mycompany.com/git/external/bitbake
git.mycompany.com/git/external/oe-core
git.mycompany.com/git/external/meta-yocto
git.mycompany.com/git/external/meta-openembedded

We then have local-assets which are based on non-company layers.  This serves
two purposed -- either as a 'snapshot in time' or occasionally you have to make
a change to a community layer to fix a critical bug.  Usually you start with the
external layer, create a local branch (and make the change there.  It then
becomes someone's job to keep that local branch up to date (which is why I don't
recommend this as general practice.)

Think of this as something like:
git.mycompany.com/git/internal/bitbake
git.mycompany.com/git/internal/oe-core
git.mycompany.com/git/internal/meta-yocto
git.mycompany.com/git/internal/meta-openembedded

(We're found using data codes in the branches helps -- since when we rebase, we
also create a new datecode based on when the branch started.  Our products have
specific 'key' files that switch the branch automatically for developers.)

We then have local layers.  These are the things that developers work on
collectively.  Every company has their own review process, maintainer roles,
etc..  but effectively -all- of your changes happen in a mixture of (local)
layers such as BSP, distribution, functional or other classification.

This is the typical tree that is company, division, and/or product specific.


And finally, each developer has a -local- layer where they can put their own
local changes, such as .bbappends, new recipes, etc and try things before
merging to the 'local layers' and going through the review process.

individual developers can upload these, but they should never be treated as
official.  An SCM can make a good backup system though.  :)

> 
> I come from a git /repo background where:
> 
> * repo init -u /<URL>/ [/<OPTIONS>/]
> 
>                * Will get the manifest.xml file with the trees
> 
> * repo sync
> 
> Will download the forest to my local HDD and I am good to go.
> 

This is very reasonable.  As I mentioned above breaking the problem into
two/three parts and treating parts as RO, controlled RW, and regular development
practice helps a lot.

You may need specific mechanisms to manage branch changes and rebases, but it
all starts with updating the control file.. and getting developers to
pull/update/rebase regularly as part of their work flows.  At some point during
product development a lead/architect needs to make the decision to 'freeze'
development and at that point everything is tagged/branched and only backports
are used from them on.  (If the number of backports gets too large, you MIGHT
decide to selectively rebase.)

> 
> I am having some problem understanding how to use Yocto/Bitbake in a development
> team and using source control/CM similar to what we had before. For example all
> out team members will use:
> 
> poky
> 
> meta-intel
> 
> meta-openembedded
> 
> meta-oracle-java

Team members need to get in the habit of updating at the start of each day,
during there development day (as long as it's not disruptive), and prior to
submitting any changes for review.  (Changes must -always- be on the latest
agreed upon version.)

> …
> 
> ….
> 
> plus a few other ones. Is the idea that each developer should download these
> necessary git trees for example meta-intel/openembedded manually using git clone
> then update build/config/bblayers.conf manually? With this approach I feel that
> each developed needs to do multiple steps to setup their build environment just
> to get started.
> 

This is where OSV products and other things come into play.  Different OSVs have
different strategies for this, but generally they wrap the manual configuration
aspects using some type of a setup wrapper.  This wrapper can be configured in
different ways, but it ensures a consist configuration.

An alternative approach is to put your build/conf/*.conf files into the SCM as
well.  When you configure it, don't use any full path names, make everything
relative from the build dir... Then the typical developer experience is:

sync down all of the layers
sync down the project config
. ./oe-init-build-env <project>

everyone has the same starting point.  If there is a reason to change the
config, it can be reviewed and controlled.  everyone just resyncs those files --
and the layers as normal.

> 
> I have tried to add my own BB file and I am able to download the repo’s by
> running bitbake XXXX-meta-intel-lr
> 
> -rw-rw-r--. 1 XXX XXX 555 Mar  3 09:08 XXXX-meta-intel-lr_1.0.bb
> 
> -rw-rw-r--. 1 XXX XXX 564 Mar  3 10:47 XXXX-meta-openembedded-lr_1.0.bb
> 
> -rw-rw-r--. 1 XXX XXX 607 Mar  3 10:25 XXXX-meta-oracle-java-lr_1.0.b
> 
> And my code end up in
> tmp/work/i586-poky-linux/XXX-meta-oracle-java-lr/1.0-r0/git/ and I am then able
> to update build/config/bblayers.conf èBut this cant be the correct way to handle
> this, path might change and tmp is what the name implies, a temporary directory……..
> 

The 'tmp' directory is just that, temporary.  You should never SCM the tmp
contents.  The OUTPUT of the build, typically in tmp/deploy can be SCMed, but it
should be explicitly copied out and SCMed as a build output.  (My opinion at least).

See above for my workflows...

--Mark

> 
> For me I would like to have the necessary meta-XXXX downloaded by yocto in the
> same way repo init/sync handles it, this would allow developers to:
> 
> 1) Clone poky from local repository
> 
> 2) execute a yocto file/script (which was part of the poky clone) to download
> (no need to build anything) all the needed meta-XXXX.
> 
> 3) Update build/config/bblayers.conf (if needed) with correct paths to meta-XXXX
> 
> 4) Build
> 
> Is this possible in Yocto or do we need to wrap our development environment in
> something like repo int/sync to be able to solve this?
> 
>  
> 
> Regards
> 
> Rikard
> 
>  
> 
>  
> 
>  
> 
> 
> 

Re: How to handle meta-intel/openembedded repos with multiple developers

[Edward Wingate]

On Thu, Mar 3, 2016 at 8:27 AM, Mark Hatle <mark.hatle@windriver.com> wrote:
>  At some point during product development a lead/architect needs to make the
> decision to 'freeze' development and at that point everything is tagged/branched
> and only backports are used from them on.  (If the number of backports gets too
> large, you MIGHT decide to selectively rebase.)

I'm currently trying to figure out with how to control external layers
in my Yocto build and found this thread.  I'm a little unclear on how
to track a release to the version used on non-company layers.  Say I'm
using poky, meta-openembedded, meta-xilinx and then my own layer,
meta-me. When I freeze development and do a release, I can tag
meta-me, but because I also treat non-company assets as RO, I
shouldn't tag poky, meta-openembedded nor meta-xilinx (or should I? Is
this where I use git's lightweight tagging as opposed to annotated
tags?) When "everything is tagged/branched", does that somehow include
tagging the non-company assets?  Thanks for any help.

Ed

Re: How to handle meta-intel/openembedded repos with multiple developers

[Chris Z.]

[-- Attachment #1: Type: text/plain, Size: 1679 bytes --]

Hi,

How you store your project configuration ? How you prepare workspace
(download each layer separately)?
Basic stuff, SW release should be reproducible (in easy way). Store
somewhere used hash of each piece or use tags. Non company assets should be
already somehow tagged or you use HEAD or master/-next ?

Br,
Chris Z

On Thu, Oct 27, 2016 at 8:22 AM, Edward Wingate <edwingate8@gmail.com>
wrote:

> On Thu, Mar 3, 2016 at 8:27 AM, Mark Hatle <mark.hatle@windriver.com>
> wrote:
> >  At some point during product development a lead/architect needs to make
> the
> > decision to 'freeze' development and at that point everything is
> tagged/branched
> > and only backports are used from them on.  (If the number of backports
> gets too
> > large, you MIGHT decide to selectively rebase.)
>
> I'm currently trying to figure out with how to control external layers
> in my Yocto build and found this thread.  I'm a little unclear on how
> to track a release to the version used on non-company layers.  Say I'm
> using poky, meta-openembedded, meta-xilinx and then my own layer,
> meta-me. When I freeze development and do a release, I can tag
> meta-me, but because I also treat non-company assets as RO, I
> shouldn't tag poky, meta-openembedded nor meta-xilinx (or should I? Is
> this where I use git's lightweight tagging as opposed to annotated
> tags?) When "everything is tagged/branched", does that somehow include
> tagging the non-company assets?  Thanks for any help.
>
> Ed
> --
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>

[-- Attachment #2: Type: text/html, Size: 2381 bytes --]

Re: How to handle meta-intel/openembedded repos with multiple developers

[Bas Mevissen]

On 07/11/2016 20:31, Chris Z. wrote:
> Hi,
> 
> How you store your project configuration ? How you prepare workspace
> (download each layer separately)?
> Basic stuff, SW release should be reproducible (in easy way). Store
> somewhere used hash of each piece or use tags. Non company assets should
> be already somehow tagged or you use HEAD or master/-next ?
> 

I think the best option is to use the Google Repo tool. You write an xml
file that specifies the layers to download and which commit, branch or
tag to checkout. The Freescale official and community project do it like
this, just like Android.

-- 
Bas.

Re: How to handle meta-intel/openembedded repos with multiple developers

[Mike Looijmans]

Git submodules work fine for this.

In general, for each project I create a top-level repo that has the OE repos 
as submodule. The project repo also contains the project-specific recipes.


On 07-11-16 20:31, Chris Z. wrote:
> Hi,
>
> How you store your project configuration ? How you prepare workspace (download
> each layer separately)?
> Basic stuff, SW release should be reproducible (in easy way). Store somewhere
> used hash of each piece or use tags. Non company assets should be already
> somehow tagged or you use HEAD or master/-next ?
>
> Br,
> Chris Z
>
> On Thu, Oct 27, 2016 at 8:22 AM, Edward Wingate <edwingate8@gmail.com
> <mailto:edwingate8@gmail.com>> wrote:
>
>     On Thu, Mar 3, 2016 at 8:27 AM, Mark Hatle <mark.hatle@windriver.com
>     <mailto:mark.hatle@windriver.com>> wrote:
>     >  At some point during product development a lead/architect needs to make the
>     > decision to 'freeze' development and at that point everything is
>     tagged/branched
>     > and only backports are used from them on.  (If the number of backports
>     gets too
>     > large, you MIGHT decide to selectively rebase.)
>
>     I'm currently trying to figure out with how to control external layers
>     in my Yocto build and found this thread.  I'm a little unclear on how
>     to track a release to the version used on non-company layers.  Say I'm
>     using poky, meta-openembedded, meta-xilinx and then my own layer,
>     meta-me. When I freeze development and do a release, I can tag
>     meta-me, but because I also treat non-company assets as RO, I
>     shouldn't tag poky, meta-openembedded nor meta-xilinx (or should I? Is
>     this where I use git's lightweight tagging as opposed to annotated
>     tags?) When "everything is tagged/branched", does that somehow include
>     tagging the non-company assets?  Thanks for any help.
>
>     Ed
>     --
>     

Kind regards,

Mike Looijmans
System Expert

TOPIC Products
Materiaalweg 4, NL-5681 RJ Best
Postbus 440, NL-5680 AK Best
Telefoon: +31 (0) 499 33 69 79
E-mail: mike.looijmans@topicproducts.com
Website: www.topicproducts.com

Please consider the environment before printing this e-mail





_______________________________________________
>     yocto mailing list
>     yocto@yoctoproject.org <mailto:yocto@yoctoproject.org>
>     https://lists.yoctoproject.org/listinfo/yocto
>     <https://lists.yoctoproject.org/listinfo/yocto>
>
>
>
>