release management

release management

[Russell Peterson]

Hello.

As I learn more about yocto and more importantly gain practical experience with it I have started to think about my release structure.  Is there a “best practices” document or something like that that speaks to this?

For example, how does everyone deal with “external” meta layer dependencies?  My software uses poky and meta-openembedded, of course.  It also relies on some recipes in meta-linaro and meta-virtualization.  I suspect there will be more as time goes by.  I have tweaked my layer priorities as well as used BBMASK to avoid unwanted bbappend files etc… works but seems slightly clunky… still better than duplicating recipes in my meta layer I think.

Also… I quickly came to the conclusion that “thou shall not modify poky or meta-openembedded”.  That is, ALWAYS use bbappend files instead of modifying external layers.  If I think that poky or some other layer has a recipe bug or want to change something in poky, for example, I plan to upstream a fix to the project and when that becomes available I rebase my poky and remove the bbappend from my meta layer. One thing about modifying poky and not using bbappend files is that I would then need to ship patches for poky instead of just directing users to to use this branch and this commit for release x.y.z.

Comments and suggestions welcome.

Regards,

Russell

Re: release management

[Gunnar Andersson]

Hey Russell

I don't claim to be an authority on best practices but I will share some
thoughts.

On Sun, 2017-08-13 at 06:58 -0400, Russell Peterson wrote:
> Hello.
> 
> As I learn more about yocto and more importantly gain practical experience
> with it I have started to think about my release structure.  Is there a
> “best practices” document or something like that that speaks to this?
> For example, how does everyone deal with “external” meta layer
> dependencies?  My software uses poky and meta-openembedded, of course.  

We simply use a parent project [1] that includes git submodules, one per
yocto layer.  I'm of the opinion that if git (only) can be used, why
introduce other tools?   But it requires you to learn and master that
feature.

For the primary layer that is unique to this project, meta-genivi-dev, we
later on decided to merge it into the parent git repository instead, but
keep all other layers as submodules.  Since there are frequent changes to
the primary layer, it reduces the complexity of having to update a submodule
for them continuously.  But the purpose is also to keep *only* unique things
there, and thereby such things that are reusable by others will be naturally
pushed to the other layers (those that are reusable, and therefore should be
separate repos).

There's a natural tradeoff between layers being developed independently (and
widely reused by other projects), vs. the complexity of hacking on your
project.

Other projects use Google's "repo" tool but I have yet to understand what
that adds compared to git-submodules (in a medium sized project at least),
other than just additional hoops to jump through.

> It also relies on some recipes in meta-linaro and meta-virtualization.  I
> suspect there will be more as time goes by.  I have tweaked my layer
> priorities as well as used BBMASK to avoid unwanted bbappend files etc…
> works but seems slightly clunky… still better than duplicating recipes in
> my meta layer I think.

I think you've got it.  Those are the tools you have.  I haven't found too
frequent needs to mask away that which is provided by other layers, but it
depends on your needs...

> 
> Also… I quickly came to the conclusion that “thou shall not modify poky or
> meta-openembedded”.  
> That is, ALWAYS use bbappend files instead of modifying external layers.  

Absolutely that's the Yocto way.  But .bbappends are a kind of patches,
so who's to say what is right or wrong?  Some people might think that git is
actually a better tool to keep track of local modifications - keeping those
on branches that can be efficiently diffed and merged when upstream changes.

But clearly most Yocto proficient developers will be used to a .bbappend
based workflow, and I would wager that you will be better received with any
support questions than if you say "I have made my own branch of poky/meta-
oe).  Strictly speaking the two option ought to be equivalent, but I'm just
guessing it will still be perceived differently.  Yocto devs might agree or
disagree.


> If I think that poky or some other layer has a recipe bug or want to
> change something in poky, for example, I plan to upstream a fix to the
> project and when that becomes available I rebase my poky and remove the
> bbappend from my meta layer. 

> One thing about modifying poky and not using bbappend files is that I
> would then need to ship patches for poky instead of just directing users
> to to use this branch and this commit for release x.y.z.

You're not really asking a question here - just drawing the standard open-
source process conclusions.  I for one agree with you.  Use bbappends.  Talk
to the upstream.  Always second-guess why you need to change anything at
all.  Then if the appends become very complex, consider if finally
overriding with a new .bb file fits better.  

In our policy that I will describe further (below) we also mostly prefer the
more explicit separate bbappend files that by virtue of having its own path
location can be selectively included or not included in the local.conf for
different project variations.  This feels less magical than having various
hardware-specific overrides and other deeply embedded tweaks.  But there's
not 100% right or wrong here -- it's clearly a case of use the right tool
depending on the situation.

> 
> Comments and suggestions welcome.

In the GENIVI community we decided on a quite specific organization for
layer modifications / bbappends.  I have been meaning to write an email to
gather community feedback on it, but it's better done in a separate thread I
think.

Best Regards
- Gunnar

[1] https://github.com//GENIVI/genivi-dev-platform

-- 
Gunnar Andersson <gandersson@genivi.org>
Development Lead
GENIVI Alliance


> 
> Regards,
> 
> Russell
> 
> 
> 
> 

Re: release management

[Russell Peterson]

Thanks very much, Gunnar, for your detailed and helpful response.

Not sure I will go the git submodule path.  My initial try is using a script in my build environment that pulls in whatever layers it needs via git clone and then using a specific commit in a specific branch for that specific build.  The cloned layers are considered “read only”.  Right now we clone directly from yocto or github but we could easily switch to local, non-public mirrors that we could populate to specific versions.

Thanks again!

Regards,

Russell


> On Aug 14, 2017, at 3:10 PM, Gunnar Andersson <gandersson@genivi.org> wrote:
> 
> 
> Hey Russell
> 
> I don't claim to be an authority on best practices but I will share some
> thoughts.
> 
> On Sun, 2017-08-13 at 06:58 -0400, Russell Peterson wrote:
>> Hello.
>> 
>> As I learn more about yocto and more importantly gain practical experience
>> with it I have started to think about my release structure.  Is there a
>> “best practices” document or something like that that speaks to this?
>> For example, how does everyone deal with “external” meta layer
>> dependencies?  My software uses poky and meta-openembedded, of course.  
> 
> We simply use a parent project [1] that includes git submodules, one per
> yocto layer.  I'm of the opinion that if git (only) can be used, why
> introduce other tools?   But it requires you to learn and master that
> feature.
> 
> For the primary layer that is unique to this project, meta-genivi-dev, we
> later on decided to merge it into the parent git repository instead, but
> keep all other layers as submodules.  Since there are frequent changes to
> the primary layer, it reduces the complexity of having to update a submodule
> for them continuously.  But the purpose is also to keep *only* unique things
> there, and thereby such things that are reusable by others will be naturally
> pushed to the other layers (those that are reusable, and therefore should be
> separate repos).
> 
> There's a natural tradeoff between layers being developed independently (and
> widely reused by other projects), vs. the complexity of hacking on your
> project.
> 
> Other projects use Google's "repo" tool but I have yet to understand what
> that adds compared to git-submodules (in a medium sized project at least),
> other than just additional hoops to jump through.
> 
>> It also relies on some recipes in meta-linaro and meta-virtualization.  I
>> suspect there will be more as time goes by.  I have tweaked my layer
>> priorities as well as used BBMASK to avoid unwanted bbappend files etc…
>> works but seems slightly clunky… still better than duplicating recipes in
>> my meta layer I think.
> 
> I think you've got it.  Those are the tools you have.  I haven't found too
> frequent needs to mask away that which is provided by other layers, but it
> depends on your needs...
> 
>> 
>> Also… I quickly came to the conclusion that “thou shall not modify poky or
>> meta-openembedded”.  
>> That is, ALWAYS use bbappend files instead of modifying external layers.  
> 
> Absolutely that's the Yocto way.  But .bbappends are a kind of patches,
> so who's to say what is right or wrong?  Some people might think that git is
> actually a better tool to keep track of local modifications - keeping those
> on branches that can be efficiently diffed and merged when upstream changes.
> 
> But clearly most Yocto proficient developers will be used to a .bbappend
> based workflow, and I would wager that you will be better received with any
> support questions than if you say "I have made my own branch of poky/meta-
> oe).  Strictly speaking the two option ought to be equivalent, but I'm just
> guessing it will still be perceived differently.  Yocto devs might agree or
> disagree.
> 
> 
>> If I think that poky or some other layer has a recipe bug or want to
>> change something in poky, for example, I plan to upstream a fix to the
>> project and when that becomes available I rebase my poky and remove the
>> bbappend from my meta layer. 
> 
>> One thing about modifying poky and not using bbappend files is that I
>> would then need to ship patches for poky instead of just directing users
>> to to use this branch and this commit for release x.y.z.
> 
> You're not really asking a question here - just drawing the standard open-
> source process conclusions.  I for one agree with you.  Use bbappends.  Talk
> to the upstream.  Always second-guess why you need to change anything at
> all.  Then if the appends become very complex, consider if finally
> overriding with a new .bb file fits better.  
> 
> In our policy that I will describe further (below) we also mostly prefer the
> more explicit separate bbappend files that by virtue of having its own path
> location can be selectively included or not included in the local.conf for
> different project variations.  This feels less magical than having various
> hardware-specific overrides and other deeply embedded tweaks.  But there's
> not 100% right or wrong here -- it's clearly a case of use the right tool
> depending on the situation.
> 
>> 
>> Comments and suggestions welcome.
> 
> In the GENIVI community we decided on a quite specific organization for
> layer modifications / bbappends.  I have been meaning to write an email to
> gather community feedback on it, but it's better done in a separate thread I
> think.
> 
> Best Regards
> - Gunnar
> 
> [1] https://github.com//GENIVI/genivi-dev-platform
> 
> -- 
> Gunnar Andersson <gandersson@genivi.org>
> Development Lead
> GENIVI Alliance
> 
> 
>> 
>> Regards,
>> 
>> Russell
>> 
>> 
>> 
>> 
> 

Re: release management

[Mike Looijmans]

On 14-08-17 21:10, Gunnar Andersson wrote:
> 
> Hey Russell
> 
> I don't claim to be an authority on best practices but I will share some
> thoughts.
> 
> On Sun, 2017-08-13 at 06:58 -0400, Russell Peterson wrote:
>> Hello.
>>
>> As I learn more about yocto and more importantly gain practical experience
>> with it I have started to think about my release structure.  Is there a
>> “best practices” document or something like that that speaks to this?
>> For example, how does everyone deal with “external” meta layer
>> dependencies?  My software uses poky and meta-openembedded, of course.
> 
> We simply use a parent project [1] that includes git submodules, one per
> yocto layer.  I'm of the opinion that if git (only) can be used, why
> introduce other tools?   But it requires you to learn and master that
> feature.
> 

We arrived at exactly the same setup. One parent repository that contains the 
project-unique recipes, and submodules for the layers it needs.

It's a lot better than attempting to script things. For one thing, git will 
tell you the state of the submodules.

If I rembemer correctly, Android builds use a tool to manage the layer 
repositories. I didn't much like it though. And "yet another tool"...


Kind regards,

Mike Looijmans
System Expert

TOPIC Products
Materiaalweg 4, NL-5681 RJ Best
Postbus 440, NL-5680 AK Best
Telefoon: +31 (0) 499 33 69 79
E-mail: mike.looijmans@topicproducts.com
Website: www.topicproducts.com

Please consider the environment before printing this e-mail

Re: release management

[Joshua Watt]

On Tue, 2017-08-15 at 09:25 +0200, Mike Looijmans wrote:
> On 14-08-17 21:10, Gunnar Andersson wrote:
> > 
> > Hey Russell
> > 
> > I don't claim to be an authority on best practices but I will share
> > some
> > thoughts.
> > 
> > On Sun, 2017-08-13 at 06:58 -0400, Russell Peterson wrote:
> > > Hello.
> > > 
> > > As I learn more about yocto and more importantly gain practical
> > > experience
> > > with it I have started to think about my release structure.  Is
> > > there a
> > > “best practices” document or something like that that speaks to
> > > this?
> > > For example, how does everyone deal with “external” meta layer
> > > dependencies?  My software uses poky and meta-openembedded, of
> > > course.
> > 
> > We simply use a parent project [1] that includes git submodules,
> > one per
> > yocto layer.  I'm of the opinion that if git (only) can be used,
> > why
> > introduce other tools?   But it requires you to learn and master
> > that
> > feature.
> > 
> 
> We arrived at exactly the same setup. One parent repository that
> contains the 
> project-unique recipes, and submodules for the layers it needs.
> 
> It's a lot better than attempting to script things. For one thing,
> git will 
> tell you the state of the submodules.
> 
> If I rembemer correctly, Android builds use a tool to manage the
> layer 
> repositories. I didn't much like it though. And "yet another tool"...

It's called "repo" https://code.google.com/archive/p/git-repo/.

In my experience git submodules work fine if you follow the core
workflow of adding new submodules and updating them to newer versions.
The workflows for deleting, renaming, changed upstream repostiory, etc.
for submodules is IHMO pretty bad. Pretty much every time I've tried
that it has resulted in a borked local repository that I either had to
manually go into the .git directory and fix, or just start over with a
fresh clone (perhaps I'm just bad at it or newer versions of git are
better?). Not to say you shouldn't use them, just be aware of some of
the catches so you can match the tool to your expected workflow.

For my $0.02: We use git submodules to manage the multiple yocto trees
we are pulling in and also keep "parallel" layers for each with the
local changes we want to make in .bbappends, where possible (following
the strategy of no modifying the original). We do (particularly for
older versions of Yocto) make changes to the core since we have local
mirrors, but we follow a strict policy of getting changes upstreamed
first, and they trying to get upstream to backport to our version (if
it is still supported).

> 
> 
> Kind regards,
> 
> Mike Looijmans
> System Expert
> 
> TOPIC Products
> Materiaalweg 4, NL-5681 RJ Best
> Postbus 440, NL-5680 AK Best
> Telefoon: +31 (0) 499 33 69 79
> E-mail: mike.looijmans@topicproducts.com
> Website: www.topicproducts.com
> 
> Please consider the environment before printing this e-mail
> 
> 
> 

Re: release management

[Chris Z.]

[-- Attachment #1: Type: text/plain, Size: 4143 bytes --]

Hi,

Regarding submodule for layers. This is good when no one beside you use it.
Especially  with yocto layers. But when dev starts to use it. You or other
support team will be faced against wall of problems with basic
understanding of git submodules and index. But google starts to invest some
funds in submodules instead repo usage. So it can be a game changer in
favor of git submodules.

From own experience (dev team of 80 persons) we decided to leave submodules
in favor of repo tool. But we had the same conclusion at first. Why to use
other tool were git provides submodules. Further development and expansion
of layers are pro the repo tool manifest file. But that is from my own exp.
And it was good change.

Br,
Chris Z

15.08.2017 15:16 "Joshua Watt" <jpewhacker@gmail.com> napisał(a):

> On Tue, 2017-08-15 at 09:25 +0200, Mike Looijmans wrote:
> > On 14-08-17 21:10, Gunnar Andersson wrote:
> > >
> > > Hey Russell
> > >
> > > I don't claim to be an authority on best practices but I will share
> > > some
> > > thoughts.
> > >
> > > On Sun, 2017-08-13 at 06:58 -0400, Russell Peterson wrote:
> > > > Hello.
> > > >
> > > > As I learn more about yocto and more importantly gain practical
> > > > experience
> > > > with it I have started to think about my release structure.  Is
> > > > there a
> > > > “best practices” document or something like that that speaks to
> > > > this?
> > > > For example, how does everyone deal with “external” meta layer
> > > > dependencies?  My software uses poky and meta-openembedded, of
> > > > course.
> > >
> > > We simply use a parent project [1] that includes git submodules,
> > > one per
> > > yocto layer.  I'm of the opinion that if git (only) can be used,
> > > why
> > > introduce other tools?   But it requires you to learn and master
> > > that
> > > feature.
> > >
> >
> > We arrived at exactly the same setup. One parent repository that
> > contains the
> > project-unique recipes, and submodules for the layers it needs.
> >
> > It's a lot better than attempting to script things. For one thing,
> > git will
> > tell you the state of the submodules.
> >
> > If I rembemer correctly, Android builds use a tool to manage the
> > layer
> > repositories. I didn't much like it though. And "yet another tool"...
>
> It's called "repo" https://code.google.com/archive/p/git-repo/.
>
> In my experience git submodules work fine if you follow the core
> workflow of adding new submodules and updating them to newer versions.
> The workflows for deleting, renaming, changed upstream repostiory, etc.
> for submodules is IHMO pretty bad. Pretty much every time I've tried
> that it has resulted in a borked local repository that I either had to
> manually go into the .git directory and fix, or just start over with a
> fresh clone (perhaps I'm just bad at it or newer versions of git are
> better?). Not to say you shouldn't use them, just be aware of some of
> the catches so you can match the tool to your expected workflow.
>
> For my $0.02: We use git submodules to manage the multiple yocto trees
> we are pulling in and also keep "parallel" layers for each with the
> local changes we want to make in .bbappends, where possible (following
> the strategy of no modifying the original). We do (particularly for
> older versions of Yocto) make changes to the core since we have local
> mirrors, but we follow a strict policy of getting changes upstreamed
> first, and they trying to get upstream to backport to our version (if
> it is still supported).
>
> >
> >
> > Kind regards,
> >
> > Mike Looijmans
> > System Expert
> >
> > TOPIC Products
> > Materiaalweg 4, NL-5681 RJ Best
> > Postbus 440, NL-5680 AK Best
> > Telefoon: +31 (0) 499 33 69 79
> > E-mail: mike.looijmans@topicproducts.com
> > Website: www.topicproducts.com
> >
> > Please consider the environment before printing this e-mail
> >
> >
> >
>
> --
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>

[-- Attachment #2: Type: text/html, Size: 5558 bytes --]

Re: release management

[Svein Seldal]

I have also been slightly puzzled over the meticulousness and 
determinism of Yocto on recipe level, but the left-to-yourself thinking 
for the top-level meta-level and configuration management.

We have chosen to make a custom configuration management (CM) and 
lightweight build system on top of Yocto. Firstly, we have mixed SCM, 
mixing both git and hg, and the CM contains a manifest of the sources 
and can handle proper checkout (which repo does too for git).

Secondly, the CM provides a standardized unified location for setting 
configuration options and control of what artifacts to generate. This is 
then independent of what sub-system, like Yocto, is being run behind the 
scenes. Included in this is organization specific systems, such as 
central defining of system version and build numbers.

The CM also contains some postprocessors for Yocto. In the versions of 
Yocto we use, Yocto cannot generate artifacts for more than one MACHINE 
for each bitbake invocation. We must pack all these output artifacts 
into a common releasable end-user image and upload it to our PLM system. 
This part I have not been able to do (cleanly) from a recipe since it 
spans multiple MACHINES.

So our conclusion is that you need to know what Yocto is good at and 
what it can do, and perhaps rely on other systems where this is 
required. YMMV

Best regards,
Svein


On 19. aug. 2017 00:58, Chris Z. wrote:
> Hi,
> 
> Regarding submodule for layers. This is good when no one beside you use 
> it. Especially  with yocto layers. But when dev starts to use it. You or 
> other support team will be faced against wall of problems with basic 
> understanding of git submodules and index. But google starts to invest 
> some funds in submodules instead repo usage. So it can be a game changer 
> in favor of git submodules.
> 
>  From own experience (dev team of 80 persons) we decided to leave 
> submodules in favor of repo tool. But we had the same conclusion at 
> first. Why to use other tool were git provides submodules. Further 
> development and expansion of layers are pro the repo tool manifest file. 
> But that is from my own exp. And it was good change.
> 
> Br,
> Chris Z
> 
> 15.08.2017 15:16 "Joshua Watt" <jpewhacker@gmail.com 
> <mailto:jpewhacker@gmail.com>> napisał(a):
> 
>     On Tue, 2017-08-15 at 09:25 +0200, Mike Looijmans wrote:
>      > On 14-08-17 21:10, Gunnar Andersson wrote:
>      > >
>      > > Hey Russell
>      > >
>      > > I don't claim to be an authority on best practices but I will share
>      > > some
>      > > thoughts.
>      > >
>      > > On Sun, 2017-08-13 at 06:58 -0400, Russell Peterson wrote:
>      > > > Hello.
>      > > >
>      > > > As I learn more about yocto and more importantly gain practical
>      > > > experience
>      > > > with it I have started to think about my release structure.  Is
>      > > > there a
>      > > > “best practices” document or something like that that speaks to
>      > > > this?
>      > > > For example, how does everyone deal with “external” meta layer
>      > > > dependencies?  My software uses poky and meta-openembedded, of
>      > > > course.
>      > >
>      > > We simply use a parent project [1] that includes git submodules,
>      > > one per
>      > > yocto layer.  I'm of the opinion that if git (only) can be used,
>      > > why
>      > > introduce other tools?   But it requires you to learn and master
>      > > that
>      > > feature.
>      > >
>      >
>      > We arrived at exactly the same setup. One parent repository that
>      > contains the
>      > project-unique recipes, and submodules for the layers it needs.
>      >
>      > It's a lot better than attempting to script things. For one thing,
>      > git will
>      > tell you the state of the submodules.
>      >
>      > If I rembemer correctly, Android builds use a tool to manage the
>      > layer
>      > repositories. I didn't much like it though. And "yet another tool"...
> 
>     It's called "repo" https://code.google.com/archive/p/git-repo/
>     <https://code.google.com/archive/p/git-repo/>.
> 
>     In my experience git submodules work fine if you follow the core
>     workflow of adding new submodules and updating them to newer versions.
>     The workflows for deleting, renaming, changed upstream repostiory, etc.
>     for submodules is IHMO pretty bad. Pretty much every time I've tried
>     that it has resulted in a borked local repository that I either had to
>     manually go into the .git directory and fix, or just start over with a
>     fresh clone (perhaps I'm just bad at it or newer versions of git are
>     better?). Not to say you shouldn't use them, just be aware of some of
>     the catches so you can match the tool to your expected workflow.
> 
>     For my $0.02: We use git submodules to manage the multiple yocto trees
>     we are pulling in and also keep "parallel" layers for each with the
>     local changes we want to make in .bbappends, where possible (following
>     the strategy of no modifying the original). We do (particularly for
>     older versions of Yocto) make changes to the core since we have local
>     mirrors, but we follow a strict policy of getting changes upstreamed
>     first, and they trying to get upstream to backport to our version (if
>     it is still supported).
> 
>      >
>      >
>      > Kind regards,
>      >
>      > Mike Looijmans
>      > System Expert
>      >
>      > TOPIC Products
>      > Materiaalweg 4, NL-5681 RJ Best
>      > Postbus 440, NL-5680 AK Best
>      > Telefoon: +31 (0) 499 33 69 79
>     <tel:%2B31%20%280%29%20499%2033%2069%2079>
>      > E-mail: mike.looijmans@topicproducts.com
>     <mailto:mike.looijmans@topicproducts.com>
>      > Website: www.topicproducts.com <http://www.topicproducts.com>
>      >
>      > Please consider the environment before printing this e-mail
>      >
>      >
>      >
> 
>     --
>     _______________________________________________
>     yocto mailing list
>     yocto@yoctoproject.org <mailto:yocto@yoctoproject.org>
>     https://lists.yoctoproject.org/listinfo/yocto
>     <https://lists.yoctoproject.org/listinfo/yocto>
> 
> 
>