Can Yocto treat layers like an external package?

Can Yocto treat layers like an external package?

[Koehler, Yannick (HPN Aruba)]

In our development with Yocto, we use vendor provided layers.  At this time we have to pull those manually (using git or submodule or other technics) to get the vendor layer on disk before bitbake can start.  Does Yocto allow to automatically pull a yocto layer given a uri, branch and other information, similar way as it can pull the source of a package?

This way, we could omit the vendor layer stuff from our git and instead have Yocto fetch the layer from its source (honoring all the premirror and mirror stuff).  The layer specification would be similar to SRC_URI format with an additional parameter to indicate locally where to put the files, maybe with a default of vendor/<layer-name> structure?

	git://git@mylayer.company.com/layer-name.git;protocol=ssh;branch=master

This would pull into 

	vendors/layer-name

Prior to parse all reccipes?

--
Yannick Koehler

Re: Can Yocto treat layers like an external package?

[Gary Thomas]

On 2017-05-24 16:39, Koehler, Yannick (HPN Aruba) wrote:
> In our development with Yocto, we use vendor provided layers.  At this time we have to pull those manually (using git or submodule or other technics) to get the vendor layer on disk before bitbake can start.  Does Yocto allow to automatically pull a yocto layer given a uri, branch and other information, similar way as it can pull the source of a package?
> 
> This way, we could omit the vendor layer stuff from our git and instead have Yocto fetch the layer from its source (honoring all the premirror and mirror stuff).  The layer specification would be similar to SRC_URI format with an additional parameter to indicate locally where to put the files, maybe with a default of vendor/<layer-name> structure?
> 
> 	git://git@mylayer.company.com/layer-name.git;protocol=ssh;branch=master
> 
> This would pull into
> 
> 	vendors/layer-name
> 
> Prior to parse all reccipes?

Not part of the default Yocto setup, but what you are asking for
seems to be a perfect fit for the [Android/Google] repo tool.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: Can Yocto treat layers like an external package?

[Trevor Woerner]

Hi Yannick,

This is a feature many people have been wanting for a while, but
getting there has been slow. So slow, in fact, that many projects have
simply gone ahead and implemented their own solutions, all of which
are different from each other, making it all that much harder to get
everyone back together to support one idea :-(

As Gary mentions, "repo" is a popular solution. See, for example, how
the Linaro people have done it with their "rpb" distro and associated
setup tools/scripts:

https://github.com/96boards/oe-rpb-manifest

The freescale project was the first such instance I saw (but I can't
say whether they were the first or not):

https://github.com/Freescale/fsl-community-bsp-platform
http://freescale.github.io/#download

Mark Hatle (windriver) has been working on and releasing a tool
they've been using internally for a while:

https://www.openembedded.org/wiki/OEDEM_2016#Windriver_.E2.80.98setup.E2.80.99_Demo

I'm sure there are better links for Mark's work, but I can't find them
at the moment. Hopefully someone jumps in and fills in the blanks :-)

I'm sure there are other such examples.

Re: Can Yocto treat layers like an external package?

[Andrea Galbusera]

[-- Attachment #1: Type: text/plain, Size: 1422 bytes --]

Il 25 mag 2017 6:12 AM, "Trevor Woerner" <twoerner@gmail.com> ha scritto:

Hi Yannick,

This is a feature many people have been wanting for a while, but
getting there has been slow. So slow, in fact, that many projects have
simply gone ahead and implemented their own solutions, all of which
are different from each other, making it all that much harder to get
everyone back together to support one idea :-(

As Gary mentions, "repo" is a popular solution. See, for example, how
the Linaro people have done it with their "rpb" distro and associated
setup tools/scripts:

https://github.com/96boards/oe-rpb-manifest

The freescale project was the first such instance I saw (but I can't
say whether they were the first or not):

https://github.com/Freescale/fsl-community-bsp-platform
http://freescale.github.io/#download

Mark Hatle (windriver) has been working on and releasing a tool
they've been using internally for a while:

https://www.openembedded.org/wiki/OEDEM_2016#Windriver_.E2.
80.98setup.E2.80.99_Demo

I'm sure there are better links for Mark's work, but I can't find them
at the moment. Hopefully someone jumps in and fills in the blanks :-)


https://github.com/Wind-River/wr-lx-setup


I'm sure there are other such examples.
--
_______________________________________________
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

[-- Attachment #2: Type: text/html, Size: 2824 bytes --]

Re: Can Yocto treat layers like an external package?

[Koehler, Yannick (HPN Aruba)]

[-- Attachment #1: Type: text/plain, Size: 3530 bytes --]

So far all reply seems to indicate that the fetching and retrieval of additional layers is not a bitbake problem and should be handled with external tools.

In my view, I fail to see how fetching a layer is different than fetching sources from a particular package.

There is in my opinion two kind of layers people will encounter:

  Private layers:
  Layer own by the current developer and likely to be heavily modified with new recipe of override of existing recipes.

  Vendor layers:
  Third-party layers, which normally should be use as is (without modification).

The case I am mostly interested about is the Vendor layers, if I pull in meta-openembedded or meta-freescale in my yocto distro, I will never touch those layer at the git level, instead whatever change I want will be done in my private layer which is the main reason for layer as I understand it (being able to change other’s vendor layer without changing them).

As such, all bitbake needs is to get a fetch location (SRC_URI) and a path to store the result (S), this is quite similar to a package, except that once the layer has been retrieve and put in place it still need to be updated and parsed.

I fail to see why people would seek a non-bitbake solution such as repo, submodules or others if bitbake was able to do that aspect.  If there is a project already for doing this, I would be interested in participating to its development and I may have one or two helper in my team to help out on this.

For private layers, I do understand and see why an external solution to bitbake would be better, since bitbake will not offer support for branch and change management which is normal as bitbake is only a build tool, not a developer tool.

--
Yannick Koehler

From: yocto-bounces@yoctoproject.org [mailto:yocto-bounces@yoctoproject.org] On Behalf Of Andrea Galbusera
Sent: Thursday, May 25, 2017 1:39 AM
To: Trevor Woerner <twoerner@gmail.com>
Cc: yocto@yoctoproject.org
Subject: Re: [yocto] Can Yocto treat layers like an external package?



Il 25 mag 2017 6:12 AM, "Trevor Woerner" <twoerner@gmail.com<mailto:twoerner@gmail.com>> ha scritto:
Hi Yannick,

This is a feature many people have been wanting for a while, but
getting there has been slow. So slow, in fact, that many projects have
simply gone ahead and implemented their own solutions, all of which
are different from each other, making it all that much harder to get
everyone back together to support one idea :-(

As Gary mentions, "repo" is a popular solution. See, for example, how
the Linaro people have done it with their "rpb" distro and associated
setup tools/scripts:

https://github.com/96boards/oe-rpb-manifest

The freescale project was the first such instance I saw (but I can't
say whether they were the first or not):

https://github.com/Freescale/fsl-community-bsp-platform
http://freescale.github.io/#download

Mark Hatle (windriver) has been working on and releasing a tool
they've been using internally for a while:

https://www.openembedded.org/wiki/OEDEM_2016#Windriver_.E2.80.98setup.E2.80.99_Demo

I'm sure there are better links for Mark's work, but I can't find them
at the moment. Hopefully someone jumps in and fills in the blanks :-)

https://github.com/Wind-River/wr-lx-setup


I'm sure there are other such examples.
--
_______________________________________________
yocto mailing list
yocto@yoctoproject.org<mailto:yocto@yoctoproject.org>
https://lists.yoctoproject.org/listinfo/yocto


[-- Attachment #2: Type: text/html, Size: 13064 bytes --]

Re: Can Yocto treat layers like an external package?

[Marcelo E. Magallon]

On Thu, May 25, 2017 at 01:13:03PM +0000, Koehler, Yannick (HPN Aruba) wrote:

> The case I am mostly interested about is the Vendor layers, if I pull
> in meta-openembedded or meta-freescale in my yocto distro, I will
> never touch those layer at the git level, instead whatever change I
> want will be done in my private layer which is the main reason for
> layer as I understand it (being able to change other’s vendor layer
> without changing them).

A "solution" that we use is have a single repo with all the layers, and
manage that using git subtrees.

I would strongly discourage other people from implementing such a
solution, as it solved one problem (creating and updating a workspace
with the correct layers is trivial), but it has created many others. The
biggest one in my view is that people feel encouraged to make
modifications to the vendor layers, which makes updating harder than it
should. Having a single repo with layers within directories also makes
it cumbersome to implement access controls (basically "you can commit to
directories A, B and C, but not X, Y or Z"). It also encourages a kind
of coupling that makes things more complicated in the long run.

> I fail to see why people would seek a non-bitbake solution such as
> repo, submodules or others if bitbake was able to do that aspect.  If
> there is a project already for doing this, I would be interested in
> participating to its development and I may have one or two helper in
> my team to help out on this.

My guess is that's because bitbake would have to read a file with the
layer metadata, fetch the layers that you want, and then read the
recipes. If the layers are handled just like recipes, (with SRC_URI,
SRCREV, S, etc), bitbake needs to be able to read two different sets of
recipes. That might work by changing BBFILES or BBPATH. At that point it
the UI becomes a bit awkward, I think.

It doesn't sound too far fetched, though. bitbake already contains most
(all?) of the code to make this work. I think you would just need to
come up with a reasonable UI to interact with that.

With our solution, getting updated layers (non-vendor) it's just a
matter of "git pull". Solutions built around repo are equally simple.

> For private layers, I do understand and see why an external solution
> to bitbake would be better, since bitbake will not offer support for
> branch and change management which is normal as bitbake is only a
> build tool, not a developer tool.

Well, if you obtain layers using recipes (or something very close to
that), you could include branch information in SRC_URI, as usual.

Marcelo

Re: Can Yocto treat layers like an external package?

[Koehler, Yannick (HPN Aruba)]

Hi Marcello,

Not sure if I understood what you meant but, I am using a single repo with submodule for vendor layers, as such, submodule kind of solve the "temptation" for our developer team to go play in there, since playing there is a minefields thanks to the complicated submodule process.  But, I do not want to rely on the difficulty of submodules to tell my dev not to go change the vendor layer, which is way I do not want them in my repo at all.

Note that private issues is a different matter and I am not addressing it here and there are multiple solution such as subtree, repo, submodules, upcoming GVFS, etc...

>It doesn't sound too far fetched, though. bitbake already contains most
>(all?) of the code to make this work. I think you would just need to come up with a reasonable UI to interact with that.

This is my opinion, I think we could alter bitbake to retrieve the SRC_URI and S information from the bblayers.conf (instead of having recipes like for package). 

    sample 
    # LAYER_CONF_VERSION is increased each time build/conf/bblayers.conf
    # changes incompatibly
    LCONF_VERSION = "6"

    BBPATH = "${TOPDIR}"
    BBFILES ?= ""

    BBLAYERS ?= " \
      ##OEROOT##/meta \
      ##OEROOT##/meta-yocto \
      ##OEROOT##/meta-yocto-bsp \
      ##OEROOT##/meta-openembedded;SRC_URI="git://git@github.com/openembedded/meta-openembedded;protocol=ssh;branch=master";SRCREV="${AUTOREV}" \
      "
    BBLAYERS_NON_REMOVABLE ?= " \
      ##OEROOT##/meta \
      ##OEROOT##/meta-yocto \
      "

Bitbake could then fetch/update the layer from the SRC_URI into the location given "##OEROOT##/meta-openembedded" prior to parsing.

--
Yannick Koehler

-----Original Message-----
From: Magallon, Marcelo 
Sent: Thursday, May 25, 2017 10:40 AM
To: Koehler, Yannick (HPN Aruba) <yannick.koehler@hpe.com>
Cc: Andrea Galbusera <gizero@gmail.com>; Trevor Woerner <twoerner@gmail.com>; yocto@yoctoproject.org
Subject: Re: [yocto] Can Yocto treat layers like an external package?

On Thu, May 25, 2017 at 01:13:03PM +0000, Koehler, Yannick (HPN Aruba) wrote:

> The case I am mostly interested about is the Vendor layers, if I pull 
> in meta-openembedded or meta-freescale in my yocto distro, I will 
> never touch those layer at the git level, instead whatever change I 
> want will be done in my private layer which is the main reason for 
> layer as I understand it (being able to change other’s vendor layer 
> without changing them).

A "solution" that we use is have a single repo with all the layers, and manage that using git subtrees.

I would strongly discourage other people from implementing such a solution, as it solved one problem (creating and updating a workspace with the correct layers is trivial), but it has created many others. The biggest one in my view is that people feel encouraged to make modifications to the vendor layers, which makes updating harder than it should. Having a single repo with layers within directories also makes it cumbersome to implement access controls (basically "you can commit to directories A, B and C, but not X, Y or Z"). It also encourages a kind of coupling that makes things more complicated in the long run.

> I fail to see why people would seek a non-bitbake solution such as 
> repo, submodules or others if bitbake was able to do that aspect.  If 
> there is a project already for doing this, I would be interested in 
> participating to its development and I may have one or two helper in 
> my team to help out on this.

My guess is that's because bitbake would have to read a file with the layer metadata, fetch the layers that you want, and then read the recipes. If the layers are handled just like recipes, (with SRC_URI, SRCREV, S, etc), bitbake needs to be able to read two different sets of recipes. That might work by changing BBFILES or BBPATH. At that point it the UI becomes a bit awkward, I think.

It doesn't sound too far fetched, though. bitbake already contains most
(all?) of the code to make this work. I think you would just need to come up with a reasonable UI to interact with that.

With our solution, getting updated layers (non-vendor) it's just a matter of "git pull". Solutions built around repo are equally simple.

> For private layers, I do understand and see why an external solution 
> to bitbake would be better, since bitbake will not offer support for 
> branch and change management which is normal as bitbake is only a 
> build tool, not a developer tool.

Well, if you obtain layers using recipes (or something very close to that), you could include branch information in SRC_URI, as usual.

Marcelo

Re: Can Yocto treat layers like an external package?

[Patrick Ohly]

On Thu, 2017-05-25 at 16:04 +0000, Koehler, Yannick (HPN Aruba) wrote:
> This is my opinion, I think we could alter bitbake to retrieve the SRC_URI and S information from the bblayers.conf (instead of having recipes like for package).
> 
>     sample 
>     # LAYER_CONF_VERSION is increased each time build/conf/bblayers.conf
>     # changes incompatibly
>     LCONF_VERSION = "6"
> 
>     BBPATH = "${TOPDIR}"
>     BBFILES ?= ""
> 
>     BBLAYERS ?= " \
>       ##OEROOT##/meta \
>       ##OEROOT##/meta-yocto \
>       ##OEROOT##/meta-yocto-bsp \
>       ##OEROOT##/meta-openembedded;SRC_URI="git://git@github.com/openembedded/meta-openembedded;protocol=ssh;branch=master";SRCREV="${AUTOREV}" \
>       "
>     BBLAYERS_NON_REMOVABLE ?= " \
>       ##OEROOT##/meta \
>       ##OEROOT##/meta-yocto \
>       "
> 
> Bitbake could then fetch/update the layer from the SRC_URI into the location given "##OEROOT##/meta-openembedded" prior to parsing.

Richard has indeed been thinking about a solution like that for 2.4.

I agree that it looks attractive at first glance. However, I see a
chicken-and-egg problem with no (easy?) solution: a developer who wants
to get started with a distro "foo" first needs to check out the repo
containing that distro's definition and local.conf.sample, then somehow
also check out a version of bitbake that works with that distro
revision, and only then can bitbake download the additional layers.

Perhaps we can make it so that bitbake has a separate tool that works
outside of a build environment and then sets up that environment.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.