Layer manifest (topic touched at ELCE)

Layer manifest (topic touched at ELCE)

[Andrei Gherzan]

[-- Attachment #1: Type: text/plain, Size: 1007 bytes --]

Hello all,

We started a discussion at the last ELCE: layer manifest in a machine
parsable format. I'll make a short summary on what is this about.

In the current yocto layers structures, the only way to describe a layer is
through README. Which provides useful information like layer dependencies,
available targets, maybe available machines etc. First of all this
information is not standardized in any way and not machine parsable. So is
not an uncommon situation to get a BSP yocto layer and have no idea what to
do without actually looking into it. As well, there is no way of
automating, for example, cloning of dependencies or any other kind of
automations.

Now the questions is, would it make sense to define a layer manifest where
a layer would be fully described in a machine parsable way? Does yocto
community feel the need of such a file and maybe additionally have a script
which would be able to parse such a manifest and setup a build environment?


--
Andrei Gherzan

[-- Attachment #2: Type: text/html, Size: 1084 bytes --]

Re: Layer manifest (topic touched at ELCE)

[Mark Hatle]

On 1/6/16 2:41 PM, Andrei Gherzan wrote:
> Hello all,
> 
> We started a discussion at the last ELCE: layer manifest in a machine parsable
> format. I'll make a short summary on what is this about.
> 
> In the current yocto layers structures, the only way to describe a layer is
> through README. Which provides useful information like layer dependencies,
> available targets, maybe available machines etc. First of all this information
> is not standardized in any way and not machine parsable. So is not an uncommon
> situation to get a BSP yocto layer and have no idea what to do without actually
> looking into it. As well, there is no way of automating, for example, cloning of
> dependencies or any other kind of automations. 
> 
> Now the questions is, would it make sense to define a layer manifest where a
> layer would be fully described in a machine parsable way? Does yocto community
> feel the need of such a file and maybe additionally have a script which would be
> able to parse such a manifest and setup a build environment?  

From a dependency perspective, there is a method to define layer dependencies.
If that is used properly it can help.  The problem I see though is the namespace
for layer dependencies is not really defined.

But using a combination of the defined dependencies and the layer index you
should be able to resolve "known" layers.  (Would also be nice if the index
looked for collisions as well...)

--Mark

> --
> Andrei Gherzan
> 
> 
> 

Re: Layer manifest (topic touched at ELCE)

[Trevor Woerner]

On 01/06/16 17:42, Mark Hatle wrote:
> But using a combination of the defined dependencies and the layer index you
> should be able to resolve "known" layers.  (Would also be nice if the index
> looked for collisions as well...)

I wrote a silly script[1] that probes the layer index and presents the
user with a list of MACHINES. The user chooses and MACHINE and, based on
the dependencies of the BSP layer defining that machine, the script
pulls in all the required layers and updates the conf/bblayer.conf file.
So basically the user just picks a MACHINE and the script does the rest
so all the user then has to do is type "bitbake <image>". It was just a
silly little proof of concept. It's not perfect, but it does work for me
and I do use it all the time. Of course, it makes assumptions with which
others might not agree ;-) ndec tried it and said it didn't work for
him; yymv. The point is the data is out there (in the layer index) and
it's not too hard to just choose a machine and let the computer pull the
layers and update your config. The thing is... IMHO the layers are
_almost_ a foreign part of the entire build process. IMHO they're
_almost_ like a second-class citizen.

I'm not trying to complain.

But it's not just layers, the whole process of setting up your
configuration is a bit of a mess! I've tried floating the idea (maybe it
was at ELCE?) that maybe the configuration files should be more unified?
Wouldn't it be great if I could give you one configuration file and with
it you could type "bitbake <image>" and it would do all the work of
pulling in the layers, the layer dependencies, maybe even specify
specific checkouts of those repositories, it would specify all the bits
you need, and then build your image? In many cases builds only succeed
when specific checkouts of specific layer repositories are used with
specific configuration incantations in specific configuration files.
It'd be nice to be able to specify all the things you need to re-create
my build in one config file.

It seems awkward that I can checkout the yocto layer and build
core-image-x11 as-is, but if I want to build core-image-wayland then I
have to edit conf/local.conf and rejigger a bunch of configuration
parameters before it'll work. If specific DISTRO_FEATURES are required
for an image to build, they should be in the image configuration file,
no? The build shouldn't rely on me correctly adjusting my local.conf to
make it work?

If I perform a build today and want to save <something> so that I can
have some hope of being able to recreate that build at some point in the
future I have to save: auto.conf, local.conf, bblayers.conf, then, for
each layer, I have to record what the HEAD was for its repository as
well as where I got every layer from (there are several github layers,
for example, with the same name).

:-)

The arguments against this panacea ;-) revolve around the need to keep
the build system as flexible as possible. It is argued that each part of
the build process should be independent and therefore be independently
configurable.

Circling back to your email, a layer manifest for a BSP should specify
all layer dependencies, specify where to find those layers (and maybe
even suggest which branches/checkouts to use?), in such a way that the
build could use it automatically without expecting the user to find,
checkout, and update bblayers.conf manually.

Maybe this discussion would be better on oe-architecture?















[1] https://github.com/twoerner/oe-layerindex-config