Re: [OE-core] [PATCH 2/5] meta/files: add layer setup JSON schema and example

[Richard Purdie <richard.purdie@linuxfoundation.org>]

On Wed, 2022-08-17 at 23:36 +0200, Alexander Kanavin wrote:
> On Wed, 17 Aug 2022 at 22:52, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > One thing that concerns me a little is that this mixes two things,
> > layer setup (as in repos) and configuration. I'm nervous about json
> > config which effectively has to duplicate the list of machines/distros
> > in layers.
> > 
> > Where is the distro/machine data used?
> 
> The use case I was thinking of is doing something useful with this
> information *before* doing actual checkout. I would even appreciate
> knowing in advance - maybe even by just looking at the json through
> gitweb ui - which machines, distros and config templates I would be
> getting, and from which layers.

The trouble is that others already went a different direction with this
which is why we have layer index and the concept of local layer indexes
as well we the ability for bblayers to talk to such indexes if I'm
remembering correctly.

An advantage it has is that it does parse the layers so it can cache
other information too.

My worry is that you have two copies of this information, one in the
json and one in the metadata and effectively by that fact, one of them
will be wrong. 

You could report it to the user with a directory listing of the remote
repo via git.

>  Also the proposed oe-setup tool could
> do useful things with this information, if it operates from the json
> definitions in the same format. You can discover and supply all needed
> information up front, and oe-setup will land you in a bitbake session.
> Without it, you have to first do the layer setup, then the
> configuration discovery, then the configuration setup, as separate
> commands - I'd say that would be an inferior experience for the users.

I think you can get a lot of this information relatively easily and
possibly without a clone, via a layer index or probably through a
remote git command too.

> > Are these the only config options people will want to add or will we
> > have others? init system? libc? Or feature information about which
> > configurations are expected to work? I worry this is a magnet for
> > future feature creep and duplication of information.
> 
> These are not the same, as machines and distros are specifically
> defined in layers as .conf files in pre-determined locations. Other
> things are defined as variables in files, and aren't as easily
> discoverable by code. I only want to allow programmatically
> discoverable options (that are guaranteed to be in lockstep with the
> layer revision) in the json; no one should ever write or modify the
> json by hand.

I'm not sure that will scale for what users want then (which I suspect
is why we ended up with the layer index instead).

> > I can see where the buildconfigs are used but again, does the json file
> > need to encode those or could you determine them by inspection of the
> > layers once setup?
> 
> In addition to the above point (knowing them in advance is useful),
> there is no fixed location for these in layers. You could perhaps walk
> the layer trees, but I'd rather place the info in the json.

I have wondered if we should define one. I kind of have a plan in mind
to turn the majority of the autobuilder configurations into config
files somewhere in OE-Core to make it easier for people to reproduce
autobuilder configs too, which may be an interesting example of that.

> > Also, if there is new version of this series, could you squash in these
> > copyright/license tweaks please:
> > 
> > https://git.yoctoproject.org/poky/commit/?h=master-next&id=45b396298c1dd638bb702f5251b4a663f07978ca
> 
> This is the easy part :-) Hope the above clarifies a bit what went on
> in my head when I was writing the code.

It does, thanks. I'm just not sure we're quite in the right place with
things. I know in your other reply you say we have to start somewhere
and we can change things. I also know the screaming if we try and
remove something later...

Cheers,

Richard