Re: [OE-core] [RFC PATCH] bitbake-layers: add layer repositories/revisions save and restore tooling (aka 'layer configuration')

[Richard Purdie <richard.purdie@linuxfoundation.org>]

On Mon, 2022-07-04 at 11:01 +0200, Adrian Freihofer wrote:
> Thank you for initiating this important discussion with the code. This
> could be one way to address this issue. However, the discussion here
> also shows how complicated the issue is and how fragmented the
> solutions and opinions are. There are already several tools out there,
> but none of them has proven to be "the only right way". I'm not sure
> that writing another tool is really the best approach. The complexity
> of the proposed tool seems to me to be already at the upper limit,
> where on the other hand Richard suggests to develop it even further and
> publish it via pip. At the very least, I see the risk of ending up with
> just another tool that is very complicated, needs maintenance, but
> still won't be accepted by the community.
> 
> Personally I really like to build software as simple as
> git clone --recursive
> bitbake my-image

This simply can't work, unless you have a different parent git repo for
each configuration you're going to share with users. It can work for a
single configuration (e.g. set of layers) only.

That works well for your single company single use case, it doesn't
work well for developers dealing with multiple layers/configurations.

> Setting up layers is basically just about fetching git repos. I don't
> see the need for creating some configuration files or other complicated
> tasks during the initial setup. So before introducing a new tool,
> please let me understand why git submodules have not been very
> successful in the past. I see some reasons for that:
>  * In the past, there were different RCS systems and the knowledge
>    about git was not everywhere. I think that has fundamentally changed
>    and the acceptance of git (and also git submodules) has massively
>    increased. Today, git may even be the only version control system
>    that needs to be officially supported to manage bitbake layers.

I think at this point we're only considering git. That said, there are
submodules, repo, subtree and outliers like combo-layer so even limited
to git, this isn't straightforward.

>  * We still use the submodule structures that Tim mentioned. In
>    general, I agree that using Git submodules is unnecessarily
>    complicated. The challenges start when multiple hierarchies of
>    submodules are used. In this use case, I miss a simple command like
>    "git checkout --recursive" that does everything I currently have to
>    do manually with multiple Git submodules sync, init and update and
>    cd commands.
>  * Probably the lack of a simple, recursive command in git is also the
>    reason why some CI implementations are in rare cases not able to
>    checkout git submodules correctly.
> 
> Do you think there would be a need for a new tool if:
>  * git submodules would be easy to use?
>  * The Yocto manual would suggest to use git submodules for managing
>    the layers and also provide an example folder and submodules
>    structure as a guide line for the users?

The Yocto manual cannot mandate submodules. They work for some people,
others despise them.

>  * If the knowledge of git had been as widespread a few years ago (when
>    the distributions Tim mentions were published) as it is today?
> I believe that today it may well be possible to establish git
> submodules as the recommended solution. (Something like an easy to use
> "git checkout --recursive" command would certainly helpful.)

If people were going to standardise on that it would have happened by
now. It hasn't and I don't think it will. They work well for some
usecases, people want/need/like other tools in other cases.

> Since the majority of mostly experienced Yocto/OE developers who are
> participating this discussion tend to develop a new tool, it makes me
> wonder if I'm missing something. I see the following use cases where
> layers need to be fetched:
>  * Initial project setup for working with bitbake.
>  * Retrieving layers from an SDK. (I'm not sure if this should remain
>    something special. The PoC which was recently posted by Alex for
>    bootstrapping the SDK directly from the bitbake environment looks
>    very promising to me).
>  * Fetching the layers on CI infrastructures which often call git fetch
>    with fancy options to improve efficiency. (That would probably not
>    work with a Yocto specific fetch tool anyway.)

There would be some value in having a common shared way of doing this
should as many people need to handle CI at some point.

> Do you see other use cases for a layer fetching tool?
> 
> What do you think about trying to optimize git submodules to handle the
> "layer fetching" use case with a simple command, rather than developing
> a new Yocto-specific git wrapper?

I think you'll end up with repo :). There are definitely Yocto specific
elements to this which is why we're seeing specific tools.

> Is it really useful to generate a configuration for KAS? A tool that
> generates a configuration for another tool that finally does a Git
> checkout seems a bit over-engineered to me. At least for us, an
> implementation based on Git submodules would be usable, which would not
> be the case for a KAS based implementation.

If the alternative is reinventing kas then it could make sense. See my
email for why I think we need something higher level than these tools,
which is implementation independent (as much as it can be).

Cheers,

Richard