Re: [PATCH RFC 00/21] Git repository sharing for kernel (and other) repos

["Richard Purdie" <richard.purdie@linuxfoundation.org>]

On Fri, 2021-04-02 at 13:15 -0400, Paul Gortmaker wrote:
> Next Steps:
> -----------
> With this being a functional implementation, it seems like a good time to
> get other people looking at it.  Ideally step #1 will be getting general
> agreement that this is something we need, something that is overdue,
> and that the implementation as shown here makes sense in the absence of
> any similar effort from anyone that does the same but in a better way.
> 
> From there, we'll want more people not just looking at it, but testing it
> as well.   I know I want to write a commit (script?) that will avoid any
> "transition tax" by prepopulating new repos with "old" already downloaded
> git objects where we can.  And to add/do tests with my own popluated mirror
> and NO_NETWORK, and also try to ensure nothing in BB_SHALLOW gets upset,
> but I wasn't going to hold up starting a review of this any longer.
> 
> I suspect I can get some co-workers using/testing it too, but Yocto gets
> used in a bunch of different ways by different groups, so we'll no doubt
> have to do some additional fixups to ensure everybody gets the benefits of
> this sharing.  But I'm hopeful that when people see the benefits above,
> they'll pitch in to help take this the final mile by ensuring it works for
> their use case as well.
> 
> I'm not too worried about pontificating out beyond that until we get past
> the acceptance/testing hurdles outlined above.  So, please do have a read
> of the commits, kick the tires, put on your bikeshedding clothes and grab
> a brush, and lets see where it goes from here...
> 
> https://github.com/paulgortmaker/poky/compare/reference-RC1

I've only briefly looked through this but the more I look, the more worried
I'm getting which is never a good sign.

I totally agree with the use case and agree we need to do something in this 
area. I'm not sure the implementation looks right though, its clever but I
don't think its very usable to end users.

The warning signs to me are:

a) Needing new/confusing "fetch only" recipes
b) A large number of new options and variables to the fetcher
c) Needing recipes to change and people to migrate, potentially with scripts
   between old and new
d) Variable namespacing needs work

I'm very worried this confuses up the git fetcher code and nobody will be able
to tell what is going on any more :/.

What I'd envisaged was git urls having something like a "mainline-linux-kernel"
tag added in the url as a parameter and a table somewhere which meant the checkout
for this would share git objects in a common pool. There would be a variable 
mapping that name to git.kernel.org/pub/scm/linux/kernel/git somewhere but that
should be all that is needed.

No, this wouldn't cover 100% of artefacts but it should cover a majority of them
and be much simpler for users to comprehend. I haven't gone into this in detail
and perhaps I'm missing some problem that prevents it? :/

The other issue here is there are no tests. The bitbake fetcher code is one of
the few pieces of the project where we do have a fairly complete test suite and
we don't add things there without tests (see bitbake-selftest and lib/bb/tests/).

Cheers,

Richard