Re: [PATCH RFC 00/21] Git repository sharing for kernel (and other) repos

["Paul Gortmaker" <paul.gortmaker@windriver.com>]

[Re: [PATCH RFC 00/21] Git repository sharing for kernel (and other) repos] On 02/04/2021 (Fri 23:14) Richard Purdie wrote:

> On Fri, 2021-04-02 at 13:15 -0400, Paul Gortmaker wrote:
> > Next Steps:
> > -----------
> > With this being a functional implementation, it seems like a good time to
> > get other people looking at it.  Ideally step #1 will be getting general
> > agreement that this is something we need, something that is overdue,
> > and that the implementation as shown here makes sense in the absence of
> > any similar effort from anyone that does the same but in a better way.
> > 
> > From there, we'll want more people not just looking at it, but testing it
> > as well.   I know I want to write a commit (script?) that will avoid any
> > "transition tax" by prepopulating new repos with "old" already downloaded
> > git objects where we can.  And to add/do tests with my own popluated mirror
> > and NO_NETWORK, and also try to ensure nothing in BB_SHALLOW gets upset,
> > but I wasn't going to hold up starting a review of this any longer.
> > 
> > I suspect I can get some co-workers using/testing it too, but Yocto gets
> > used in a bunch of different ways by different groups, so we'll no doubt
> > have to do some additional fixups to ensure everybody gets the benefits of
> > this sharing.  But I'm hopeful that when people see the benefits above,
> > they'll pitch in to help take this the final mile by ensuring it works for
> > their use case as well.
> > 
> > I'm not too worried about pontificating out beyond that until we get past
> > the acceptance/testing hurdles outlined above.  So, please do have a read
> > of the commits, kick the tires, put on your bikeshedding clothes and grab
> > a brush, and lets see where it goes from here...
> > 
> > https://github.com/paulgortmaker/poky/compare/reference-RC1
> 
> I've only briefly looked through this but the more I look, the more worried
> I'm getting which is never a good sign.

Well, firstly thanks for the quick response, and I hope you'll give it
another look when you have more time, because I'm not quite convinced
I've conveyed how it is to be used in a way that got through as intended.

> I totally agree with the use case and agree we need to do something in this 
> area. I'm not sure the implementation looks right though, its clever but I
> don't think its very usable to end users.

Not sure who you consider "end users".  I don't really consider myself
or Bruce or anyone else crafting up a kernel recipe from scratch as an
end user.  But even if an "end user" wants to do that, everything here
is opt-in.  If you want to go clone 3G of stuff from mysuperkernel.org
in your new recipe, then feel free.  Nothing stops them.

> The warning signs to me are:
> 
> a) Needing new/confusing "fetch only" recipes

Not sure how they are confusing - and not really new either; I stole the
idea of zeroing out all the tasks from kernel-devsrc if I recall...

> b) A large number of new options and variables to the fetcher

I'd disagree with the characterization of "large" - but if you do have
the time, please go look over the commit logs for each again.  They
all clearly indicate a necessary use case we don't handle well (or at
all) -- and we'll have to solve them one way or another.  I'm open to
considering alternate solutions for any of them.

> c) Needing recipes to change and people to migrate, potentially with scripts
>    between old and new

It feels like we are looking at two different things.  The users or
"people" don't really migrate and the only people changing things would
be a small subset of people who maintain kernel recipes - which is a
small group to begin with.  Recycling old git objects from existing
downloads would most likey be done in a transparent and hands free way.

> d) Variable namespacing needs work

OK - if given specifics it can and will be addressed.

> I'm very worried this confuses up the git fetcher code and nobody will be able
> to tell what is going on any more :/.

That kind of surprises me, to hear that after having looked at the shallow clone
changes to the fetcher, but again - I'm listening - what can we change
in the fetcher?  What is off limits?  The "pool" idea below doesn't address
the fact that "--mirror" is simply not viable for some repos, and that
is hard-coded into the fetcher.   Similar for the other fetcher changes.
And the "pool" idea is still going to have to teach the fetcher about
some kind of "--reference" because that *has* to be added at fetch/clone
time and not as some later afterthought post-fetch.

I could simply take over do_fetch for the kernel and leave the fetcher
untouched, but when the problems being solved weren't really kernel
specific, that didn't seem like the right approach.

> What I'd envisaged was git urls having something like a "mainline-linux-kernel"
> tag added in the url as a parameter and a table somewhere which meant the checkout
> for this would share git objects in a common pool. There would be a variable 
> mapping that name to git.kernel.org/pub/scm/linux/kernel/git somewhere but that
> should be all that is needed.

I saw this idea floated in the earlier thread that Randy pointed me at,
but I just don't see it being viable or the right solution even if one
could make it work.  How do you decide what lives in the pool?  If it is
just mainline, then it really isn't a pool.  Are we only solving for the
kernel, or are we making a solution for any repo that is unweildy?  What
triggers populating the pool?  Do you stuff all mainline and -rt and all
of stable in the pool?  Or just chunks of rt and stable?  If you aren't
tracking what is in there via encapsulating chunks somehow, how do you
expire stuff from the pool?  If it is one giant blob covering mainline
and linux-yocto objects, with no fetch-on-demand properties, then we are
back to the same old 3G-downloads-from-the-mirror problem.  How do you
split that monster?  The devil is in the details.

> No, this wouldn't cover 100% of artefacts but it should cover a majority of them
> and be much simpler for users to comprehend. I haven't gone into this in detail
> and perhaps I'm missing some problem that prevents it? :/

Well, I've thought about this a bunch, and I didn't come up with a whole
bunch of options to choose from that deliver what this delivers.

A mainline-only kernel-specific non-pool "pool" would be better than
nothing, but I'd still want to hear you flesh out how you thought it
would look and operate before I went out and tried to guess what you had
in mind.

Also, I'm still confused by this use of  "users" - what is here all
happens transparently in the background and users aren't aware of it any
more than they are aware of what the fetcher is currently doing today.

> The other issue here is there are no tests. The bitbake fetcher code is one of
> the few pieces of the project where we do have a fairly complete test suite and
> we don't add things there without tests (see bitbake-selftest and lib/bb/tests/).

Yes, of course; and I've been told that by several people in advance.
But as you can imagine, I wasn't going to write tests without 1st
floating the general concept as an RFC and seeing how that went.

Anyway, thanks for the initial scan and I do hope you have a chance to
revisit it in more detail and consider the various problems it solves.

Paul.
--

> 
> Cheers,
> 
> Richard
>