Re: [OE-core] [PATCH RFC] mirrors: Switch glibc and binutils to use shallow mirror tarballs

["Richard Purdie" <richard.purdie@linuxfoundation.org>]

On Thu, 2022-03-31 at 15:45 +0000, Peter Kjellerstedt wrote:
> > -----Original Message-----
> > From: Richard Purdie <richard.purdie@linuxfoundation.org>
> > Sent: den 31 mars 2022 16:16
> > To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>; openembedded-
> > core@lists.openembedded.org
> > Subject: Re: [OE-core] [PATCH RFC] mirrors: Switch glibc and binutils to
> > use shallow mirror tarballs
> > 
> > On Thu, 2022-03-31 at 14:09 +0000, Peter Kjellerstedt wrote:
> > > > -----Original Message-----
> > > > From: openembedded-core@lists.openembedded.org
> > > > <openembedded-core@lists.openembedded.org> On Behalf Of Richard Purdie
> > > > Sent: den 30 mars 2022 14:24
> > > > To: openembedded-core@lists.openembedded.org
> > > > Subject: [OE-core] [PATCH RFC] mirrors: Switch glibc and binutils to use
> > > > shallow mirror tarballs
> > > > 
> > > > These two repositories are large and overload our downloads server as a
> > > > premirror but the recipes are easier to maintain as git urls. Compromise
> > > > and use shallow clones for them.
> > > > 
> > > > In order to be effective, we need premirror entries on where to find
> > > > the shallow mirror tarballs.
> > > > 
> > > > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> > > > ---
> > > >  meta/classes/mirrors.bbclass | 11 +++++++++++
> > > >  1 file changed, 11 insertions(+)
> > > > 
> > > > diff --git a/meta/classes/mirrors.bbclass b/meta/classes/mirrors.bbclass
> > > > index 37dc449ade7..ffdccff5fb4 100644
> > > > --- a/meta/classes/mirrors.bbclass
> > > > +++ b/meta/classes/mirrors.bbclass
> > > > @@ -76,3 +76,14 @@ git://git.gnome.org/.*
> > > > git://gitlab.gnome.org/GNOME/PATH;protocol=https \
> > > >  git://.*/.*                   git://HOST/PATH;protocol=https \
> > > >  git://.*/.*                   git://HOST/git/PATH;protocol=https \
> > > >  "
> > > > +
> > > > +# Switch glibc and binutils recipes to use shallow clones as they're
> > > > large and this
> > > > +# improves user experience whilst allowing the flexibility of git urls
> > > > in the recipes
> > > > +BB_GIT_SHALLOW:pn-binutils = "1"
> > > > +BB_GIT_SHALLOW:pn-binutils-cross-${TARGET_ARCH} = "1"
> > > > +BB_GIT_SHALLOW:pn-binutils-cross-canadian-${TRANSLATED_TARGET_ARCH} =
> > > > "1"
> > > > +BB_GIT_SHALLOW:pn-binutils-cross-testsuite = "1"
> > > > +BB_GIT_SHALLOW:pn-binutils-crosssdk-${SDK_SYS} = "1"
> > > > +BB_GIT_SHALLOW:pn-glibc = "1"
> > > > +PREMIRRORS += "git://sourceware.org/git/glibc.git
> > > > https://downloads.yoctoproject.org/mirror/sources/ \
> > > > +              git://sourceware.org/git/binutils-gdb.git
> > > > https://downloads.yoctoproject.org/mirror/sources/"
> > > 
> > > Please use PREMIRRORS:append here, or you will wipe away any
> > > default configuration of PREMIRRORS defined using "?=".
> > 
> > No, I'm not using append here since I do want users to be able to reset
> > this.
> 
> That can easily be achieved by instead doing something like:
> 
> SHALLOW_PREMIRRORS = " \
>     git://sourceware.org/git/glibc.git https://downloads.yoctoproject.org/mirror/sources/ \
>     git://sourceware.org/git/binutils-gdb.git https://downloads.yoctoproject.org/mirror/sources/ \
> "
> PREMIRRORS:append = " ${SHALLOW_PREMIRRORS}"

This implies that any variable alteration should always use indirection via
another variable. I am not going to support moves in that direction as it is
horrible. If our syntax is causing issues we need to resolve that but I'm not
going to compound it with moves like the above, sorry.

I'd also note this matches how MIRRORS is being handled.

> which I think is a far better solution anyway since it is very 
> hard to reset something in PREMIRRORS as one cannot use :remove.
> With this solution it is possible to set SHALLOW_PREMIRRORS = "" 
> in case one do not want the mirrors OE-Core adds by default.
> 
> Otherwise you are effectively saying that one can no longer define 
> a default for PREMIRRORS using ?=, which was possible up until the 
> above change. And unless I can persuade you to switch to :append, 
> then this needs to be mentioned in the migration notes for 4.0.

Lets release note it then.

Cheers,

Richard