RE: [OE-core] [PATCH RFC] mirrors: Switch glibc and binutils to use shallow mirror tarballs

[Peter Kjellerstedt <peter.kjellerstedt@axis.com>]

> -----Original Message-----
> From: Richard Purdie <richard.purdie@linuxfoundation.org>
> Sent: den 31 mars 2022 16:16
> To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>; openembedded-
> core@lists.openembedded.org
> Subject: Re: [OE-core] [PATCH RFC] mirrors: Switch glibc and binutils to
> use shallow mirror tarballs
> 
> On Thu, 2022-03-31 at 14:09 +0000, Peter Kjellerstedt wrote:
> > > -----Original Message-----
> > > From: openembedded-core@lists.openembedded.org
> > > <openembedded-core@lists.openembedded.org> On Behalf Of Richard Purdie
> > > Sent: den 30 mars 2022 14:24
> > > To: openembedded-core@lists.openembedded.org
> > > Subject: [OE-core] [PATCH RFC] mirrors: Switch glibc and binutils to use
> > > shallow mirror tarballs
> > >
> > > These two repositories are large and overload our downloads server as a
> > > premirror but the recipes are easier to maintain as git urls. Compromise
> > > and use shallow clones for them.
> > >
> > > In order to be effective, we need premirror entries on where to find
> > > the shallow mirror tarballs.
> > >
> > > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> > > ---
> > >  meta/classes/mirrors.bbclass | 11 +++++++++++
> > >  1 file changed, 11 insertions(+)
> > >
> > > diff --git a/meta/classes/mirrors.bbclass b/meta/classes/mirrors.bbclass
> > > index 37dc449ade7..ffdccff5fb4 100644
> > > --- a/meta/classes/mirrors.bbclass
> > > +++ b/meta/classes/mirrors.bbclass
> > > @@ -76,3 +76,14 @@ git://git.gnome.org/.* git://gitlab.gnome.org/GNOME/PATH;protocol=https \
> > >  git://.*/.*                   git://HOST/PATH;protocol=https \
> > >  git://.*/.*                   git://HOST/git/PATH;protocol=https \
> > >  "
> > > +
> > > +# Switch glibc and binutils recipes to use shallow clones as they're large and this
> > > +# improves user experience whilst allowing the flexibility of git urls in the recipes
> > > +BB_GIT_SHALLOW:pn-binutils = "1"
> > > +BB_GIT_SHALLOW:pn-binutils-cross-${TARGET_ARCH} = "1"
> > > +BB_GIT_SHALLOW:pn-binutils-cross-canadian-${TRANSLATED_TARGET_ARCH} = "1"
> > > +BB_GIT_SHALLOW:pn-binutils-cross-testsuite = "1"
> > > +BB_GIT_SHALLOW:pn-binutils-crosssdk-${SDK_SYS} = "1"
> > > +BB_GIT_SHALLOW:pn-glibc = "1"
> > > +PREMIRRORS += "git://sourceware.org/git/glibc.git https://downloads.yoctoproject.org/mirror/sources/ \
> > > +              git://sourceware.org/git/binutils-gdb.git https://downloads.yoctoproject.org/mirror/sources/"
> >
> > Please use PREMIRRORS:append here, or you will wipe away any
> > default configuration of PREMIRRORS defined using "?=".
> 
> No, I'm not using append here since I do want users to be able to reset this.

That can easily be achieved by instead doing something like:

SHALLOW_PREMIRRORS = " \
    git://sourceware.org/git/glibc.git https://downloads.yoctoproject.org/mirror/sources/ \
    git://sourceware.org/git/binutils-gdb.git https://downloads.yoctoproject.org/mirror/sources/ \
"
PREMIRRORS:append = " ${SHALLOW_PREMIRRORS}"

which I think is a far better solution anyway since it is very 
hard to reset something in PREMIRRORS as one cannot use :remove.
With this solution it is possible to set SHALLOW_PREMIRRORS = "" 
in case one do not want the mirrors OE-Core adds by default.

Otherwise you are effectively saying that one can no longer define 
a default for PREMIRRORS using ?=, which was possible up until the 
above change. And unless I can persuade you to switch to :append, 
then this needs to be mentioned in the migration notes for 4.0.

> We should change the default to "=" if that is causing an issue.

There is no longer any default in OE-Core (there used to be one 
in poky.conf defined using ??=, but you removed it the other day). 
However, due to that preexisting definition, we use PREMIRRORS ?= 
in our site.conf so that it takes precedence over the default from 
OE-Core (if it exists), while it still defines a default 
configuration.

> Cheers,
> 
> Richard

//Peter