[PATCH RFC] mirrors: Switch glibc and binutils to use shallow mirror tarballs

[PATCH RFC] mirrors: Switch glibc and binutils to use shallow mirror tarballs

[Richard Purdie]

These two repositories are large and overload our downloads server as a
premirror but the recipes are easier to maintain as git urls. Compromise
and use shallow clones for them.

In order to be effective, we need premirror entries on where to find
the shallow mirror tarballs.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/classes/mirrors.bbclass | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/meta/classes/mirrors.bbclass b/meta/classes/mirrors.bbclass
index 37dc449ade7..ffdccff5fb4 100644
--- a/meta/classes/mirrors.bbclass
+++ b/meta/classes/mirrors.bbclass
@@ -76,3 +76,14 @@ git://git.gnome.org/.*        git://gitlab.gnome.org/GNOME/PATH;protocol=https \
 git://.*/.*                   git://HOST/PATH;protocol=https \
 git://.*/.*                   git://HOST/git/PATH;protocol=https \
 "
+
+# Switch glibc and binutils recipes to use shallow clones as they're large and this
+# improves user experience whilst allowing the flexibility of git urls in the recipes
+BB_GIT_SHALLOW:pn-binutils = "1"
+BB_GIT_SHALLOW:pn-binutils-cross-${TARGET_ARCH} = "1"
+BB_GIT_SHALLOW:pn-binutils-cross-canadian-${TRANSLATED_TARGET_ARCH} = "1"
+BB_GIT_SHALLOW:pn-binutils-cross-testsuite = "1"
+BB_GIT_SHALLOW:pn-binutils-crosssdk-${SDK_SYS} = "1"
+BB_GIT_SHALLOW:pn-glibc = "1"
+PREMIRRORS += "git://sourceware.org/git/glibc.git https://downloads.yoctoproject.org/mirror/sources/ \
+              git://sourceware.org/git/binutils-gdb.git https://downloads.yoctoproject.org/mirror/sources/"
-- 
2.32.0

RE: [OE-core] [PATCH RFC] mirrors: Switch glibc and binutils to use shallow mirror tarballs

[Peter Kjellerstedt]

> -----Original Message-----
> From: openembedded-core@lists.openembedded.org <openembedded-core@lists.openembedded.org> On Behalf Of Richard Purdie
> Sent: den 30 mars 2022 14:24
> To: openembedded-core@lists.openembedded.org
> Subject: [OE-core] [PATCH RFC] mirrors: Switch glibc and binutils to use shallow mirror tarballs
> 
> These two repositories are large and overload our downloads server as a
> premirror but the recipes are easier to maintain as git urls. Compromise
> and use shallow clones for them.
> 
> In order to be effective, we need premirror entries on where to find
> the shallow mirror tarballs.
> 
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
>  meta/classes/mirrors.bbclass | 11 +++++++++++
>  1 file changed, 11 insertions(+)
> 
> diff --git a/meta/classes/mirrors.bbclass b/meta/classes/mirrors.bbclass
> index 37dc449ade7..ffdccff5fb4 100644
> --- a/meta/classes/mirrors.bbclass
> +++ b/meta/classes/mirrors.bbclass
> @@ -76,3 +76,14 @@ git://git.gnome.org/.* git://gitlab.gnome.org/GNOME/PATH;protocol=https \
>  git://.*/.*                   git://HOST/PATH;protocol=https \
>  git://.*/.*                   git://HOST/git/PATH;protocol=https \
>  "
> +
> +# Switch glibc and binutils recipes to use shallow clones as they're large and this
> +# improves user experience whilst allowing the flexibility of git urls in the recipes
> +BB_GIT_SHALLOW:pn-binutils = "1"
> +BB_GIT_SHALLOW:pn-binutils-cross-${TARGET_ARCH} = "1"
> +BB_GIT_SHALLOW:pn-binutils-cross-canadian-${TRANSLATED_TARGET_ARCH} = "1"
> +BB_GIT_SHALLOW:pn-binutils-cross-testsuite = "1"
> +BB_GIT_SHALLOW:pn-binutils-crosssdk-${SDK_SYS} = "1"
> +BB_GIT_SHALLOW:pn-glibc = "1"
> +PREMIRRORS += "git://sourceware.org/git/glibc.git https://downloads.yoctoproject.org/mirror/sources/ \
> +              git://sourceware.org/git/binutils-gdb.git https://downloads.yoctoproject.org/mirror/sources/"

Please use PREMIRRORS:append here, or you will wipe away any 
default configuration of PREMIRRORS defined using "?=".

> --
> 2.32.0

//Peter

Re: [OE-core] [PATCH RFC] mirrors: Switch glibc and binutils to use shallow mirror tarballs

[Richard Purdie]

On Thu, 2022-03-31 at 14:09 +0000, Peter Kjellerstedt wrote:
> > -----Original Message-----
> > From: openembedded-core@lists.openembedded.org
> > <openembedded-core@lists.openembedded.org> On Behalf Of Richard Purdie
> > Sent: den 30 mars 2022 14:24
> > To: openembedded-core@lists.openembedded.org
> > Subject: [OE-core] [PATCH RFC] mirrors: Switch glibc and binutils to use
> > shallow mirror tarballs
> > 
> > These two repositories are large and overload our downloads server as a
> > premirror but the recipes are easier to maintain as git urls. Compromise
> > and use shallow clones for them.
> > 
> > In order to be effective, we need premirror entries on where to find
> > the shallow mirror tarballs.
> > 
> > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> > ---
> >  meta/classes/mirrors.bbclass | 11 +++++++++++
> >  1 file changed, 11 insertions(+)
> > 
> > diff --git a/meta/classes/mirrors.bbclass b/meta/classes/mirrors.bbclass
> > index 37dc449ade7..ffdccff5fb4 100644
> > --- a/meta/classes/mirrors.bbclass
> > +++ b/meta/classes/mirrors.bbclass
> > @@ -76,3 +76,14 @@ git://git.gnome.org/.*
> > git://gitlab.gnome.org/GNOME/PATH;protocol=https \
> >  git://.*/.*                   git://HOST/PATH;protocol=https \
> >  git://.*/.*                   git://HOST/git/PATH;protocol=https \
> >  "
> > +
> > +# Switch glibc and binutils recipes to use shallow clones as they're large
> > and this
> > +# improves user experience whilst allowing the flexibility of git urls in
> > the recipes
> > +BB_GIT_SHALLOW:pn-binutils = "1"
> > +BB_GIT_SHALLOW:pn-binutils-cross-${TARGET_ARCH} = "1"
> > +BB_GIT_SHALLOW:pn-binutils-cross-canadian-${TRANSLATED_TARGET_ARCH} = "1"
> > +BB_GIT_SHALLOW:pn-binutils-cross-testsuite = "1"
> > +BB_GIT_SHALLOW:pn-binutils-crosssdk-${SDK_SYS} = "1"
> > +BB_GIT_SHALLOW:pn-glibc = "1"
> > +PREMIRRORS += "git://sourceware.org/git/glibc.git
> > https://downloads.yoctoproject.org/mirror/sources/ \
> > +              git://sourceware.org/git/binutils-gdb.git
> > https://downloads.yoctoproject.org/mirror/sources/"
> 
> Please use PREMIRRORS:append here, or you will wipe away any 
> default configuration of PREMIRRORS defined using "?=".

No, I'm not using append here since I do want users to be able to reset this. We
should change the default to "=" if that is causing an issue.

Cheers,

Richard

RE: [OE-core] [PATCH RFC] mirrors: Switch glibc and binutils to use shallow mirror tarballs

[Peter Kjellerstedt]

> -----Original Message-----
> From: Richard Purdie <richard.purdie@linuxfoundation.org>
> Sent: den 31 mars 2022 16:16
> To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>; openembedded-
> core@lists.openembedded.org
> Subject: Re: [OE-core] [PATCH RFC] mirrors: Switch glibc and binutils to
> use shallow mirror tarballs
> 
> On Thu, 2022-03-31 at 14:09 +0000, Peter Kjellerstedt wrote:
> > > -----Original Message-----
> > > From: openembedded-core@lists.openembedded.org
> > > <openembedded-core@lists.openembedded.org> On Behalf Of Richard Purdie
> > > Sent: den 30 mars 2022 14:24
> > > To: openembedded-core@lists.openembedded.org
> > > Subject: [OE-core] [PATCH RFC] mirrors: Switch glibc and binutils to use
> > > shallow mirror tarballs
> > >
> > > These two repositories are large and overload our downloads server as a
> > > premirror but the recipes are easier to maintain as git urls. Compromise
> > > and use shallow clones for them.
> > >
> > > In order to be effective, we need premirror entries on where to find
> > > the shallow mirror tarballs.
> > >
> > > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> > > ---
> > >  meta/classes/mirrors.bbclass | 11 +++++++++++
> > >  1 file changed, 11 insertions(+)
> > >
> > > diff --git a/meta/classes/mirrors.bbclass b/meta/classes/mirrors.bbclass
> > > index 37dc449ade7..ffdccff5fb4 100644
> > > --- a/meta/classes/mirrors.bbclass
> > > +++ b/meta/classes/mirrors.bbclass
> > > @@ -76,3 +76,14 @@ git://git.gnome.org/.* git://gitlab.gnome.org/GNOME/PATH;protocol=https \
> > >  git://.*/.*                   git://HOST/PATH;protocol=https \
> > >  git://.*/.*                   git://HOST/git/PATH;protocol=https \
> > >  "
> > > +
> > > +# Switch glibc and binutils recipes to use shallow clones as they're large and this
> > > +# improves user experience whilst allowing the flexibility of git urls in the recipes
> > > +BB_GIT_SHALLOW:pn-binutils = "1"
> > > +BB_GIT_SHALLOW:pn-binutils-cross-${TARGET_ARCH} = "1"
> > > +BB_GIT_SHALLOW:pn-binutils-cross-canadian-${TRANSLATED_TARGET_ARCH} = "1"
> > > +BB_GIT_SHALLOW:pn-binutils-cross-testsuite = "1"
> > > +BB_GIT_SHALLOW:pn-binutils-crosssdk-${SDK_SYS} = "1"
> > > +BB_GIT_SHALLOW:pn-glibc = "1"
> > > +PREMIRRORS += "git://sourceware.org/git/glibc.git https://downloads.yoctoproject.org/mirror/sources/ \
> > > +              git://sourceware.org/git/binutils-gdb.git https://downloads.yoctoproject.org/mirror/sources/"
> >
> > Please use PREMIRRORS:append here, or you will wipe away any
> > default configuration of PREMIRRORS defined using "?=".
> 
> No, I'm not using append here since I do want users to be able to reset this.

That can easily be achieved by instead doing something like:

SHALLOW_PREMIRRORS = " \
    git://sourceware.org/git/glibc.git https://downloads.yoctoproject.org/mirror/sources/ \
    git://sourceware.org/git/binutils-gdb.git https://downloads.yoctoproject.org/mirror/sources/ \
"
PREMIRRORS:append = " ${SHALLOW_PREMIRRORS}"

which I think is a far better solution anyway since it is very 
hard to reset something in PREMIRRORS as one cannot use :remove.
With this solution it is possible to set SHALLOW_PREMIRRORS = "" 
in case one do not want the mirrors OE-Core adds by default.

Otherwise you are effectively saying that one can no longer define 
a default for PREMIRRORS using ?=, which was possible up until the 
above change. And unless I can persuade you to switch to :append, 
then this needs to be mentioned in the migration notes for 4.0.

> We should change the default to "=" if that is causing an issue.

There is no longer any default in OE-Core (there used to be one 
in poky.conf defined using ??=, but you removed it the other day). 
However, due to that preexisting definition, we use PREMIRRORS ?= 
in our site.conf so that it takes precedence over the default from 
OE-Core (if it exists), while it still defines a default 
configuration.

> Cheers,
> 
> Richard

//Peter

Re: [OE-core] [PATCH RFC] mirrors: Switch glibc and binutils to use shallow mirror tarballs

[Richard Purdie]

On Thu, 2022-03-31 at 15:45 +0000, Peter Kjellerstedt wrote:
> > -----Original Message-----
> > From: Richard Purdie <richard.purdie@linuxfoundation.org>
> > Sent: den 31 mars 2022 16:16
> > To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>; openembedded-
> > core@lists.openembedded.org
> > Subject: Re: [OE-core] [PATCH RFC] mirrors: Switch glibc and binutils to
> > use shallow mirror tarballs
> > 
> > On Thu, 2022-03-31 at 14:09 +0000, Peter Kjellerstedt wrote:
> > > > -----Original Message-----
> > > > From: openembedded-core@lists.openembedded.org
> > > > <openembedded-core@lists.openembedded.org> On Behalf Of Richard Purdie
> > > > Sent: den 30 mars 2022 14:24
> > > > To: openembedded-core@lists.openembedded.org
> > > > Subject: [OE-core] [PATCH RFC] mirrors: Switch glibc and binutils to use
> > > > shallow mirror tarballs
> > > > 
> > > > These two repositories are large and overload our downloads server as a
> > > > premirror but the recipes are easier to maintain as git urls. Compromise
> > > > and use shallow clones for them.
> > > > 
> > > > In order to be effective, we need premirror entries on where to find
> > > > the shallow mirror tarballs.
> > > > 
> > > > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> > > > ---
> > > >  meta/classes/mirrors.bbclass | 11 +++++++++++
> > > >  1 file changed, 11 insertions(+)
> > > > 
> > > > diff --git a/meta/classes/mirrors.bbclass b/meta/classes/mirrors.bbclass
> > > > index 37dc449ade7..ffdccff5fb4 100644
> > > > --- a/meta/classes/mirrors.bbclass
> > > > +++ b/meta/classes/mirrors.bbclass
> > > > @@ -76,3 +76,14 @@ git://git.gnome.org/.*
> > > > git://gitlab.gnome.org/GNOME/PATH;protocol=https \
> > > >  git://.*/.*                   git://HOST/PATH;protocol=https \
> > > >  git://.*/.*                   git://HOST/git/PATH;protocol=https \
> > > >  "
> > > > +
> > > > +# Switch glibc and binutils recipes to use shallow clones as they're
> > > > large and this
> > > > +# improves user experience whilst allowing the flexibility of git urls
> > > > in the recipes
> > > > +BB_GIT_SHALLOW:pn-binutils = "1"
> > > > +BB_GIT_SHALLOW:pn-binutils-cross-${TARGET_ARCH} = "1"
> > > > +BB_GIT_SHALLOW:pn-binutils-cross-canadian-${TRANSLATED_TARGET_ARCH} =
> > > > "1"
> > > > +BB_GIT_SHALLOW:pn-binutils-cross-testsuite = "1"
> > > > +BB_GIT_SHALLOW:pn-binutils-crosssdk-${SDK_SYS} = "1"
> > > > +BB_GIT_SHALLOW:pn-glibc = "1"
> > > > +PREMIRRORS += "git://sourceware.org/git/glibc.git
> > > > https://downloads.yoctoproject.org/mirror/sources/ \
> > > > +              git://sourceware.org/git/binutils-gdb.git
> > > > https://downloads.yoctoproject.org/mirror/sources/"
> > > 
> > > Please use PREMIRRORS:append here, or you will wipe away any
> > > default configuration of PREMIRRORS defined using "?=".
> > 
> > No, I'm not using append here since I do want users to be able to reset
> > this.
> 
> That can easily be achieved by instead doing something like:
> 
> SHALLOW_PREMIRRORS = " \
>     git://sourceware.org/git/glibc.git https://downloads.yoctoproject.org/mirror/sources/ \
>     git://sourceware.org/git/binutils-gdb.git https://downloads.yoctoproject.org/mirror/sources/ \
> "
> PREMIRRORS:append = " ${SHALLOW_PREMIRRORS}"

This implies that any variable alteration should always use indirection via
another variable. I am not going to support moves in that direction as it is
horrible. If our syntax is causing issues we need to resolve that but I'm not
going to compound it with moves like the above, sorry.

I'd also note this matches how MIRRORS is being handled.

> which I think is a far better solution anyway since it is very 
> hard to reset something in PREMIRRORS as one cannot use :remove.
> With this solution it is possible to set SHALLOW_PREMIRRORS = "" 
> in case one do not want the mirrors OE-Core adds by default.
> 
> Otherwise you are effectively saying that one can no longer define 
> a default for PREMIRRORS using ?=, which was possible up until the 
> above change. And unless I can persuade you to switch to :append, 
> then this needs to be mentioned in the migration notes for 4.0.

Lets release note it then.

Cheers,

Richard