[Buildroot] [PATCH 1/1] package/mpfr: Fix GNU Mirror Path

[Buildroot] [PATCH 1/1] package/mpfr: Fix GNU Mirror Path

[Zhangjin Wu]

Replacing http://www.mpfr.org with BR2_GNU_MIRROR variable

Signed-off-by: Zhangjin Wu <falcon@tinylab.org>
---
 package/mpfr/mpfr.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package/mpfr/mpfr.mk b/package/mpfr/mpfr.mk
index 8d12c7502b..a5e26404d1 100644
--- a/package/mpfr/mpfr.mk
+++ b/package/mpfr/mpfr.mk
@@ -5,7 +5,7 @@
 ################################################################################
 
 MPFR_VERSION = 4.1.1
-MPFR_SITE = http://www.mpfr.org/mpfr-$(MPFR_VERSION)
+MPFR_SITE = $(BR2_GNU_MIRROR)/mpfr
 MPFR_SOURCE = mpfr-$(MPFR_VERSION).tar.xz
 MPFR_LICENSE = LGPL-3.0+
 MPFR_LICENSE_FILES = COPYING.LESSER
-- 
2.34.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/1] package/mpfr: Fix GNU Mirror Path

[Yann E. MORIN]

Zhangjin Wu, All,

On 2023-12-10 00:21 +0800, Zhangjin Wu spake thusly:
> Replacing http://www.mpfr.org with BR2_GNU_MIRROR variable

The upstream for MPFR still references their own location as the
download location:

    https://www.mpfr.org/mpfr-4.1.1/

E.g.:

    https://www.mpfr.org/mpfr-4.1.1/mpfr-4.1.1.tar.xz

What is the rationale for switching over to the GNU mirror rather than
use the official upstream location?

Regards,
Yann E. MORIN.

> Signed-off-by: Zhangjin Wu <falcon@tinylab.org>
> ---
>  package/mpfr/mpfr.mk | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/package/mpfr/mpfr.mk b/package/mpfr/mpfr.mk
> index 8d12c7502b..a5e26404d1 100644
> --- a/package/mpfr/mpfr.mk
> +++ b/package/mpfr/mpfr.mk
> @@ -5,7 +5,7 @@
>  ################################################################################
>  
>  MPFR_VERSION = 4.1.1
> -MPFR_SITE = http://www.mpfr.org/mpfr-$(MPFR_VERSION)
> +MPFR_SITE = $(BR2_GNU_MIRROR)/mpfr
>  MPFR_SOURCE = mpfr-$(MPFR_VERSION).tar.xz
>  MPFR_LICENSE = LGPL-3.0+
>  MPFR_LICENSE_FILES = COPYING.LESSER
> -- 
> 2.34.1
> 
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/1] package/mpfr: Fix GNU Mirror Path

[Zhangjin Wu]

Hi Yann

> Zhangjin Wu, All,
> 
> On 2023-12-10 00:21 +0800, Zhangjin Wu spake thusly:
> > Replacing http://www.mpfr.org with BR2_GNU_MIRROR variable
> 
> The upstream for MPFR still references their own location as the
> download location:
> 
>     https://www.mpfr.org/mpfr-4.1.1/
> 
> E.g.:
> 
>     https://www.mpfr.org/mpfr-4.1.1/mpfr-4.1.1.tar.xz
> 
> What is the rationale for switching over to the GNU mirror rather than
> use the official upstream location?
>

I have found that the downloading of mpfr from its official site is very
slow in our region, sometimes, costs several hours and even fails at
last. But the downloading from one of its local GNU mirror sites is fast
and deterministic.

And I have checked that the latest version of mpfr has been stored in
the GNU mirror site too:

http://ftpmirror.gnu.org/mpfr-4.2.1.tar.xz

So, it should be reasonable and safe for us to do such switch ;-)

Regards,
Zhangjin Wu

> Regards,
> Yann E. MORIN.
> 
> > Signed-off-by: Zhangjin Wu <falcon@tinylab.org>
> > ---
> >  package/mpfr/mpfr.mk | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/package/mpfr/mpfr.mk b/package/mpfr/mpfr.mk
> > index 8d12c7502b..a5e26404d1 100644
> > --- a/package/mpfr/mpfr.mk
> > +++ b/package/mpfr/mpfr.mk
> > @@ -5,7 +5,7 @@
> >  ################################################################################
> >  
> >  MPFR_VERSION = 4.1.1
> > -MPFR_SITE = http://www.mpfr.org/mpfr-$(MPFR_VERSION)
> > +MPFR_SITE = $(BR2_GNU_MIRROR)/mpfr
> >  MPFR_SOURCE = mpfr-$(MPFR_VERSION).tar.xz
> >  MPFR_LICENSE = LGPL-3.0+
> >  MPFR_LICENSE_FILES = COPYING.LESSER
> > -- 
> > 2.34.1
> > 
> > _______________________________________________
> > buildroot mailing list
> > buildroot@buildroot.org
> > https://lists.buildroot.org/mailman/listinfo/buildroot
> 
> -- 
> .-----------------.--------------------.------------------.--------------------.
> |  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
> | +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
> | +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
> | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
> '------------------------------^-------^------------------^--------------------'
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/1] package/mpfr: Fix GNU Mirror Path

[Yann E. MORIN]

Zhangjin Wu, All,

On 2023-12-10 09:36 +0800, Zhangjin Wu spake thusly:
> > On 2023-12-10 00:21 +0800, Zhangjin Wu spake thusly:
> > > Replacing http://www.mpfr.org with BR2_GNU_MIRROR variable
[--SNIP--]
> > What is the rationale for switching over to the GNU mirror rather than
> > use the official upstream location?
> I have found that the downloading of mpfr from its official site is very
> slow in our region, sometimes, costs several hours and even fails at
> last. But the downloading from one of its local GNU mirror sites is fast
> and deterministic.
> And I have checked that the latest version of mpfr has been stored in
> the GNU mirror site too:
> http://ftpmirror.gnu.org/mpfr-4.2.1.tar.xz

    $ wget http://ftpmirror.gnu.org/mpfr-4.2.1.tar.xz
    HTTP request sent, awaiting response... 404 Not Found
    2023-12-10 11:54:02 ERROR 404: Not Found.

(it's missing the /mpfr/ directory component in the path ;-) ).

> So, it should be reasonable and safe for us to do such switch ;-)

The thing is, we can't really switch away from the official site, just
because it is slow for some people, as the alternative might be the
other way around for others: for me, the GNU mirror is ~8 times slower
than the official MPFR site: 8.80MBps vs 1.06MBps (mpfr is small, so
that's not much consequential, but that's on principle).

So, I think you would have a few (complementary) options:

  - keep a local cache of your download directory: if BR2_DL_DIR is set
    in the environment, then Buildroot will use that directory to look
    for, and store, downloaded archives. That directory is never removed
    by Buildroot (unless it is the default value of $(TOPDIR)/dl);

  - similar to the above, populate a directory on a machine in your
    local network, serve that directory with an HTTP(S) server, and
    point BR2_PRIMARY_SITE to that location;

  - change BR2_WGET to include one of the wget timeout options (see the
    wget manual for the details): --timeout=seconds --dns-timeout=seconds
    --connect-timeout=seconds --read-timeout=seconds. In your case, maybe
    --read-timeout with a small-ish number of seconds would be enough to
    trigger a failed download, and Buildroot will fallback to
    downloading from BR2_BACKUP_SITE.

I know that the above might not be entirely satisfactory, but we really
prefer to get the sources from the official location, unless there is a
very strong reason not to. In this case, I don't think it is
appropriate, especially since Buildroot has mechanisms in place to cope
with such situations already.

Regards,
Yann E. MORIN.

> Regards,
> Zhangjin Wu
> 
> > Regards,
> > Yann E. MORIN.
> > 
> > > Signed-off-by: Zhangjin Wu <falcon@tinylab.org>
> > > ---
> > >  package/mpfr/mpfr.mk | 2 +-
> > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > 
> > > diff --git a/package/mpfr/mpfr.mk b/package/mpfr/mpfr.mk
> > > index 8d12c7502b..a5e26404d1 100644
> > > --- a/package/mpfr/mpfr.mk
> > > +++ b/package/mpfr/mpfr.mk
> > > @@ -5,7 +5,7 @@
> > >  ################################################################################
> > >  
> > >  MPFR_VERSION = 4.1.1
> > > -MPFR_SITE = http://www.mpfr.org/mpfr-$(MPFR_VERSION)
> > > +MPFR_SITE = $(BR2_GNU_MIRROR)/mpfr
> > >  MPFR_SOURCE = mpfr-$(MPFR_VERSION).tar.xz
> > >  MPFR_LICENSE = LGPL-3.0+
> > >  MPFR_LICENSE_FILES = COPYING.LESSER
> > > -- 
> > > 2.34.1
> > > 
> > > _______________________________________________
> > > buildroot mailing list
> > > buildroot@buildroot.org
> > > https://lists.buildroot.org/mailman/listinfo/buildroot
> > 
> > -- 
> > .-----------------.--------------------.------------------.--------------------.
> > |  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
> > | +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
> > | +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
> > | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
> > '------------------------------^-------^------------------^--------------------'
> > _______________________________________________
> > buildroot mailing list
> > buildroot@buildroot.org
> > https://lists.buildroot.org/mailman/listinfo/buildroot
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/1] package/mpfr: Fix GNU Mirror Path

[Zhangjin Wu]

> Zhangjin Wu, All,
> 
> On 2023-12-10 09:36 +0800, Zhangjin Wu spake thusly:
> > > On 2023-12-10 00:21 +0800, Zhangjin Wu spake thusly:
> > > > Replacing http://www.mpfr.org with BR2_GNU_MIRROR variable
> [--SNIP--]
> > > What is the rationale for switching over to the GNU mirror rather than
> > > use the official upstream location?
> > I have found that the downloading of mpfr from its official site is very
> > slow in our region, sometimes, costs several hours and even fails at
> > last. But the downloading from one of its local GNU mirror sites is fast
> > and deterministic.
> > And I have checked that the latest version of mpfr has been stored in
> > the GNU mirror site too:
> > http://ftpmirror.gnu.org/mpfr-4.2.1.tar.xz
> 
>     $ wget http://ftpmirror.gnu.org/mpfr-4.2.1.tar.xz
>     HTTP request sent, awaiting response... 404 Not Found
>     2023-12-10 11:54:02 ERROR 404: Not Found.
> 
> (it's missing the /mpfr/ directory component in the path ;-) ).
>

Yeah, /mpfr/ is missing (but it is there for the patch, my typo issue), thanks ;-)

> > So, it should be reasonable and safe for us to do such switch ;-)
> 
> The thing is, we can't really switch away from the official site, just
> because it is slow for some people, as the alternative might be the
> other way around for others: for me, the GNU mirror is ~8 times slower
> than the official MPFR site: 8.80MBps vs 1.06MBps (mpfr is small, so
> that's not much consequential, but that's on principle).
>

For the speed issue above, perhaps the automatic selection of
http://ftpmirror.gnu.org didn't provide the best mirror instance or the select
logic itself costed some time, what about manually choose one from the
following list or another one even not in this list?

    https://www.gnu.org/prep/ftp.html

Of course, it still depends on the service quanlity of the local mirror site.

As you suggested, it is better to use the official site of a target package, I
understand the advantage, but to be honest, I'm still curious about that is it
possible to benefit from the GNU mirror variable for the packages that have
already been 'released' in the GNU mirror sites, mpfr is only one of such
packages, and for better availability with enough tolerance (some of the
official sites may down sometimes) and balanced average download speed (there
should be at least one mirror site there with a not that bad speed), the GNU
mirror like services may be a download candidate for such packages.

The only issue here may be how to guarantee the GNU mirror *always* store such
packages, keep update, or at least make sure the storing policy is long-term, I
didn't find any official document about this yet.

> So, I think you would have a few (complementary) options:
> 
>   - keep a local cache of your download directory: if BR2_DL_DIR is set
>     in the environment, then Buildroot will use that directory to look
>     for, and store, downloaded archives. That directory is never removed
>     by Buildroot (unless it is the default value of $(TOPDIR)/dl);
> 
>   - similar to the above, populate a directory on a machine in your
>     local network, serve that directory with an HTTP(S) server, and
>     point BR2_PRIMARY_SITE to that location;
> 
>   - change BR2_WGET to include one of the wget timeout options (see the
>     wget manual for the details): --timeout=seconds --dns-timeout=seconds
>     --connect-timeout=seconds --read-timeout=seconds. In your case, maybe
>     --read-timeout with a small-ish number of seconds would be enough to
>     trigger a failed download, and Buildroot will fallback to
>     downloading from BR2_BACKUP_SITE.
>

Thanks very much, dl/, BR2_PRIMARY_SITE and BR2_BACKUP_SITE are good solutions,
but all of them require further work locally. It is possible to build some
other mirror sites for dl/, but the GNU mirror sites have been there for many
years, perhaps we can let them work as a fallback (like BR2_BACKUP_SITE)?
 
> I know that the above might not be entirely satisfactory, but we really
> prefer to get the sources from the official location, unless there is a
> very strong reason not to. In this case, I don't think it is
> appropriate, especially since Buildroot has mechanisms in place to cope
> with such situations already.
>

Get it, thanks.

Best regards,
Zhangjin Wu
 
> Regards,
> Yann E. MORIN.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot