[Buildroot] BR2_SOURCEFORGE_MIRROR issue in package .mk files

[Buildroot] BR2_SOURCEFORGE_MIRROR issue in package .mk files

[Aras Vaichas]

I was rebuilding my Buildroot when I noticed that the
kent.dl.sourceforge.net mirror is down and my build fails.

I encountered this same problem the other day with crosstool-ng, and
soon afterwards a patch was posted:
http://sourceware.org/ml/crossgcc/2012-08/msg00066.html

I looked at changing BR2_SOURCEFORGE_MIRROR but unfortunately
Buildroot only allows BR2_SOURCEFORGE_MIRROR to replace *part* of the
Sourceforge URL

e.g. in buildroot/package/boost/boost.mk:
BOOST_SITE = http://$(BR2_SOURCEFORGE_MIRROR).dl.sourceforge.net/sourceforge/boost

Should it be changed to:
http://$(BR2_SOURCEFORGE_MIRROR).sourceforge.net/sourceforge/boost

Comments?

regards,

Aras Vaichas

[Buildroot] BR2_SOURCEFORGE_MIRROR issue in package .mk files

[Yann E. MORIN]

Aras, All,

On Tuesday 21 August 2012 17:51:07 Aras Vaichas wrote:
> I was rebuilding my Buildroot when I noticed that the
> kent.dl.sourceforge.net mirror is down and my build fails.
> 
> I encountered this same problem the other day with crosstool-ng, and
> soon afterwards a patch was posted:
> http://sourceware.org/ml/crossgcc/2012-08/msg00066.html

.. and has been applied and pushed! ;-)

> I looked at changing BR2_SOURCEFORGE_MIRROR but unfortunately
> Buildroot only allows BR2_SOURCEFORGE_MIRROR to replace *part* of the
> Sourceforge URL
> 
> e.g. in buildroot/package/boost/boost.mk:
> BOOST_SITE = http://$(BR2_SOURCEFORGE_MIRROR).dl.sourceforge.net/sourceforge/boost
> 
> Should it be changed to:
> http://$(BR2_SOURCEFORGE_MIRROR).sourceforge.net/sourceforge/boost

I guess the correct solution would be to use their mirror-selector (as is
done for ct-ng, as you point out):
  http://downloads.sourceforge.net/blablabla

This will redirect to an active mirror, and we can get rid of this
SF mirror selection.

To be noted, the SF mirrors can come and go without notice. kent, who used
to be a reliable one, has disapeared a few days ago.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] BR2_SOURCEFORGE_MIRROR issue in package .mk files

[Stefan Fröberg]

21.8.2012 18:51, Aras Vaichas kirjoitti:
> I was rebuilding my Buildroot when I noticed that the
> kent.dl.sourceforge.net mirror is down and my build fails.
>
> I encountered this same problem the other day with crosstool-ng, and
> soon afterwards a patch was posted:
> http://sourceware.org/ml/crossgcc/2012-08/msg00066.html
>
> I looked at changing BR2_SOURCEFORGE_MIRROR but unfortunately
> Buildroot only allows BR2_SOURCEFORGE_MIRROR to replace *part* of the
> Sourceforge URL
>
> e.g. in buildroot/package/boost/boost.mk:
> BOOST_SITE = http://$(BR2_SOURCEFORGE_MIRROR).dl.sourceforge.net/sourceforge/boost
>
> Should it be changed to:
> http://$(BR2_SOURCEFORGE_MIRROR).sourceforge.net/sourceforge/boost
>
> Comments?

Yeah, kent is definetely down.
Checked with this:

http://kent.dl.sourceforge.net/project/boost/boost/1.49.0/boost_1_49_0.tar.bz2

But this works:

http://tenet.dl.sourceforge.net/project/boost/boost/1.49.0/boost_1_49_0.tar.bz2

If that South African mirror is too far away/slow  then you could change
the BR2_SOURCEFORGE_MIRROR
to some other, short name value from
http://sourceforge.net/apps/trac/sourceforge/wiki/Mirrors

In related note: Maybe this BR2_SOURCEFORGE_MIRROR stuff could be
automated somehow ?

Like using some mirrorselect tool (or something like that) if
BR2_SOURCEFORGE_MIRROR is empty and picking the fastest/closest mirror.
And if BR2_SOURCEFORGE_MIRROR is set in Config.in then use that value no
matter what.

Just thoughts....

Stefan

>
> regards,
>
> Aras Vaichas
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

[Buildroot] BR2_SOURCEFORGE_MIRROR issue in package .mk files

[Stefan Fröberg]

21.8.2012 20:07, Stefan Fr?berg kirjoitti:
> 21.8.2012 18:51, Aras Vaichas kirjoitti:
>> I was rebuilding my Buildroot when I noticed that the
>> kent.dl.sourceforge.net mirror is down and my build fails.
>>
>> I encountered this same problem the other day with crosstool-ng, and
>> soon afterwards a patch was posted:
>> http://sourceware.org/ml/crossgcc/2012-08/msg00066.html
>>
>> I looked at changing BR2_SOURCEFORGE_MIRROR but unfortunately
>> Buildroot only allows BR2_SOURCEFORGE_MIRROR to replace *part* of the
>> Sourceforge URL
>>
>> e.g. in buildroot/package/boost/boost.mk:
>> BOOST_SITE = http://$(BR2_SOURCEFORGE_MIRROR).dl.sourceforge.net/sourceforge/boost
>>
>> Should it be changed to:
>> http://$(BR2_SOURCEFORGE_MIRROR).sourceforge.net/sourceforge/boost
>>
>> Comments?
> Yeah, kent is definetely down.
> Checked with this:
>
> http://kent.dl.sourceforge.net/project/boost/boost/1.49.0/boost_1_49_0.tar.bz2
>
> But this works:
>
> http://tenet.dl.sourceforge.net/project/boost/boost/1.49.0/boost_1_49_0.tar.bz2
>
> If that South African mirror is too far away/slow  then you could change
> the BR2_SOURCEFORGE_MIRROR
> to some other, short name value from
> http://sourceforge.net/apps/trac/sourceforge/wiki/Mirrors
>
> In related note: Maybe this BR2_SOURCEFORGE_MIRROR stuff could be
> automated somehow ?
>
> Like using some mirrorselect tool (or something like that) if
> BR2_SOURCEFORGE_MIRROR is empty and picking the fastest/closest mirror.
> And if BR2_SOURCEFORGE_MIRROR is set in Config.in then use that value no
> matter what.
>
> Just thoughts....
>
> Stefan

Update:

It seems that it is possible to let sourceforge pick the right mirror
automatically with wget

I used this:

wget
http://sourceforge.net/projects/boost/files/boost/1.49.0/boost_1_49_0.tar.bz2/download

and it produced something like this:

--2012-08-21 21:21:40-- 
http://sourceforge.net/projects/boost/files/boost/1.49.0/boost_1_49_0.tar.bz2/download
Resolving sourceforge.net... 216.34.181.60
Connecting to sourceforge.net|216.34.181.60|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location:
http://downloads.sourceforge.net/project/boost/boost/1.49.0/boost_1_49_0.tar.bz2?r=&ts=1345573301&use_mirror=garr
[following]
--2012-08-21 21:21:41-- 
http://downloads.sourceforge.net/project/boost/boost/1.49.0/boost_1_49_0.tar.bz2?r=&ts=1345573301&use_mirror=garr
Resolving downloads.sourceforge.net... 216.34.181.59
Connecting to downloads.sourceforge.net|216.34.181.59|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location:
http://garr.dl.sourceforge.net/project/boost/boost/1.49.0/boost_1_49_0.tar.bz2
[following]
--2012-08-21 21:21:41-- 
http://garr.dl.sourceforge.net/project/boost/boost/1.49.0/boost_1_49_0.tar.bz2
Resolving garr.dl.sourceforge.net... 193.206.140.34, 2001:760:ffff:b0::34
Connecting to garr.dl.sourceforge.net|193.206.140.34|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 48499961 (46M) [application/octet-stream]
Saving to: `boost_1_49_0.tar.bz2.1'

     0K .......... .......... .......... .......... ..........  0% 3.90M 12s
    50K .......... .......... .......... .......... ..........  0%  776K 36s
   100K .......... .......... .......... .......... ..........  0%  654K 48s
   150K .......... .......... .......... .......... ..........  0%  457K 62s
   200K .......... .......... .......... .......... ..........  0%  579K 66s
   250K .......... .......... .......... .......... ..........  0%  152M 55s
   300K .......... .......... .......... .......... ..........  0%  895K 54s
   350K .......... .......... .......... .......... ..........  0%  753K 55s
   400K .......... .......... .......... .......... ..........  0%  776K 56s
   450K .......... .......... .......... .......... ..........  1%  763K 56s
   500K .......... .......... .......... .......... ..........  1%  314K 65s
   550K .......... .......... .......... .......... ..........  1% 8.01M 60s

etc....

So it seems that wget is able to pick mirror (garr in this case)
automatically.

So automatic sourceforge mirror selection should be trivial to do if all
the other sourceforge packages also follow the same syntax:
http://sourceforge.net/projects/${PACKAGE_NAME}/files/${PACKAGE_NAME}/${PACKAGE_VERSION}/${PACKAGE_FILENAME}/download


Stefan


>> regards,
>>
>> Aras Vaichas
>> _______________________________________________
>> buildroot mailing list
>> buildroot at busybox.net
>> http://lists.busybox.net/mailman/listinfo/buildroot
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

[Buildroot] BR2_SOURCEFORGE_MIRROR issue in package .mk files

[Yann E. MORIN]

Aras, Stefan, All,

On Tuesday 21 August 2012 20:29:43 Stefan Fr?berg wrote:
> 21.8.2012 20:07, Stefan Fr?berg kirjoitti:
> > 21.8.2012 18:51, Aras Vaichas kirjoitti:
> >> I was rebuilding my Buildroot when I noticed that the
> >> kent.dl.sourceforge.net mirror is down and my build fails.
[--SNIP--]
> It seems that it is possible to let sourceforge pick the right mirror
> automatically with wget

> wget
> http://sourceforge.net/projects/boost/files/boost/1.49.0/boost_1_49_0.tar.bz2/download
> 
> and it produced something like this:
> 
> --2012-08-21 21:21:40-- 
> http://sourceforge.net/projects/boost/files/boost/1.49.0/boost_1_49_0.tar.bz2/download
> Resolving sourceforge.net... 216.34.181.60
> Connecting to sourceforge.net|216.34.181.60|:80... connected.
> HTTP request sent, awaiting response... 302 Found
> Location:
> http://downloads.sourceforge.net/project/boost/boost/1.49.0/boost_1_49_0.tar.bz2?r=&ts=1345573301&use_mirror=garr
         ^^^^^^^^^
> [following]

[--SNIP--]

> So automatic sourceforge mirror selection should be trivial to do if all
> the other sourceforge packages also follow the same syntax:
> http://sourceforge.net/projects/${PACKAGE_NAME}/files/${PACKAGE_NAME}/${PACKAGE_VERSION}/${PACKAGE_FILENAME}/download

Well, to avoid one 302-redirect, it's possible to directly use their mirror
selection mechanism. For boost, that'd simply be:

$ wget http://downloads.sourceforge.net/project/boost/boost/1.49.0/boost_1_49_0.tar.bz2

--2012-08-21 21:52:21--  http://downloads.sourceforge.net/project/boost/boost/1.49.0/boost_1_49_0.tar.bz2
Resolving downloads.sourceforge.net... 216.34.181.59
Connecting to downloads.sourceforge.net|216.34.181.59|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://ignum.dl.sourceforge.net/project/boost/boost/1.49.0/boost_1_49_0.tar.bz2 [following]
--2012-08-21 21:52:21--  http://ignum.dl.sourceforge.net/project/boost/boost/1.49.0/boost_1_49_0.tar.bz2
Resolving ignum.dl.sourceforge.net... 62.109.128.11, 2001:1ab0:7e1f:1:230:48ff:fed1:9c0a
Connecting to ignum.dl.sourceforge.net|62.109.128.11|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 48499961 (46M) [application/x-bzip2]
Saving to: ?boost_1_49_0.tar.bz2?
[--SNIP--]

In buildroot, it could be written thus:
BOOST_SITE = sf://boost/boost/$(BOOST_VERSION)

with 'sf://' automatically turned by the pkg-infa into the canonical SF
download base URL:
  http://downloads.sourceforge.net/project/

and thus giving the full URL:
  http://downloads.sourceforge.net/project/boost/boost/$(BOOST_VERSION)/$(BOOST_SOURCE)

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] BR2_SOURCEFORGE_MIRROR issue in package .mk files

[Stefan Fröberg]

21.8.2012 23:04, Yann E. MORIN kirjoitti:
> Aras, Stefan, All,
>
> On Tuesday 21 August 2012 20:29:43 Stefan Fr?berg wrote:
>> 21.8.2012 20:07, Stefan Fr?berg kirjoitti:
>>> 21.8.2012 18:51, Aras Vaichas kirjoitti:
>>>> I was rebuilding my Buildroot when I noticed that the
>>>> kent.dl.sourceforge.net mirror is down and my build fails.
> [--SNIP--]
>> It seems that it is possible to let sourceforge pick the right mirror
>> automatically with wget
>> wget
>> http://sourceforge.net/projects/boost/files/boost/1.49.0/boost_1_49_0.tar.bz2/download
>>
>> and it produced something like this:
>>
>> --2012-08-21 21:21:40-- 
>> http://sourceforge.net/projects/boost/files/boost/1.49.0/boost_1_49_0.tar.bz2/download
>> Resolving sourceforge.net... 216.34.181.60
>> Connecting to sourceforge.net|216.34.181.60|:80... connected.
>> HTTP request sent, awaiting response... 302 Found
>> Location:
>> http://downloads.sourceforge.net/project/boost/boost/1.49.0/boost_1_49_0.tar.bz2?r=&ts=1345573301&use_mirror=garr
>          ^^^^^^^^^
>> [following]
> [--SNIP--]
>
>> So automatic sourceforge mirror selection should be trivial to do if all
>> the other sourceforge packages also follow the same syntax:
>> http://sourceforge.net/projects/${PACKAGE_NAME}/files/${PACKAGE_NAME}/${PACKAGE_VERSION}/${PACKAGE_FILENAME}/download
> Well, to avoid one 302-redirect, it's possible to directly use their mirror
> selection mechanism. For boost, that'd simply be:
>
> $ wget http://downloads.sourceforge.net/project/boost/boost/1.49.0/boost_1_49_0.tar.bz2
>
> --2012-08-21 21:52:21--  http://downloads.sourceforge.net/project/boost/boost/1.49.0/boost_1_49_0.tar.bz2
> Resolving downloads.sourceforge.net... 216.34.181.59
> Connecting to downloads.sourceforge.net|216.34.181.59|:80... connected.
> HTTP request sent, awaiting response... 302 Found
> Location: http://ignum.dl.sourceforge.net/project/boost/boost/1.49.0/boost_1_49_0.tar.bz2 [following]
> --2012-08-21 21:52:21--  http://ignum.dl.sourceforge.net/project/boost/boost/1.49.0/boost_1_49_0.tar.bz2
> Resolving ignum.dl.sourceforge.net... 62.109.128.11, 2001:1ab0:7e1f:1:230:48ff:fed1:9c0a
> Connecting to ignum.dl.sourceforge.net|62.109.128.11|:80... connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 48499961 (46M) [application/x-bzip2]
> Saving to: ?boost_1_49_0.tar.bz2?
> [--SNIP--]
>
> In buildroot, it could be written thus:
> BOOST_SITE = sf://boost/boost/$(BOOST_VERSION)
>
> with 'sf://' automatically turned by the pkg-infa into the canonical SF
> download base URL:
>   http://downloads.sourceforge.net/project/
>
> and thus giving the full URL:
>   http://downloads.sourceforge.net/project/boost/boost/$(BOOST_VERSION)/$(BOOST_SOURCE)
Thank you!

That sf:// thing did not work for me but I tried this with libpng and
changed LIBPNG_SITE to:

LIBPNG_SITE =
http://downloads.sourceforge.net/project/libpng/libpng${LIBPNG_SERIES}/${LIBPNG_VERSION}

And when I run "make source" it gives:

--2012-08-21 23:46:14-- 
http://downloads.sourceforge.net/project/libpng/libpng14/1.4.12/libpng-1.4.12.tar.bz2
Resolving downloads.sourceforge.net... 216.34.181.59
Connecting to downloads.sourceforge.net|216.34.181.59|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location:
http://garr.dl.sourceforge.net/project/libpng/libpng14/1.4.12/libpng-1.4.12.tar.bz2
[following]
--2012-08-21 23:46:15-- 
http://garr.dl.sourceforge.net/project/libpng/libpng14/1.4.12/libpng-1.4.12.tar.bz2
Resolving garr.dl.sourceforge.net... 193.206.140.34, 2001:760:ffff:b0::34
Connecting to garr.dl.sourceforge.net|193.206.140.34|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 652462 (637K) [application/octet-stream]
Saving to: `/work/buildroot/dl/libpng-1.4.12.tar.bz2.tmp'

     0K .......... .......... .......... .......... ..........  7%  177K 3s
    50K .......... .......... .......... .......... .......... 15%  341K 2s
   100K .......... .......... .......... .......... .......... 23%  368K 2s
   150K .......... .......... .......... .......... .......... 31%  523K 1s

Works nicely !  :-)
So that BR2_SOURCEFORGE_MIRROR variable could be now (unless somebody
really want's to force mirror) considered deprecated ?

Thanks again!

Stefan

> Regards,
> Yann E. MORIN.
>

[Buildroot] BR2_SOURCEFORGE_MIRROR issue in package .mk files

[Yann E. MORIN]

Stefan, All,

On Tuesday 21 August 2012 22:55:52 Stefan Fr?berg wrote:
> 21.8.2012 23:04, Yann E. MORIN kirjoitti:
[--SNIP--]
> > In buildroot, it could be written thus:
> > BOOST_SITE = sf://boost/boost/$(BOOST_VERSION)
> >
> > with 'sf://' automatically turned by the pkg-infa into the canonical SF
> > download base URL:
> >   http://downloads.sourceforge.net/project/
> >
> > and thus giving the full URL:
> >   http://downloads.sourceforge.net/project/boost/boost/$(BOOST_VERSION)/$(BOOST_SOURCE)
> Thank you!
> 
> That sf:// thing did not work for me

Of course not! Maybe I was not clear, but that was *only* a suggestion! ;-)

> but I tried this with libpng and
> changed LIBPNG_SITE to:
> 
> LIBPNG_SITE =
> http://downloads.sourceforge.net/project/libpng/libpng${LIBPNG_SERIES}/${LIBPNG_VERSION}
[--SNIP--]
> Works nicely !  :-)

What's even better, is that downloads.sf.net should pick a mirror that is
near to you (in terms of network), and that is not over-loaded. So I think
it would better to use than use a fixed mirror.

I'll submit a patch shortly to switch all packages to use that. I'll even
see if I can implement my sf:// scheme, just as a proof-of-concept. ;-)

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'