[PATCH] base.bbclass: introduce OE source mirror network as last resort for do_fetch

[PATCH] base.bbclass: introduce OE source mirror network as last resort for do_fetch

[Rolf Leggewie]

Updating recipes and checksums.ini for vanishing upstream hosts or simple 
restructuring can quickly become a chore for OE devs if taken seriously. 
The OE user experience suffers severely when at any given point in time 4 
to 5 recipes on average fail do_fetch for a simple image.

The solution is for OE to provide its own caching of upstream sources. 
mirrors.openembedded.org is a swarm of possibly incomplete source mirrors 
provided by the community and load-balanced via DNS round robin.  They 
act as a first line of defense and lower bandwidth load on melo which 
hosts sources.openembedded.org.  The melo mirror aims to have a complete 
copy of free-to-publish sources used by the current org.oe.dev branch.

Some distributions have provided source mirrors in the past for their 
users. I think it's time for a general and all-encompassing solution.

The infrastructure is being currently put in place.  This patch will
turn on the OE sources mirror network by default.
---
 classes/base.bbclass |    4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)

diff --git a/classes/base.bbclass b/classes/base.bbclass
index 5363e5c..51648da 100644
--- a/classes/base.bbclass
+++ b/classes/base.bbclass
@@ -1312,6 +1312,10 @@ ftp://lsof.itap.purdue.edu/pub/tools/unix/lsof/  ftp://gd.tuwien.ac.at/utils/adm
 ftp://lsof.itap.purdue.edu/pub/tools/unix/lsof/  ftp://sunsite.ualberta.ca/pub/Mirror/lsof/
 ftp://lsof.itap.purdue.edu/pub/tools/unix/lsof/  ftp://the.wiretapped.net/pub/security/host-security/lsof/
 http://www.apache.org/dist  http://archive.apache.org/dist
+ftp://.*/.*     http://mirrors.openembedded.org/
+https?$://.*/.* http://mirrors.openembedded.org/
+ftp://.*/.*     http://sources.openembedded.org/
+https?$://.*/.* http://sources.openembedded.org/
 
 }
 
-- 
1.5.4.3

Re: [PATCH] base.bbclass: introduce OE source mirror network as last resort for do_fetch

[Holger Hans Peter Freyther]

On Friday 14 August 2009 16:29:22 Rolf Leggewie wrote:
> Updating recipes and checksums.ini for vanishing upstream hosts or simple
> restructuring can quickly become a chore for OE devs if taken seriously.
> The OE user experience suffers severely when at any given point in time 4
> to 5 recipes on average fail do_fetch for a simple image.
>
> The solution is for OE to provide its own caching of upstream sources.
> mirrors.openembedded.org is a swarm of possibly incomplete source mirrors
> provided by the community and load-balanced via DNS round robin.  They
> act as a first line of defense and lower bandwidth load on melo which
> hosts sources.openembedded.org.  The melo mirror aims to have a complete
> copy of free-to-publish sources used by the current org.oe.dev branch.
>
> Some distributions have provided source mirrors in the past for their
> users. I think it's time for a general and all-encompassing solution.
>
> The infrastructure is being currently put in place.  This patch will
> turn on the OE sources mirror network by default.

Acked-by: Holger Freyther <zecke@selfish.org>

I think having a last resort is good to have. Some recipes are unfetchable 
faster than you can commit it (fakeroot I look like you...giving me that hong 
kong feeling)


> ---
>  classes/base.bbclass |    4 ++++
>  1 files changed, 4 insertions(+), 0 deletions(-)
>
> diff --git a/classes/base.bbclass b/classes/base.bbclass
> index 5363e5c..51648da 100644
> --- a/classes/base.bbclass
> +++ b/classes/base.bbclass
> @@ -1312,6 +1312,10 @@ ftp://lsof.itap.purdue.edu/pub/tools/unix/lsof/ 
> ftp://gd.tuwien.ac.at/utils/adm
> ftp://lsof.itap.purdue.edu/pub/tools/unix/lsof/ 
> ftp://sunsite.ualberta.ca/pub/Mirror/lsof/
> ftp://lsof.itap.purdue.edu/pub/tools/unix/lsof/ 
> ftp://the.wiretapped.net/pub/security/host-security/lsof/
> http://www.apache.org/dist  http://archive.apache.org/dist
> +ftp://.*/.*     http://mirrors.openembedded.org/
> +https?$://.*/.* http://mirrors.openembedded.org/
> +ftp://.*/.*     http://sources.openembedded.org/
> +https?$://.*/.* http://sources.openembedded.org/
>
>  }

Re: [PATCH] base.bbclass: introduce OE source mirror network as last resort for do_fetch

[Chris Larson]

On Fri, Aug 14, 2009 at 7:29 AM, Rolf
Leggewie<oe-devel@rolf.leggewie.biz> wrote:
> Updating recipes and checksums.ini for vanishing upstream hosts or simple
> restructuring can quickly become a chore for OE devs if taken seriously.
> The OE user experience suffers severely when at any given point in time 4
> to 5 recipes on average fail do_fetch for a simple image.
>
> The solution is for OE to provide its own caching of upstream sources.
> mirrors.openembedded.org is a swarm of possibly incomplete source mirrors
> provided by the community and load-balanced via DNS round robin.  They
> act as a first line of defense and lower bandwidth load on melo which
> hosts sources.openembedded.org.  The melo mirror aims to have a complete
> copy of free-to-publish sources used by the current org.oe.dev branch.
>
> Some distributions have provided source mirrors in the past for their
> users. I think it's time for a general and all-encompassing solution.
>
> The infrastructure is being currently put in place.  This patch will
> turn on the OE sources mirror network by default.

Looks like a good idea to me.  404s cause a great deal of difficulty
for the userbase.

Acked-by: Chris Larson <clarson@kergoth.com>
-- 
Chris Larson
clarson at kergoth dot com
clarson at mvista dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Software Engineer
MontaVista Software, Inc.

Re: [PATCH] base.bbclass: introduce OE source mirror network as last resort for do_fetch

[Martyn Welch]

Chris Larson wrote:
> On Fri, Aug 14, 2009 at 7:29 AM, Rolf
> Leggewie<oe-devel@rolf.leggewie.biz> wrote:
>   
>> Updating recipes and checksums.ini for vanishing upstream hosts or simple
>> restructuring can quickly become a chore for OE devs if taken seriously.
>> The OE user experience suffers severely when at any given point in time 4
>> to 5 recipes on average fail do_fetch for a simple image.
>>
>> The solution is for OE to provide its own caching of upstream sources.
>> mirrors.openembedded.org is a swarm of possibly incomplete source mirrors
>> provided by the community and load-balanced via DNS round robin.  They
>> act as a first line of defense and lower bandwidth load on melo which
>> hosts sources.openembedded.org.  The melo mirror aims to have a complete
>> copy of free-to-publish sources used by the current org.oe.dev branch.
>>
>> Some distributions have provided source mirrors in the past for their
>> users. I think it's time for a general and all-encompassing solution.
>>
>> The infrastructure is being currently put in place.  This patch will
>> turn on the OE sources mirror network by default.
>>     
>
> Looks like a good idea to me.  404s cause a great deal of difficulty
> for the userbase
On that note we just ran a "quick" test on OE, basically:

find openembedded -name "*.bb" -exec bitbake -c fetch -b {} \;

The following files seem to be pulling "index.html" files instead of the 
intended packages, I assume because the file/project account has 
moved/vanished:

recipes/balsa/balsa_2.0.17.bb
recipes/zd1211/zd1211_r83.bb
recipes/zd1211/zd1211_r85.bb
recipes/zd1211/zd1211_r77.bb
recipes/mtools/mtools_3.9.11.bb
recipes/mtools/mtools-native_3.9.11.bb
recipes/qxmp/qxmp_1.0.bb
recipes/murasaki/murasaki_0.8.8.bb
recipes/uim/uim-native_1.3.1.bb
recipes/uim/uim_1.3.1.bb
recipes/tmsnc/tmsnc_0.3.2.bb
recipes/ttf-fonts/ttf-sazanami_20040629.bb
recipes/qpe-gaim/qpe-libgaim_1.5.0.bb

I haven't had a chance to get any further.

Martyn

-- 
Martyn Welch MEng MPhil MIET (Principal Software Engineer)   T:+44(0)1327322748
GE Fanuc Intelligent Platforms Ltd,        |Registered in England and Wales
Tove Valley Business Park, Towcester,      |(3828642) at 100 Barbirolli Square,
Northants, NN12 6PF, UK T:+44(0)1327359444 |Manchester,M2 3AB  VAT:GB 927559189

Re: [PATCH] base.bbclass: introduce OE source mirror network as last resort for do_fetch

[Rolf Leggewie]

Martyn Welch wrote:
> On that note we just ran a "quick" test on OE

Thanks for testing.  Did you test this against the stable or the dev branch?

Re: [PATCH] base.bbclass: introduce OE source mirror network as last resort for do_fetch

[Martyn Welch]

Rolf Leggewie wrote:
> Martyn Welch wrote:
>   
>> On that note we just ran a "quick" test on OE
>>     
>
> Thanks for testing.  Did you test this against the stable or the dev branch?
>
>   
I believe it was the stable branch,

Martyn

-- 
Martyn Welch MEng MPhil MIET (Principal Software Engineer)   T:+44(0)1327322748
GE Fanuc Intelligent Platforms Ltd,        |Registered in England and Wales
Tove Valley Business Park, Towcester,      |(3828642) at 100 Barbirolli Square,
Northants, NN12 6PF, UK T:+44(0)1327359444 |Manchester,M2 3AB  VAT:GB 927559189

Re: [PATCH] base.bbclass: introduce OE source mirror network as last resort for do_fetch

[Leon Woestenberg]

Hello,

On Wed, Aug 19, 2009 at 3:11 PM, Martyn Welch<martyn.welch@gefanuc.com> wrote:
> Rolf Leggewie wrote:
>> Martyn Welch wrote:
>>> On that note we just ran a "quick" test on OE
>> Thanks for testing.  Did you test this against the stable or the dev
>> branch?
> I believe it was the stable branch,
>
Do you still have the disk usage numbers for the (almost) complete
downloads/, so that I can see into running a mirror here?

Also, the contrib/witpa/ package allows to Google for a package by
name and sha256 and md5sum. I use it for the really hard to find
packages in case of 404.


Regards,
-- 
Leon