[RFC] DISTRO_MIRRORS / DISTRO_PREMIRRORS

[RFC] DISTRO_MIRRORS / DISTRO_PREMIRRORS

[Saul Wold]

Folks,

Recently we have seen a number of fetch failures, due to upstream sites 
being down or deprecated (tinylogin, pscmic and others).

The issue is that oe-core as a non-distro does not have a default set of 
mirrors, which causes the fetcher to fail as there is no fall-back 
mirror to distrubte the source code.

As has been pointed out, different projects and distributions (Yocto 
Project and Angstrom) both have upstream source mirrors which could be 
used. Currently the Yocto Project defines it's MIRRORS and PREMIRRORS in 
meta-yocto/conf/distro/poky.conf.

I propose that we add a list of DISTO_MIRRORS and DISTRO_PREMIRRORS with 
default settings pointing to the current autobuilder.yoctoproject.org 
source site.  These would be added to the mirrors.bbclass, so that 
oe-core will have a default set of mirrors that can be overridden by a 
distro or layer.

diff --git a/meta/classes/mirrors.bbclass b/meta/classes/mirrors.bbclass
index 2146087..a52c6d5 100644
--- a/meta/classes/mirrors.bbclass
+++ b/meta/classes/mirrors.bbclass
@@ -1,4 +1,22 @@
+DISTRO_PREMIRRORS ?= "\
+bzr://.*/.*   http://autobuilder.yoctoproject.org/sources/ \n \
+cvs://.*/.*   http://autobuilder.yoctoproject.org/sources/ \n \
+git://.*/.*   http://autobuilder.yoctoproject.org/sources/ \n \
+hg://.*/.*    http://autobuilder.yoctoproject.org/sources/ \n \
+osc://.*/.*   http://autobuilder.yoctoproject.org/sources/ \n \
+p4://.*/.*    http://autobuilder.yoctoproject.org/sources/ \n \
+svk://.*/.*   http://autobuilder.yoctoproject.org/sources/ \n \
+svn://.*/.*   http://autobuilder.yoctoproject.org/sources/ \n"
+
+PREMIRRORS += "${DISTRO_PREMIRRORS}"
+
+DISTRO_MIRRORS ?= "\
+ftp://.*/.*      http://autobuilder.yoctoproject.org/sources/ \n \
+http://.*/.*     http://autobuilder.yoctoproject.org/sources/ \n \
+https://.*/.*    http://autobuilder.yoctoproject.org/sources/ \n"
+
  MIRRORS += "\
+${DISTRO_MIRRORS} \
  ${DEBIAN_MIRROR}/main  http://snapshot.debian.net/archive/pool \n \
  ${DEBIAN_MIRROR}       ftp://ftp.de.debian.org/debian/pool \n \
  ${DEBIAN_MIRROR}       ftp://ftp.au.debian.org/debian/pool \n \

Thoughts, ACKs, NAKs??


--
     Sau!

Saul Wold
Yocto Component Wrangler @ Intel
Yocto Project / Poky Build System

Re: [RFC] DISTRO_MIRRORS / DISTRO_PREMIRRORS

[Mark Hatle]

On 6/15/11 11:05 AM, Saul Wold wrote:
> 
> Folks,
> 
> Recently we have seen a number of fetch failures, due to upstream sites 
> being down or deprecated (tinylogin, pscmic and others).
> 
> The issue is that oe-core as a non-distro does not have a default set of 
> mirrors, which causes the fetcher to fail as there is no fall-back 
> mirror to distrubte the source code.
> 
> As has been pointed out, different projects and distributions (Yocto 
> Project and Angstrom) both have upstream source mirrors which could be 
> used. Currently the Yocto Project defines it's MIRRORS and PREMIRRORS in 
> meta-yocto/conf/distro/poky.conf.
> 
> I propose that we add a list of DISTO_MIRRORS and DISTRO_PREMIRRORS with 
> default settings pointing to the current autobuilder.yoctoproject.org 
> source site.  These would be added to the mirrors.bbclass, so that 
> oe-core will have a default set of mirrors that can be overridden by a 
> distro or layer.
> 

...

> 
> Thoughts, ACKs, NAKs??

The only concern I have is that we may miss when an upstream source location
changes, thus not fixing a recipe when a package moves, disappears and otherwise
can't be downloaded.

--Mark

> --
>      Sau!
> 
> Saul Wold
> Yocto Component Wrangler @ Intel
> Yocto Project / Poky Build System
> 
> 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: [RFC] DISTRO_MIRRORS / DISTRO_PREMIRRORS

[Khem Raj]

On 06/15/2011 09:11 AM, Mark Hatle wrote:
> On 6/15/11 11:05 AM, Saul Wold wrote:
>>
>> Folks,
>>
>> Recently we have seen a number of fetch failures, due to upstream sites
>> being down or deprecated (tinylogin, pscmic and others).
>>
>> The issue is that oe-core as a non-distro does not have a default set of
>> mirrors, which causes the fetcher to fail as there is no fall-back
>> mirror to distrubte the source code.
>>
>> As has been pointed out, different projects and distributions (Yocto
>> Project and Angstrom) both have upstream source mirrors which could be
>> used. Currently the Yocto Project defines it's MIRRORS and PREMIRRORS in
>> meta-yocto/conf/distro/poky.conf.
>>
>> I propose that we add a list of DISTO_MIRRORS and DISTRO_PREMIRRORS with
>> default settings pointing to the current autobuilder.yoctoproject.org
>> source site.  These would be added to the mirrors.bbclass, so that
>> oe-core will have a default set of mirrors that can be overridden by a
>> distro or layer.
>>
>
> ...
>
>>
>> Thoughts, ACKs, NAKs??
>
> The only concern I have is that we may miss when an upstream source location
> changes, thus not fixing a recipe when a package moves, disappears and otherwise
> can't be downloaded.
>

yeah thats why mirror.openembedded.org usually contains the sources that 
are not found anywhere.


> --Mark
>
>> --
>>       Sau!
>>
>> Saul Wold
>> Yocto Component Wrangler @ Intel
>> Yocto Project / Poky Build System
>>
>>
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: [RFC] DISTRO_MIRRORS / DISTRO_PREMIRRORS

[Khem Raj]

On 06/15/2011 09:05 AM, Saul Wold wrote:
>
> Folks,
>
> Recently we have seen a number of fetch failures, due to upstream sites
> being down or deprecated (tinylogin, pscmic and others).
>
> The issue is that oe-core as a non-distro does not have a default set of
> mirrors, which causes the fetcher to fail as there is no fall-back
> mirror to distrubte the source code.
>
> As has been pointed out, different projects and distributions (Yocto
> Project and Angstrom) both have upstream source mirrors which could be
> used. Currently the Yocto Project defines it's MIRRORS and PREMIRRORS in
> meta-yocto/conf/distro/poky.conf.
>
> I propose that we add a list of DISTO_MIRRORS and DISTRO_PREMIRRORS with
> default settings pointing to the current autobuilder.yoctoproject.org
> source site. These would be added to the mirrors.bbclass, so that
> oe-core will have a default set of mirrors that can be overridden by a
> distro or layer.
>
> diff --git a/meta/classes/mirrors.bbclass b/meta/classes/mirrors.bbclass
> index 2146087..a52c6d5 100644
> --- a/meta/classes/mirrors.bbclass
> +++ b/meta/classes/mirrors.bbclass
> @@ -1,4 +1,22 @@
> +DISTRO_PREMIRRORS ?= "\
> +bzr://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
> +cvs://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
> +git://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
> +hg://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
> +osc://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
> +p4://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
> +svk://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
> +svn://.*/.* http://autobuilder.yoctoproject.org/sources/ \n"
> +
> +PREMIRRORS += "${DISTRO_PREMIRRORS}"
> +
> +DISTRO_MIRRORS ?= "\
> +ftp://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
> +http://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
> +https://.*/.* http://autobuilder.yoctoproject.org/sources/ \n"
> +
> MIRRORS += "\
> +${DISTRO_MIRRORS} \
> ${DEBIAN_MIRROR}/main http://snapshot.debian.net/archive/pool \n \
> ${DEBIAN_MIRROR} ftp://ftp.de.debian.org/debian/pool \n \
> ${DEBIAN_MIRROR} ftp://ftp.au.debian.org/debian/pool \n \
>
> Thoughts, ACKs, NAKs??
>

FWIW http://mirrors.openembedded.org/ also exists.
>
> --
> Sau!
>
> Saul Wold
> Yocto Component Wrangler @ Intel
> Yocto Project / Poky Build System
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: [RFC] DISTRO_MIRRORS / DISTRO_PREMIRRORS

[Saul Wold]

On 06/15/2011 09:13 AM, Khem Raj wrote:
> On 06/15/2011 09:05 AM, Saul Wold wrote:
>>
>> Folks,
>>
>> Recently we have seen a number of fetch failures, due to upstream sites
>> being down or deprecated (tinylogin, pscmic and others).
>>
>> The issue is that oe-core as a non-distro does not have a default set of
>> mirrors, which causes the fetcher to fail as there is no fall-back
>> mirror to distrubte the source code.
>>
>> As has been pointed out, different projects and distributions (Yocto
>> Project and Angstrom) both have upstream source mirrors which could be
>> used. Currently the Yocto Project defines it's MIRRORS and PREMIRRORS in
>> meta-yocto/conf/distro/poky.conf.
>>
>> I propose that we add a list of DISTO_MIRRORS and DISTRO_PREMIRRORS with
>> default settings pointing to the current autobuilder.yoctoproject.org
>> source site. These would be added to the mirrors.bbclass, so that
>> oe-core will have a default set of mirrors that can be overridden by a
>> distro or layer.
>>
>> diff --git a/meta/classes/mirrors.bbclass b/meta/classes/mirrors.bbclass
>> index 2146087..a52c6d5 100644
>> --- a/meta/classes/mirrors.bbclass
>> +++ b/meta/classes/mirrors.bbclass
>> @@ -1,4 +1,22 @@
>> +DISTRO_PREMIRRORS ?= "\
>> +bzr://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
>> +cvs://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
>> +git://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
>> +hg://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
>> +osc://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
>> +p4://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
>> +svk://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
>> +svn://.*/.* http://autobuilder.yoctoproject.org/sources/ \n"
>> +
>> +PREMIRRORS += "${DISTRO_PREMIRRORS}"
>> +
>> +DISTRO_MIRRORS ?= "\
>> +ftp://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
>> +http://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
>> +https://.*/.* http://autobuilder.yoctoproject.org/sources/ \n"
>> +
>> MIRRORS += "\
>> +${DISTRO_MIRRORS} \
>> ${DEBIAN_MIRROR}/main http://snapshot.debian.net/archive/pool \n \
>> ${DEBIAN_MIRROR} ftp://ftp.de.debian.org/debian/pool \n \
>> ${DEBIAN_MIRROR} ftp://ftp.au.debian.org/debian/pool \n \
>>
>> Thoughts, ACKs, NAKs??
>>
>
> FWIW http://mirrors.openembedded.org/ also exists.
>>
How up to date is it and how is it maintained? Is it possible for this 
mirror site to fall behind oe-core?

Sau!

>> --
>> Sau!
>>
>> Saul Wold
>> Yocto Component Wrangler @ Intel
>> Yocto Project / Poky Build System
>>
>>
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>

Re: [RFC] DISTRO_MIRRORS / DISTRO_PREMIRRORS

[Khem Raj]

On 06/15/2011 09:26 AM, Saul Wold wrote:
> On 06/15/2011 09:13 AM, Khem Raj wrote:
>> On 06/15/2011 09:05 AM, Saul Wold wrote:
>>>
>>> Folks,
>>>
>>> Recently we have seen a number of fetch failures, due to upstream sites
>>> being down or deprecated (tinylogin, pscmic and others).
>>>
>>> The issue is that oe-core as a non-distro does not have a default set of
>>> mirrors, which causes the fetcher to fail as there is no fall-back
>>> mirror to distrubte the source code.
>>>
>>> As has been pointed out, different projects and distributions (Yocto
>>> Project and Angstrom) both have upstream source mirrors which could be
>>> used. Currently the Yocto Project defines it's MIRRORS and PREMIRRORS in
>>> meta-yocto/conf/distro/poky.conf.
>>>
>>> I propose that we add a list of DISTO_MIRRORS and DISTRO_PREMIRRORS with
>>> default settings pointing to the current autobuilder.yoctoproject.org
>>> source site. These would be added to the mirrors.bbclass, so that
>>> oe-core will have a default set of mirrors that can be overridden by a
>>> distro or layer.
>>>
>>> diff --git a/meta/classes/mirrors.bbclass b/meta/classes/mirrors.bbclass
>>> index 2146087..a52c6d5 100644
>>> --- a/meta/classes/mirrors.bbclass
>>> +++ b/meta/classes/mirrors.bbclass
>>> @@ -1,4 +1,22 @@
>>> +DISTRO_PREMIRRORS ?= "\
>>> +bzr://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
>>> +cvs://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
>>> +git://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
>>> +hg://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
>>> +osc://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
>>> +p4://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
>>> +svk://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
>>> +svn://.*/.* http://autobuilder.yoctoproject.org/sources/ \n"
>>> +
>>> +PREMIRRORS += "${DISTRO_PREMIRRORS}"
>>> +
>>> +DISTRO_MIRRORS ?= "\
>>> +ftp://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
>>> +http://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
>>> +https://.*/.* http://autobuilder.yoctoproject.org/sources/ \n"
>>> +
>>> MIRRORS += "\
>>> +${DISTRO_MIRRORS} \
>>> ${DEBIAN_MIRROR}/main http://snapshot.debian.net/archive/pool \n \
>>> ${DEBIAN_MIRROR} ftp://ftp.de.debian.org/debian/pool \n \
>>> ${DEBIAN_MIRROR} ftp://ftp.au.debian.org/debian/pool \n \
>>>
>>> Thoughts, ACKs, NAKs??
>>>
>>
>> FWIW http://mirrors.openembedded.org/ also exists.
>>>
> How up to date is it and how is it maintained? Is it possible for this
> mirror site to fall behind oe-core?
>

oe.dev falls back to it by default and whenever there is a report of non 
existing tars they are put up there.

> Sau!
>
>>> --
>>> Sau!
>>>
>>> Saul Wold
>>> Yocto Component Wrangler @ Intel
>>> Yocto Project / Poky Build System
>>>
>>>
>>> _______________________________________________
>>> Openembedded-core mailing list
>>> Openembedded-core@lists.openembedded.org
>>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>>
>>
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>>

Re: [RFC] DISTRO_MIRRORS / DISTRO_PREMIRRORS

[Tom Rini]

On 06/15/2011 09:13 AM, Khem Raj wrote:
> On 06/15/2011 09:05 AM, Saul Wold wrote:
>>
>> Folks,
>>
>> Recently we have seen a number of fetch failures, due to upstream sites
>> being down or deprecated (tinylogin, pscmic and others).
>>
>> The issue is that oe-core as a non-distro does not have a default set of
>> mirrors, which causes the fetcher to fail as there is no fall-back
>> mirror to distrubte the source code.
>>
>> As has been pointed out, different projects and distributions (Yocto
>> Project and Angstrom) both have upstream source mirrors which could be
>> used. Currently the Yocto Project defines it's MIRRORS and PREMIRRORS in
>> meta-yocto/conf/distro/poky.conf.
>>
>> I propose that we add a list of DISTO_MIRRORS and DISTRO_PREMIRRORS with
>> default settings pointing to the current autobuilder.yoctoproject.org
>> source site. These would be added to the mirrors.bbclass, so that
>> oe-core will have a default set of mirrors that can be overridden by a
>> distro or layer.
>>
>> diff --git a/meta/classes/mirrors.bbclass b/meta/classes/mirrors.bbclass
>> index 2146087..a52c6d5 100644
>> --- a/meta/classes/mirrors.bbclass
>> +++ b/meta/classes/mirrors.bbclass
>> @@ -1,4 +1,22 @@
>> +DISTRO_PREMIRRORS ?= "\
>> +bzr://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
>> +cvs://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
>> +git://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
>> +hg://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
>> +osc://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
>> +p4://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
>> +svk://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
>> +svn://.*/.* http://autobuilder.yoctoproject.org/sources/ \n"
>> +
>> +PREMIRRORS += "${DISTRO_PREMIRRORS}"
>> +
>> +DISTRO_MIRRORS ?= "\
>> +ftp://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
>> +http://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
>> +https://.*/.* http://autobuilder.yoctoproject.org/sources/ \n"
>> +
>> MIRRORS += "\
>> +${DISTRO_MIRRORS} \
>> ${DEBIAN_MIRROR}/main http://snapshot.debian.net/archive/pool \n \
>> ${DEBIAN_MIRROR} ftp://ftp.de.debian.org/debian/pool \n \
>> ${DEBIAN_MIRROR} ftp://ftp.au.debian.org/debian/pool \n \
>>
>> Thoughts, ACKs, NAKs??
>>
> 
> FWIW http://mirrors.openembedded.org/ also exists.

To be clear, mirrors.openembedded.org is a "last resort" type of mirror
(eg the tinylogin case) whereas autobuilder.yoctoproject.org and the
angstrom mirrors are complete (modulo fetch2 git issues, on the angstrom
side) mirrors.

Both have their uses and I think for oe-core we want just the former (so
that when tzcode/tzdata get their weekly upgrade, we know) while poky,
angstrom, etc, etc, will continue to want what they have now.

-- 
Tom Rini
Mentor Graphics Corporation