PREMIRRORS in yocto, do they actually work?

PREMIRRORS in yocto, do they actually work?

[Koen Kooi]

Hi,

I would like to have the fetchter try my PREMIRROS first before trying to fetch from git. In OE I have the following in a bbclass:

PREMIRRORS_append () {
git://.*/.* http://www.angstrom-distribution.org/unstable/sources/
}

When used in OE that works, but in yocto it doesn't. So I tried to do it poky style in a .inc:

PREMIRRORS_append = "\
git://.*/.*   http:/www.angstrom-distribution.org/sources/ \n"

Which also doesn't work since it doesn't try to fetch from there, but it does mention "http:/www.angstrom-distribution.org/sources/" in the fetch error message when git read-tree fails.

So my question: how do I make it work in yocto like it works in OE?

regards,

Koen

Re: PREMIRRORS in yocto, do they actually work?

[Gary Thomas]

On 01/02/2011 10:08 AM, Koen Kooi wrote:
> Hi,
>
> I would like to have the fetchter try my PREMIRROS first before trying to fetch from git. In OE I have the following in a bbclass:
>
> PREMIRRORS_append () {
> git://.*/.* http://www.angstrom-distribution.org/unstable/sources/
> }
>
> When used in OE that works, but in yocto it doesn't. So I tried to do it poky style in a .inc:
>
> PREMIRRORS_append = "\
> git://.*/.*   http:/www.angstrom-distribution.org/sources/ \n"
>
> Which also doesn't work since it doesn't try to fetch from there, but it does mention "http:/www.angstrom-distribution.org/sources/" in the fetch error message when git read-tree fails.
>
> So my question: how do I make it work in yocto like it works in OE?

I too have been struggling with this as I'd like to have
a local repository to hold the current source set.  Sadly,
as you've observed, it doesn't work, at least not as expected.

What error(s) do you get when a GIT based package needs to
be fetched?

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: PREMIRRORS in yocto, do they actually work?

[Koen Kooi]

Op 2 jan 2011, om 18:08 heeft Koen Kooi het volgende geschreven:

> Hi,
> 
> I would like to have the fetchter try my PREMIRROS first before trying to fetch from git. In OE I have the following in a bbclass:
> 
> PREMIRRORS_append () {
> git://.*/.* http://www.angstrom-distribution.org/unstable/sources/
> }
> 
> When used in OE that works, but in yocto it doesn't. So I tried to do it poky style in a .inc:
> 
> PREMIRRORS_append = "\
> git://.*/.*   http:/www.angstrom-distribution.org/sources/ \n"
> 
> Which also doesn't work since it doesn't try to fetch from there, but it does mention "http:/www.angstrom-distribution.org/sources/" in the fetch error message when git read-tree fails.

And the actual error message:

DEBUG: Parsing /OE/tentacle/sources/angstrom-layers/BSP/TI/recipes-bsp/x-load/x-load_git.bb (full)
DEBUG: Executing task do_fetch
NOTE: package x-load-1_1.44+r16+gitr1c9276af4d6a5b7014a7630a1abeddf3b3177563-r16: task do_fetch: Started
DEBUG: Running export HOME="/home/koen"; export GIT_CONFIG="/OE/tentacle/build/tmp-angstrom_2010_x/sysroots/x86_64-linux/usr/etc/gitconfig"; export PATH="/OE/tentacle/build/tmp-angstrom_2010_x/sysroots/x86_64-linux/usr/bin/armv7a-angstrom-linux-gnueabi:/OE/tentacle/build/tmp-angstrom_2010_x/sysroots/armv7a-angstrom-linux-gnueabi/usr/bin/crossscripts:/OE/tentacle/build/tmp-angstrom_2010_x/sysroots/x86_64-linux/usr/sbin:/OE/tentacle/build/tmp-angstrom_2010_x/sysroots/x86_64-linux/usr/bin:/OE/tentacle/build/tmp-angstrom_2010_x/sysroots/x86_64-linux/sbin:/OE/tentacle/build/tmp-angstrom_2010_x/sysroots/x86_64-linux//bin:/OE/tentacle/sources/openembedded/scripts:/OE/tentacle/sources/bitbake/bin:/OE/tentacle/sources/bitbake/bin:/OE/bitbake/bin:/usr/local/bin:/usr/bin:/bin:/usr/games:/usr/X11/bin:/usr/local/bin:/usr/local/enlightenment/bin:/usr/local/j2re1.4.1/bin:/usr/X11/bin:/usr/local/bin:/usr/local/enlightenment/bin:/usr/local/j2re1.4.1/bin:/OE/tentacle/sources/openembedded/scripts"; git log --pretty=oneline -n 1 1c9276af4d6a5b7014a7630a1abeddf3b3177563 -- 2> /dev/null | wc -l
ERROR: Task failed: Unknown fetch Error: http:/www.angstrom-distribution.org/sources/
NOTE: package x-load-1_1.44+r16+gitr1c9276af4d6a5b7014a7630a1abeddf3b3177563-r16: task do_fetch: Failed

I wiped before the build, DL_DIR, so the 'git log' is always going to fail.

regards,

Koen

Re: PREMIRRORS in yocto, do they actually work?

[Koen Kooi]

Op 2 jan 2011, om 18:53 heeft Esben Haabendal het volgende geschreven:

> On Sun, 2011-01-02 at 18:08 +0100, Koen Kooi wrote:
>> Hi,
>> 
>> I would like to have the fetchter try my PREMIRROS first before trying to fetch from git. In OE I have the following in a bbclass:
>> 
>> PREMIRRORS_append () {
>> git://.*/.* http://www.angstrom-distribution.org/unstable/sources/
>> }
>> 
>> When used in OE that works, but in yocto it doesn't. So I tried to do it poky style in a .inc:
>> 
>> PREMIRRORS_append = "\
>> git://.*/.*   http:/www.angstrom-distribution.org/sources/ \n"
>> 
>> Which also doesn't work since it doesn't try to fetch from there, but it does mention "http:/www.angstrom-distribution.org/sources/" in the fetch error message when git read-tree fails.
>> 
>> So my question: how do I make it work in yocto like it works in OE?
> 
> What about
> 
> PREMIRRORS_append = "\
> git://.*/.*   http:/www.angstrom-distribution.org/sources/\n"
> 
> ie. without the space before \n
> 
> I have no idea if it makes sense, but might be worth a try.

Same thing:

DEBUG: Executing task do_fetch
NOTE: package x-load-1_1.44+r17+gitr6f3a26101303051e0f91b6213735b68ce804e94e-r17: task do_fetch: Started
DEBUG: Running export HOME="/home/koen"; export GIT_CONFIG="/OE/tentacle/build/tmp-angstrom_2010_x/sysroots/x86_64-linux/usr/etc/gitconfig"; export PATH="/OE/tentacle/build/tmp-angstrom_2010_x/sysroots/x86_64-linux/usr/bin/armv7a-angstrom-linux-gnueabi:/OE/tentacle/build/tmp-angstrom_2010_x/sysroots/armv7a-angstrom-linux-gnueabi/usr/bin/crossscripts:/OE/tentacle/build/tmp-angstrom_2010_x/sysroots/x86_64-linux/usr/sbin:/OE/tentacle/build/tmp-angstrom_2010_x/sysroots/x86_64-linux/usr/bin:/OE/tentacle/build/tmp-angstrom_2010_x/sysroots/x86_64-linux/sbin:/OE/tentacle/build/tmp-angstrom_2010_x/sysroots/x86_64-linux//bin:/OE/tentacle/sources/openembedded/scripts:/OE/tentacle/sources/bitbake/bin:/OE/tentacle/sources/bitbake/bin:/OE/bitbake/bin:/usr/local/bin:/usr/bin:/bin:/usr/games:/usr/X11/bin:/usr/local/bin:/usr/local/enlightenment/bin:/usr/local/j2re1.4.1/bin:/usr/X11/bin:/usr/local/bin:/usr/local/enlightenment/bin:/usr/local/j2re1.4.1/bin:/OE/tentacle/sources/openembedded/scripts"; git log --pretty=oneline -n 1 6f3a26101303051e0f91b6213735b68ce804e94e -- 2> /dev/null | wc -l
ERROR: Task failed: Unknown fetch Error: http:/www.angstrom-distribution.org/sources/
NOTE: package x-load-1_1.44+r17+gitr6f3a26101303051e0f91b6213735b68ce804e94e-r17: task do_fetch: Failed

As a work around I fetch it using OE into a shared DL_DIR first and then try a yocto build.....

Re: PREMIRRORS in yocto, do they actually work?

[Yu Ke]

On Jan 02, 20:01, Koen Kooi wrote:
> 
> Op 2 jan 2011, om 18:53 heeft Esben Haabendal het volgende geschreven:
> 
> > On Sun, 2011-01-02 at 18:08 +0100, Koen Kooi wrote:
> >> Hi,
> >> 
> >> I would like to have the fetchter try my PREMIRROS first before trying to fetch from git. In OE I have the following in a bbclass:
> >> 
> >> PREMIRRORS_append () {
> >> git://.*/.* http://www.angstrom-distribution.org/unstable/sources/
> >> }
> >> 
> >> When used in OE that works, but in yocto it doesn't. So I tried to do it poky style in a .inc:
> >> 
> >> PREMIRRORS_append = "\
> >> git://.*/.*   http:/www.angstrom-distribution.org/sources/ \n"
> >> 
> >> Which also doesn't work since it doesn't try to fetch from there, but it does mention "http:/www.angstrom-distribution.org/sources/" in the fetch error message when git read-tree fails.
> >> 
> >> So my question: how do I make it work in yocto like it works in OE?
> > 
> > What about
> > 
> > PREMIRRORS_append = "\
> > git://.*/.*   http:/www.angstrom-distribution.org/sources/\n"
> > 
> > ie. without the space before \n
> > 
> > I have no idea if it makes sense, but might be worth a try.
> 
> Same thing:
> 
> DEBUG: Executing task do_fetch
> NOTE: package x-load-1_1.44+r17+gitr6f3a26101303051e0f91b6213735b68ce804e94e-r17: task do_fetch: Started
> DEBUG: Running export HOME="/home/koen"; export GIT_CONFIG="/OE/tentacle/build/tmp-angstrom_2010_x/sysroots/x86_64-linux/usr/etc/gitconfig"; export PATH="/OE/tentacle/build/tmp-angstrom_2010_x/sysroots/x86_64-linux/usr/bin/armv7a-angstrom-linux-gnueabi:/OE/tentacle/build/tmp-angstrom_2010_x/sysroots/armv7a-angstrom-linux-gnueabi/usr/bin/crossscripts:/OE/tentacle/build/tmp-angstrom_2010_x/sysroots/x86_64-linux/usr/sbin:/OE/tentacle/build/tmp-angstrom_2010_x/sysroots/x86_64-linux/usr/bin:/OE/tentacle/build/tmp-angstrom_2010_x/sysroots/x86_64-linux/sbin:/OE/tentacle/build/tmp-angstrom_2010_x/sysroots/x86_64-linux//bin:/OE/tentacle/sources/openembedded/scripts:/OE/tentacle/sources/bitbake/bin:/OE/tentacle/sources/bitbake/bin:/OE/bitbake/bin:/usr/local/bin:/usr/bin:/bin:/usr/games:/usr/X11/bin:/usr/local/bin:/usr/local/enlightenment/bin:/usr/local/j2re1.4.1/bin:/usr/X11/bin:/usr/local/bin:/usr/local/enlightenment/bin:/usr/local/j2re1.4.1/bin:/OE/tentacle/sources/openembedded/scrip
>  ts"; git log --pretty=oneline -n 1 6f3a26101303051e0f91b6213735b68ce804e94e -- 2> /dev/null | wc -l
> ERROR: Task failed: Unknown fetch Error: http:/www.angstrom-distribution.org/sources/
> NOTE: package x-load-1_1.44+r17+gitr6f3a26101303051e0f91b6213735b68ce804e94e-r17: task do_fetch: Failed

I try to reproduce the issue by:
1. add the following to meta/conf/distro/poky.conf

PREMIRRORS_append = "\
 git://.*/.*   http://www.angstrom-distribution.org/sources/\n"

Note: your original URL "http:/www.angstrom-distribution.org/sources/" miss a slash after the "http:/", which cause the "Unknown fetch Error".

2. bitbake -f -c fetch x-load
then the fetcher will try to fetch http://www.angstrom-distribution.org/sources/git_www.sakoman.net.git.x-load-omap3.git_65ed00323f3807197a83abc75d62ed2a8d3f60de.tar.gz, and failed, because this tar ball does not exist in this mirror.

so the could you add the corresponding tar ball to the web site, and try again?

Regards
Ke

> 
> As a work around I fetch it using OE into a shared DL_DIR first and then try a yocto build.....
> 
> _______________________________________________
> poky mailing list
> poky@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/poky

Re: PREMIRRORS in yocto, do they actually work?

[Yu Ke]

On Jan 02, 10:15, Gary Thomas wrote:
> On 01/02/2011 10:08 AM, Koen Kooi wrote:
> >Hi,
> >
> >I would like to have the fetchter try my PREMIRROS first before trying to fetch from git. In OE I have the following in a bbclass:
> >
> >PREMIRRORS_append () {
> >git://.*/.* http://www.angstrom-distribution.org/unstable/sources/
> >}
> >
> >When used in OE that works, but in yocto it doesn't. So I tried to do it poky style in a .inc:
> >
> >PREMIRRORS_append = "\
> >git://.*/.*   http:/www.angstrom-distribution.org/sources/ \n"
> >
> >Which also doesn't work since it doesn't try to fetch from there, but it does mention "http:/www.angstrom-distribution.org/sources/" in the fetch error message when git read-tree fails.
> >
> >So my question: how do I make it work in yocto like it works in OE?
> 
> I too have been struggling with this as I'd like to have
> a local repository to hold the current source set.  Sadly,
> as you've observed, it doesn't work, at least not as expected.

I've tried to use file:///some/dir as PREMIRROR, and found it does has issue. The main reason is that the local.py fetcher does not do real copy in go() method. Although it make sense in normal fetch case, it does cause trouble in try_mirror() case.

I don't have simple idea to address this issue yet and I will take care of it in the coming fetcher change. As a quick workaround, could you try this approach: use http://local/some/dir as PREMIRROR? e.g. if you use /opt/source as mirror, you can add to meta/conf/distro/poky.conf:
PREMIRRORS_append = "\
git://.*/.*   http://local/opt/source/ \n \
svn://.*/.*   http://local/opt/source/ \n \
...
"

in this case, wget.py fetcher will be used, and it works fine in try_mirror() case. Also the /opt/source can be read-only.

Regards
Ke

> 
> What error(s) do you get when a GIT based package needs to
> be fetched?
> 
> -- 
> ------------------------------------------------------------
> Gary Thomas                 |  Consulting for the
> MLB Associates              |    Embedded world
> ------------------------------------------------------------
> _______________________________________________
> poky mailing list
> poky@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/poky

Re: PREMIRRORS in yocto, do they actually work?

[Koen Kooi]

Op 6 jan 2011, om 08:00 heeft Yu Ke het volgende geschreven:

> On Jan 02, 20:01, Koen Kooi wrote:
>> 
>> Op 2 jan 2011, om 18:53 heeft Esben Haabendal het volgende geschreven:
>> 
>>> On Sun, 2011-01-02 at 18:08 +0100, Koen Kooi wrote:
>>>> Hi,
>>>> 
>>>> I would like to have the fetchter try my PREMIRROS first before trying to fetch from git. In OE I have the following in a bbclass:
>>>> 
>>>> PREMIRRORS_append () {
>>>> git://.*/.* http://www.angstrom-distribution.org/unstable/sources/
>>>> }
>>>> 
>>>> When used in OE that works, but in yocto it doesn't. So I tried to do it poky style in a .inc:
>>>> 
>>>> PREMIRRORS_append = "\
>>>> git://.*/.*   http:/www.angstrom-distribution.org/sources/ \n"
>>>> 
>>>> Which also doesn't work since it doesn't try to fetch from there, but it does mention "http:/www.angstrom-distribution.org/sources/" in the fetch error message when git read-tree fails.
>>>> 
>>>> So my question: how do I make it work in yocto like it works in OE?
>>> 
>>> What about
>>> 
>>> PREMIRRORS_append = "\
>>> git://.*/.*   http:/www.angstrom-distribution.org/sources/\n"
>>> 
>>> ie. without the space before \n
>>> 
>>> I have no idea if it makes sense, but might be worth a try.
>> 
>> Same thing:
>> 
>> DEBUG: Executing task do_fetch
>> NOTE: package x-load-1_1.44+r17+gitr6f3a26101303051e0f91b6213735b68ce804e94e-r17: task do_fetch: Started
>> DEBUG: Running export HOME="/home/koen"; export GIT_CONFIG="/OE/tentacle/build/tmp-angstrom_2010_x/sysroots/x86_64-linux/usr/etc/gitconfig"; export PATH="/OE/tentacle/build/tmp-angstrom_2010_x/sysroots/x86_64-linux/usr/bin/armv7a-angstrom-linux-gnueabi:/OE/tentacle/build/tmp-angstrom_2010_x/sysroots/armv7a-angstrom-linux-gnueabi/usr/bin/crossscripts:/OE/tentacle/build/tmp-angstrom_2010_x/sysroots/x86_64-linux/usr/sbin:/OE/tentacle/build/tmp-angstrom_2010_x/sysroots/x86_64-linux/usr/bin:/OE/tentacle/build/tmp-angstrom_2010_x/sysroots/x86_64-linux/sbin:/OE/tentacle/build/tmp-angstrom_2010_x/sysroots/x86_64-linux//bin:/OE/tentacle/sources/openembedded/scripts:/OE/tentacle/sources/bitbake/bin:/OE/tentacle/sources/bitbake/bin:/OE/bitbake/bin:/usr/local/bin:/usr/bin:/bin:/usr/games:/usr/X11/bin:/usr/local/bin:/usr/local/enlightenment/bin:/usr/local/j2re1.4.1/bin:/usr/X11/bin:/usr/local/bin:/usr/local/enlightenment/bin:/usr/local/j2re1.4.1/bin:/OE/tentacle/sources/openembedded/scrip
>> ts"; git log --pretty=oneline -n 1 6f3a26101303051e0f91b6213735b68ce804e94e -- 2> /dev/null | wc -l
>> ERROR: Task failed: Unknown fetch Error: http:/www.angstrom-distribution.org/sources/
>> NOTE: package x-load-1_1.44+r17+gitr6f3a26101303051e0f91b6213735b68ce804e94e-r17: task do_fetch: Failed
> 
> I try to reproduce the issue by:
> 1. add the following to meta/conf/distro/poky.conf
> 
> PREMIRRORS_append = "\
> git://.*/.*   http://www.angstrom-distribution.org/sources/\n"
> 
> Note: your original URL "http:/www.angstrom-distribution.org/sources/" miss a slash after the "http:/", which cause the "Unknown fetch Error".
> 
> 2. bitbake -f -c fetch x-load
> then the fetcher will try to fetch http://www.angstrom-distribution.org/sources/git_www.sakoman.net.git.x-load-omap3.git_65ed00323f3807197a83abc75d62ed2a8d3f60de.tar.gz, and failed, because this tar ball does not exist in this mirror.
> 
> so the could you add the corresponding tar ball to the web site, and try again?

I can confirm the following works: http://gitorious.org/angstrom/angstrom-layers/commit/e78d521cf0f9310867b660df7743087efdc2642a

regards,

Koen

Re: PREMIRRORS in yocto, do they actually work?

[Frans Meulenbroeks]

2011/1/6 Yu Ke <ke.yu@intel.com>:
> On Jan 02, 10:15, Gary Thomas wrote:
>> On 01/02/2011 10:08 AM, Koen Kooi wrote:
>> >Hi,
>> >
>> >I would like to have the fetchter try my PREMIRROS first before trying to fetch from git. In OE I have the following in a bbclass:
>> >
>> >PREMIRRORS_append () {
>> >git://.*/.* http://www.angstrom-distribution.org/unstable/sources/
>> >}
>> >
>> >When used in OE that works, but in yocto it doesn't. So I tried to do it poky style in a .inc:
>> >
>> >PREMIRRORS_append = "\
>> >git://.*/.*   http:/www.angstrom-distribution.org/sources/ \n"
>> >
>> >Which also doesn't work since it doesn't try to fetch from there, but it does mention "http:/www.angstrom-distribution.org/sources/" in the fetch error message when git read-tree fails.
>> >
>> >So my question: how do I make it work in yocto like it works in OE?
>>
>> I too have been struggling with this as I'd like to have
>> a local repository to hold the current source set.  Sadly,
>> as you've observed, it doesn't work, at least not as expected.
>
> I've tried to use file:///some/dir as PREMIRROR, and found it does has issue. The main reason is that the local.py fetcher does not do real copy in go() method. Although it make sense in normal fetch case, it does cause trouble in try_mirror() case.
>
> I don't have simple idea to address this issue yet and I will take care of it in the coming fetcher change. As a quick workaround, could you try this approach: use http://local/some/dir as PREMIRROR? e.g. if you use /opt/source as mirror, you can add to meta/conf/distro/poky.conf:
> PREMIRRORS_append = "\
> git://.*/.*   http://local/opt/source/ \n \
> svn://.*/.*   http://local/opt/source/ \n \
> ...
> "
>
> in this case, wget.py fetcher will be used, and it works fine in try_mirror() case. Also the /opt/source can be read-only.
>
> Regards
> Ke

Thanks for the info Ke.
One additional question on this topic.
is it also possible to block fetching from the original source?
I'm looking for an easy, automated way to verify that all sources for
our products are indeed on our public server (to verify GPL
compliance).

Frans

Re: PREMIRRORS in yocto, do they actually work?

[Yu Ke]

On Jan 06, 08:18, Koen Kooi wrote:
> 
> Op 6 jan 2011, om 08:00 heeft Yu Ke het volgende geschreven:
> 
> > On Jan 02, 20:01, Koen Kooi wrote:
> >> 
> >> Op 2 jan 2011, om 18:53 heeft Esben Haabendal het volgende geschreven:
> >> 
> >>> On Sun, 2011-01-02 at 18:08 +0100, Koen Kooi wrote:
> >>>> Hi,
> >>>> 
> >>>> I would like to have the fetchter try my PREMIRROS first before trying to fetch from git. In OE I have the following in a bbclass:
> >>>> 
> >>>> PREMIRRORS_append () {
> >>>> git://.*/.* http://www.angstrom-distribution.org/unstable/sources/
> >>>> }
> >>>> 
> >>>> When used in OE that works, but in yocto it doesn't. So I tried to do it poky style in a .inc:
> >>>> 
> >>>> PREMIRRORS_append = "\
> >>>> git://.*/.*   http:/www.angstrom-distribution.org/sources/ \n"
> >>>> 
> >>>> Which also doesn't work since it doesn't try to fetch from there, but it does mention "http:/www.angstrom-distribution.org/sources/" in the fetch error message when git read-tree fails.
> >>>> 
> >>>> So my question: how do I make it work in yocto like it works in OE?
> >>> 
> >>> What about
> >>> 
> >>> PREMIRRORS_append = "\
> >>> git://.*/.*   http:/www.angstrom-distribution.org/sources/\n"
> >>> 
> >>> ie. without the space before \n
> >>> 
> >>> I have no idea if it makes sense, but might be worth a try.
> >> 
> >> Same thing:
> >> 
> >> DEBUG: Executing task do_fetch
> >> NOTE: package x-load-1_1.44+r17+gitr6f3a26101303051e0f91b6213735b68ce804e94e-r17: task do_fetch: Started
> >> DEBUG: Running export HOME="/home/koen"; export GIT_CONFIG="/OE/tentacle/build/tmp-angstrom_2010_x/sysroots/x86_64-linux/usr/etc/gitconfig"; export PATH="/OE/tentacle/build/tmp-angstrom_2010_x/sysroots/x86_64-linux/usr/bin/armv7a-angstrom-linux-gnueabi:/OE/tentacle/build/tmp-angstrom_2010_x/sysroots/armv7a-angstrom-linux-gnueabi/usr/bin/crossscripts:/OE/tentacle/build/tmp-angstrom_2010_x/sysroots/x86_64-linux/usr/sbin:/OE/tentacle/build/tmp-angstrom_2010_x/sysroots/x86_64-linux/usr/bin:/OE/tentacle/build/tmp-angstrom_2010_x/sysroots/x86_64-linux/sbin:/OE/tentacle/build/tmp-angstrom_2010_x/sysroots/x86_64-linux//bin:/OE/tentacle/sources/openembedded/scripts:/OE/tentacle/sources/bitbake/bin:/OE/tentacle/sources/bitbake/bin:/OE/bitbake/bin:/usr/local/bin:/usr/bin:/bin:/usr/games:/usr/X11/bin:/usr/local/bin:/usr/local/enlightenment/bin:/usr/local/j2re1.4.1/bin:/usr/X11/bin:/usr/local/bin:/usr/local/enlightenment/bin:/usr/local/j2re1.4.1/bin:/OE/tentacle/sources/openembedded/sc
>  rip
> >> ts"; git log --pretty=oneline -n 1 6f3a26101303051e0f91b6213735b68ce804e94e -- 2> /dev/null | wc -l
> >> ERROR: Task failed: Unknown fetch Error: http:/www.angstrom-distribution.org/sources/
> >> NOTE: package x-load-1_1.44+r17+gitr6f3a26101303051e0f91b6213735b68ce804e94e-r17: task do_fetch: Failed
> > 
> > I try to reproduce the issue by:
> > 1. add the following to meta/conf/distro/poky.conf
> > 
> > PREMIRRORS_append = "\
> > git://.*/.*   http://www.angstrom-distribution.org/sources/\n"
> > 
> > Note: your original URL "http:/www.angstrom-distribution.org/sources/" miss a slash after the "http:/", which cause the "Unknown fetch Error".
> > 
> > 2. bitbake -f -c fetch x-load
> > then the fetcher will try to fetch http://www.angstrom-distribution.org/sources/git_www.sakoman.net.git.x-load-omap3.git_65ed00323f3807197a83abc75d62ed2a8d3f60de.tar.gz, and failed, because this tar ball does not exist in this mirror.
> > 
> > so the could you add the corresponding tar ball to the web site, and try again?
> 
> I can confirm the following works: http://gitorious.org/angstrom/angstrom-layers/commit/e78d521cf0f9310867b660df7743087efdc2642a

Good to see that PREMIRRORS works in your side. one thing i am not quite sure is that the above commit use http://www.angstrom-distribution.org/unstable/sources/ as PREMIRRORS, but http://www.angstrom-distribution.org/unstable/sources/ only contains a webcam-tools svn repo, which looks not a valid mirror, so i am not sure how it works. or maybe i miss something here?

Regards
Ke

> 
> regards,
> 
> Koen
> 
> _______________________________________________
> poky mailing list
> poky@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/poky

Re: PREMIRRORS in yocto, do they actually work?

[Yu Ke]

On Jan 06, 08:39, Frans Meulenbroeks wrote:
> 2011/1/6 Yu Ke <ke.yu@intel.com>:
> > On Jan 02, 10:15, Gary Thomas wrote:
> >> On 01/02/2011 10:08 AM, Koen Kooi wrote:
> >> >Hi,
> >> >
> >> >I would like to have the fetchter try my PREMIRROS first before trying to fetch from git. In OE I have the following in a bbclass:
> >> >
> >> >PREMIRRORS_append () {
> >> >git://.*/.* http://www.angstrom-distribution.org/unstable/sources/
> >> >}
> >> >
> >> >When used in OE that works, but in yocto it doesn't. So I tried to do it poky style in a .inc:
> >> >
> >> >PREMIRRORS_append = "\
> >> >git://.*/.*   http:/www.angstrom-distribution.org/sources/ \n"
> >> >
> >> >Which also doesn't work since it doesn't try to fetch from there, but it does mention "http:/www.angstrom-distribution.org/sources/" in the fetch error message when git read-tree fails.
> >> >
> >> >So my question: how do I make it work in yocto like it works in OE?
> >>
> >> I too have been struggling with this as I'd like to have
> >> a local repository to hold the current source set.  Sadly,
> >> as you've observed, it doesn't work, at least not as expected.
> >
> > I've tried to use file:///some/dir as PREMIRROR, and found it does has issue. The main reason is that the local.py fetcher does not do real copy in go() method. Although it make sense in normal fetch case, it does cause trouble in try_mirror() case.
> >
> > I don't have simple idea to address this issue yet and I will take care of it in the coming fetcher change. As a quick workaround, could you try this approach: use http://local/some/dir as PREMIRROR? e.g. if you use /opt/source as mirror, you can add to meta/conf/distro/poky.conf:
> > PREMIRRORS_append = "\
> > git://.*/.*   http://local/opt/source/ \n \
> > svn://.*/.*   http://local/opt/source/ \n \
> > ...
> > "
> >
> > in this case, wget.py fetcher will be used, and it works fine in try_mirror() case. Also the /opt/source can be read-only.
> >
> > Regards
> > Ke
> 
> Thanks for the info Ke.
> One additional question on this topic.
> is it also possible to block fetching from the original source?
> I'm looking for an easy, automated way to verify that all sources for
> our products are indeed on our public server (to verify GPL
> compliance).

sorry taht I am not aware of such way, maybe other people has ideas.

If it is for one time verification, probably you can try the following patch. But I am not sure if it is desirable for upstream checked in.

Regards
Ke

diff --git a/bitbake/lib/bb/fetch/__init__.py b/bitbake/lib/bb/fetch/__init__.py
index 67e5add..eb053e1 100644
--- a/bitbake/lib/bb/fetch/__init__.py
+++ b/bitbake/lib/bb/fetch/__init__.py
@@ -286,6 +286,8 @@ def go(d, urls = None):
         if m.forcefetch(u, ud, d) or not localpath:
             # Next try fetching from the original uri, u
             try:
+                if bb.data.getVar("BB_FETCH_MIRROR_ONLY", d, True) == "1":
+                    raise FetchError("Only allow to fetch from mirror")
                 m.go(u, ud, d)
                 localpath = ud.localpath
             except FetchError:

> 
> Frans
> _______________________________________________
> poky mailing list
> poky@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/poky

Re: PREMIRRORS in yocto, do they actually work?

[Frans Meulenbroeks]

2011/1/6 Yu Ke <ke.yu@intel.com>:
> On Jan 06, 08:39, Frans Meulenbroeks wrote:
>> 2011/1/6 Yu Ke <ke.yu@intel.com>:
>> > On Jan 02, 10:15, Gary Thomas wrote:
>> >> On 01/02/2011 10:08 AM, Koen Kooi wrote:
>> >> >Hi,
>> >> >
>> >> >I would like to have the fetchter try my PREMIRROS first before trying to fetch from git. In OE I have the following in a bbclass:
>> >> >
>> >> >PREMIRRORS_append () {
>> >> >git://.*/.* http://www.angstrom-distribution.org/unstable/sources/
>> >> >}
>> >> >
>> >> >When used in OE that works, but in yocto it doesn't. So I tried to do it poky style in a .inc:
>> >> >
>> >> >PREMIRRORS_append = "\
>> >> >git://.*/.*   http:/www.angstrom-distribution.org/sources/ \n"
>> >> >
>> >> >Which also doesn't work since it doesn't try to fetch from there, but it does mention "http:/www.angstrom-distribution.org/sources/" in the fetch error message when git read-tree fails.
>> >> >
>> >> >So my question: how do I make it work in yocto like it works in OE?
>> >>
>> >> I too have been struggling with this as I'd like to have
>> >> a local repository to hold the current source set.  Sadly,
>> >> as you've observed, it doesn't work, at least not as expected.
>> >
>> > I've tried to use file:///some/dir as PREMIRROR, and found it does has issue. The main reason is that the local.py fetcher does not do real copy in go() method. Although it make sense in normal fetch case, it does cause trouble in try_mirror() case.
>> >
>> > I don't have simple idea to address this issue yet and I will take care of it in the coming fetcher change. As a quick workaround, could you try this approach: use http://local/some/dir as PREMIRROR? e.g. if you use /opt/source as mirror, you can add to meta/conf/distro/poky.conf:
>> > PREMIRRORS_append = "\
>> > git://.*/.*   http://local/opt/source/ \n \
>> > svn://.*/.*   http://local/opt/source/ \n \
>> > ...
>> > "
>> >
>> > in this case, wget.py fetcher will be used, and it works fine in try_mirror() case. Also the /opt/source can be read-only.
>> >
>> > Regards
>> > Ke
>>
>> Thanks for the info Ke.
>> One additional question on this topic.
>> is it also possible to block fetching from the original source?
>> I'm looking for an easy, automated way to verify that all sources for
>> our products are indeed on our public server (to verify GPL
>> compliance).
>
> sorry taht I am not aware of such way, maybe other people has ideas.
>
> If it is for one time verification, probably you can try the following patch. But I am not sure if it is desirable for upstream checked in.
>
> Regards
> Ke
>
> diff --git a/bitbake/lib/bb/fetch/__init__.py b/bitbake/lib/bb/fetch/__init__.py
> index 67e5add..eb053e1 100644
> --- a/bitbake/lib/bb/fetch/__init__.py
> +++ b/bitbake/lib/bb/fetch/__init__.py
> @@ -286,6 +286,8 @@ def go(d, urls = None):
>         if m.forcefetch(u, ud, d) or not localpath:
>             # Next try fetching from the original uri, u
>             try:
> +                if bb.data.getVar("BB_FETCH_MIRROR_ONLY", d, True) == "1":
> +                    raise FetchError("Only allow to fetch from mirror")
>                 m.go(u, ud, d)
>                 localpath = ud.localpath
>             except FetchError:
>

that looks like a nice patch to me.
Richard, how do you feel about such a patch ?

I can always of course do this in a local patch.

Thanks! Frans

Re: PREMIRRORS in yocto, do they actually work?

[Gary Thomas]

On 01/06/2011 12:14 AM, Yu Ke wrote:
> On Jan 02, 10:15, Gary Thomas wrote:
>> On 01/02/2011 10:08 AM, Koen Kooi wrote:
>>> Hi,
>>>
>>> I would like to have the fetchter try my PREMIRROS first before trying to fetch from git. In OE I have the following in a bbclass:
>>>
>>> PREMIRRORS_append () {
>>> git://.*/.* http://www.angstrom-distribution.org/unstable/sources/
>>> }
>>>
>>> When used in OE that works, but in yocto it doesn't. So I tried to do it poky style in a .inc:
>>>
>>> PREMIRRORS_append = "\
>>> git://.*/.*   http:/www.angstrom-distribution.org/sources/ \n"
>>>
>>> Which also doesn't work since it doesn't try to fetch from there, but it does mention "http:/www.angstrom-distribution.org/sources/" in the fetch error message when git read-tree fails.
>>>
>>> So my question: how do I make it work in yocto like it works in OE?
>>
>> I too have been struggling with this as I'd like to have
>> a local repository to hold the current source set.  Sadly,
>> as you've observed, it doesn't work, at least not as expected.
>
> I've tried to use file:///some/dir as PREMIRROR, and found it does has issue. The main reason is that the local.py fetcher does not do real copy in go() method. Although it make sense in normal fetch case, it does cause trouble in try_mirror() case.
>
> I don't have simple idea to address this issue yet and I will take care of it in the coming fetcher change. As a quick workaround, could you try this approach: use http://local/some/dir as PREMIRROR? e.g. if you use /opt/source as mirror, you can add to meta/conf/distro/poky.conf:
> PREMIRRORS_append = "\
> git://.*/.*   http://local/opt/source/ \n \
> svn://.*/.*   http://local/opt/source/ \n \
> ...
> "
>
> in this case, wget.py fetcher will be used, and it works fine in try_mirror() case. Also the /opt/source can be read-only.

Per my example, I set both PREMIRRORS and MIRRORS to use these paths.

This _works_ (in that it doesn't fail), but it doesn't always
seem to use my local MIRROR copy.  After an abreviated build from
scratch, I found that these files _were_ copied from my local
MIRROR:
   automake-1.11.1.tar.bz2
   m4-1.4.15.tar.gz
   autoconf-2.65.tar.bz2
   libtool-2.4.tar.gz
   ncurses-5.4.tar.gz
   gdbm-1.8.3.tar.gz
   gzip-1.4.tar.gz
   readline-6.1.tar.gz
   gmp-5.0.1.tar.bz2
   bison-2.4.3.tar.gz
   gettext-0.17.tar.gz
   binutils-2.20.1.tar.bz2

but these were [re]fetched over the network from their original
sources (my MIRROR does have local copies):
   db-4.2.52.tar.gz
   expat-2.0.1.tar.gz
   flex-2.5.35.tar.bz2
   perl-5.8.8.tar.gz
   quilt-0.48.tar.gz
   zlib-1.2.5.tar.bz2
   mpc-0.8.2.tar.gz
   pkg-config-0.25.tar.gz
   mpfr-3.0.0.tar.bz2
   elfutils-0.148.tar.bz2
   tcl8.5.9-src.tar.gz
   config_cvs.sv.gnu.org__20080123.tar.gz
   curl-7.21.2.tar.bz2
   linux-2.6.36.tar.bz2
   git-1.7.3.2.tar.bz2
   sqlite-3.7.3.tar.gz
   openssl-0.9.8p.tar.gz
   git_github.com.wrpseudo.pseudo.git_c9792c7acdb1cac90549ff08db12a8bf0c53cdcf.tar.gz

Not quite what I was hoping for.  I don't mind getting a
[temporary] copy of the files into ${BUILD}/downloads, but
I really need to find a way to avoid ever going to the internet
if the file is already available (which I plan to guarantee
for my customers)

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: PREMIRRORS in yocto, do they actually work?

[Yu Ke]

On Jan 06, 08:06, Gary Thomas wrote:
> On 01/06/2011 12:14 AM, Yu Ke wrote:
> >On Jan 02, 10:15, Gary Thomas wrote:
> >>On 01/02/2011 10:08 AM, Koen Kooi wrote:
> >>>Hi,
> >>>
> >>>I would like to have the fetchter try my PREMIRROS first before trying to fetch from git. In OE I have the following in a bbclass:
> >>>
> >>>PREMIRRORS_append () {
> >>>git://.*/.* http://www.angstrom-distribution.org/unstable/sources/
> >>>}
> >>>
> >>>When used in OE that works, but in yocto it doesn't. So I tried to do it poky style in a .inc:
> >>>
> >>>PREMIRRORS_append = "\
> >>>git://.*/.*   http:/www.angstrom-distribution.org/sources/ \n"
> >>>
> >>>Which also doesn't work since it doesn't try to fetch from there, but it does mention "http:/www.angstrom-distribution.org/sources/" in the fetch error message when git read-tree fails.
> >>>
> >>>So my question: how do I make it work in yocto like it works in OE?
> >>
> >>I too have been struggling with this as I'd like to have
> >>a local repository to hold the current source set.  Sadly,
> >>as you've observed, it doesn't work, at least not as expected.
> >
> >I've tried to use file:///some/dir as PREMIRROR, and found it does has issue. The main reason is that the local.py fetcher does not do real copy in go() method. Although it make sense in normal fetch case, it does cause trouble in try_mirror() case.
> >
> >I don't have simple idea to address this issue yet and I will take care of it in the coming fetcher change. As a quick workaround, could you try this approach: use http://local/some/dir as PREMIRROR? e.g. if you use /opt/source as mirror, you can add to meta/conf/distro/poky.conf:
> >PREMIRRORS_append = "\
> >git://.*/.*   http://local/opt/source/ \n \
> >svn://.*/.*   http://local/opt/source/ \n \
> >...
> >"
> >
> >in this case, wget.py fetcher will be used, and it works fine in try_mirror() case. Also the /opt/source can be read-only.
> 
> Per my example, I set both PREMIRRORS and MIRRORS to use these paths.
> 
> This _works_ (in that it doesn't fail), but it doesn't always
> seem to use my local MIRROR copy.  After an abreviated build from
> scratch, I found that these files _were_ copied from my local
> MIRROR:
>   automake-1.11.1.tar.bz2
>   m4-1.4.15.tar.gz
>   autoconf-2.65.tar.bz2
>   libtool-2.4.tar.gz
>   ncurses-5.4.tar.gz
>   gdbm-1.8.3.tar.gz
>   gzip-1.4.tar.gz
>   readline-6.1.tar.gz
>   gmp-5.0.1.tar.bz2
>   bison-2.4.3.tar.gz
>   gettext-0.17.tar.gz
>   binutils-2.20.1.tar.bz2
> 
> but these were [re]fetched over the network from their original
> sources (my MIRROR does have local copies):
>   db-4.2.52.tar.gz
>   expat-2.0.1.tar.gz
>   flex-2.5.35.tar.bz2
>   perl-5.8.8.tar.gz
>   quilt-0.48.tar.gz
>   zlib-1.2.5.tar.bz2
>   mpc-0.8.2.tar.gz
>   pkg-config-0.25.tar.gz
>   mpfr-3.0.0.tar.bz2
>   elfutils-0.148.tar.bz2
>   tcl8.5.9-src.tar.gz
>   config_cvs.sv.gnu.org__20080123.tar.gz
>   curl-7.21.2.tar.bz2
>   linux-2.6.36.tar.bz2
>   git-1.7.3.2.tar.bz2
>   sqlite-3.7.3.tar.gz
>   openssl-0.9.8p.tar.gz
>   git_github.com.wrpseudo.pseudo.git_c9792c7acdb1cac90549ff08db12a8bf0c53cdcf.tar.gz
> 
> Not quite what I was hoping for.  I don't mind getting a
> [temporary] copy of the files into ${BUILD}/downloads, but
> I really need to find a way to avoid ever going to the internet
> if the file is already available (which I plan to guarantee
> for my customers)

yes, It is unexpected. After digging more, I find meta/classes/mirrors.bbclass also provide some internet mirror, and bitbake may use them for fetching from internet. So could you remove those mirrors and have a quick try? This somehow does not explain why bitbake ignore PREMIRROR, so there must be wrong somewhere else. I am still looking. Will send out if there is more finding.

Regares
Ke

> 
> -- 
> ------------------------------------------------------------
> Gary Thomas                 |  Consulting for the
> MLB Associates              |    Embedded world
> ------------------------------------------------------------
> _______________________________________________
> poky mailing list
> poky@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/poky

Re: PREMIRRORS in yocto, do they actually work?

[Gary Thomas]

On 01/07/2011 12:55 AM, Yu Ke wrote:
> On Jan 06, 08:06, Gary Thomas wrote:
>> On 01/06/2011 12:14 AM, Yu Ke wrote:
>>> On Jan 02, 10:15, Gary Thomas wrote:
>>>> On 01/02/2011 10:08 AM, Koen Kooi wrote:
>>>>> Hi,
>>>>>
>>>>> I would like to have the fetchter try my PREMIRROS first before trying to fetch from git. In OE I have the following in a bbclass:
>>>>>
>>>>> PREMIRRORS_append () {
>>>>> git://.*/.* http://www.angstrom-distribution.org/unstable/sources/
>>>>> }
>>>>>
>>>>> When used in OE that works, but in yocto it doesn't. So I tried to do it poky style in a .inc:
>>>>>
>>>>> PREMIRRORS_append = "\
>>>>> git://.*/.*   http:/www.angstrom-distribution.org/sources/ \n"
>>>>>
>>>>> Which also doesn't work since it doesn't try to fetch from there, but it does mention "http:/www.angstrom-distribution.org/sources/" in the fetch error message when git read-tree fails.
>>>>>
>>>>> So my question: how do I make it work in yocto like it works in OE?
>>>>
>>>> I too have been struggling with this as I'd like to have
>>>> a local repository to hold the current source set.  Sadly,
>>>> as you've observed, it doesn't work, at least not as expected.
>>>
>>> I've tried to use file:///some/dir as PREMIRROR, and found it does has issue. The main reason is that the local.py fetcher does not do real copy in go() method. Although it make sense in normal fetch case, it does cause trouble in try_mirror() case.
>>>
>>> I don't have simple idea to address this issue yet and I will take care of it in the coming fetcher change. As a quick workaround, could you try this approach: use http://local/some/dir as PREMIRROR? e.g. if you use /opt/source as mirror, you can add to meta/conf/distro/poky.conf:
>>> PREMIRRORS_append = "\
>>> git://.*/.*   http://local/opt/source/ \n \
>>> svn://.*/.*   http://local/opt/source/ \n \
>>> ...
>>> "
>>>
>>> in this case, wget.py fetcher will be used, and it works fine in try_mirror() case. Also the /opt/source can be read-only.
>>
>> Per my example, I set both PREMIRRORS and MIRRORS to use these paths.
>>
>> This _works_ (in that it doesn't fail), but it doesn't always
>> seem to use my local MIRROR copy.  After an abreviated build from
>> scratch, I found that these files _were_ copied from my local
>> MIRROR:
>>    automake-1.11.1.tar.bz2
>>    m4-1.4.15.tar.gz
>>    autoconf-2.65.tar.bz2
>>    libtool-2.4.tar.gz
>>    ncurses-5.4.tar.gz
>>    gdbm-1.8.3.tar.gz
>>    gzip-1.4.tar.gz
>>    readline-6.1.tar.gz
>>    gmp-5.0.1.tar.bz2
>>    bison-2.4.3.tar.gz
>>    gettext-0.17.tar.gz
>>    binutils-2.20.1.tar.bz2
>>
>> but these were [re]fetched over the network from their original
>> sources (my MIRROR does have local copies):
>>    db-4.2.52.tar.gz
>>    expat-2.0.1.tar.gz
>>    flex-2.5.35.tar.bz2
>>    perl-5.8.8.tar.gz
>>    quilt-0.48.tar.gz
>>    zlib-1.2.5.tar.bz2
>>    mpc-0.8.2.tar.gz
>>    pkg-config-0.25.tar.gz
>>    mpfr-3.0.0.tar.bz2
>>    elfutils-0.148.tar.bz2
>>    tcl8.5.9-src.tar.gz
>>    config_cvs.sv.gnu.org__20080123.tar.gz
>>    curl-7.21.2.tar.bz2
>>    linux-2.6.36.tar.bz2
>>    git-1.7.3.2.tar.bz2
>>    sqlite-3.7.3.tar.gz
>>    openssl-0.9.8p.tar.gz
>>    git_github.com.wrpseudo.pseudo.git_c9792c7acdb1cac90549ff08db12a8bf0c53cdcf.tar.gz
>>
>> Not quite what I was hoping for.  I don't mind getting a
>> [temporary] copy of the files into ${BUILD}/downloads, but
>> I really need to find a way to avoid ever going to the internet
>> if the file is already available (which I plan to guarantee
>> for my customers)
>
> yes, It is unexpected. After digging more, I find meta/classes/mirrors.bbclass also provide some internet mirror, and bitbake may use them for fetching from internet. So could you remove those mirrors and have a quick try? This somehow does not explain why bitbake ignore PREMIRROR, so there must be wrong somewhere else. I am still looking. Will send out if there is more finding.

Any progress on this?  I just tried my original (file://)
setup with the latest master that has fetch2 enabled and
it still fails miserably.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------