Re: Fetch problems with PREMIRRORS

[Gary Thomas <gary@mlbassoc.com>]

On 02/10/2011 07:03 AM, Richard Purdie wrote:
> On Wed, 2011-02-09 at 06:40 -0700, Gary Thomas wrote:
>> Per Richard's suggestion, I have added this to my DISTRO.conf:
>>
>> # Provide pre-staged sources
>> PREMIRRORS = "\
>> http://.*/.*   file:///work/misc/Poky/sources/ \n \
>> https://.*/.*   file:///work/misc/Poky/sources/ \n \
>> ftp://.*/.*   file:///work/misc/Poky/sources/ \n \
>> bzr://.*/.*   file:///work/misc/Poky/sources/ \n \
>> cvs://.*/.*   file:///work/misc/Poky/sources/ \n \
>> git://.*/.*   file:///work/misc/Poky/sources/ \n \
>> hg://.*/.*    file:///work/misc/Poky/sources/ \n \
>> osc://.*/.*   file:///work/misc/Poky/sources/ \n \
>> p4://.*/.*    file:///work/misc/Poky/sources/ \n \
>> svk://.*/.*   file:///work/misc/Poky/sources/ \n \
>> svn://.*/.*   file:///work/misc/Poky/sources/ \n"
>>
>> The tree '/work/misc/Poky/sources' is a fully populated source
>> cache.
>>
>> This all works great as long as the needed sources are in
>> the cache.  However, there seems to be no fallback if they
>> are not present.  I get errors like this:
>> | NOTE: fetch http://pokylinux.org/sources/git2_github.com.wrpseudo.pseudo.git.tar.gz
>> | NOTE: fetch http://autobuilder.pokylinux.org/sources/git2_github.com.wrpseudo.pseudo.git.tar.gz
>> | ERROR: Function 'Fetcher failure for URL: 'git://github.com/wrpseudo/pseudo.git;protocol=git'. Unable to fetch URL git://github.com/wrpseudo/pseudo.git;protocol=git from any
>> source.' failed
>> NOTE: package pseudo-native-0.0+git1+5434325fc887817ebb2bad36313d8277992cef1d-r18: task Fetcher failure for URL: 'git://github.com/wrpseudo/pseudo.git;protocol=git'. Unable to
>> fetch URL git://github.com/wrpseudo/pseudo.git;protocol=git from any source.: Failed
>>
>> In order to actually fetch, I have to remove the PREMIRRORS
>> from my conf.
>>
>> Is there some way to get it to use the source cache but
>> still fetch when needed?
>>
>> Thanks
>>
>> Note: I also normally have BB_NO_NETWORK="1", but that is not
>> applicable here (no message about it) and setting it false does
>> not change the behaviour.
>
> It is applicable.
>
> Try not setting that variable at all, not setting it false. I suspect
> there are two issues here:
>
> a) BB_NO_NETWORK is having an effect but the log messages are being
>     swallowed...
> b) "false" handling for the variable might not be working properly so
>     leave it unset.
>
> We can fix those things but I'd like to confirm these are the problems
> first.

Correct, it does seem to be the case.  When BB_NO_NETWORK="0",
the fetcher works properly.  When set to "1", it fails, but
no explicit error about why is generated, neither on the console
nor in the log.do_fetch file.

I made this small change to check it out:
diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py
index ef9d75f..57c4f20 100644
--- a/bitbake/lib/bb/fetch2/__init__.py
+++ b/bitbake/lib/bb/fetch2/__init__.py
@@ -395,6 +395,7 @@ def check_network_access(d, info = ""):
      log remote network access, and error if BB_NO_NETWORK is set
      """
      if bb.data.getVar("BB_NO_NETWORK", d, True) == "1":
+        bb.error("BB_NO_NETWORK is set, but the fetcher code attempted network access with the command %s" % info)
          raise FetchError("BB_NO_NETWORK is set, but the fetcher code attempted network access with the command %s" % inf
      else:
          logger.debug(1, "Fetcher accessed the network with the command %s" % info)

This shows that this _is_ the error case, with the messages making it to
the console and log file.  I can see that this exception is being thrown
on line 886, but the reason is lost in the exception data & I'm unclear
how to retrieve it and present it to the user.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------