how to set up PREMIRRORS_prepend to avoid constant git clones?

how to set up PREMIRRORS_prepend to avoid constant git clones?

[Robert P. J. Day]

  over the years, i've built up a sizable collection of source
tarballs for use in various projects -- top-level directory called
"~/dl", about 5G in size with lots and lots of .tgz and .bz2 files and
so on, and i'd like to confirm the best way to take advantage of all
that source whenever i configure and build a yocto project.

  based on my reading, whenever i create a new project, i immediately
add the following to the bottom of its local.conf file:

PREMIRRORS_prepend = "\
git://.*/.* file:///home/rpjday/dl/ \n \
svn://.*/.* file:///home/rpjday/dl/ \n \
cvs://.*/.* file:///home/rpjday/dl/ \n \
ftp://.*/.* file:///home/rpjday/dl/ \n \
http://.*/.* file:///home/rpjday/dl/ \n \
https://.*/.* file:///home/rpjday/dl/ \n"

and that certainly makes a massive difference -- whenever the fetching
starts, matching tarballs will be located in my "dl" directory, and
the project's downloads directory will be populated with symlinks to
the actual tarballs, as in:

  vala-0.14.2.tar.xz -> /home/rpjday/dl/vala-0.14.2.tar.xz

and after the download, i sneak a peek at whatever newer tarballs
needed to be downloaded, and quietly copy them over to "dl" to avoid
ever downloading them again, and that's worked well so far.  so, the
questions:

1) does this sound reasonable?  i know about using DL_DIR and perhaps
sharing that across projects, but i rather like the idea of an
application-agnostic repository of straight tarballs and zip files
that any build system is allowed to pilfer from (especially since i
can make it read-only).

2) am i missing any fetch protocols from that list?

3) how do i set it up to avoid repeated git clones?

  it seems that, while i can avoid fetching simple tarballs, i still
end up doing the same "git clone" operations over and over.  am i just
missing something obvious?

rday

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

Re: how to set up PREMIRRORS_prepend to avoid constant git clones?

[Gary Thomas]

On 2012-03-10 03:35, Robert P. J. Day wrote:
>
>    over the years, i've built up a sizable collection of source
> tarballs for use in various projects -- top-level directory called
> "~/dl", about 5G in size with lots and lots of .tgz and .bz2 files and
> so on, and i'd like to confirm the best way to take advantage of all
> that source whenever i configure and build a yocto project.
>
>    based on my reading, whenever i create a new project, i immediately
> add the following to the bottom of its local.conf file:
>
> PREMIRRORS_prepend = "\
> git://.*/.* file:///home/rpjday/dl/ \n \
> svn://.*/.* file:///home/rpjday/dl/ \n \
> cvs://.*/.* file:///home/rpjday/dl/ \n \
> ftp://.*/.* file:///home/rpjday/dl/ \n \
> http://.*/.* file:///home/rpjday/dl/ \n \
> https://.*/.* file:///home/rpjday/dl/ \n"
>
> and that certainly makes a massive difference -- whenever the fetching
> starts, matching tarballs will be located in my "dl" directory, and
> the project's downloads directory will be populated with symlinks to
> the actual tarballs, as in:
>
>    vala-0.14.2.tar.xz ->  /home/rpjday/dl/vala-0.14.2.tar.xz
>
> and after the download, i sneak a peek at whatever newer tarballs
> needed to be downloaded, and quietly copy them over to "dl" to avoid
> ever downloading them again, and that's worked well so far.  so, the
> questions:
>
> 1) does this sound reasonable?  i know about using DL_DIR and perhaps
> sharing that across projects, but i rather like the idea of an
> application-agnostic repository of straight tarballs and zip files
> that any build system is allowed to pilfer from (especially since i
> can make it read-only).
>
> 2) am i missing any fetch protocols from that list?
>
> 3) how do i set it up to avoid repeated git clones?
>
>    it seems that, while i can avoid fetching simple tarballs, i still
> end up doing the same "git clone" operations over and over.  am i just
> missing something obvious?

All you really need are these lines:
   SOURCE_MIRROR_URL ?= "file://${COREBASE}/sources/"
   INHERIT += "own-mirrors"
Adjust the SOURCE_MIRROR_URL to wherever you want the files.  I also always add
   BB_GENERATE_MIRROR_TARBALLS = "1"
This will let the fetcher make a git2_XXX tarball, once the repo has been cloned.

My understanding is that the git (and other SCM) fetcher works like this:
   * If there is a git2_XXX tar file in the DL_DIR (or mirror), unpack it
     into ${BUILD}/downloads/git2
   * Check to see if the desired ${SRCREV} is present.  If not, do a 'git pull'
     to update the local clone and then rebuild the git2_XXX tarball.
This way, you only have to clone once - just copy any generated tarballs back
to your source mirror.

Finally, if you set
   BB_NO_NETWORK = "1"
the build will fail if bitbake *ever* has to go to the network, i.e. not find
the desired files in the source mirror or DL_DIR.  This is a great way to insure
that you've got what you need locally.  I use this all the time to make sure I
have a fully populated local cache of sources.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: how to set up PREMIRRORS_prepend to avoid constant git clones?

[Gary Thomas]

On 2012-03-10 03:52, Gary Thomas wrote:
> On 2012-03-10 03:35, Robert P. J. Day wrote:
>>
>> over the years, i've built up a sizable collection of source
>> tarballs for use in various projects -- top-level directory called
>> "~/dl", about 5G in size with lots and lots of .tgz and .bz2 files and
>> so on, and i'd like to confirm the best way to take advantage of all
>> that source whenever i configure and build a yocto project.
>>
>> based on my reading, whenever i create a new project, i immediately
>> add the following to the bottom of its local.conf file:
>>
>> PREMIRRORS_prepend = "\
>> git://.*/.* file:///home/rpjday/dl/ \n \
>> svn://.*/.* file:///home/rpjday/dl/ \n \
>> cvs://.*/.* file:///home/rpjday/dl/ \n \
>> ftp://.*/.* file:///home/rpjday/dl/ \n \
>> http://.*/.* file:///home/rpjday/dl/ \n \
>> https://.*/.* file:///home/rpjday/dl/ \n"
>>
>> and that certainly makes a massive difference -- whenever the fetching
>> starts, matching tarballs will be located in my "dl" directory, and
>> the project's downloads directory will be populated with symlinks to
>> the actual tarballs, as in:
>>
>> vala-0.14.2.tar.xz -> /home/rpjday/dl/vala-0.14.2.tar.xz
>>
>> and after the download, i sneak a peek at whatever newer tarballs
>> needed to be downloaded, and quietly copy them over to "dl" to avoid
>> ever downloading them again, and that's worked well so far. so, the
>> questions:
>>
>> 1) does this sound reasonable? i know about using DL_DIR and perhaps
>> sharing that across projects, but i rather like the idea of an
>> application-agnostic repository of straight tarballs and zip files
>> that any build system is allowed to pilfer from (especially since i
>> can make it read-only).
>>
>> 2) am i missing any fetch protocols from that list?
>>
>> 3) how do i set it up to avoid repeated git clones?
>>
>> it seems that, while i can avoid fetching simple tarballs, i still
>> end up doing the same "git clone" operations over and over. am i just
>> missing something obvious?
>
> All you really need are these lines:
> SOURCE_MIRROR_URL ?= "file://${COREBASE}/sources/"
> INHERIT += "own-mirrors"

Note: using this class makes sure all the needed fetcher types are filled in
as you don't have to think about it :-)

> Adjust the SOURCE_MIRROR_URL to wherever you want the files. I also always add
> BB_GENERATE_MIRROR_TARBALLS = "1"
> This will let the fetcher make a git2_XXX tarball, once the repo has been cloned.
>
> My understanding is that the git (and other SCM) fetcher works like this:
> * If there is a git2_XXX tar file in the DL_DIR (or mirror), unpack it
> into ${BUILD}/downloads/git2
> * Check to see if the desired ${SRCREV} is present. If not, do a 'git pull'
> to update the local clone and then rebuild the git2_XXX tarball.
> This way, you only have to clone once - just copy any generated tarballs back
> to your source mirror.
>
> Finally, if you set
> BB_NO_NETWORK = "1"
> the build will fail if bitbake *ever* has to go to the network, i.e. not find
> the desired files in the source mirror or DL_DIR. This is a great way to insure
> that you've got what you need locally. I use this all the time to make sure I
> have a fully populated local cache of sources.
>

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: how to set up PREMIRRORS_prepend to avoid constant git clones?

[Robert P. J. Day]

On Sat, 10 Mar 2012, Gary Thomas wrote:

> All you really need are these lines:
>   SOURCE_MIRROR_URL ?= "file://${COREBASE}/sources/"

  ah ... and i see no mention of that in the variable glossary in the
reference manual. :-P  i'm assuming that that value is nothing more
than the name of my massive tarballs directory, right?

>   INHERIT += "own-mirrors"

> Adjust the SOURCE_MIRROR_URL to wherever you want the files.
>
> I also always add
>
>   BB_GENERATE_MIRROR_TARBALLS = "1"
> This will let the fetcher make a git2_XXX tarball, once the repo has been
> cloned.
>
> My understanding is that the git (and other SCM) fetcher works like this:
>   * If there is a git2_XXX tar file in the DL_DIR (or mirror), unpack it
>     into ${BUILD}/downloads/git2
>   * Check to see if the desired ${SRCREV} is present.  If not, do a 'git pull'
>     to update the local clone and then rebuild the git2_XXX tarball.
> This way, you only have to clone once - just copy any generated tarballs back
> to your source mirror.

  yup, that's what i was after -- the ability to avoid constant
cloning.  and i see i still have to manually copy the git-based
tarball back to my own mirror, that's fine, i would have expected
that.

> Finally, if you set
>   BB_NO_NETWORK = "1"
> the build will fail if bitbake *ever* has to go to the network, i.e. not find
> the desired files in the source mirror or DL_DIR.  This is a great way to
> insure
> that you've got what you need locally.  I use this all the time to make sure I
> have a fully populated local cache of sources.

  i knew about that setting, i can use that for, as you say,
verification that i have all local sources.

  ok, i'm off to try this, i'll let you know.  thanks.

rday

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

Re: how to set up PREMIRRORS_prepend to avoid constant git clones?

[Robert P. J. Day]

On Sat, 10 Mar 2012, Gary Thomas wrote:

> All you really need are these lines:
>   SOURCE_MIRROR_URL ?= "file://${COREBASE}/sources/"
>   INHERIT += "own-mirrors"
> Adjust the SOURCE_MIRROR_URL to wherever you want the files.  I also always
> add
>   BB_GENERATE_MIRROR_TARBALLS = "1"
> This will let the fetcher make a git2_XXX tarball, once the repo has been
> cloned.
>
> My understanding is that the git (and other SCM) fetcher works like this:
>   * If there is a git2_XXX tar file in the DL_DIR (or mirror), unpack it
>     into ${BUILD}/downloads/git2
>   * Check to see if the desired ${SRCREV} is present.  If not, do a 'git pull'
>     to update the local clone and then rebuild the git2_XXX tarball.
> This way, you only have to clone once - just copy any generated tarballs back
> to your source mirror.

  one last point -- what i typically do as a test is always

  $ bitbake -c fetchall <target>

before doing the actual build, just to see that everything is
fetchable and gets fetched.  is there a bitbake task that would tell
me what needs actual fetching before i start that?

rday

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

Re: how to set up PREMIRRORS_prepend to avoid constant git clones?

[Robert P. J. Day]

On Sat, 10 Mar 2012, Gary Thomas wrote:

> All you really need are these lines:
>   SOURCE_MIRROR_URL ?= "file://${COREBASE}/sources/"
>   INHERIT += "own-mirrors"
> Adjust the SOURCE_MIRROR_URL to wherever you want the files.  I also always
> add
>   BB_GENERATE_MIRROR_TARBALLS = "1"
> This will let the fetcher make a git2_XXX tarball, once the repo has been
> cloned.
>
> My understanding is that the git (and other SCM) fetcher works like this:
>   * If there is a git2_XXX tar file in the DL_DIR (or mirror), unpack it
>     into ${BUILD}/downloads/git2
>   * Check to see if the desired ${SRCREV} is present.  If not, do a 'git pull'
>     to update the local clone and then rebuild the git2_XXX tarball.
> This way, you only have to clone once - just copy any generated tarballs back
> to your source mirror.
>
> Finally, if you set
>   BB_NO_NETWORK = "1"
> the build will fail if bitbake *ever* has to go to the network,

  well, that (sort of) worked so let me summarize for the benefit of
anyone else interested.

  i configured for a new build (beagleboard), and added the following
to the bottom of local.conf:

SOURCE_MIRROR_URL ?= "file:///home/rpjday/dl/"
INHERIT += "own-mirrors"
BB_GENERATE_MIRROR_TARBALLS = "1"
# BB_NO_NETWORK = "1"

  (since i know i'll need to do some downloading, i won't ban network
access, obviously, not for this first run.)

  did a "bitbake -c fetchall core-image-minimal", then checked my
downloads/ directory for any content that actually got downloaded and
tarball-ed, and found one obvious and sizable download:

... 87770604 2012-03-10 06:45 gcc-4_6-branch_gcc.gnu.org_.svn.gcc.branches_184847_.tar.gz

  i copied that over to my own mirror dl/ directory, then configured a
whole new build (also beagleboard), added same mirror directives to
local.conf except (of course) banning network access this time, and
once again did

  $ bitbake -c fetchall core-image-minimal

and got something unexpected:

NOTE: package bash-4.2-r1: task do_fetch: Started
ERROR: Function failed: Network access disabled through BB_NO_NETWORK
but access rquested with command /usr/bin/env wget -t 5 -nv
--passive-ftp --no-check-certificate -P
/home/rpjday/yocto/builds/beag2/downloads
'ftp://ftp.gnu.org/gnu/bash/bash-4.2-patches/bash42-001' (for url
None)
ERROR: Logfile of failure stored in:
/home/rpjday/yocto/builds/beag2/tmp/work/armv7a-vfp-neon-poky-linux-gnueabi/bash-4.2-r1/temp/log.do_fetch.32479
Log data follows:
| DEBUG: Trying PREMIRRORS
... snip ...

  i can clearly see the problem -- while i have the bash tarball
itself in my own mirror, i don't have the patches to go with it, as
defined in the recipes file:

SRC_URI = "${GNU_MIRROR}/bash/${BPN}-${PV}.tar.gz;name=tarball \

${GNU_MIRROR}/bash/bash-4.2-patches/bash42-001;apply=yes;striplevel=0;name=patch001 \
${GNU_MIRROR}/bash/bash-4.2-patches/bash42-002;apply=yes;striplevel=0;name=patch002 \
${GNU_MIRROR}/bash/bash-4.2-patches/bash42-003;apply=yes;striplevel=0;name=patch003 \
... etc etc ...

  when i went looking for actual downloads in step 1, i just searched
for sizable tarballs or zip files, it never occurred to me to check
for patch files.  so i guess i just have to be more observant and
check for patch files as well (or anything else a recipe might want to
download).

  so far, so good?

rday

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

Re: how to set up PREMIRRORS_prepend to avoid constant git clones?

[Gary Thomas]

On 2012-03-10 06:39, Robert P. J. Day wrote:
> On Sat, 10 Mar 2012, Gary Thomas wrote:
>
>> All you really need are these lines:
>>    SOURCE_MIRROR_URL ?= "file://${COREBASE}/sources/"
>>    INHERIT += "own-mirrors"
>> Adjust the SOURCE_MIRROR_URL to wherever you want the files.  I also always
>> add
>>    BB_GENERATE_MIRROR_TARBALLS = "1"
>> This will let the fetcher make a git2_XXX tarball, once the repo has been
>> cloned.
>>
>> My understanding is that the git (and other SCM) fetcher works like this:
>>    * If there is a git2_XXX tar file in the DL_DIR (or mirror), unpack it
>>      into ${BUILD}/downloads/git2
>>    * Check to see if the desired ${SRCREV} is present.  If not, do a 'git pull'
>>      to update the local clone and then rebuild the git2_XXX tarball.
>> This way, you only have to clone once - just copy any generated tarballs back
>> to your source mirror.
>>
>> Finally, if you set
>>    BB_NO_NETWORK = "1"
>> the build will fail if bitbake *ever* has to go to the network,
>
>    well, that (sort of) worked so let me summarize for the benefit of
> anyone else interested.
>
>    i configured for a new build (beagleboard), and added the following
> to the bottom of local.conf:
>
> SOURCE_MIRROR_URL ?= "file:///home/rpjday/dl/"
> INHERIT += "own-mirrors"
> BB_GENERATE_MIRROR_TARBALLS = "1"
> # BB_NO_NETWORK = "1"
>
>    (since i know i'll need to do some downloading, i won't ban network
> access, obviously, not for this first run.)
>
>    did a "bitbake -c fetchall core-image-minimal", then checked my
> downloads/ directory for any content that actually got downloaded and
> tarball-ed, and found one obvious and sizable download:
>
> ... 87770604 2012-03-10 06:45 gcc-4_6-branch_gcc.gnu.org_.svn.gcc.branches_184847_.tar.gz
>
>    i copied that over to my own mirror dl/ directory, then configured a
> whole new build (also beagleboard), added same mirror directives to
> local.conf except (of course) banning network access this time, and
> once again did
>
>    $ bitbake -c fetchall core-image-minimal
>
> and got something unexpected:
>
> NOTE: package bash-4.2-r1: task do_fetch: Started
> ERROR: Function failed: Network access disabled through BB_NO_NETWORK
> but access rquested with command /usr/bin/env wget -t 5 -nv
> --passive-ftp --no-check-certificate -P
> /home/rpjday/yocto/builds/beag2/downloads
> 'ftp://ftp.gnu.org/gnu/bash/bash-4.2-patches/bash42-001' (for url
> None)
> ERROR: Logfile of failure stored in:
> /home/rpjday/yocto/builds/beag2/tmp/work/armv7a-vfp-neon-poky-linux-gnueabi/bash-4.2-r1/temp/log.do_fetch.32479
> Log data follows:
> | DEBUG: Trying PREMIRRORS
> ... snip ...
>
>    i can clearly see the problem -- while i have the bash tarball
> itself in my own mirror, i don't have the patches to go with it, as
> defined in the recipes file:
>
> SRC_URI = "${GNU_MIRROR}/bash/${BPN}-${PV}.tar.gz;name=tarball \
>
> ${GNU_MIRROR}/bash/bash-4.2-patches/bash42-001;apply=yes;striplevel=0;name=patch001 \
> ${GNU_MIRROR}/bash/bash-4.2-patches/bash42-002;apply=yes;striplevel=0;name=patch002 \
> ${GNU_MIRROR}/bash/bash-4.2-patches/bash42-003;apply=yes;striplevel=0;name=patch003 \
> ... etc etc ...
>
>    when i went looking for actual downloads in step 1, i just searched
> for sizable tarballs or zip files, it never occurred to me to check
> for patch files.  so i guess i just have to be more observant and
> check for patch files as well (or anything else a recipe might want to
> download).
>
>    so far, so good?

I didn't check but this is often the recipe's fault.  It must specify
an exact git SRCREV, not a tag, for the NO_NETWORK to be successful.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: how to set up PREMIRRORS_prepend to avoid constant git clones?

[Robert P. J. Day]

On Sat, 10 Mar 2012, Gary Thomas wrote:

> On 2012-03-10 06:39, Robert P. J. Day wrote:

... snip ...

> >    $ bitbake -c fetchall core-image-minimal
> >
> > and got something unexpected:
> >
> > NOTE: package bash-4.2-r1: task do_fetch: Started
> > ERROR: Function failed: Network access disabled through BB_NO_NETWORK
> > but access rquested with command /usr/bin/env wget -t 5 -nv
> > --passive-ftp --no-check-certificate -P
> > /home/rpjday/yocto/builds/beag2/downloads
> > 'ftp://ftp.gnu.org/gnu/bash/bash-4.2-patches/bash42-001' (for url
> > None)
> > ERROR: Logfile of failure stored in:
> > /home/rpjday/yocto/builds/beag2/tmp/work/armv7a-vfp-neon-poky-linux-gnueabi/bash-4.2-r1/temp/log.do_fetch.32479
> > Log data follows:
> > | DEBUG: Trying PREMIRRORS
> > ... snip ...
> >
> >    i can clearly see the problem -- while i have the bash tarball
> > itself in my own mirror, i don't have the patches to go with it, as
> > defined in the recipes file:
> >
> > SRC_URI = "${GNU_MIRROR}/bash/${BPN}-${PV}.tar.gz;name=tarball \
> >
> > ${GNU_MIRROR}/bash/bash-4.2-patches/bash42-001;apply=yes;striplevel=0;name=patch001
> > \
> > ${GNU_MIRROR}/bash/bash-4.2-patches/bash42-002;apply=yes;striplevel=0;name=patch002
> > \
> > ${GNU_MIRROR}/bash/bash-4.2-patches/bash42-003;apply=yes;striplevel=0;name=patch003
> > \
> > ... etc etc ...
> >
> >    when i went looking for actual downloads in step 1, i just searched
> > for sizable tarballs or zip files, it never occurred to me to check
> > for patch files.  so i guess i just have to be more observant and
> > check for patch files as well (or anything else a recipe might want to
> > download).
> >
> >    so far, so good?
>
> I didn't check but this is often the recipe's fault.  It must specify
> an exact git SRCREV, not a tag, for the NO_NETWORK to be successful.

  i'm not sure what you're saying here.  the patches are simple files,
nothing to do with git.  the bash source itself is properly being
fetched from my mirror -- it's the numerous patch files i was unaware
of that still require network access; that is, until i manually copied
them over to my mirror directory, and that solved the problem.

  i just had to notice that the bash recipe downloaded a number of
patch files, then add them to my local mirror.

rday

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

Re: how to set up PREMIRRORS_prepend to avoid constant git clones?

[Gary Thomas]

On 2012-03-10 08:29, Robert P. J. Day wrote:
> On Sat, 10 Mar 2012, Gary Thomas wrote:
>
>> On 2012-03-10 06:39, Robert P. J. Day wrote:
>
> ... snip ...
>
>>>     $ bitbake -c fetchall core-image-minimal
>>>
>>> and got something unexpected:
>>>
>>> NOTE: package bash-4.2-r1: task do_fetch: Started
>>> ERROR: Function failed: Network access disabled through BB_NO_NETWORK
>>> but access rquested with command /usr/bin/env wget -t 5 -nv
>>> --passive-ftp --no-check-certificate -P
>>> /home/rpjday/yocto/builds/beag2/downloads
>>> 'ftp://ftp.gnu.org/gnu/bash/bash-4.2-patches/bash42-001' (for url
>>> None)
>>> ERROR: Logfile of failure stored in:
>>> /home/rpjday/yocto/builds/beag2/tmp/work/armv7a-vfp-neon-poky-linux-gnueabi/bash-4.2-r1/temp/log.do_fetch.32479
>>> Log data follows:
>>> | DEBUG: Trying PREMIRRORS
>>> ... snip ...
>>>
>>>     i can clearly see the problem -- while i have the bash tarball
>>> itself in my own mirror, i don't have the patches to go with it, as
>>> defined in the recipes file:
>>>
>>> SRC_URI = "${GNU_MIRROR}/bash/${BPN}-${PV}.tar.gz;name=tarball \
>>>
>>> ${GNU_MIRROR}/bash/bash-4.2-patches/bash42-001;apply=yes;striplevel=0;name=patch001
>>> \
>>> ${GNU_MIRROR}/bash/bash-4.2-patches/bash42-002;apply=yes;striplevel=0;name=patch002
>>> \
>>> ${GNU_MIRROR}/bash/bash-4.2-patches/bash42-003;apply=yes;striplevel=0;name=patch003
>>> \
>>> ... etc etc ...
>>>
>>>     when i went looking for actual downloads in step 1, i just searched
>>> for sizable tarballs or zip files, it never occurred to me to check
>>> for patch files.  so i guess i just have to be more observant and
>>> check for patch files as well (or anything else a recipe might want to
>>> download).
>>>
>>>     so far, so good?
>>
>> I didn't check but this is often the recipe's fault.  It must specify
>> an exact git SRCREV, not a tag, for the NO_NETWORK to be successful.
>
>    i'm not sure what you're saying here.  the patches are simple files,
> nothing to do with git.  the bash source itself is properly being
> fetched from my mirror -- it's the numerous patch files i was unaware
> of that still require network access; that is, until i manually copied
> them over to my mirror directory, and that solved the problem.
>
>    i just had to notice that the bash recipe downloaded a number of
> patch files, then add them to my local mirror.

Sorry, I misread.  Yes, you do need those patch files in your local mirror.
In fact, if you look at ${BUILD}/downloads, every _file_ (not symlink) that's
at the top level needs to be copied.  I manage this by running a script
like this:
   (cd downloads;find . -type f -maxdepth 1 | tar -cT - | tar -x -C ${DL_DIR})

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: how to set up PREMIRRORS_prepend to avoid constant git clones?

[Robert P. J. Day]

On Sat, 10 Mar 2012, Gary Thomas wrote:

> On 2012-03-10 08:29, Robert P. J. Day wrote:
> > On Sat, 10 Mar 2012, Gary Thomas wrote:
> >
> > > On 2012-03-10 06:39, Robert P. J. Day wrote:
> >
> > ... snip ...
> >
> > > >     $ bitbake -c fetchall core-image-minimal
> > > >
> > > > and got something unexpected:
> > > >
> > > > NOTE: package bash-4.2-r1: task do_fetch: Started
> > > > ERROR: Function failed: Network access disabled through BB_NO_NETWORK
> > > > but access rquested with command /usr/bin/env wget -t 5 -nv
> > > > --passive-ftp --no-check-certificate -P
> > > > /home/rpjday/yocto/builds/beag2/downloads
> > > > 'ftp://ftp.gnu.org/gnu/bash/bash-4.2-patches/bash42-001' (for url
> > > > None)
> > > > ERROR: Logfile of failure stored in:
> > > > /home/rpjday/yocto/builds/beag2/tmp/work/armv7a-vfp-neon-poky-linux-gnueabi/bash-4.2-r1/temp/log.do_fetch.32479
> > > > Log data follows:
> > > > | DEBUG: Trying PREMIRRORS
> > > > ... snip ...
> > > >
> > > >     i can clearly see the problem -- while i have the bash tarball
> > > > itself in my own mirror, i don't have the patches to go with it, as
> > > > defined in the recipes file:
> > > >
> > > > SRC_URI = "${GNU_MIRROR}/bash/${BPN}-${PV}.tar.gz;name=tarball \
> > > >
> > > > ${GNU_MIRROR}/bash/bash-4.2-patches/bash42-001;apply=yes;striplevel=0;name=patch001
> > > > \
> > > > ${GNU_MIRROR}/bash/bash-4.2-patches/bash42-002;apply=yes;striplevel=0;name=patch002
> > > > \
> > > > ${GNU_MIRROR}/bash/bash-4.2-patches/bash42-003;apply=yes;striplevel=0;name=patch003
> > > > \
> > > > ... etc etc ...
> > > >
> > > >     when i went looking for actual downloads in step 1, i just searched
> > > > for sizable tarballs or zip files, it never occurred to me to check
> > > > for patch files.  so i guess i just have to be more observant and
> > > > check for patch files as well (or anything else a recipe might want to
> > > > download).
> > > >
> > > >     so far, so good?
> > >
> > > I didn't check but this is often the recipe's fault.  It must specify
> > > an exact git SRCREV, not a tag, for the NO_NETWORK to be successful.
> >
> >    i'm not sure what you're saying here.  the patches are simple files,
> > nothing to do with git.  the bash source itself is properly being
> > fetched from my mirror -- it's the numerous patch files i was unaware
> > of that still require network access; that is, until i manually copied
> > them over to my mirror directory, and that solved the problem.
> >
> >    i just had to notice that the bash recipe downloaded a number of
> > patch files, then add them to my local mirror.
>
> Sorry, I misread.  Yes, you do need those patch files in your local mirror.
> In fact, if you look at ${BUILD}/downloads, every _file_ (not symlink) that's
> at the top level needs to be copied.  I manage this by running a script
> like this:
>   (cd downloads;find . -type f -maxdepth 1 | tar -cT - | tar -x -C ${DL_DIR})

  but doesn't that pick up all the ".done" files as well?  doesn't
hurt, but certainly unnecessary.

rday

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

Re: how to set up PREMIRRORS_prepend to avoid constant git clones?

[Gary Thomas]

On 2012-03-10 08:46, Robert P. J. Day wrote:
> On Sat, 10 Mar 2012, Gary Thomas wrote:
>
>> On 2012-03-10 08:29, Robert P. J. Day wrote:
>>> On Sat, 10 Mar 2012, Gary Thomas wrote:
>>>
>>>> On 2012-03-10 06:39, Robert P. J. Day wrote:
>>>
>>> ... snip ...
>>>
>>>>>      $ bitbake -c fetchall core-image-minimal
>>>>>
>>>>> and got something unexpected:
>>>>>
>>>>> NOTE: package bash-4.2-r1: task do_fetch: Started
>>>>> ERROR: Function failed: Network access disabled through BB_NO_NETWORK
>>>>> but access rquested with command /usr/bin/env wget -t 5 -nv
>>>>> --passive-ftp --no-check-certificate -P
>>>>> /home/rpjday/yocto/builds/beag2/downloads
>>>>> 'ftp://ftp.gnu.org/gnu/bash/bash-4.2-patches/bash42-001' (for url
>>>>> None)
>>>>> ERROR: Logfile of failure stored in:
>>>>> /home/rpjday/yocto/builds/beag2/tmp/work/armv7a-vfp-neon-poky-linux-gnueabi/bash-4.2-r1/temp/log.do_fetch.32479
>>>>> Log data follows:
>>>>> | DEBUG: Trying PREMIRRORS
>>>>> ... snip ...
>>>>>
>>>>>      i can clearly see the problem -- while i have the bash tarball
>>>>> itself in my own mirror, i don't have the patches to go with it, as
>>>>> defined in the recipes file:
>>>>>
>>>>> SRC_URI = "${GNU_MIRROR}/bash/${BPN}-${PV}.tar.gz;name=tarball \
>>>>>
>>>>> ${GNU_MIRROR}/bash/bash-4.2-patches/bash42-001;apply=yes;striplevel=0;name=patch001
>>>>> \
>>>>> ${GNU_MIRROR}/bash/bash-4.2-patches/bash42-002;apply=yes;striplevel=0;name=patch002
>>>>> \
>>>>> ${GNU_MIRROR}/bash/bash-4.2-patches/bash42-003;apply=yes;striplevel=0;name=patch003
>>>>> \
>>>>> ... etc etc ...
>>>>>
>>>>>      when i went looking for actual downloads in step 1, i just searched
>>>>> for sizable tarballs or zip files, it never occurred to me to check
>>>>> for patch files.  so i guess i just have to be more observant and
>>>>> check for patch files as well (or anything else a recipe might want to
>>>>> download).
>>>>>
>>>>>      so far, so good?
>>>>
>>>> I didn't check but this is often the recipe's fault.  It must specify
>>>> an exact git SRCREV, not a tag, for the NO_NETWORK to be successful.
>>>
>>>     i'm not sure what you're saying here.  the patches are simple files,
>>> nothing to do with git.  the bash source itself is properly being
>>> fetched from my mirror -- it's the numerous patch files i was unaware
>>> of that still require network access; that is, until i manually copied
>>> them over to my mirror directory, and that solved the problem.
>>>
>>>     i just had to notice that the bash recipe downloaded a number of
>>> patch files, then add them to my local mirror.
>>
>> Sorry, I misread.  Yes, you do need those patch files in your local mirror.
>> In fact, if you look at ${BUILD}/downloads, every _file_ (not symlink) that's
>> at the top level needs to be copied.  I manage this by running a script
>> like this:
>>    (cd downloads;find . -type f -maxdepth 1 | tar -cT - | tar -x -C ${DL_DIR})
>
>    but doesn't that pick up all the ".done" files as well?  doesn't
> hurt, but certainly unnecessary.

I said "something like" - add in "| grep -v .done" if you want to skip those.
I actually use a more complex script as there are some things I simply never
want to copy...

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: how to set up PREMIRRORS_prepend to avoid constant git clones?

[David Nyström]

On 03/10/2012 04:51 PM, Gary Thomas wrote:
> On 2012-03-10 08:46, Robert P. J. Day wrote:
>> On Sat, 10 Mar 2012, Gary Thomas wrote:
>>
>>> On 2012-03-10 08:29, Robert P. J. Day wrote:
>>>> On Sat, 10 Mar 2012, Gary Thomas wrote:
>>>>
>>>>> On 2012-03-10 06:39, Robert P. J. Day wrote:
>>>>
>>>> ... snip ...
>>>>
>>>>>>      $ bitbake -c fetchall core-image-minimal
>>>>>>
>>>>>> and got something unexpected:
>>>>>>
>>>>>> NOTE: package bash-4.2-r1: task do_fetch: Started
>>>>>> ERROR: Function failed: Network access disabled through 
>>>>>> BB_NO_NETWORK
>>>>>> but access rquested with command /usr/bin/env wget -t 5 -nv
>>>>>> --passive-ftp --no-check-certificate -P
>>>>>> /home/rpjday/yocto/builds/beag2/downloads
>>>>>> 'ftp://ftp.gnu.org/gnu/bash/bash-4.2-patches/bash42-001' (for url
>>>>>> None)
>>>>>> ERROR: Logfile of failure stored in:
>>>>>> /home/rpjday/yocto/builds/beag2/tmp/work/armv7a-vfp-neon-poky-linux-gnueabi/bash-4.2-r1/temp/log.do_fetch.32479 
>>>>>>
>>>>>> Log data follows:
>>>>>> | DEBUG: Trying PREMIRRORS
>>>>>> ... snip ...
>>>>>>
>>>>>>      i can clearly see the problem -- while i have the bash tarball
>>>>>> itself in my own mirror, i don't have the patches to go with it, as
>>>>>> defined in the recipes file:
>>>>>>
>>>>>> SRC_URI = "${GNU_MIRROR}/bash/${BPN}-${PV}.tar.gz;name=tarball \
>>>>>>
>>>>>> ${GNU_MIRROR}/bash/bash-4.2-patches/bash42-001;apply=yes;striplevel=0;name=patch001 
>>>>>>
>>>>>> \
>>>>>> ${GNU_MIRROR}/bash/bash-4.2-patches/bash42-002;apply=yes;striplevel=0;name=patch002 
>>>>>>
>>>>>> \
>>>>>> ${GNU_MIRROR}/bash/bash-4.2-patches/bash42-003;apply=yes;striplevel=0;name=patch003 
>>>>>>
>>>>>> \
>>>>>> ... etc etc ...
>>>>>>
>>>>>>      when i went looking for actual downloads in step 1, i just 
>>>>>> searched
>>>>>> for sizable tarballs or zip files, it never occurred to me to check
>>>>>> for patch files.  so i guess i just have to be more observant and
>>>>>> check for patch files as well (or anything else a recipe might 
>>>>>> want to
>>>>>> download).
>>>>>>
>>>>>>      so far, so good?
>>>>>
>>>>> I didn't check but this is often the recipe's fault.  It must specify
>>>>> an exact git SRCREV, not a tag, for the NO_NETWORK to be successful.
>>>>
>>>>     i'm not sure what you're saying here.  the patches are simple 
>>>> files,
>>>> nothing to do with git.  the bash source itself is properly being
>>>> fetched from my mirror -- it's the numerous patch files i was unaware
>>>> of that still require network access; that is, until i manually copied
>>>> them over to my mirror directory, and that solved the problem.
>>>>
>>>>     i just had to notice that the bash recipe downloaded a number of
>>>> patch files, then add them to my local mirror.
>>>
>>> Sorry, I misread.  Yes, you do need those patch files in your local 
>>> mirror.
>>> In fact, if you look at ${BUILD}/downloads, every _file_ (not 
>>> symlink) that's
>>> at the top level needs to be copied.  I manage this by running a script
>>> like this:
>>>    (cd downloads;find . -type f -maxdepth 1 | tar -cT - | tar -x -C 
>>> ${DL_DIR})
>>
>>    but doesn't that pick up all the ".done" files as well?  doesn't
>> hurt, but certainly unnecessary.
>
> I said "something like" - add in "| grep -v .done" if you want to skip 
> those.
> I actually use a more complex script as there are some things I simply 
> never
> want to copy...
>
I added a small how-do-i on this subject, since I have not found it 
documented elsewhere.
https://wiki.yoctoproject.org/wiki/How_do_I

Please remove/add if something is incorrect.

Br,
David