Re: how to set up PREMIRRORS_prepend to avoid constant git clones?

[Gary Thomas <gary@mlbassoc.com>]

On 2012-03-10 03:35, Robert P. J. Day wrote:
>
>    over the years, i've built up a sizable collection of source
> tarballs for use in various projects -- top-level directory called
> "~/dl", about 5G in size with lots and lots of .tgz and .bz2 files and
> so on, and i'd like to confirm the best way to take advantage of all
> that source whenever i configure and build a yocto project.
>
>    based on my reading, whenever i create a new project, i immediately
> add the following to the bottom of its local.conf file:
>
> PREMIRRORS_prepend = "\
> git://.*/.* file:///home/rpjday/dl/ \n \
> svn://.*/.* file:///home/rpjday/dl/ \n \
> cvs://.*/.* file:///home/rpjday/dl/ \n \
> ftp://.*/.* file:///home/rpjday/dl/ \n \
> http://.*/.* file:///home/rpjday/dl/ \n \
> https://.*/.* file:///home/rpjday/dl/ \n"
>
> and that certainly makes a massive difference -- whenever the fetching
> starts, matching tarballs will be located in my "dl" directory, and
> the project's downloads directory will be populated with symlinks to
> the actual tarballs, as in:
>
>    vala-0.14.2.tar.xz ->  /home/rpjday/dl/vala-0.14.2.tar.xz
>
> and after the download, i sneak a peek at whatever newer tarballs
> needed to be downloaded, and quietly copy them over to "dl" to avoid
> ever downloading them again, and that's worked well so far.  so, the
> questions:
>
> 1) does this sound reasonable?  i know about using DL_DIR and perhaps
> sharing that across projects, but i rather like the idea of an
> application-agnostic repository of straight tarballs and zip files
> that any build system is allowed to pilfer from (especially since i
> can make it read-only).
>
> 2) am i missing any fetch protocols from that list?
>
> 3) how do i set it up to avoid repeated git clones?
>
>    it seems that, while i can avoid fetching simple tarballs, i still
> end up doing the same "git clone" operations over and over.  am i just
> missing something obvious?

All you really need are these lines:
   SOURCE_MIRROR_URL ?= "file://${COREBASE}/sources/"
   INHERIT += "own-mirrors"
Adjust the SOURCE_MIRROR_URL to wherever you want the files.  I also always add
   BB_GENERATE_MIRROR_TARBALLS = "1"
This will let the fetcher make a git2_XXX tarball, once the repo has been cloned.

My understanding is that the git (and other SCM) fetcher works like this:
   * If there is a git2_XXX tar file in the DL_DIR (or mirror), unpack it
     into ${BUILD}/downloads/git2
   * Check to see if the desired ${SRCREV} is present.  If not, do a 'git pull'
     to update the local clone and then rebuild the git2_XXX tarball.
This way, you only have to clone once - just copy any generated tarballs back
to your source mirror.

Finally, if you set
   BB_NO_NETWORK = "1"
the build will fail if bitbake *ever* has to go to the network, i.e. not find
the desired files in the source mirror or DL_DIR.  This is a great way to insure
that you've got what you need locally.  I use this all the time to make sure I
have a fully populated local cache of sources.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------