linux-yocto fetch bottleneck

linux-yocto fetch bottleneck

[Jate S]

I am running into a bottleneck with the linux-yocto recipe on dora
1.5.1. The fetch retrieves git2_git.yoctoproject.org.git.tar.gz, but
then it hits a bottleneck when it must do a git fetch to update to the
required references. The git server does not have high bandwdith.

One workaround that I've found is to archive the git repository and
replace the tar.gz in the downloads directory. The recipe then finds
the required references.

Would BB_GENERATE_MIRROR_TARBALLS actually take care of this? Any
other recommendations?


- Jate S.

Re: linux-yocto fetch bottleneck

[Gary Thomas]

On 2014-03-21 15:27, Jate S wrote:
> I am running into a bottleneck with the linux-yocto recipe on dora
> 1.5.1. The fetch retrieves git2_git.yoctoproject.org.git.tar.gz, but
> then it hits a bottleneck when it must do a git fetch to update to the
> required references. The git server does not have high bandwdith.
> 
> One workaround that I've found is to archive the git repository and
> replace the tar.gz in the downloads directory. The recipe then finds
> the required references.
> 
> Would BB_GENERATE_MIRROR_TARBALLS actually take care of this? Any
> other recommendations?

Yes, that's exactly what BB_GENERATE_MIRROR_TARBALLS does.  The updated
tarball will end up in your downloads directory.  If you use any sort of
local mirror, you'll want to push the new tarball to it for the next time.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: linux-yocto fetch bottleneck

[Gary Thomas]

On 2014-03-21 15:54, Jate S wrote:
> The documentation has warnings of space usage. Will it start using
> lots of space? If so, can I use it on a per mirror basis?

Yes, the tarballs can be quite large, but they are incredibly useful
with a local mirror.  Especially when you consider that updating a GIT
checkout, e.g. the Linux kernel. can take a *really* long time.

> I am using a local mirror, so maybe the space usage will not be a problem.

The space needed will not be terribly different than what you've already
encountered without generating the tarballs.

n.b. it's polite to keep replies on the mailing list so that everyone benefits.

> On Fri, Mar 21, 2014 at 5:33 PM, Gary Thomas <gary@mlbassoc.com> wrote:
>> On 2014-03-21 15:27, Jate S wrote:
>>> I am running into a bottleneck with the linux-yocto recipe on dora
>>> 1.5.1. The fetch retrieves git2_git.yoctoproject.org.git.tar.gz, but
>>> then it hits a bottleneck when it must do a git fetch to update to the
>>> required references. The git server does not have high bandwdith.
>>>
>>> One workaround that I've found is to archive the git repository and
>>> replace the tar.gz in the downloads directory. The recipe then finds
>>> the required references.
>>>
>>> Would BB_GENERATE_MIRROR_TARBALLS actually take care of this? Any
>>> other recommendations?
>>
>> Yes, that's exactly what BB_GENERATE_MIRROR_TARBALLS does.  The updated
>> tarball will end up in your downloads directory.  If you use any sort of
>> local mirror, you'll want to push the new tarball to it for the next time.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: linux-yocto fetch bottleneck

[Jate S]

What is the name of the mirror tarball that's created? I am trying to
see if it worked. I saw no log messages stating "Creating tarball of
git repository".


On Fri, Mar 21, 2014 at 6:04 PM, Gary Thomas <gary@mlbassoc.com> wrote:
> On 2014-03-21 15:54, Jate S wrote:
>> The documentation has warnings of space usage. Will it start using
>> lots of space? If so, can I use it on a per mirror basis?
>
> Yes, the tarballs can be quite large, but they are incredibly useful
> with a local mirror.  Especially when you consider that updating a GIT
> checkout, e.g. the Linux kernel. can take a *really* long time.
>
>> I am using a local mirror, so maybe the space usage will not be a problem.
>
> The space needed will not be terribly different than what you've already
> encountered without generating the tarballs.
>
> n.b. it's polite to keep replies on the mailing list so that everyone benefits.
>
>> On Fri, Mar 21, 2014 at 5:33 PM, Gary Thomas <gary@mlbassoc.com> wrote:
>>> On 2014-03-21 15:27, Jate S wrote:
>>>> I am running into a bottleneck with the linux-yocto recipe on dora
>>>> 1.5.1. The fetch retrieves git2_git.yoctoproject.org.git.tar.gz, but
>>>> then it hits a bottleneck when it must do a git fetch to update to the
>>>> required references. The git server does not have high bandwdith.
>>>>
>>>> One workaround that I've found is to archive the git repository and
>>>> replace the tar.gz in the downloads directory. The recipe then finds
>>>> the required references.
>>>>
>>>> Would BB_GENERATE_MIRROR_TARBALLS actually take care of this? Any
>>>> other recommendations?
>>>
>>> Yes, that's exactly what BB_GENERATE_MIRROR_TARBALLS does.  The updated
>>> tarball will end up in your downloads directory.  If you use any sort of
>>> local mirror, you'll want to push the new tarball to it for the next time.
>
> --
> ------------------------------------------------------------
> Gary Thomas                 |  Consulting for the
> MLB Associates              |    Embedded world
> ------------------------------------------------------------

Re: linux-yocto fetch bottleneck

[Gary Thomas]

On 2014-03-24 17:18, Jate S wrote:
> What is the name of the mirror tarball that's created? I am trying to
> see if it worked. I saw no log messages stating "Creating tarball of
> git repository".

It will look something like this (for the opkg-utils_git recipe)
  downloads/git2_git.yoctoproject.org.opkg-utils.tar.gz

You should then  arrange to move/copy any of these tarballs to your mirror
so you benefit from this the next time around (when building in a new/different
build tree)

> On Fri, Mar 21, 2014 at 6:04 PM, Gary Thomas <gary@mlbassoc.com> wrote:
>> On 2014-03-21 15:54, Jate S wrote:
>>> The documentation has warnings of space usage. Will it start using
>>> lots of space? If so, can I use it on a per mirror basis?
>>
>> Yes, the tarballs can be quite large, but they are incredibly useful
>> with a local mirror.  Especially when you consider that updating a GIT
>> checkout, e.g. the Linux kernel. can take a *really* long time.
>>
>>> I am using a local mirror, so maybe the space usage will not be a problem.
>>
>> The space needed will not be terribly different than what you've already
>> encountered without generating the tarballs.
>>
>> n.b. it's polite to keep replies on the mailing list so that everyone benefits.
>>
>>> On Fri, Mar 21, 2014 at 5:33 PM, Gary Thomas <gary@mlbassoc.com> wrote:
>>>> On 2014-03-21 15:27, Jate S wrote:
>>>>> I am running into a bottleneck with the linux-yocto recipe on dora
>>>>> 1.5.1. The fetch retrieves git2_git.yoctoproject.org.git.tar.gz, but
>>>>> then it hits a bottleneck when it must do a git fetch to update to the
>>>>> required references. The git server does not have high bandwdith.
>>>>>
>>>>> One workaround that I've found is to archive the git repository and
>>>>> replace the tar.gz in the downloads directory. The recipe then finds
>>>>> the required references.
>>>>>
>>>>> Would BB_GENERATE_MIRROR_TARBALLS actually take care of this? Any
>>>>> other recommendations?
>>>>
>>>> Yes, that's exactly what BB_GENERATE_MIRROR_TARBALLS does.  The updated
>>>> tarball will end up in your downloads directory.  If you use any sort of
>>>> local mirror, you'll want to push the new tarball to it for the next time.
>>
>> --
>> ------------------------------------------------------------
>> Gary Thomas                 |  Consulting for the
>> MLB Associates              |    Embedded world
>> ------------------------------------------------------------

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------