* tar ball vs. git development questions
@ 2012-01-22 20:19 James Abernathy
2012-01-23 1:12 ` Gary Thomas
0 siblings, 1 reply; 6+ messages in thread
From: James Abernathy @ 2012-01-22 20:19 UTC (permalink / raw)
To: yocto
[-- Attachment #1: Type: text/plain, Size: 1788 bytes --]
I have used both git and the tarball methods of bitbaking projects, all of them derivatives of the examples in the Yocto documentation. I was having issues using the local clone of the Yocto kernel git repository this weekend. I had successfully done that before, but I was rebuilding the PC workstation, and getting everything setup and tested some of the meta-intel BSPs to make sure I had everything right. Cloning the linux-yocto-3.0 repository was successful, but the bakes against it failed. I made sure I had poky-extras setup right, but I still had problems. To isolate the problem, I changed to building with the tarballs and everything worked fine.
So that got me thinking what are the differences between the 2 methods:
I assume that if I use the tarball method, bitbake, using the recipes, pulls down files from the online repositories and puts those files into the centralized local download directory ($DL_DIR), allowing reuse instead of re-downloading each time. The content downloaded for linux-yocto-3.0 is exactly what would be pulled from the local repository if I used a local clone of the git repository for linux-yocto-3.0.
If my assumption above is correct, if I'm not modifying the source code of the kernel (only changing config parameters), then once you've run at least one build with the tarball method, the $DL_DIR directory contains all the files you'll need to build any image with linux-yocto-3.0. So there is no need to have a local clone of the kernel repository for speeding up development. Am I right?
If I have a successful creation of a bare clone of linux-yocto-3.0.git, how could builds of Edison packages be failing? That makes me concerned about using git and successfully repeating builds of stable branches like Edison.
Jim A
[-- Attachment #2: Type: text/html, Size: 2082 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: tar ball vs. git development questions
2012-01-22 20:19 tar ball vs. git development questions James Abernathy
@ 2012-01-23 1:12 ` Gary Thomas
2012-01-23 12:51 ` jfabernathy
0 siblings, 1 reply; 6+ messages in thread
From: Gary Thomas @ 2012-01-23 1:12 UTC (permalink / raw)
To: James Abernathy; +Cc: yocto
On 2012-01-22 13:19, James Abernathy wrote:
> I have used both git and the tarball methods of bitbaking projects, all of them derivatives of the examples in the Yocto documentation. I was having issues using the local clone of
> the Yocto kernel git repository this weekend. I had successfully done that before, but I was rebuilding the PC workstation, and getting everything setup and tested some of the
> meta-intel BSPs to make sure I had everything right. Cloning the linux-yocto-3.0 repository was successful, but the bakes against it failed. I made sure I had poky-extras setup
> right, but I still had problems. To isolate the problem, I changed to building with the tarballs and everything worked fine.
>
> So that got me thinking what are the differences between the 2 methods:
>
> * I assume that if I use the tarball method, bitbake, using the recipes, pulls down files from the online repositories and puts those files into the centralized local download
> directory ($DL_DIR), allowing reuse instead of re-downloading each time. The content downloaded for linux-yocto-3.0 is exactly what would be pulled from the local repository if
> I used a local clone of the git repository for linux-yocto-3.0.
> * If my assumption above is correct, if I'm not modifying the source code of the kernel (only changing config parameters), then once you've run at least one build with the
> tarball method, the $DL_DIR directory contains all the files you'll need to build any image with linux-yocto-3.0. So there is no need to have a local clone of the kernel
> repository for speeding up development. Am I right?
> * If I have a successful creation of a bare clone of linux-yocto-3.0.git, how could builds of Edison packages be failing? That makes me concerned about using git and successfully
> repeating builds of stable branches like Edison.
If you set BB_GENERATE_MIRROR_TARBALLS = "1" (e.g. in local.conf)
then you'll get tarballs which hold the git repositories after
download. You can then reuse these (by sharing the DL_DIR or
using a local mirror). Does that help with the issue you're seeing?
--
------------------------------------------------------------
Gary Thomas | Consulting for the
MLB Associates | Embedded world
------------------------------------------------------------
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: tar ball vs. git development questions
2012-01-23 1:12 ` Gary Thomas
@ 2012-01-23 12:51 ` jfabernathy
2012-01-23 13:10 ` Gary Thomas
0 siblings, 1 reply; 6+ messages in thread
From: jfabernathy @ 2012-01-23 12:51 UTC (permalink / raw)
To: Gary Thomas; +Cc: yocto
On 01/22/2012 08:12 PM, Gary Thomas wrote:
> On 2012-01-22 13:19, James Abernathy wrote:
>> I have used both git and the tarball methods of bitbaking projects,
>> all of them derivatives of the examples in the Yocto documentation. I
>> was having issues using the local clone of
>> the Yocto kernel git repository this weekend. I had successfully done
>> that before, but I was rebuilding the PC workstation, and getting
>> everything setup and tested some of the
>> meta-intel BSPs to make sure I had everything right. Cloning the
>> linux-yocto-3.0 repository was successful, but the bakes against it
>> failed. I made sure I had poky-extras setup
>> right, but I still had problems. To isolate the problem, I changed to
>> building with the tarballs and everything worked fine.
>>
>> So that got me thinking what are the differences between the 2 methods:
>>
>> * I assume that if I use the tarball method, bitbake, using the
>> recipes, pulls down files from the online repositories and puts those
>> files into the centralized local download
>> directory ($DL_DIR), allowing reuse instead of re-downloading
>> each time. The content downloaded for linux-yocto-3.0 is exactly what
>> would be pulled from the local repository if
>> I used a local clone of the git repository for linux-yocto-3.0.
>> * If my assumption above is correct, if I'm not modifying the
>> source code of the kernel (only changing config parameters), then
>> once you've run at least one build with the
>> tarball method, the $DL_DIR directory contains all the files
>> you'll need to build any image with linux-yocto-3.0. So there is no
>> need to have a local clone of the kernel
>> repository for speeding up development. Am I right?
>> * If I have a successful creation of a bare clone of
>> linux-yocto-3.0.git, how could builds of Edison packages be failing?
>> That makes me concerned about using git and successfully
>> repeating builds of stable branches like Edison.
>
> If you set BB_GENERATE_MIRROR_TARBALLS = "1" (e.g. in local.conf)
> then you'll get tarballs which hold the git repositories after
> download. You can then reuse these (by sharing the DL_DIR or
> using a local mirror). Does that help with the issue you're seeing?
>
I'm not sure it does. I don't want poky to do more work. I have my
download directory, $DL_DIR, outside my build directory so I can keep it
run to run, as mentioned in the comments in local.conf. I was trying to
understand 2 basic questions:
1. what could be causing build failures using a freshly created bare
clone yesterday vs. using the poky-edison-6.0 tarball. It would be
scary if you could clone the linux-yocto-3.0 successfully one day and
have it be used in a build successfully, but clone it another day and it
not work. I figured that bitbake/poky pulled the same information into
DL_DIR regardless of whether you pulled from the bare clone locally or
straight from the on-line repository.
2. I was trying to look at items to speed up build runs. I thought that
if the downloads in DL_DIR were reused if they existed, it would speed
up a run. It seems to. So the question is, after the first build, the
files I need for linux-yocto-3.0 are in DL_DIR regardless of whether
they came from the online repository or the local bare clone. right?
Jim A
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: tar ball vs. git development questions
2012-01-23 12:51 ` jfabernathy
@ 2012-01-23 13:10 ` Gary Thomas
2012-01-23 14:01 ` Bruce Ashfield
0 siblings, 1 reply; 6+ messages in thread
From: Gary Thomas @ 2012-01-23 13:10 UTC (permalink / raw)
To: jfabernathy; +Cc: yocto
On 2012-01-23 05:51, jfabernathy wrote:
> On 01/22/2012 08:12 PM, Gary Thomas wrote:
>> On 2012-01-22 13:19, James Abernathy wrote:
>>> I have used both git and the tarball methods of bitbaking projects, all of them derivatives of the examples in the Yocto documentation. I was having issues using the local clone of
>>> the Yocto kernel git repository this weekend. I had successfully done that before, but I was rebuilding the PC workstation, and getting everything setup and tested some of the
>>> meta-intel BSPs to make sure I had everything right. Cloning the linux-yocto-3.0 repository was successful, but the bakes against it failed. I made sure I had poky-extras setup
>>> right, but I still had problems. To isolate the problem, I changed to building with the tarballs and everything worked fine.
>>>
>>> So that got me thinking what are the differences between the 2 methods:
>>>
>>> * I assume that if I use the tarball method, bitbake, using the recipes, pulls down files from the online repositories and puts those files into the centralized local download
>>> directory ($DL_DIR), allowing reuse instead of re-downloading each time. The content downloaded for linux-yocto-3.0 is exactly what would be pulled from the local repository if
>>> I used a local clone of the git repository for linux-yocto-3.0.
>>> * If my assumption above is correct, if I'm not modifying the source code of the kernel (only changing config parameters), then once you've run at least one build with the
>>> tarball method, the $DL_DIR directory contains all the files you'll need to build any image with linux-yocto-3.0. So there is no need to have a local clone of the kernel
>>> repository for speeding up development. Am I right?
>>> * If I have a successful creation of a bare clone of linux-yocto-3.0.git, how could builds of Edison packages be failing? That makes me concerned about using git and successfully
>>> repeating builds of stable branches like Edison.
>>
>> If you set BB_GENERATE_MIRROR_TARBALLS = "1" (e.g. in local.conf)
>> then you'll get tarballs which hold the git repositories after
>> download. You can then reuse these (by sharing the DL_DIR or
>> using a local mirror). Does that help with the issue you're seeing?
>>
> I'm not sure it does. I don't want poky to do more work. I have my download directory, $DL_DIR, outside my build directory so I can keep it run to run, as mentioned in the comments
> in local.conf. I was trying to understand 2 basic questions:
>
> 1. what could be causing build failures using a freshly created bare clone yesterday vs. using the poky-edison-6.0 tarball. It would be scary if you could clone the linux-yocto-3.0
> successfully one day and have it be used in a build successfully, but clone it another day and it not work. I figured that bitbake/poky pulled the same information into DL_DIR
> regardless of whether you pulled from the bare clone locally or straight from the on-line repository.
What kind of errors? Some details might help understand what the problem is.
> 2. I was trying to look at items to speed up build runs. I thought that if the downloads in DL_DIR were reused if they existed, it would speed up a run. It seems to. So the
> question is, after the first build, the files I need for linux-yocto-3.0 are in DL_DIR regardless of whether they came from the online repository or the local bare clone. right?
I think so, but I'm not 100% sure.
The way I have my system set up, I never download anything more
than once :-) I have a shared source repository which I point to
using MIRRORS and then all my builds are relative to that. If there
is new code, it gets downloaded and saved (as a tarball) in my sources
repository to be used by subsequent builds. To do this, I just have
these lines in <distro>.conf
# Provide pre-staged sources
SOURCE_MIRROR_URL ?= "file://${COREBASE}/sources/"
INHERIT += "own-mirrors"
BB_GENERATE_MIRROR_TARBALLS = "1"
I don't define DL_DIR in local.conf so files only come from my mirror.
This process is so successful that I almost always run with BB_NO_NETWORK="1"
which causes the fetcher to die if it can't find the file locally. If I find
that I do need to download something new, I disable this and let the fetcher
download it. I'll then have a saved tarball (either downloaded directly or
synthesized) in <build_dir>/downloads that I can push to my mirror.
--
------------------------------------------------------------
Gary Thomas | Consulting for the
MLB Associates | Embedded world
------------------------------------------------------------
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: tar ball vs. git development questions
2012-01-23 13:10 ` Gary Thomas
@ 2012-01-23 14:01 ` Bruce Ashfield
2012-01-24 21:24 ` James Abernathy
0 siblings, 1 reply; 6+ messages in thread
From: Bruce Ashfield @ 2012-01-23 14:01 UTC (permalink / raw)
To: Gary Thomas; +Cc: yocto
On 12-01-23 08:10 AM, Gary Thomas wrote:
> On 2012-01-23 05:51, jfabernathy wrote:
>> On 01/22/2012 08:12 PM, Gary Thomas wrote:
>>> On 2012-01-22 13:19, James Abernathy wrote:
>>>> I have used both git and the tarball methods of bitbaking projects,
>>>> all of them derivatives of the examples in the Yocto documentation.
>>>> I was having issues using the local clone of
>>>> the Yocto kernel git repository this weekend. I had successfully
>>>> done that before, but I was rebuilding the PC workstation, and
>>>> getting everything setup and tested some of the
>>>> meta-intel BSPs to make sure I had everything right. Cloning the
>>>> linux-yocto-3.0 repository was successful, but the bakes against it
>>>> failed. I made sure I had poky-extras setup
>>>> right, but I still had problems. To isolate the problem, I changed
>>>> to building with the tarballs and everything worked fine.
>>>>
>>>> So that got me thinking what are the differences between the 2 methods:
>>>>
>>>> * I assume that if I use the tarball method, bitbake, using the
>>>> recipes, pulls down files from the online repositories and puts
>>>> those files into the centralized local download
>>>> directory ($DL_DIR), allowing reuse instead of re-downloading each
>>>> time. The content downloaded for linux-yocto-3.0 is exactly what
>>>> would be pulled from the local repository if
>>>> I used a local clone of the git repository for linux-yocto-3.0.
>>>> * If my assumption above is correct, if I'm not modifying the source
>>>> code of the kernel (only changing config parameters), then once
>>>> you've run at least one build with the
>>>> tarball method, the $DL_DIR directory contains all the files you'll
>>>> need to build any image with linux-yocto-3.0. So there is no need to
>>>> have a local clone of the kernel
>>>> repository for speeding up development. Am I right?
>>>> * If I have a successful creation of a bare clone of
>>>> linux-yocto-3.0.git, how could builds of Edison packages be failing?
>>>> That makes me concerned about using git and successfully
>>>> repeating builds of stable branches like Edison.
>>>
>>> If you set BB_GENERATE_MIRROR_TARBALLS = "1" (e.g. in local.conf)
>>> then you'll get tarballs which hold the git repositories after
>>> download. You can then reuse these (by sharing the DL_DIR or
>>> using a local mirror). Does that help with the issue you're seeing?
>>>
>> I'm not sure it does. I don't want poky to do more work. I have my
>> download directory, $DL_DIR, outside my build directory so I can keep
>> it run to run, as mentioned in the comments
>> in local.conf. I was trying to understand 2 basic questions:
>>
>> 1. what could be causing build failures using a freshly created bare
>> clone yesterday vs. using the poky-edison-6.0 tarball. It would be
>> scary if you could clone the linux-yocto-3.0
>> successfully one day and have it be used in a build successfully, but
>> clone it another day and it not work. I figured that bitbake/poky
>> pulled the same information into DL_DIR
>> regardless of whether you pulled from the bare clone locally or
>> straight from the on-line repository.
>
> What kind of errors? Some details might help understand what the problem
> is.
>
>> 2. I was trying to look at items to speed up build runs. I thought
>> that if the downloads in DL_DIR were reused if they existed, it would
>> speed up a run. It seems to. So the
>> question is, after the first build, the files I need for
>> linux-yocto-3.0 are in DL_DIR regardless of whether they came from the
>> online repository or the local bare clone. right?
>
> I think so, but I'm not 100% sure.
From my experience, this is true. Regardless of the SRCI_URI, the
downloads directory contains everything that poky/bitbake need to
do builds after the initial fetch has been performed. And that's
where subsequent updates are pullled.
Cheers,
Bruce
>
> The way I have my system set up, I never download anything more
> than once :-) I have a shared source repository which I point to
> using MIRRORS and then all my builds are relative to that. If there
> is new code, it gets downloaded and saved (as a tarball) in my sources
> repository to be used by subsequent builds. To do this, I just have
> these lines in <distro>.conf
> # Provide pre-staged sources
> SOURCE_MIRROR_URL ?= "file://${COREBASE}/sources/"
> INHERIT += "own-mirrors"
> BB_GENERATE_MIRROR_TARBALLS = "1"
> I don't define DL_DIR in local.conf so files only come from my mirror.
>
> This process is so successful that I almost always run with
> BB_NO_NETWORK="1"
> which causes the fetcher to die if it can't find the file locally. If I
> find
> that I do need to download something new, I disable this and let the
> fetcher
> download it. I'll then have a saved tarball (either downloaded directly or
> synthesized) in <build_dir>/downloads that I can push to my mirror.
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: tar ball vs. git development questions
2012-01-23 14:01 ` Bruce Ashfield
@ 2012-01-24 21:24 ` James Abernathy
0 siblings, 0 replies; 6+ messages in thread
From: James Abernathy @ 2012-01-24 21:24 UTC (permalink / raw)
To: Bruce Ashfield; +Cc: yocto
On Jan 23, 2012, at 9:01 AM, Bruce Ashfield wrote:
> On 12-01-23 08:10 AM, Gary Thomas wrote:
>> On 2012-01-23 05:51, jfabernathy wrote:
>>> On 01/22/2012 08:12 PM, Gary Thomas wrote:
>>>> On 2012-01-22 13:19, James Abernathy wrote:
>>>>> I have used both git and the tarball methods of bitbaking projects,
>>>>> all of them derivatives of the examples in the Yocto documentation.
>>>>> I was having issues using the local clone of
>>>>> the Yocto kernel git repository this weekend. I had successfully
>>>>> done that before, but I was rebuilding the PC workstation, and
>>>>> getting everything setup and tested some of the
>>>>> meta-intel BSPs to make sure I had everything right. Cloning the
>>>>> linux-yocto-3.0 repository was successful, but the bakes against it
>>>>> failed. I made sure I had poky-extras setup
>>>>> right, but I still had problems. To isolate the problem, I changed
>>>>> to building with the tarballs and everything worked fine.
>>>>>
>>>>> So that got me thinking what are the differences between the 2 methods:
>>>>>
>>>>> * I assume that if I use the tarball method, bitbake, using the
>>>>> recipes, pulls down files from the online repositories and puts
>>>>> those files into the centralized local download
>>>>> directory ($DL_DIR), allowing reuse instead of re-downloading each
>>>>> time. The content downloaded for linux-yocto-3.0 is exactly what
>>>>> would be pulled from the local repository if
>>>>> I used a local clone of the git repository for linux-yocto-3.0.
>>>>> * If my assumption above is correct, if I'm not modifying the source
>>>>> code of the kernel (only changing config parameters), then once
>>>>> you've run at least one build with the
>>>>> tarball method, the $DL_DIR directory contains all the files you'll
>>>>> need to build any image with linux-yocto-3.0. So there is no need to
>>>>> have a local clone of the kernel
>>>>> repository for speeding up development. Am I right?
>>>>> * If I have a successful creation of a bare clone of
>>>>> linux-yocto-3.0.git, how could builds of Edison packages be failing?
>>>>> That makes me concerned about using git and successfully
>>>>> repeating builds of stable branches like Edison.
>>>>
>>>> If you set BB_GENERATE_MIRROR_TARBALLS = "1" (e.g. in local.conf)
>>>> then you'll get tarballs which hold the git repositories after
>>>> download. You can then reuse these (by sharing the DL_DIR or
>>>> using a local mirror). Does that help with the issue you're seeing?
>>>>
>>> I'm not sure it does. I don't want poky to do more work. I have my
>>> download directory, $DL_DIR, outside my build directory so I can keep
>>> it run to run, as mentioned in the comments
>>> in local.conf. I was trying to understand 2 basic questions:
>>>
>>> 1. what could be causing build failures using a freshly created bare
>>> clone yesterday vs. using the poky-edison-6.0 tarball. It would be
>>> scary if you could clone the linux-yocto-3.0
>>> successfully one day and have it be used in a build successfully, but
>>> clone it another day and it not work. I figured that bitbake/poky
>>> pulled the same information into DL_DIR
>>> regardless of whether you pulled from the bare clone locally or
>>> straight from the on-line repository.
>>
>> What kind of errors? Some details might help understand what the problem
>> is.
>>
>>> 2. I was trying to look at items to speed up build runs. I thought
>>> that if the downloads in DL_DIR were reused if they existed, it would
>>> speed up a run. It seems to. So the
>>> question is, after the first build, the files I need for
>>> linux-yocto-3.0 are in DL_DIR regardless of whether they came from the
>>> online repository or the local bare clone. right?
>>
>> I think so, but I'm not 100% sure.
>
> From my experience, this is true. Regardless of the SRCI_URI, the
> downloads directory contains everything that poky/bitbake need to
> do builds after the initial fetch has been performed. And that's
> where subsequent updates are pullled.
>
> Cheers,
>
> Bruce
>
I'm going to retest all this and try some of these tricks when I have some time. If I get any more errors with the local repository, I'll post them.
Thanks, Jim A
>>
>> The way I have my system set up, I never download anything more
>> than once :-) I have a shared source repository which I point to
>> using MIRRORS and then all my builds are relative to that. If there
>> is new code, it gets downloaded and saved (as a tarball) in my sources
>> repository to be used by subsequent builds. To do this, I just have
>> these lines in <distro>.conf
>> # Provide pre-staged sources
>> SOURCE_MIRROR_URL ?= "file://${COREBASE}/sources/"
>> INHERIT += "own-mirrors"
>> BB_GENERATE_MIRROR_TARBALLS = "1"
>> I don't define DL_DIR in local.conf so files only come from my mirror.
>>
>> This process is so successful that I almost always run with
>> BB_NO_NETWORK="1"
>> which causes the fetcher to die if it can't find the file locally. If I
>> find
>> that I do need to download something new, I disable this and let the
>> fetcher
>> download it. I'll then have a saved tarball (either downloaded directly or
>> synthesized) in <build_dir>/downloads that I can push to my mirror.
>>
>
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2012-01-24 21:24 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-01-22 20:19 tar ball vs. git development questions James Abernathy
2012-01-23 1:12 ` Gary Thomas
2012-01-23 12:51 ` jfabernathy
2012-01-23 13:10 ` Gary Thomas
2012-01-23 14:01 ` Bruce Ashfield
2012-01-24 21:24 ` James Abernathy
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.