* sharing DL_DIR?
@ 2012-02-22 23:33 Rich Pixley
2012-02-22 23:39 ` Tom King
2012-02-22 23:41 ` Saul Wold
0 siblings, 2 replies; 11+ messages in thread
From: Rich Pixley @ 2012-02-22 23:33 UTC (permalink / raw)
To: Patches and discussions about the oe-core layer
[-- Attachment #1: Type: text/plain, Size: 1271 bytes --]
Is it reasonable to expect to share DL_DIR between multiple builds?
That is, are downloads properly locked so that multiple concurrent
downloads of the same file won't collide? And if so, are they NFS safe
locks?
Or must each build download it's own copies of every component?
I ask because our, (Palm), branch has been, but it's a bit of a nuisance
to pick a suitable locking mechanism that is both functional and performant.
* Flock/lockf aren't reliably supported in heterogeneous environments.
* The old lockfiles don't work over NFS. Lock directories apparently
do, but trying to clean these up after interruptions or failures is
a losing battle so if we do use these, we end up with orphan locks
periodically.
* With NFS as a possibility, we can't assume any kernel local IPC
mechanism, so sysV ipc is out.
* It seems like a pretty huge overhead to try to create any sort of
zeroconf overhead.
Our solution so far has been to use NFS lock directories for multiple
machine builds and flock/lockf for single machine builds. This mostly
works, though it's not ideal, and it's significantly faster than
downloading/copying all of the component source multiple times even over
local mirrors.
--rich
[-- Attachment #2: Type: text/html, Size: 1660 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread* Re: sharing DL_DIR?
2012-02-22 23:33 sharing DL_DIR? Rich Pixley
@ 2012-02-22 23:39 ` Tom King
2012-02-22 23:41 ` Christopher Larson
2012-02-22 23:41 ` Saul Wold
1 sibling, 1 reply; 11+ messages in thread
From: Tom King @ 2012-02-22 23:39 UTC (permalink / raw)
To: Patches and discussions about the oe-core layer
[-- Attachment #1: Type: text/plain, Size: 1783 bytes --]
I have a single downloads directory with symlinks for every target build I
have to that directory. As long as I have the source already in there I
don't fetch it again
Tom
On Wed, Feb 22, 2012 at 1:33 PM, Rich Pixley <rich.pixley@palm.com> wrote:
> Is it reasonable to expect to share DL_DIR between multiple builds?
>
> That is, are downloads properly locked so that multiple concurrent
> downloads of the same file won't collide? And if so, are they NFS safe
> locks?
>
> Or must each build download it's own copies of every component?
>
> I ask because our, (Palm), branch has been, but it's a bit of a nuisance
> to pick a suitable locking mechanism that is both functional and performant.
>
> - Flock/lockf aren't reliably supported in heterogeneous environments.
> - The old lockfiles don't work over NFS. Lock directories apparently
> do, but trying to clean these up after interruptions or failures is a
> losing battle so if we do use these, we end up with orphan locks
> periodically.
> - With NFS as a possibility, we can't assume any kernel local IPC
> mechanism, so sysV ipc is out.
> - It seems like a pretty huge overhead to try to create any sort of
> zeroconf overhead.
>
> Our solution so far has been to use NFS lock directories for multiple
> machine builds and flock/lockf for single machine builds. This mostly
> works, though it's not ideal, and it's significantly faster than
> downloading/copying all of the component source multiple times even over
> local mirrors.
> --rich
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>
>
[-- Attachment #2: Type: text/html, Size: 2591 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: sharing DL_DIR?
2012-02-22 23:39 ` Tom King
@ 2012-02-22 23:41 ` Christopher Larson
2012-02-23 0:47 ` Rich Pixley
0 siblings, 1 reply; 11+ messages in thread
From: Christopher Larson @ 2012-02-22 23:41 UTC (permalink / raw)
To: Patches and discussions about the oe-core layer
[-- Attachment #1: Type: text/plain, Size: 2380 bytes --]
I share it as well. I don't see locks being a big issue for most things, as the worst case is a re-download, not use of a corrupt or incomplete file, due to the completion stamps.
--
Christopher Larson
On Wednesday, February 22, 2012 at 4:39 PM, Tom King wrote:
> I have a single downloads directory with symlinks for every target build I have to that directory. As long as I have the source already in there I don't fetch it again
>
> Tom
>
> On Wed, Feb 22, 2012 at 1:33 PM, Rich Pixley <rich.pixley@palm.com (mailto:rich.pixley@palm.com)> wrote:
> > Is it reasonable to expect to share DL_DIR between multiple builds?
> >
> > That is, are downloads properly locked so that multiple concurrent downloads of the same file won't collide? And if so, are they NFS safe locks?
> >
> > Or must each build download it's own copies of every component?
> >
> > I ask because our, (Palm), branch has been, but it's a bit of a nuisance to pick a suitable locking mechanism that is both functional and performant.
> > Flock/lockf aren't reliably supported in heterogeneous environments.
> > The old lockfiles don't work over NFS. Lock directories apparently do, but trying to clean these up after interruptions or failures is a losing battle so if we do use these, we end up with orphan locks periodically.
> > With NFS as a possibility, we can't assume any kernel local IPC mechanism, so sysV ipc is out.
> > It seems like a pretty huge overhead to try to create any sort of zeroconf overhead.
> >
> > Our solution so far has been to use NFS lock directories for multiple machine builds and flock/lockf for single machine builds. This mostly works, though it's not ideal, and it's significantly faster than downloading/copying all of the component source multiple times even over local mirrors.
> > --rich
> >
> > _______________________________________________
> > Openembedded-core mailing list
> > Openembedded-core@lists.openembedded.org (mailto:Openembedded-core@lists.openembedded.org)
> > http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
> >
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org (mailto:Openembedded-core@lists.openembedded.org)
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>
>
[-- Attachment #2: Type: text/html, Size: 3730 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: sharing DL_DIR?
2012-02-22 23:41 ` Christopher Larson
@ 2012-02-23 0:47 ` Rich Pixley
2012-02-23 10:16 ` Richard Purdie
0 siblings, 1 reply; 11+ messages in thread
From: Rich Pixley @ 2012-02-23 0:47 UTC (permalink / raw)
To: openembedded-core
[-- Attachment #1: Type: text/plain, Size: 2698 bytes --]
What happens if two separate bitbake invocations both simultaneously
attempt to download the same file?
--rich
On 2/22/12 15:41 , Christopher Larson wrote:
> I share it as well. I don't see locks being a big issue for most
> things, as the worst case is a re-download, not use of a corrupt or
> incomplete file, due to the completion stamps.
>
> --
> Christopher Larson
>
> On Wednesday, February 22, 2012 at 4:39 PM, Tom King wrote:
>
>> I have a single downloads directory with symlinks for every target
>> build I have to that directory. As long as I have the source already
>> in there I don't fetch it again
>>
>> Tom
>>
>> On Wed, Feb 22, 2012 at 1:33 PM, Rich Pixley <rich.pixley@palm.com
>> <mailto:rich.pixley@palm.com>> wrote:
>>> Is it reasonable to expect to share DL_DIR between multiple builds?
>>>
>>> That is, are downloads properly locked so that multiple concurrent
>>> downloads of the same file won't collide? And if so, are they NFS
>>> safe locks?
>>>
>>> Or must each build download it's own copies of every component?
>>>
>>> I ask because our, (Palm), branch has been, but it's a bit of a
>>> nuisance to pick a suitable locking mechanism that is both
>>> functional and performant.
>>>
>>> * Flock/lockf aren't reliably supported in heterogeneous environments.
>>> * The old lockfiles don't work over NFS. Lock directories
>>> apparently do, but trying to clean these up after interruptions
>>> or failures is a losing battle so if we do use these, we end up
>>> with orphan locks periodically.
>>> * With NFS as a possibility, we can't assume any kernel local IPC
>>> mechanism, so sysV ipc is out.
>>> * It seems like a pretty huge overhead to try to create any sort
>>> of zeroconf overhead.
>>>
>>> Our solution so far has been to use NFS lock directories for
>>> multiple machine builds and flock/lockf for single machine builds.
>>> This mostly works, though it's not ideal, and it's significantly
>>> faster than downloading/copying all of the component source multiple
>>> times even over local mirrors.
>>>
>>> --rich
>>>
>>> _______________________________________________
>>> Openembedded-core mailing list
>>> Openembedded-core@lists.openembedded.org
>>> <mailto:Openembedded-core@lists.openembedded.org>
>>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>>>
>>
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> <mailto:Openembedded-core@lists.openembedded.org>
>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>
[-- Attachment #2: Type: text/html, Size: 5775 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: sharing DL_DIR?
2012-02-23 0:47 ` Rich Pixley
@ 2012-02-23 10:16 ` Richard Purdie
2012-02-23 19:21 ` Rich Pixley
0 siblings, 1 reply; 11+ messages in thread
From: Richard Purdie @ 2012-02-23 10:16 UTC (permalink / raw)
To: Patches and discussions about the oe-core layer
On Wed, 2012-02-22 at 16:47 -0800, Rich Pixley wrote:
> What happens if two separate bitbake invocations both simultaneously
> attempt to download the same file?
The first will download it, the second will wait for it to finish, then
both will continue with the downloaded file.
Cheers,
Richard
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: sharing DL_DIR?
2012-02-23 10:16 ` Richard Purdie
@ 2012-02-23 19:21 ` Rich Pixley
2012-02-23 20:11 ` Richard Purdie
0 siblings, 1 reply; 11+ messages in thread
From: Rich Pixley @ 2012-02-23 19:21 UTC (permalink / raw)
To: openembedded-core
On 2/23/12 02:16 , Richard Purdie wrote:
> On Wed, 2012-02-22 at 16:47 -0800, Rich Pixley wrote:
>> What happens if two separate bitbake invocations both simultaneously
>> attempt to download the same file?
> The first will download it, the second will wait for it to finish, then
> both will continue with the downloaded file.
How does the second know that the first is in progress?
--rich
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: sharing DL_DIR?
2012-02-23 19:21 ` Rich Pixley
@ 2012-02-23 20:11 ` Richard Purdie
2012-02-23 20:36 ` Rich Pixley
0 siblings, 1 reply; 11+ messages in thread
From: Richard Purdie @ 2012-02-23 20:11 UTC (permalink / raw)
To: Patches and discussions about the oe-core layer
On Thu, 2012-02-23 at 11:21 -0800, Rich Pixley wrote:
> On 2/23/12 02:16 , Richard Purdie wrote:
> > On Wed, 2012-02-22 at 16:47 -0800, Rich Pixley wrote:
> >> What happens if two separate bitbake invocations both simultaneously
> >> attempt to download the same file?
> > The first will download it, the second will wait for it to finish, then
> > both will continue with the downloaded file.
> How does the second know that the first is in progress?
It won't be able to take the lock and the done stamp file doesn't exist
yet.
Cheers,
Richard
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: sharing DL_DIR?
2012-02-23 20:11 ` Richard Purdie
@ 2012-02-23 20:36 ` Rich Pixley
2012-02-23 20:41 ` Richard Purdie
0 siblings, 1 reply; 11+ messages in thread
From: Rich Pixley @ 2012-02-23 20:36 UTC (permalink / raw)
To: openembedded-core
On 2/23/12 12:11 , Richard Purdie wrote:
> On Thu, 2012-02-23 at 11:21 -0800, Rich Pixley wrote:
>> On 2/23/12 02:16 , Richard Purdie wrote:
>>> On Wed, 2012-02-22 at 16:47 -0800, Rich Pixley wrote:
>>>> What happens if two separate bitbake invocations both simultaneously
>>>> attempt to download the same file?
>>> The first will download it, the second will wait for it to finish, then
>>> both will continue with the downloaded file.
>> How does the second know that the first is in progress?
> It won't be able to take the lock and the done stamp file doesn't exist
> yet.
What lock?
I've just read through lib/bb/fetch2/__init__.py but I'm not seeing a
lock. What am I missing?
--rich
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: sharing DL_DIR?
2012-02-23 20:36 ` Rich Pixley
@ 2012-02-23 20:41 ` Richard Purdie
0 siblings, 0 replies; 11+ messages in thread
From: Richard Purdie @ 2012-02-23 20:41 UTC (permalink / raw)
To: Patches and discussions about the oe-core layer
On Thu, 2012-02-23 at 12:36 -0800, Rich Pixley wrote:
> On 2/23/12 12:11 , Richard Purdie wrote:
> > On Thu, 2012-02-23 at 11:21 -0800, Rich Pixley wrote:
> >> On 2/23/12 02:16 , Richard Purdie wrote:
> >>> On Wed, 2012-02-22 at 16:47 -0800, Rich Pixley wrote:
> >>>> What happens if two separate bitbake invocations both simultaneously
> >>>> attempt to download the same file?
> >>> The first will download it, the second will wait for it to finish, then
> >>> both will continue with the downloaded file.
> >> How does the second know that the first is in progress?
> > It won't be able to take the lock and the done stamp file doesn't exist
> > yet.
> What lock?
>
> I've just read through lib/bb/fetch2/__init__.py but I'm not seeing a
> lock. What am I missing?
Try searching that file for expressions like "lock" and "lockfile" ;-)
Cheers,
Richard
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: sharing DL_DIR?
2012-02-22 23:33 sharing DL_DIR? Rich Pixley
2012-02-22 23:39 ` Tom King
@ 2012-02-22 23:41 ` Saul Wold
2012-02-23 0:50 ` Rich Pixley
1 sibling, 1 reply; 11+ messages in thread
From: Saul Wold @ 2012-02-22 23:41 UTC (permalink / raw)
To: Patches and discussions about the oe-core layer
On 02/22/2012 03:33 PM, Rich Pixley wrote:
> Is it reasonable to expect to share DL_DIR between multiple builds?
>
Absolutely, We share DL_DIR via NFS across multiple machines and builds.
> That is, are downloads properly locked so that multiple concurrent
> downloads of the same file won't collide? And if so, are they NFS safe
> locks?
>
Yes, as far as I know.
> Or must each build download it's own copies of every component?
>
No need.
> I ask because our, (Palm), branch has been, but it's a bit of a nuisance
> to pick a suitable locking mechanism that is both functional and performant.
>
> * Flock/lockf aren't reliably supported in heterogeneous environments.
> * The old lockfiles don't work over NFS. Lock directories apparently
> do, but trying to clean these up after interruptions or failures is
> a losing battle so if we do use these, we end up with orphan locks
> periodically.
> * With NFS as a possibility, we can't assume any kernel local IPC
> mechanism, so sysV ipc is out.
> * It seems like a pretty huge overhead to try to create any sort of
> zeroconf overhead.
>
> Our solution so far has been to use NFS lock directories for multiple
> machine builds and flock/lockf for single machine builds. This mostly
> works, though it's not ideal, and it's significantly faster than
> downloading/copying all of the component source multiple times even over
> local mirrors.
>
I think we are using a very basic locking mechism that creates the
lockfile and a donefile when complete. It seems to work well enough.
Sau!
> --rich
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: sharing DL_DIR?
2012-02-22 23:41 ` Saul Wold
@ 2012-02-23 0:50 ` Rich Pixley
0 siblings, 0 replies; 11+ messages in thread
From: Rich Pixley @ 2012-02-23 0:50 UTC (permalink / raw)
To: Saul Wold; +Cc: Patches and discussions about the oe-core layer
Excellent, thanks. That's the news I was hoping for.
--rich
On 2/22/12 15:41 , Saul Wold wrote:
> On 02/22/2012 03:33 PM, Rich Pixley wrote:
>> Is it reasonable to expect to share DL_DIR between multiple builds?
>>
> Absolutely, We share DL_DIR via NFS across multiple machines and builds.
>
>> That is, are downloads properly locked so that multiple concurrent
>> downloads of the same file won't collide? And if so, are they NFS safe
>> locks?
>>
> Yes, as far as I know.
>
>> Or must each build download it's own copies of every component?
>>
> No need.
>
>> I ask because our, (Palm), branch has been, but it's a bit of a nuisance
>> to pick a suitable locking mechanism that is both functional and performant.
>>
>> * Flock/lockf aren't reliably supported in heterogeneous environments.
>> * The old lockfiles don't work over NFS. Lock directories apparently
>> do, but trying to clean these up after interruptions or failures is
>> a losing battle so if we do use these, we end up with orphan locks
>> periodically.
>> * With NFS as a possibility, we can't assume any kernel local IPC
>> mechanism, so sysV ipc is out.
>> * It seems like a pretty huge overhead to try to create any sort of
>> zeroconf overhead.
>>
>> Our solution so far has been to use NFS lock directories for multiple
>> machine builds and flock/lockf for single machine builds. This mostly
>> works, though it's not ideal, and it's significantly faster than
>> downloading/copying all of the component source multiple times even over
>> local mirrors.
>>
> I think we are using a very basic locking mechism that creates the
> lockfile and a donefile when complete. It seems to work well enough.
>
> Sau!
>
>> --rich
>>
>>
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2012-02-23 20:49 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-02-22 23:33 sharing DL_DIR? Rich Pixley
2012-02-22 23:39 ` Tom King
2012-02-22 23:41 ` Christopher Larson
2012-02-23 0:47 ` Rich Pixley
2012-02-23 10:16 ` Richard Purdie
2012-02-23 19:21 ` Rich Pixley
2012-02-23 20:11 ` Richard Purdie
2012-02-23 20:36 ` Rich Pixley
2012-02-23 20:41 ` Richard Purdie
2012-02-22 23:41 ` Saul Wold
2012-02-23 0:50 ` Rich Pixley
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox