From: Saul Wold <sgw@linux.intel.com>
To: Patches and discussions about the oe-core layer
<openembedded-core@lists.openembedded.org>
Subject: Re: sharing DL_DIR?
Date: Wed, 22 Feb 2012 15:41:14 -0800 [thread overview]
Message-ID: <4F457D1A.8010707@linux.intel.com> (raw)
In-Reply-To: <4F457B4B.4000507@palm.com>
On 02/22/2012 03:33 PM, Rich Pixley wrote:
> Is it reasonable to expect to share DL_DIR between multiple builds?
>
Absolutely, We share DL_DIR via NFS across multiple machines and builds.
> That is, are downloads properly locked so that multiple concurrent
> downloads of the same file won't collide? And if so, are they NFS safe
> locks?
>
Yes, as far as I know.
> Or must each build download it's own copies of every component?
>
No need.
> I ask because our, (Palm), branch has been, but it's a bit of a nuisance
> to pick a suitable locking mechanism that is both functional and performant.
>
> * Flock/lockf aren't reliably supported in heterogeneous environments.
> * The old lockfiles don't work over NFS. Lock directories apparently
> do, but trying to clean these up after interruptions or failures is
> a losing battle so if we do use these, we end up with orphan locks
> periodically.
> * With NFS as a possibility, we can't assume any kernel local IPC
> mechanism, so sysV ipc is out.
> * It seems like a pretty huge overhead to try to create any sort of
> zeroconf overhead.
>
> Our solution so far has been to use NFS lock directories for multiple
> machine builds and flock/lockf for single machine builds. This mostly
> works, though it's not ideal, and it's significantly faster than
> downloading/copying all of the component source multiple times even over
> local mirrors.
>
I think we are using a very basic locking mechism that creates the
lockfile and a donefile when complete. It seems to work well enough.
Sau!
> --rich
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
next prev parent reply other threads:[~2012-02-22 23:49 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-02-22 23:33 sharing DL_DIR? Rich Pixley
2012-02-22 23:39 ` Tom King
2012-02-22 23:41 ` Christopher Larson
2012-02-23 0:47 ` Rich Pixley
2012-02-23 10:16 ` Richard Purdie
2012-02-23 19:21 ` Rich Pixley
2012-02-23 20:11 ` Richard Purdie
2012-02-23 20:36 ` Rich Pixley
2012-02-23 20:41 ` Richard Purdie
2012-02-22 23:41 ` Saul Wold [this message]
2012-02-23 0:50 ` Rich Pixley
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4F457D1A.8010707@linux.intel.com \
--to=sgw@linux.intel.com \
--cc=openembedded-core@lists.openembedded.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox