From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp-relay2.palm.com ([64.28.152.243]) by linuxtogo.org with esmtp (Exim 4.72) (envelope-from ) id 1S0Lpp-0004Uc-BA for openembedded-core@lists.openembedded.org; Thu, 23 Feb 2012 00:42:57 +0100 X-IronPort-AV: E=Sophos;i="4.73,466,1325491200"; d="scan'208,217";a="11845253" Received: from unknown (HELO ushqusdns4.palm.com) ([148.92.223.164]) by smtp-relay2.palm.com with ESMTP; 22 Feb 2012 15:33:31 -0800 Received: from fuji.noir.com ([10.100.2.13]) by ushqusdns4.palm.com (8.14.4/8.14.4) with ESMTP id q1MNXULm019417; Wed, 22 Feb 2012 15:33:30 -0800 Message-ID: <4F457B4B.4000507@palm.com> Date: Wed, 22 Feb 2012 15:33:31 -0800 From: Rich Pixley User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:10.0.2) Gecko/20120216 Thunderbird/10.0.2 MIME-Version: 1.0 To: Patches and discussions about the oe-core layer Subject: sharing DL_DIR? X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.11 Precedence: list Reply-To: Patches and discussions about the oe-core layer List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Feb 2012 23:42:57 -0000 Content-Type: multipart/alternative; boundary="------------070108090307090607000505" --------------070108090307090607000505 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Is it reasonable to expect to share DL_DIR between multiple builds? That is, are downloads properly locked so that multiple concurrent downloads of the same file won't collide? And if so, are they NFS safe locks? Or must each build download it's own copies of every component? I ask because our, (Palm), branch has been, but it's a bit of a nuisance to pick a suitable locking mechanism that is both functional and performant. * Flock/lockf aren't reliably supported in heterogeneous environments. * The old lockfiles don't work over NFS. Lock directories apparently do, but trying to clean these up after interruptions or failures is a losing battle so if we do use these, we end up with orphan locks periodically. * With NFS as a possibility, we can't assume any kernel local IPC mechanism, so sysV ipc is out. * It seems like a pretty huge overhead to try to create any sort of zeroconf overhead. Our solution so far has been to use NFS lock directories for multiple machine builds and flock/lockf for single machine builds. This mostly works, though it's not ideal, and it's significantly faster than downloading/copying all of the component source multiple times even over local mirrors. --rich --------------070108090307090607000505 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Is it reasonable to expect to share DL_DIR between multiple builds?

That is, are downloads properly locked so that multiple concurrent downloads of the same file won't collide?  And if so, are they NFS safe locks?

Or must each build download it's own copies of every component?

I ask because our, (Palm), branch has been, but it's a bit of a nuisance to pick a suitable locking mechanism that is both functional and performant.
  • Flock/lockf aren't reliably supported in heterogeneous environments.
  • The old lockfiles don't work over NFS. Lock directories apparently do, but trying to clean these up after interruptions or failures is a losing battle so if we do use these, we end up with orphan locks periodically.
  • With NFS as a possibility, we can't assume any kernel local IPC mechanism, so sysV ipc is out.
  • It seems like a pretty huge overhead to try to create any sort of zeroconf overhead.

Our solution so far has been to use NFS lock directories for multiple machine builds and flock/lockf for single machine builds.  This mostly works, though it's not ideal, and it's significantly faster than downloading/copying all of the component source multiple times even over local mirrors.

--rich
--------------070108090307090607000505-- From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-iy0-f175.google.com ([209.85.210.175]) by linuxtogo.org with esmtp (Exim 4.72) (envelope-from ) id 1S0Luy-0004ef-Ji for openembedded-core@lists.openembedded.org; Thu, 23 Feb 2012 00:48:16 +0100 Received: by iaby12 with SMTP id y12so801680iab.6 for ; Wed, 22 Feb 2012 15:39:55 -0800 (PST) Received-SPF: pass (google.com: domain of ka6sox@gmail.com designates 10.50.184.168 as permitted sender) client-ip=10.50.184.168; Authentication-Results: mr.google.com; spf=pass (google.com: domain of ka6sox@gmail.com designates 10.50.184.168 as permitted sender) smtp.mail=ka6sox@gmail.com; dkim=pass header.i=ka6sox@gmail.com Received: from mr.google.com ([10.50.184.168]) by 10.50.184.168 with SMTP id ev8mr521968igc.29.1329953995272 (num_hops = 1); Wed, 22 Feb 2012 15:39:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=OTAppf7dE0//EyVx8ERIjDQD+qMl/hULpy5UstvRDCQ=; b=SH8CCQ9kF4217vbYzMIeRCGX5I4AgSGkieP4SYzCtAOgws6rfR4wY80eiMN4AaJG5Y Pp/qk+bbwPkENn1kRJ7nfR5Ce5htkjBgw4/DNS+KVvDmQhOKrbKlSgLc007QrIRectGO e06MVOKW36DVhBP0scxydk5xJrBCmmEtSkSjQ= MIME-Version: 1.0 Received: by 10.50.184.168 with SMTP id ev8mr447004igc.29.1329953995214; Wed, 22 Feb 2012 15:39:55 -0800 (PST) Received: by 10.50.42.9 with HTTP; Wed, 22 Feb 2012 15:39:55 -0800 (PST) In-Reply-To: <4F457B4B.4000507@palm.com> References: <4F457B4B.4000507@palm.com> Date: Wed, 22 Feb 2012 13:39:55 -1000 Message-ID: From: Tom King To: Patches and discussions about the oe-core layer Subject: Re: sharing DL_DIR? X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.11 Precedence: list Reply-To: Patches and discussions about the oe-core layer List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Feb 2012 23:48:16 -0000 Content-Type: multipart/alternative; boundary=14dae934044973cf7104b9960bb0 --14dae934044973cf7104b9960bb0 Content-Type: text/plain; charset=ISO-8859-1 I have a single downloads directory with symlinks for every target build I have to that directory. As long as I have the source already in there I don't fetch it again Tom On Wed, Feb 22, 2012 at 1:33 PM, Rich Pixley wrote: > Is it reasonable to expect to share DL_DIR between multiple builds? > > That is, are downloads properly locked so that multiple concurrent > downloads of the same file won't collide? And if so, are they NFS safe > locks? > > Or must each build download it's own copies of every component? > > I ask because our, (Palm), branch has been, but it's a bit of a nuisance > to pick a suitable locking mechanism that is both functional and performant. > > - Flock/lockf aren't reliably supported in heterogeneous environments. > - The old lockfiles don't work over NFS. Lock directories apparently > do, but trying to clean these up after interruptions or failures is a > losing battle so if we do use these, we end up with orphan locks > periodically. > - With NFS as a possibility, we can't assume any kernel local IPC > mechanism, so sysV ipc is out. > - It seems like a pretty huge overhead to try to create any sort of > zeroconf overhead. > > Our solution so far has been to use NFS lock directories for multiple > machine builds and flock/lockf for single machine builds. This mostly > works, though it's not ideal, and it's significantly faster than > downloading/copying all of the component source multiple times even over > local mirrors. > --rich > > _______________________________________________ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org > http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core > > --14dae934044973cf7104b9960bb0 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable I have a single downloads directory with symlinks for every target build I = have to that directory. =A0As long as I have the source already in there I = don't fetch =A0it again

Tom

On Wed, Feb 22, 2012 at 1:33 PM, Rich Pixley <rich.pixley@palm.com> wrote:=
=20 =20 =20
Is it reasonable to expect to share DL_DIR between multiple builds?

That is, are downloads properly locked so that multiple concurrent downloads of the same file won't collide?=A0 And if so, are they NF= S safe locks?

Or must each build download it's own copies of every component?

I ask because our, (Palm), branch has been, but it's a bit of a nuisance to pick a suitable locking mechanism that is both functional and performant.
  • Flock/lockf aren't reliably supported in heterogeneous environments.
  • The old lockfiles don't work over NFS. Lock directories apparently do, but trying to clean these up after interruptions or failures is a losing battle so if we do use these, we end up with orphan locks periodically.
  • With NFS as a possibility, we can't assume any kernel local IPC mechanism, so sysV ipc is out.
  • It seems like a pretty huge overhead to try to create any sort of zeroconf overhead.

Our solution so far has been to use NFS lock directories for multiple machine builds and flock/lockf for single machine builds.=A0 This mostly works, though it's not ideal, and it's significantly faster than downloading/copying all of the component source multiple times even over local mirrors.=

--rich

_______________________________________________
Openembedded-core mailing list
Openembedded-co= re@lists.openembedded.org
http://lists.linuxtogo.org/cgi-bin/mailman/listinf= o/openembedded-core


--14dae934044973cf7104b9960bb0-- From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mga02.intel.com ([134.134.136.20]) by linuxtogo.org with esmtp (Exim 4.72) (envelope-from ) id 1S0LwG-0004hh-Es for openembedded-core@lists.openembedded.org; Thu, 23 Feb 2012 00:49:36 +0100 Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga101.jf.intel.com with ESMTP; 22 Feb 2012 15:41:14 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="4.67,351,1309762800"; d="scan'208";a="113615227" Received: from unknown (HELO [10.255.15.72]) ([10.255.15.72]) by orsmga002.jf.intel.com with ESMTP; 22 Feb 2012 15:41:14 -0800 Message-ID: <4F457D1A.8010707@linux.intel.com> Date: Wed, 22 Feb 2012 15:41:14 -0800 From: Saul Wold User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20120131 Thunderbird/10.0 MIME-Version: 1.0 To: Patches and discussions about the oe-core layer References: <4F457B4B.4000507@palm.com> In-Reply-To: <4F457B4B.4000507@palm.com> Subject: Re: sharing DL_DIR? X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.11 Precedence: list Reply-To: Patches and discussions about the oe-core layer List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Feb 2012 23:49:36 -0000 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit On 02/22/2012 03:33 PM, Rich Pixley wrote: > Is it reasonable to expect to share DL_DIR between multiple builds? > Absolutely, We share DL_DIR via NFS across multiple machines and builds. > That is, are downloads properly locked so that multiple concurrent > downloads of the same file won't collide? And if so, are they NFS safe > locks? > Yes, as far as I know. > Or must each build download it's own copies of every component? > No need. > I ask because our, (Palm), branch has been, but it's a bit of a nuisance > to pick a suitable locking mechanism that is both functional and performant. > > * Flock/lockf aren't reliably supported in heterogeneous environments. > * The old lockfiles don't work over NFS. Lock directories apparently > do, but trying to clean these up after interruptions or failures is > a losing battle so if we do use these, we end up with orphan locks > periodically. > * With NFS as a possibility, we can't assume any kernel local IPC > mechanism, so sysV ipc is out. > * It seems like a pretty huge overhead to try to create any sort of > zeroconf overhead. > > Our solution so far has been to use NFS lock directories for multiple > machine builds and flock/lockf for single machine builds. This mostly > works, though it's not ideal, and it's significantly faster than > downloading/copying all of the component source multiple times even over > local mirrors. > I think we are using a very basic locking mechism that creates the lockfile and a donefile when complete. It seems to work well enough. Sau! > --rich > > > _______________________________________________ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org > http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pw0-f47.google.com ([209.85.160.47]) by linuxtogo.org with esmtp (Exim 4.72) (envelope-from ) id 1S0LwH-0004hk-NB for openembedded-core@lists.openembedded.org; Thu, 23 Feb 2012 00:49:38 +0100 Received: by pbbjt11 with SMTP id jt11so68592pbb.6 for ; Wed, 22 Feb 2012 15:41:16 -0800 (PST) Received-SPF: pass (google.com: domain of kergoth@gmail.com designates 10.68.131.98 as permitted sender) client-ip=10.68.131.98; Authentication-Results: mr.google.com; spf=pass (google.com: domain of kergoth@gmail.com designates 10.68.131.98 as permitted sender) smtp.mail=kergoth@gmail.com; dkim=pass header.i=kergoth@gmail.com Received: from mr.google.com ([10.68.131.98]) by 10.68.131.98 with SMTP id ol2mr86717796pbb.138.1329954076389 (num_hops = 1); Wed, 22 Feb 2012 15:41:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=date:from:to:message-id:in-reply-to:references:subject:x-mailer :mime-version:content-type; bh=de+obIAlFMOFf1ggfAApQlvYxoDi0SKWXPZX95stJFg=; b=pyGgdpKGlhZCK+E70O+ZAPQpZeahRoU5gKn3iLUXB8eRraCoge+kDWbRooRhXD9sIW fufLnfOGpojK3RqW8FlUvuuRgKzAZViItIk7spjAYiDmv1/UWMsQTpUBV229ixo56Zmc mQb3cTtkHEiBdgR5thJLFtU4Fe83JNoL2KgBA= Received: by 10.68.131.98 with SMTP id ol2mr71741800pbb.138.1329954076294; Wed, 22 Feb 2012 15:41:16 -0800 (PST) Received: from frey.local (ip24-251-167-38.ph.ph.cox.net. [24.251.167.38]) by mx.google.com with ESMTPS id q10sm31754284pbb.10.2012.02.22.15.41.13 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 22 Feb 2012 15:41:14 -0800 (PST) Date: Wed, 22 Feb 2012 16:41:11 -0700 From: Christopher Larson To: Patches and discussions about the oe-core layer Message-ID: In-Reply-To: References: <4F457B4B.4000507@palm.com> X-Mailer: sparrow 1.5 (build 1043.1) MIME-Version: 1.0 Subject: Re: sharing DL_DIR? X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.11 Precedence: list Reply-To: Patches and discussions about the oe-core layer List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Feb 2012 23:49:38 -0000 Content-Type: multipart/alternative; boundary="4f457d17_507ed7ab_1049f" --4f457d17_507ed7ab_1049f Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Content-Disposition: inline I share it as well. I don't see locks being a big issue for most things, as the worst case is a re-download, not use of a corrupt or incomplete file, due to the completion stamps. -- Christopher Larson On Wednesday, February 22, 2012 at 4:39 PM, Tom King wrote: > I have a single downloads directory with symlinks for every target build I have to that directory. As long as I have the source already in there I don't fetch it again > > Tom > > On Wed, Feb 22, 2012 at 1:33 PM, Rich Pixley wrote: > > Is it reasonable to expect to share DL_DIR between multiple builds? > > > > That is, are downloads properly locked so that multiple concurrent downloads of the same file won't collide? And if so, are they NFS safe locks? > > > > Or must each build download it's own copies of every component? > > > > I ask because our, (Palm), branch has been, but it's a bit of a nuisance to pick a suitable locking mechanism that is both functional and performant. > > Flock/lockf aren't reliably supported in heterogeneous environments. > > The old lockfiles don't work over NFS. Lock directories apparently do, but trying to clean these up after interruptions or failures is a losing battle so if we do use these, we end up with orphan locks periodically. > > With NFS as a possibility, we can't assume any kernel local IPC mechanism, so sysV ipc is out. > > It seems like a pretty huge overhead to try to create any sort of zeroconf overhead. > > > > Our solution so far has been to use NFS lock directories for multiple machine builds and flock/lockf for single machine builds. This mostly works, though it's not ideal, and it's significantly faster than downloading/copying all of the component source multiple times even over local mirrors. > > --rich > > > > _______________________________________________ > > Openembedded-core mailing list > > Openembedded-core@lists.openembedded.org (mailto:Openembedded-core@lists.openembedded.org) > > http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core > > > > _______________________________________________ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org (mailto:Openembedded-core@lists.openembedded.org) > http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core > > --4f457d17_507ed7ab_1049f Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline
I share it as well. I don't see locks being a big issue for most thin= gs, as the worst case is a re-download, not use of a corrupt or incomplet= e file, due to the completion stamps.

-- 
Christopher L= arson

=20

On Wednesday, =46ebrua= ry 22, 2012 at 4:39 PM, Tom King wrote:

I have a single downloads directory w= ith symlinks for every target build I have to that directory.  As lo= ng as I have the source already in there I don't fetch  it again
Tom

On Wed, =46eb 22, 2012 at 1:33 PM, Rich Pixley <= rich.pixley=40palm.com&= gt; wrote:
=20 =20 =20
Is it reasonable to expect to share DL=5FDIR between multiple builds=3F=

That is, are downloads properly locked so that multiple concurrent downloads of the same file won't collide=3F  And if so, are they= N=46S safe locks=3F

Or must each build download it's own copies of every component=3F
=
I ask because our, (Palm), branch has been, but it's a bit of a nuisance to pick a suitable locking mechanism that is both functional and performant.
  • =46lock/lockf aren't reliably supported in heterogeneous environments.
  • The old lockfiles don't work over N=46S. Lock directories apparently do, but trying to clean these up after interruptions or failures is a losing battle so if we do use these, we end up with orphan locks periodically.
  • With N=46S as a possibility, we can't assume any kernel local IPC mechanism, so sysV ipc is out.
  • It seems like a pretty huge overhead to try to create any sort of zeroconf overhead.

Our solution so far has been to use N=46S lock directories for multiple machine builds and flock/lockf for single machine builds.  This mostly works, though it's not ideal, and it's significantly faster than downloading/copying all of the component source multiple times even over local mirrors.

--rich

=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F<= br> Openembedded-core mailing list
Openemb= edded-core=40lists.openembedded.org
http://lists.linuxtogo.org/cgi-bin/m= ailman/listinfo/openembedded-core


=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F
Openembedded-core mailing list
=20 =20 =20 =20
=20

--4f457d17_507ed7ab_1049f-- From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp-relay2.palm.com ([64.28.152.243]) by linuxtogo.org with esmtp (Exim 4.72) (envelope-from ) id 1S0MzY-000770-LH for openembedded-core@lists.openembedded.org; Thu, 23 Feb 2012 01:57:05 +0100 X-IronPort-AV: E=Sophos;i="4.73,466,1325491200"; d="scan'208,217";a="11846353" Received: from unknown (HELO ushqusdns4.palm.com) ([148.92.223.164]) by smtp-relay2.palm.com with ESMTP; 22 Feb 2012 16:47:38 -0800 Received: from fuji.noir.com ([10.100.2.13]) by ushqusdns4.palm.com (8.14.4/8.14.4) with ESMTP id q1N0lb1L027665; Wed, 22 Feb 2012 16:47:37 -0800 Message-ID: <4F458CAA.7040107@palm.com> Date: Wed, 22 Feb 2012 16:47:38 -0800 From: Rich Pixley User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:10.0.2) Gecko/20120216 Thunderbird/10.0.2 MIME-Version: 1.0 To: openembedded-core@lists.openembedded.org References: <4F457B4B.4000507@palm.com> In-Reply-To: Subject: Re: sharing DL_DIR? X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.11 Precedence: list Reply-To: Patches and discussions about the oe-core layer List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Feb 2012 00:57:05 -0000 Content-Type: multipart/alternative; boundary="------------060805080208090009000806" --------------060805080208090009000806 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit What happens if two separate bitbake invocations both simultaneously attempt to download the same file? --rich On 2/22/12 15:41 , Christopher Larson wrote: > I share it as well. I don't see locks being a big issue for most > things, as the worst case is a re-download, not use of a corrupt or > incomplete file, due to the completion stamps. > > -- > Christopher Larson > > On Wednesday, February 22, 2012 at 4:39 PM, Tom King wrote: > >> I have a single downloads directory with symlinks for every target >> build I have to that directory. As long as I have the source already >> in there I don't fetch it again >> >> Tom >> >> On Wed, Feb 22, 2012 at 1:33 PM, Rich Pixley > > wrote: >>> Is it reasonable to expect to share DL_DIR between multiple builds? >>> >>> That is, are downloads properly locked so that multiple concurrent >>> downloads of the same file won't collide? And if so, are they NFS >>> safe locks? >>> >>> Or must each build download it's own copies of every component? >>> >>> I ask because our, (Palm), branch has been, but it's a bit of a >>> nuisance to pick a suitable locking mechanism that is both >>> functional and performant. >>> >>> * Flock/lockf aren't reliably supported in heterogeneous environments. >>> * The old lockfiles don't work over NFS. Lock directories >>> apparently do, but trying to clean these up after interruptions >>> or failures is a losing battle so if we do use these, we end up >>> with orphan locks periodically. >>> * With NFS as a possibility, we can't assume any kernel local IPC >>> mechanism, so sysV ipc is out. >>> * It seems like a pretty huge overhead to try to create any sort >>> of zeroconf overhead. >>> >>> Our solution so far has been to use NFS lock directories for >>> multiple machine builds and flock/lockf for single machine builds. >>> This mostly works, though it's not ideal, and it's significantly >>> faster than downloading/copying all of the component source multiple >>> times even over local mirrors. >>> >>> --rich >>> >>> _______________________________________________ >>> Openembedded-core mailing list >>> Openembedded-core@lists.openembedded.org >>> >>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core >>> >> >> _______________________________________________ >> Openembedded-core mailing list >> Openembedded-core@lists.openembedded.org >> >> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core > --------------060805080208090009000806 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit What happens if two separate bitbake invocations both simultaneously attempt to download the same file?

--rich

On 2/22/12 15:41 , Christopher Larson wrote:
I share it as well. I don't see locks being a big issue for most things, as the worst case is a re-download, not use of a corrupt or incomplete file, due to the completion stamps.

-- 
Christopher Larson

On Wednesday, February 22, 2012 at 4:39 PM, Tom King wrote:

I have a single downloads directory with symlinks for every target build I have to that directory.  As long as I have the source already in there I don't fetch  it again

Tom

On Wed, Feb 22, 2012 at 1:33 PM, Rich Pixley <rich.pixley@palm.com> wrote:
Is it reasonable to expect to share DL_DIR between multiple builds?

That is, are downloads properly locked so that multiple concurrent downloads of the same file won't collide?  And if so, are they NFS safe locks?

Or must each build download it's own copies of every component?

I ask because our, (Palm), branch has been, but it's a bit of a nuisance to pick a suitable locking mechanism that is both functional and performant.
  • Flock/lockf aren't reliably supported in heterogeneous environments.
  • The old lockfiles don't work over NFS. Lock directories apparently do, but trying to clean these up after interruptions or failures is a losing battle so if we do use these, we end up with orphan locks periodically.
  • With NFS as a possibility, we can't assume any kernel local IPC mechanism, so sysV ipc is out.
  • It seems like a pretty huge overhead to try to create any sort of zeroconf overhead.

Our solution so far has been to use NFS lock directories for multiple machine builds and flock/lockf for single machine builds.  This mostly works, though it's not ideal, and it's significantly faster than downloading/copying all of the component source multiple times even over local mirrors.

--rich

_______________________________________________
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core


_______________________________________________
Openembedded-core mailing list

--------------060805080208090009000806-- From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp-relay2.palm.com ([64.28.152.243]) by linuxtogo.org with esmtp (Exim 4.72) (envelope-from ) id 1S0N0u-0007Eh-OD for openembedded-core@lists.openembedded.org; Thu, 23 Feb 2012 01:58:28 +0100 X-IronPort-AV: E=Sophos;i="4.73,466,1325491200"; d="scan'208";a="11846391" Received: from unknown (HELO ushqusdns4.palm.com) ([148.92.223.164]) by smtp-relay2.palm.com with ESMTP; 22 Feb 2012 16:50:08 -0800 Received: from fuji.noir.com ([10.100.2.13]) by ushqusdns4.palm.com (8.14.4/8.14.4) with ESMTP id q1N0o7gu029757; Wed, 22 Feb 2012 16:50:07 -0800 Message-ID: <4F458D3F.9020602@palm.com> Date: Wed, 22 Feb 2012 16:50:07 -0800 From: Rich Pixley User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:10.0.2) Gecko/20120216 Thunderbird/10.0.2 MIME-Version: 1.0 To: Saul Wold References: <4F457B4B.4000507@palm.com> <4F457D1A.8010707@linux.intel.com> In-Reply-To: <4F457D1A.8010707@linux.intel.com> Cc: Patches and discussions about the oe-core layer Subject: Re: sharing DL_DIR? X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.11 Precedence: list Reply-To: Patches and discussions about the oe-core layer List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Feb 2012 00:58:29 -0000 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Excellent, thanks. That's the news I was hoping for. --rich On 2/22/12 15:41 , Saul Wold wrote: > On 02/22/2012 03:33 PM, Rich Pixley wrote: >> Is it reasonable to expect to share DL_DIR between multiple builds? >> > Absolutely, We share DL_DIR via NFS across multiple machines and builds. > >> That is, are downloads properly locked so that multiple concurrent >> downloads of the same file won't collide? And if so, are they NFS safe >> locks? >> > Yes, as far as I know. > >> Or must each build download it's own copies of every component? >> > No need. > >> I ask because our, (Palm), branch has been, but it's a bit of a nuisance >> to pick a suitable locking mechanism that is both functional and performant. >> >> * Flock/lockf aren't reliably supported in heterogeneous environments. >> * The old lockfiles don't work over NFS. Lock directories apparently >> do, but trying to clean these up after interruptions or failures is >> a losing battle so if we do use these, we end up with orphan locks >> periodically. >> * With NFS as a possibility, we can't assume any kernel local IPC >> mechanism, so sysV ipc is out. >> * It seems like a pretty huge overhead to try to create any sort of >> zeroconf overhead. >> >> Our solution so far has been to use NFS lock directories for multiple >> machine builds and flock/lockf for single machine builds. This mostly >> works, though it's not ideal, and it's significantly faster than >> downloading/copying all of the component source multiple times even over >> local mirrors. >> > I think we are using a very basic locking mechism that creates the > lockfile and a donefile when complete. It seems to work well enough. > > Sau! > >> --rich >> >> >> _______________________________________________ >> Openembedded-core mailing list >> Openembedded-core@lists.openembedded.org >> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from 93-97-173-237.zone5.bethere.co.uk ([93.97.173.237] helo=tim.rpsys.net) by linuxtogo.org with esmtp (Exim 4.72) (envelope-from ) id 1S0Vqi-0005tG-A2 for openembedded-core@lists.openembedded.org; Thu, 23 Feb 2012 11:24:32 +0100 Received: from localhost (localhost [127.0.0.1]) by tim.rpsys.net (8.13.6/8.13.8) with ESMTP id q1NAGBht009170 for ; Thu, 23 Feb 2012 10:16:11 GMT Received: from tim.rpsys.net ([127.0.0.1]) by localhost (tim.rpsys.net [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 07301-10 for ; Thu, 23 Feb 2012 10:16:07 +0000 (GMT) Received: from [192.168.3.10] ([192.168.3.10]) (authenticated bits=0) by tim.rpsys.net (8.13.6/8.13.8) with ESMTP id q1NAG2Qb009164 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Thu, 23 Feb 2012 10:16:03 GMT Message-ID: <1329992162.32110.53.camel@ted> From: Richard Purdie To: Patches and discussions about the oe-core layer Date: Thu, 23 Feb 2012 10:16:02 +0000 In-Reply-To: <4F458CAA.7040107@palm.com> References: <4F457B4B.4000507@palm.com> <4F458CAA.7040107@palm.com> X-Mailer: Evolution 3.2.2- Mime-Version: 1.0 X-Virus-Scanned: amavisd-new at rpsys.net Subject: Re: sharing DL_DIR? X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.11 Precedence: list Reply-To: Patches and discussions about the oe-core layer List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Feb 2012 10:24:32 -0000 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit On Wed, 2012-02-22 at 16:47 -0800, Rich Pixley wrote: > What happens if two separate bitbake invocations both simultaneously > attempt to download the same file? The first will download it, the second will wait for it to finish, then both will continue with the downloaded file. Cheers, Richard From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp-relay2.palm.com ([64.28.152.243]) by linuxtogo.org with esmtp (Exim 4.72) (envelope-from ) id 1S0eM2-0003QZ-1X for openembedded-core@lists.openembedded.org; Thu, 23 Feb 2012 20:29:26 +0100 X-IronPort-AV: E=Sophos;i="4.73,471,1325491200"; d="scan'208";a="11866440" Received: from unknown (HELO ushqusdns3.palm.com) ([148.92.223.90]) by smtp-relay2.palm.com with ESMTP; 23 Feb 2012 11:21:03 -0800 Received: from joso.noir.com ([10.100.2.14]) by ushqusdns3.palm.com (8.14.4/8.14.4) with ESMTP id q1NJL3D0005665; Thu, 23 Feb 2012 11:21:03 -0800 Message-ID: <4F46919E.2050204@palm.com> Date: Thu, 23 Feb 2012 11:21:02 -0800 From: Rich Pixley User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:10.0.2) Gecko/20120216 Thunderbird/10.0.2 MIME-Version: 1.0 To: openembedded-core@lists.openembedded.org References: <4F457B4B.4000507@palm.com> <4F458CAA.7040107@palm.com> <1329992162.32110.53.camel@ted> In-Reply-To: <1329992162.32110.53.camel@ted> Subject: Re: sharing DL_DIR? X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.11 Precedence: list Reply-To: Patches and discussions about the oe-core layer List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Feb 2012 19:29:26 -0000 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit On 2/23/12 02:16 , Richard Purdie wrote: > On Wed, 2012-02-22 at 16:47 -0800, Rich Pixley wrote: >> What happens if two separate bitbake invocations both simultaneously >> attempt to download the same file? > The first will download it, the second will wait for it to finish, then > both will continue with the downloaded file. How does the second know that the first is in progress? --rich From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from 93-97-173-237.zone5.bethere.co.uk ([93.97.173.237] helo=tim.rpsys.net) by linuxtogo.org with esmtp (Exim 4.72) (envelope-from ) id 1S0f93-0004IQ-7E for openembedded-core@lists.openembedded.org; Thu, 23 Feb 2012 21:20:05 +0100 Received: from localhost (localhost [127.0.0.1]) by tim.rpsys.net (8.13.6/8.13.8) with ESMTP id q1NKBhLt024685 for ; Thu, 23 Feb 2012 20:11:43 GMT Received: from tim.rpsys.net ([127.0.0.1]) by localhost (tim.rpsys.net [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 18053-10 for ; Thu, 23 Feb 2012 20:11:39 +0000 (GMT) Received: from [192.168.3.10] ([192.168.3.10]) (authenticated bits=0) by tim.rpsys.net (8.13.6/8.13.8) with ESMTP id q1NKBYht024672 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Thu, 23 Feb 2012 20:11:36 GMT Message-ID: <1330027895.32110.84.camel@ted> From: Richard Purdie To: Patches and discussions about the oe-core layer Date: Thu, 23 Feb 2012 20:11:35 +0000 In-Reply-To: <4F46919E.2050204@palm.com> References: <4F457B4B.4000507@palm.com> <4F458CAA.7040107@palm.com> <1329992162.32110.53.camel@ted> <4F46919E.2050204@palm.com> X-Mailer: Evolution 3.2.2- Mime-Version: 1.0 X-Virus-Scanned: amavisd-new at rpsys.net Subject: Re: sharing DL_DIR? X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.11 Precedence: list Reply-To: Patches and discussions about the oe-core layer List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Feb 2012 20:20:05 -0000 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit On Thu, 2012-02-23 at 11:21 -0800, Rich Pixley wrote: > On 2/23/12 02:16 , Richard Purdie wrote: > > On Wed, 2012-02-22 at 16:47 -0800, Rich Pixley wrote: > >> What happens if two separate bitbake invocations both simultaneously > >> attempt to download the same file? > > The first will download it, the second will wait for it to finish, then > > both will continue with the downloaded file. > How does the second know that the first is in progress? It won't be able to take the lock and the done stamp file doesn't exist yet. Cheers, Richard From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp-relay2.palm.com ([64.28.152.243]) by linuxtogo.org with esmtp (Exim 4.72) (envelope-from ) id 1S0fWw-0004Vc-2a for openembedded-core@lists.openembedded.org; Thu, 23 Feb 2012 21:44:46 +0100 X-IronPort-AV: E=Sophos;i="4.73,471,1325491200"; d="scan'208";a="11868511" Received: from unknown (HELO ushqusdns4.palm.com) ([148.92.223.164]) by smtp-relay2.palm.com with ESMTP; 23 Feb 2012 12:36:23 -0800 Received: from joso.noir.com ([10.100.2.14]) by ushqusdns4.palm.com (8.14.4/8.14.4) with ESMTP id q1NKaMP6029072; Thu, 23 Feb 2012 12:36:23 -0800 Message-ID: <4F46A346.3050809@palm.com> Date: Thu, 23 Feb 2012 12:36:22 -0800 From: Rich Pixley User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:10.0.2) Gecko/20120216 Thunderbird/10.0.2 MIME-Version: 1.0 To: openembedded-core@lists.openembedded.org References: <4F457B4B.4000507@palm.com> <4F458CAA.7040107@palm.com> <1329992162.32110.53.camel@ted> <4F46919E.2050204@palm.com> <1330027895.32110.84.camel@ted> In-Reply-To: <1330027895.32110.84.camel@ted> Subject: Re: sharing DL_DIR? X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.11 Precedence: list Reply-To: Patches and discussions about the oe-core layer List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Feb 2012 20:44:46 -0000 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit On 2/23/12 12:11 , Richard Purdie wrote: > On Thu, 2012-02-23 at 11:21 -0800, Rich Pixley wrote: >> On 2/23/12 02:16 , Richard Purdie wrote: >>> On Wed, 2012-02-22 at 16:47 -0800, Rich Pixley wrote: >>>> What happens if two separate bitbake invocations both simultaneously >>>> attempt to download the same file? >>> The first will download it, the second will wait for it to finish, then >>> both will continue with the downloaded file. >> How does the second know that the first is in progress? > It won't be able to take the lock and the done stamp file doesn't exist > yet. What lock? I've just read through lib/bb/fetch2/__init__.py but I'm not seeing a lock. What am I missing? --rich From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from 93-97-173-237.zone5.bethere.co.uk ([93.97.173.237] helo=tim.rpsys.net) by linuxtogo.org with esmtp (Exim 4.72) (envelope-from ) id 1S0fbY-0004ad-GC for openembedded-core@lists.openembedded.org; Thu, 23 Feb 2012 21:49:32 +0100 Received: from localhost (localhost [127.0.0.1]) by tim.rpsys.net (8.13.6/8.13.8) with ESMTP id q1NKfB4f025350 for ; Thu, 23 Feb 2012 20:41:11 GMT Received: from tim.rpsys.net ([127.0.0.1]) by localhost (tim.rpsys.net [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 22892-04 for ; Thu, 23 Feb 2012 20:41:07 +0000 (GMT) Received: from [192.168.3.10] ([192.168.3.10]) (authenticated bits=0) by tim.rpsys.net (8.13.6/8.13.8) with ESMTP id q1NKf2eQ025343 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Thu, 23 Feb 2012 20:41:03 GMT Message-ID: <1330029663.32110.85.camel@ted> From: Richard Purdie To: Patches and discussions about the oe-core layer Date: Thu, 23 Feb 2012 20:41:03 +0000 In-Reply-To: <4F46A346.3050809@palm.com> References: <4F457B4B.4000507@palm.com> <4F458CAA.7040107@palm.com> <1329992162.32110.53.camel@ted> <4F46919E.2050204@palm.com> <1330027895.32110.84.camel@ted> <4F46A346.3050809@palm.com> X-Mailer: Evolution 3.2.2- Mime-Version: 1.0 X-Virus-Scanned: amavisd-new at rpsys.net Subject: Re: sharing DL_DIR? X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.11 Precedence: list Reply-To: Patches and discussions about the oe-core layer List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Feb 2012 20:49:32 -0000 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit On Thu, 2012-02-23 at 12:36 -0800, Rich Pixley wrote: > On 2/23/12 12:11 , Richard Purdie wrote: > > On Thu, 2012-02-23 at 11:21 -0800, Rich Pixley wrote: > >> On 2/23/12 02:16 , Richard Purdie wrote: > >>> On Wed, 2012-02-22 at 16:47 -0800, Rich Pixley wrote: > >>>> What happens if two separate bitbake invocations both simultaneously > >>>> attempt to download the same file? > >>> The first will download it, the second will wait for it to finish, then > >>> both will continue with the downloaded file. > >> How does the second know that the first is in progress? > > It won't be able to take the lock and the done stamp file doesn't exist > > yet. > What lock? > > I've just read through lib/bb/fetch2/__init__.py but I'm not seeing a > lock. What am I missing? Try searching that file for expressions like "lock" and "lockfile" ;-) Cheers, Richard