[PATCH] Add locking on tasks that add package_update_index_ipk

[PATCH] Add locking on tasks that add package_update_index_ipk

[Tom Rini]

We add a lockfile of do_populate_sdk and do_rootfs (in rootfs_ipk) that 
is also the lockfile that package_update_index_ipk uses.  With enough 
threads it is possible that one image (or meta-toolchain) is attempting 
to use the package index while another has only reached the point of 
generating the index leading to an empty index being seen later on.

diff --git a/classes/rootfs_ipk.bbclass b/classes/rootfs_ipk.bbclass
index 9fa1d5e..0303c54 100644
--- a/classes/rootfs_ipk.bbclass
+++ b/classes/rootfs_ipk.bbclass
@@ -6,6 +6,7 @@
  #

  do_rootfs[depends] += "opkg-native:do_populate_sysroot"
+do_rootfs[lockfiles] = "${DEPLOY_DIR_IPK}.lock"

  IPKG_TMP_DIR = "${IMAGE_ROOTFS}-tmp"
  IPKG_ARGS = "-f ${IPKGCONF_TARGET} -o ${IMAGE_ROOTFS} -t 
${IPKG_TMP_DIR} ${@base_conditional("PACKAGE_INSTALL_NO_DEPS", "1", 
"-nodeps", "", d)}"
diff --git a/recipes/meta/canadian-sdk.bb b/recipes/meta/canadian-sdk.bb
index 47529a2..2160441 100644
--- a/recipes/meta/canadian-sdk.bb
+++ b/recipes/meta/canadian-sdk.bb
@@ -169,5 +169,6 @@ do_populate_sdk() {
  }

  do_populate_sdk[nostamp] = "1"
+do_populate_sdk[lockfiles] = "${DEPLOY_DIR_IPK}.lock"
  addtask package_update_index_ipk before do_populate_sdk
  addtask populate_sdk before do_build after do_install
diff --git a/recipes/meta/meta-toolchain.bb b/recipes/meta/meta-toolchain.bb
index 064fc3b..82918ad 100644
--- a/recipes/meta/meta-toolchain.bb
+++ b/recipes/meta/meta-toolchain.bb
@@ -177,5 +177,6 @@ do_populate_sdk() {
  }

  do_populate_sdk[nostamp] = "1"
+do_populate_sdk[lockfiles] = "${DEPLOY_DIR_IPK}.lock"
  addtask package_update_index_ipk before do_populate_sdk
  addtask populate_sdk before do_build after do_install

-- 
Tom Rini
Mentor Graphics Corporation

Re: [PATCH] Add locking on tasks that add package_update_index_ipk

[Chris Larson]

On Fri, Jun 4, 2010 at 5:03 PM, Tom Rini <tom_rini@mentor.com> wrote:

> We add a lockfile of do_populate_sdk and do_rootfs (in rootfs_ipk) that is
> also the lockfile that package_update_index_ipk uses.  With enough threads
> it is possible that one image (or meta-toolchain) is attempting to use the
> package index while another has only reached the point of generating the
> index leading to an empty index being seen later on.


Acked-by: Chris Larson <chris_larson@mentor.com>
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

Re: [PATCH] Add locking on tasks that add package_update_index_ipk

[Roman I Khimov]

[-- Attachment #1: Type: Text/Plain, Size: 1429 bytes --]

В сообщении от Суббота 05 июня 2010 04:03:59 автор Tom Rini написал:
> We add a lockfile of do_populate_sdk and do_rootfs (in rootfs_ipk) that
> is also the lockfile that package_update_index_ipk uses.  With enough
> threads it is possible that one image (or meta-toolchain) is attempting
> to use the package index while another has only reached the point of
> generating the index leading to an empty index being seen later on.

Am I right that effectively it makes rootfs creation sequential? It's bad for 
performance. Probably it's time to fix ipkg-make-index to not touch index 
files if no new packages are found as was suggested some time ago:

http://patchwork.openembedded.org/patch/744/

But thinking about it a bit more, ipkg-make-index hack won't really fix the 
issue in this case, as we can have image A with a set of packages, image B 
with a similar set of packages plus additional package P and then A would do 
package_update_index_ipk and enter do_rootfs, then P gets completed, 
package_update_index_ipk for B is started and the package index really needs 
an update, thus things can break again.

Maybe it's easier to make those indexes per-image for performance.

Well,

Acked-by: Roman I Khimov <khimov@altell.ru>

for now.

-- 
 http://roman.khimov.ru
mailto: roman@khimov.ru
gpg --keyserver hkp://subkeys.pgp.net --recv-keys 0xE5E055C3

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: [PATCH] Add locking on tasks that add package_update_index_ipk

[Tom Rini]

Roman I Khimov wrote:
> В сообщении от Суббота 05 июня 2010 04:03:59 автор Tom Rini написал:
>> We add a lockfile of do_populate_sdk and do_rootfs (in rootfs_ipk) that
>> is also the lockfile that package_update_index_ipk uses.  With enough
>> threads it is possible that one image (or meta-toolchain) is attempting
>> to use the package index while another has only reached the point of
>> generating the index leading to an empty index being seen later on.
> 
> Am I right that effectively it makes rootfs creation sequential? It's bad for 
> performance.

Yes, this does introduce a bottleneck at these points.

> Probably it's time to fix ipkg-make-index to not touch index 
> files if no new packages are found as was suggested some time ago:
> 
> http://patchwork.openembedded.org/patch/744/
> 
> But thinking about it a bit more, ipkg-make-index hack won't really fix the 
> issue in this case, as we can have image A with a set of packages, image B 
> with a similar set of packages plus additional package P and then A would do 
> package_update_index_ipk and enter do_rootfs, then P gets completed, 
> package_update_index_ipk for B is started and the package index really needs 
> an update, thus things can break again.
 >
 > Maybe it's easier to make those indexes per-image for performance.

Right.  The problem is that we really need the index to be per user of 
the feed.  I don't know enough about ipkg to say if we could move the 
indices to ${WORKDIR} and keep the packages where they are (and then, 
what do folks that populate a feed need?) but that would solve this 
particular problem.

-- 
Tom Rini
Mentor Graphics Corporation