[PATCH 0/1] Fix race while building external modules

[PATCH 0/1] Fix race while building external modules

[Anders Darander]

When building an image with several external modules, a race is likely to
occur. This race is when building scripts in the staged kernel tree (sysroot).

To prevent this, we promote the do_make_scripts() function to a task, which we
protect by a lock. Tip on how to make this work was received by Richard in
[1]. To avoid new problems, introduced by this promotion, we make this new
task depend on do_populate_sysroot from the modules dependencies
(i.e. virtual/kernel).

This patch has been extensively tested on the 2011-1 release branch, on a
custom ARM-machine and distro with several external modules.

[1] http://lists.linuxtogo.org/pipermail/openembedded-core/2011-October/011163.html

The following changes since commit e31dd9b65f3b03f79cabab25eca157532de3bd9c:

  fontconfig: fix fix-pkgconfig.patch (2011-10-18 18:13:47 +0100)

are available in the git repository at:
  git://github.com/darander/oe-core ext-mod-race
  https://github.com/darander/oe-core/tree/ext-mod-race

Anders Darander (1):
  module.bbclass: add lock to prevent error bulding ext modules

 meta/classes/module.bbclass |    5 ++++-
 1 files changed, 4 insertions(+), 1 deletions(-)

-- 
1.7.7

[PATCH 1/1] module.bbclass: add lock to prevent error bulding ext modules

[Anders Darander]

When external modules are built, files in $STAGING_KERNEL_DIR/scripts/basic will/can get
rebuilt.
This raises a potential race condition. Prevent this by adding a lock around the
do_make_scripts() function. Further, make sure that the kernel has been installed
to the sysroot, prior to executing this new task.

Signed-off-by: Anders Darander <anders@chargestorm.se>
---
 meta/classes/module.bbclass |    5 ++++-
 1 files changed, 4 insertions(+), 1 deletions(-)

diff --git a/meta/classes/module.bbclass b/meta/classes/module.bbclass
index 572df0d..53c16b7 100644
--- a/meta/classes/module.bbclass
+++ b/meta/classes/module.bbclass
@@ -14,8 +14,11 @@ do_make_scripts() {
 	           -C ${STAGING_KERNEL_DIR} scripts
 }
 
+addtask make_scripts before do_compile
+do_make_scripts[lockfiles] = "${TMPDIR}/kernel-scripts.lock"
+do_make_scripts[deptask] = "do_populate_sysroot"
+
 module_do_compile() {
-	do_make_scripts
 	unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS
 	oe_runmake KERNEL_PATH=${STAGING_KERNEL_DIR}   \
 		   KERNEL_SRC=${STAGING_KERNEL_DIR}    \
-- 
1.7.7

Re: [PATCH 1/1] module.bbclass: add lock to prevent error bulding ext modules

[Koen Kooi]

Op 19 okt. 2011, om 13:15 heeft Anders Darander het volgende geschreven:

> When external modules are built, files in $STAGING_KERNEL_DIR/scripts/basic will/can get
> rebuilt.
> This raises a potential race condition. Prevent this by adding a lock around the
> do_make_scripts() function. Further, make sure that the kernel has been installed
> to the sysroot, prior to executing this new task.

So what do external module recipe that aren't using do_compile from module.bbclass need to change to keep working?

Re: [PATCH 1/1] module.bbclass: add lock to prevent error bulding ext modules

[Anders Darander]

* Koen Kooi <koen@dominion.thruhere.net> [111019 13:38]:

> Op 19 okt. 2011, om 13:15 heeft Anders Darander het volgende geschreven:

> > When external modules are built, files in $STAGING_KERNEL_DIR/scripts/basic will/can get
> > rebuilt.
> > This raises a potential race condition. Prevent this by adding a lock around the
> > do_make_scripts() function. Further, make sure that the kernel has been installed
> > to the sysroot, prior to executing this new task.

> So what do external module recipe that aren't using do_compile from
> module.bbclass need to change to keep working?

I haven't tried any, do you know any widely available ones that I could
check?

If they aren't using the do_compile from module.bbclass, are they
inheriting module.bbclass anyway? If they aren't inheriting
module.bbclass, then they shouldn't be affected.

If they inherit module.bbclass, than I guess that we should remove the
call to do_make_scripts from their do_compile. _But_ this is untested,
as I've got no such recipe (that I'm aware of). If my assumptions hold,
we should probably rephrase the comment regarding do_make_scripts.

Cheers,
Anders

-- 
Anders Darander
ChargeStorm AB / eStorm AB

Re: [PATCH 0/1] Fix race while building external modules

[Saul Wold]

On 10/19/2011 04:15 AM, Anders Darander wrote:
>
> When building an image with several external modules, a race is likely to
> occur. This race is when building scripts in the staged kernel tree (sysroot).
>
> To prevent this, we promote the do_make_scripts() function to a task, which we
> protect by a lock. Tip on how to make this work was received by Richard in
> [1]. To avoid new problems, introduced by this promotion, we make this new
> task depend on do_populate_sysroot from the modules dependencies
> (i.e. virtual/kernel).
>
> This patch has been extensively tested on the 2011-1 release branch, on a
> custom ARM-machine and distro with several external modules.
>
> [1] http://lists.linuxtogo.org/pipermail/openembedded-core/2011-October/011163.html
>
> The following changes since commit e31dd9b65f3b03f79cabab25eca157532de3bd9c:
>
>    fontconfig: fix fix-pkgconfig.patch (2011-10-18 18:13:47 +0100)
>
> are available in the git repository at:
>    git://github.com/darander/oe-core ext-mod-race
>    https://github.com/darander/oe-core/tree/ext-mod-race
>
> Anders Darander (1):
>    module.bbclass: add lock to prevent error bulding ext modules
>
>   meta/classes/module.bbclass |    5 ++++-
>   1 files changed, 4 insertions(+), 1 deletions(-)
>
Merged into OE-Core

Thanks
	Sau!