coreutils-native race

coreutils-native race

[Uli Luckas]

hi all,
I just tracked down a problem in bitbake, oe-stable and parallel builds.
My oe-stable is a few days old but I suppose this problem might not be version 
dependent so let's see if this is known and fixed already.

What happens (once in a while) is that some build process uses commands from 
coreutils-native while coreutils-native is being staged.
Let's say the command is 'rm' - so coreutils-native stages 'rm' while another 
build process trys to delete some file. This leads to "rm: text file busy" 
because rm is still open for writing.

Has anyone seen this? And is there a known fix or workaround?

Thanks
Uli


-- 

------- ROAD ...the handyPC Company - - -  ) ) )

Uli Luckas
Head of Software Development

ROAD GmbH
Bennigsenstr. 14 | 12159 Berlin | Germany
fon: +49 (30) 230069 - 62 | fax: +49 (30) 230069 - 69
url: www.road.de

Amtsgericht Charlottenburg: HRB 96688 B
Managing director: Hans-Peter Constien

Re: coreutils-native race

[Phil Blundell]

On Fri, 2009-08-14 at 17:10 +0200, Uli Luckas wrote:
> What happens (once in a while) is that some build process uses commands from 
> coreutils-native while coreutils-native is being staged.
> Let's say the command is 'rm' - so coreutils-native stages 'rm' while another 
> build process trys to delete some file. This leads to "rm: text file busy" 
> because rm is still open for writing.
> 
> Has anyone seen this? And is there a known fix or workaround?

I guess the fix would be to make coreutils-native install its binaries
atomically (i.e. install them to ${STAGING_BINDIR}/rm.new and then mv
them into place) rather than installing them directly into their final
location.  Alternatively you could probably patch it to just not install
things like rm at all: if your host system doesn't have a functioning
"rm" then you're probably hosed anyway.

p.

[PATCH] Re: coreutils-native race

[Uli Luckas]

On Friday, 14. August 2009, Phil Blundell wrote:
> On Fri, 2009-08-14 at 17:10 +0200, Uli Luckas wrote:
> > What happens (once in a while) is that some build process uses commands
> > from coreutils-native while coreutils-native is being staged.
> > Let's say the command is 'rm' - so coreutils-native stages 'rm' while
> > another build process trys to delete some file. This leads to "rm: text
> > file busy" because rm is still open for writing.
> >
> > Has anyone seen this? And is there a known fix or workaround?
>
> I guess the fix would be to make coreutils-native install its binaries
> atomically (i.e. install them to ${STAGING_BINDIR}/rm.new and then mv
> them into place) rather than installing them directly into their final
> location.  Alternatively you could probably patch it to just not install
> things like rm at all: if your host system doesn't have a functioning
> "rm" then you're probably hosed anyway.
>
Hi Phil,
I went for your first sugestion. Attached is a patch that does atomic 
installation. The patch is against a snapshot of the stable branch but 
should not be to hard to forward port to other branches for people who
have trees of these other branches available.

regards,
Uli

commit 59171bc7bc35eabab3bd35730f77d194aa802a53
Author: Uli Luckas <u.luckas@road.de>
Date:   Mon Aug 17 15:53:06 2009 +0200

    - Fix "text file busy" build bug.
    
    Signed-off-by: Uli Luckas <u.luckas@road.de>

diff --git a/packages/coreutils/coreutils-native.inc b/packages/coreutils/coreutils-native.inc
new file mode 100644
index 0000000..f6a6dc2
--- /dev/null
+++ b/packages/coreutils/coreutils-native.inc
@@ -0,0 +1,24 @@
+# binaries from coreutils-native are usually provided by the host os and already
+# used by parallel build threads before coreutils-native is staged.
+#
+# The regular autotools_stage_dir installs non atomically and can cause parallel
+# build threads to find busy binaries.
+#
+autotools_stage_dir() {
+	from="$1"
+	to="$2"
+	# This will remove empty directories so we can ignore them
+	rmdir "$from" 2> /dev/null || true
+	if [ -d "$from" ]; then
+		mkdir -p "$to"
+		(cd "$from"; find . -print ) | while read object; do
+			if [ -d "$from/$object" ]; then
+				mkdir -p "$to/$object"
+			else
+				tmpname="`mktemp "$to/$object".XXXXXXXX`"
+				cp -fpP "$from/$object" "$tmpname"
+				mv "$tmpname" "$to/$object"
+			fi
+		done
+	fi
+}
diff --git a/packages/coreutils/coreutils-native_5.3.0.bb b/packages/coreutils/coreutils-native_5.3.0.bb
index ee90981..3a278b6 100644
--- a/packages/coreutils/coreutils-native_5.3.0.bb
+++ b/packages/coreutils/coreutils-native_5.3.0.bb
@@ -5,3 +5,4 @@ S = "${WORKDIR}/coreutils-${PV}"
 
 require coreutils_${PV}.bb
 inherit native
+require coreutils-native.inc
diff --git a/packages/coreutils/coreutils-native_6.0.bb b/packages/coreutils/coreutils-native_6.0.bb
index ee90981..3a278b6 100644
--- a/packages/coreutils/coreutils-native_6.0.bb
+++ b/packages/coreutils/coreutils-native_6.0.bb
@@ -5,3 +5,4 @@ S = "${WORKDIR}/coreutils-${PV}"
 
 require coreutils_${PV}.bb
 inherit native
+require coreutils-native.inc


-- 

------- ROAD ...the handyPC Company - - -  ) ) )

Uli Luckas
Head of Software Development

ROAD GmbH
Bennigsenstr. 14 | 12159 Berlin | Germany
fon: +49 (30) 230069 - 62 | fax: +49 (30) 230069 - 69
url: www.road.de

Amtsgericht Charlottenburg: HRB 96688 B
Managing director: Hans-Peter Constien

Re: coreutils-native race

[Leon Woestenberg]

Hello Uli,

On Fri, Aug 14, 2009 at 5:10 PM, Uli Luckas <u.luckas@road.de> wrote:
> build process trys to delete some file. This leads to "rm: text file busy"
> because rm is still open for writing.
>
> Has anyone seen this? And is there a known fix or workaround?
>
Yes I have seen this (every 1 out of 50 builds or so). Thanks for
diving into this one.

(Of course, only seen when building from scratch, which I now default
to because of these kind of bugs.)


Cheers,
-- 
Leon