[PATCH 0/1 V2] telepathy-mission-control: do_compile failed (race issue)

[PATCH 0/1 V2] telepathy-mission-control: do_compile failed (race issue)

[Robert Yang]

* Changes of V2:
  - Use the similar fix from telepathy-glib to fix the problem as Ross suggested.

// Robert

The following changes since commit bf6b0d95db3863a2cfbccb154c0c7ad5b3bbf0b0:

  glib-2.0: fix broken python script header on machines using buildtools (2013-09-22 12:23:35 +0100)

are available in the git repository at:

  git://git.pokylinux.org/poky-contrib robert/tmc
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=robert/tmc

Robert Yang (1):
  telepathy-mission-control: do_compile failed (race issue)

 .../tmc-Makefile-fix-race.patch                    |   76 ++++++++++++++++++++
 .../telepathy/telepathy-mission-control_5.15.0.bb  |    4 +-
 2 files changed, 79 insertions(+), 1 deletion(-)
 create mode 100644 meta/recipes-connectivity/telepathy/telepathy-mission-control/tmc-Makefile-fix-race.patch

-- 
1.7.10.4

[PATCH 1/1] telepathy-mission-control: do_compile failed (race issue)

[Robert Yang]

There might be an error when parallel build:

[snip]
Traceback (most recent call last):
  File "/path/to/tools/glib-gtypes-generator.py", line 304, in <module>
    GTypesGenerator(dom, argv[1], argv[2])()
  File "/path/to/tools/glib-gtypes-generator.py", line 295, in __call__
    file_set_contents(self.output + '.h', ''.join(self.header))
  File "/path/to/tools/libtpcodegen.py", line 42, in file_set_contents
    os.rename(filename + '.tmp', filename)
OSError: [Errno 2] No such file or directory
[snip]

This is a race issue, the _gen/gtypes.h and _gen/gtypes-body.h may
write(remove/rename) _gen/gtypes.tmp at the same time, then there would
be the error.

There was a similar bug in telepathy-glib which was already fixed, we use the
similar patch to fix it here.

[YOCTO #5184]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 .../tmc-Makefile-fix-race.patch                    |   76 ++++++++++++++++++++
 .../telepathy/telepathy-mission-control_5.15.0.bb  |    4 +-
 2 files changed, 79 insertions(+), 1 deletion(-)
 create mode 100644 meta/recipes-connectivity/telepathy/telepathy-mission-control/tmc-Makefile-fix-race.patch

diff --git a/meta/recipes-connectivity/telepathy/telepathy-mission-control/tmc-Makefile-fix-race.patch b/meta/recipes-connectivity/telepathy/telepathy-mission-control/tmc-Makefile-fix-race.patch
new file mode 100644
index 0000000..ece1da6
--- /dev/null
+++ b/meta/recipes-connectivity/telepathy/telepathy-mission-control/tmc-Makefile-fix-race.patch
@@ -0,0 +1,76 @@
+From: Robert Yang <liezhi.yang@windriver.com>
+Date: Sun, 22 Sep 2013 23:21:01 -0400
+Subject: [PATCH] src/Makefile.am: fix race issue for _gen/gtypes.h and _gen/gtypes-body.h
+
+There might be an error when parallel build:
+
+[snip]
+Traceback (most recent call last):
+  File "/path/to/tools/glib-gtypes-generator.py", line 304, in <module>
+    GTypesGenerator(dom, argv[1], argv[2])()
+  File "/path/to/tools/glib-gtypes-generator.py", line 295, in __call__
+    file_set_contents(self.output + '.h', ''.join(self.header))
+  File "/path/to/tools/libtpcodegen.py", line 42, in file_set_contents
+    os.rename(filename + '.tmp', filename)
+OSError: [Errno 2] No such file or directory
+[snip]
+
+This is a race issue, the _gen/gtypes.h and _gen/gtypes-body.h may
+write(remove/rename) _gen/gtypes.tmp at the same time, then there would
+be the error.
+
+There was a similar bug in telepathy-glib which was already fixed, we use the
+similar patch to fix it.
+
+Upstream-Status: Pending
+
+Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
+---
+ src/Makefile.am |   18 +++++++++++++++---
+ 1 file changed, 15 insertions(+), 3 deletions(-)
+
+diff --git a/src/Makefile.am b/src/Makefile.am
+--- a/src/Makefile.am
++++ b/src/Makefile.am
+@@ -288,7 +288,11 @@ _gen/interfaces-body.h: _gen/mcd.xml \
+ 		$(tools_dir)/glib-interfaces-body-generator.xsl \
+ 		$< > $@
+ 
+-_gen/gtypes.h _gen/gtypes-body.h: _gen/mcd.xml \
++# do nothing, output as a side-effect
++_gen/gtypes.h: _gen/gtypes-body.h
++	@:
++
++_gen/gtypes-body.h: _gen/mcd.xml \
+ 	$(top_srcdir)/tools/glib-gtypes-generator.py
+ 	$(AM_V_GEN)$(PYTHON) $(top_srcdir)/tools/glib-gtypes-generator.py \
+ 		$< _gen/gtypes mc
+@@ -309,7 +313,11 @@ _gen/%.xml: %.xml $(wildcard $(top_srcdir)/xml/*.xml) Makefile.am
+ 	$(AM_V_GEN)$(XSLTPROC) $(XSLTPROCFLAGS) --xinclude $(tools_dir)/identity.xsl \
+ 		$< > $@
+ 
+-_gen/cli-%-body.h _gen/cli-%.h: _gen/%.xml \
++# do nothing, output as a side-effect
++_gen/cli-%.h: _gen/cli-%-body.h
++	@:
++
++_gen/cli-%-body.h: _gen/%.xml \
+ 	$(tools_dir)/glib-client-gen.py Makefile.am
+ 	$(AM_V_GEN)$(PYTHON) $(tools_dir)/glib-client-gen.py \
+ 		--group=`echo $* | tr x- x_` \
+@@ -317,7 +325,11 @@ _gen/cli-%-body.h _gen/cli-%.h: _gen/%.xml \
+ 		--tp-proxy-api=0.7.6 \
+ 		$< Mc_Cli _gen/cli-$*
+ 
+-_gen/svc-%.c _gen/svc-%.h: _gen/%.xml \
++# do nothing, output as a side-effect
++_gen/svc-%.h: _gen/svc-%.c
++	@:
++
++_gen/svc-%.c: _gen/%.xml \
+ 	$(tools_dir)/glib-ginterface-gen.py Makefile.am
+ 	$(AM_V_GEN)$(PYTHON) $(tools_dir)/glib-ginterface-gen.py \
+ 		--filename=_gen/svc-$* \
+-- 
+1.7.10.4
+
diff --git a/meta/recipes-connectivity/telepathy/telepathy-mission-control_5.15.0.bb b/meta/recipes-connectivity/telepathy/telepathy-mission-control_5.15.0.bb
index b2736ca..aa776a1 100644
--- a/meta/recipes-connectivity/telepathy/telepathy-mission-control_5.15.0.bb
+++ b/meta/recipes-connectivity/telepathy/telepathy-mission-control_5.15.0.bb
@@ -6,7 +6,9 @@ LIC_FILES_CHKSUM = "file://COPYING;md5=d8045f3b8f929c1cb29a1e3fd737b499 \
 SECTION = "libs"
 DEPENDS = "libtelepathy dbus-glib gconf libxslt-native"
 
-SRC_URI = "http://telepathy.freedesktop.org/releases/telepathy-mission-control/telepathy-mission-control-${PV}.tar.gz"
+SRC_URI = "http://telepathy.freedesktop.org/releases/telepathy-mission-control/telepathy-mission-control-${PV}.tar.gz \
+	file://tmc-Makefile-fix-race.patch \
+	"
 SRC_URI[md5sum] = "0efd7d9043f6e189e8ed8111d043b52c"
 SRC_URI[sha256sum] = "88800d1b5f385e94b34721319ff4fe65c493632954d7229b36e57af5b4a7e1b5"
 
-- 
1.7.10.4

Re: [PATCH 1/1] telepathy-mission-control: do_compile failed (race issue)

[Burton, Ross]

On 23 September 2013 06:54, Robert Yang <liezhi.yang@windriver.com> wrote:
> This is a race issue, the _gen/gtypes.h and _gen/gtypes-body.h may
> write(remove/rename) _gen/gtypes.tmp at the same time, then there would
> be the error.

Looks good.  Can you submit that patch against the upstream bug at
https://bugs.freedesktop.org/show_bug.cgi?id=69446 too?

Acked-By: Ross Burton <ross.burton@intel.com>

Ross

Re: [PATCH 1/1] telepathy-mission-control: do_compile failed (race issue)

[Robert Yang]

On 09/23/2013 10:07 PM, Burton, Ross wrote:
> On 23 September 2013 06:54, Robert Yang <liezhi.yang@windriver.com> wrote:
>> This is a race issue, the _gen/gtypes.h and _gen/gtypes-body.h may
>> write(remove/rename) _gen/gtypes.tmp at the same time, then there would
>> be the error.
>
> Looks good.  Can you submit that patch against the upstream bug at
> https://bugs.freedesktop.org/show_bug.cgi?id=69446 too?

thanks, I will.

// Robert

>
> Acked-By: Ross Burton <ross.burton@intel.com>
>
> Ross
>
>

[PATCH 0/1] telepathy-mission-control: do_compile failed (race issue)

[Robert Yang]

The following changes since commit 7c85b0d6dad9410b43842a18a372ed3331c96e27:

  linux-libc-headers: do_install: fix "Argument list too long" error (2013-09-13 22:28:46 -0400)

are available in the git repository at:

  git://git.pokylinux.org/poky-contrib robert/tmc
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=robert/tmc

Robert Yang (1):
  telepathy-mission-control: do_compile failed (race issue)

 .../tmc-Makefile-fix-race.patch                    |   47 ++++++++++++++++++++
 .../telepathy/telepathy-mission-control_5.15.0.bb  |    5 ++-
 2 files changed, 51 insertions(+), 1 deletion(-)
 create mode 100644 meta/recipes-connectivity/telepathy/telepathy-mission-control/tmc-Makefile-fix-race.patch

-- 
1.7.10.4

[PATCH 1/1] telepathy-mission-control: do_compile failed (race issue)

[Robert Yang]

There might be an error when parallel build:

[snip]
Traceback (most recent call last):
  File "/path/to/tools/glib-gtypes-generator.py", line 304, in <module>
    GTypesGenerator(dom, argv[1], argv[2])()
  File "/path/to/tools/glib-gtypes-generator.py", line 295, in __call__
    file_set_contents(self.output + '.h', ''.join(self.header))
  File "/path/to/tools/libtpcodegen.py", line 42, in file_set_contents
    os.rename(filename + '.tmp', filename)
OSError: [Errno 2] No such file or directory
[snip]

This is a race issue, the _gen/gtypes.h and _gen/gtypes-body.h may
write(remove/rename) _gen/gtypes.tmp at the same time, then there would
be the error.

Use ".NOTPARALLEL: _gen/gtypes.h _gen/gtypes-body.h" will fix the
problem.

[YOCTO #5184]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 .../tmc-Makefile-fix-race.patch                    |   47 ++++++++++++++++++++
 .../telepathy/telepathy-mission-control_5.15.0.bb  |    5 ++-
 2 files changed, 51 insertions(+), 1 deletion(-)
 create mode 100644 meta/recipes-connectivity/telepathy/telepathy-mission-control/tmc-Makefile-fix-race.patch

diff --git a/meta/recipes-connectivity/telepathy/telepathy-mission-control/tmc-Makefile-fix-race.patch b/meta/recipes-connectivity/telepathy/telepathy-mission-control/tmc-Makefile-fix-race.patch
new file mode 100644
index 0000000..537e1d7
--- /dev/null
+++ b/meta/recipes-connectivity/telepathy/telepathy-mission-control/tmc-Makefile-fix-race.patch
@@ -0,0 +1,47 @@
+From 029fcd600707f25a73b0171497140dab264f56b7 Mon Sep 17 00:00:00 2001
+From: Robert Yang <liezhi.yang@windriver.com>
+Date: Sat, 14 Sep 2013 04:26:46 -0400
+Subject: [PATCH] src/Makefile.am: let _gen/gtypes.h and _gen/gtypes-body.h run serially
+
+There might be an error when parallel build:
+
+[snip]
+Traceback (most recent call last):
+  File "/path/to/tools/glib-gtypes-generator.py", line 304, in <module>
+    GTypesGenerator(dom, argv[1], argv[2])()
+  File "/path/to/tools/glib-gtypes-generator.py", line 295, in __call__
+    file_set_contents(self.output + '.h', ''.join(self.header))
+  File "/path/to/tools/libtpcodegen.py", line 42, in file_set_contents
+    os.rename(filename + '.tmp', filename)
+OSError: [Errno 2] No such file or directory
+[snip]
+
+This is a race issue, the _gen/gtypes.h and _gen/gtypes-body.h may
+write(remove/rename) _gen/gtypes.tmp at the same time, then there would
+be the error.
+
+Use ".NOTPARALLEL: _gen/gtypes.h _gen/gtypes-body.h" will fix the
+problem.
+
+Upstream-Status: Pending
+
+Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
+---
+ src/Makefile.am |    2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/src/Makefile.am b/src/Makefile.am
+--- a/src/Makefile.am
++++ b/src/Makefile.am
+@@ -293,7 +293,7 @@ _gen/gtypes.h _gen/gtypes-body.h: _gen/mcd.xml \
+ 	$(AM_V_GEN)$(PYTHON) $(top_srcdir)/tools/glib-gtypes-generator.py \
+ 		$< _gen/gtypes mc
+ 
+-
++.NOTPARALLEL: _gen/gtypes.h _gen/gtypes-body.h
+ 
+ # Generated files which must be generated per "class".
+ # (Currently the only "class" is nmc4, but the new API will need "classes"
+-- 
+1.7.10.4
+
diff --git a/meta/recipes-connectivity/telepathy/telepathy-mission-control_5.15.0.bb b/meta/recipes-connectivity/telepathy/telepathy-mission-control_5.15.0.bb
index b2736ca..91159e1 100644
--- a/meta/recipes-connectivity/telepathy/telepathy-mission-control_5.15.0.bb
+++ b/meta/recipes-connectivity/telepathy/telepathy-mission-control_5.15.0.bb
@@ -6,7 +6,10 @@ LIC_FILES_CHKSUM = "file://COPYING;md5=d8045f3b8f929c1cb29a1e3fd737b499 \
 SECTION = "libs"
 DEPENDS = "libtelepathy dbus-glib gconf libxslt-native"
 
-SRC_URI = "http://telepathy.freedesktop.org/releases/telepathy-mission-control/telepathy-mission-control-${PV}.tar.gz"
+SRC_URI = "http://telepathy.freedesktop.org/releases/telepathy-mission-control/telepathy-mission-control-${PV}.tar.gz \
+	   file://tmc-Makefile-fix-race.patch \
+	"
+
 SRC_URI[md5sum] = "0efd7d9043f6e189e8ed8111d043b52c"
 SRC_URI[sha256sum] = "88800d1b5f385e94b34721319ff4fe65c493632954d7229b36e57af5b4a7e1b5"
 
-- 
1.7.10.4

Re: [PATCH 1/1] telepathy-mission-control: do_compile failed (race issue)

[Burton, Ross]

On 14 September 2013 09:49, Robert Yang <liezhi.yang@windriver.com> wrote:
> There might be an error when parallel build:
>
> [snip]
> Traceback (most recent call last):
>   File "/path/to/tools/glib-gtypes-generator.py", line 304, in <module>
>     GTypesGenerator(dom, argv[1], argv[2])()
>   File "/path/to/tools/glib-gtypes-generator.py", line 295, in __call__
>     file_set_contents(self.output + '.h', ''.join(self.header))
>   File "/path/to/tools/libtpcodegen.py", line 42, in file_set_contents
>     os.rename(filename + '.tmp', filename)
> OSError: [Errno 2] No such file or directory
> [snip]
>
> This is a race issue, the _gen/gtypes.h and _gen/gtypes-body.h may
> write(remove/rename) _gen/gtypes.tmp at the same time, then there would
> be the error.
>
> Use ".NOTPARALLEL: _gen/gtypes.h _gen/gtypes-body.h" will fix the
> problem.

This is going to be a game of whack-a-mole as the root cause is the
bad temporary filename assignment in libtpcodegen.py.  Is there an
upstream bug for this?

Ross

Re: [PATCH 1/1] telepathy-mission-control: do_compile failed (race issue)

[Burton, Ross]

On 17 September 2013 00:08, Burton, Ross <ross.burton@intel.com> wrote:
>> This is a race issue, the _gen/gtypes.h and _gen/gtypes-body.h may
>> write(remove/rename) _gen/gtypes.tmp at the same time, then there would
>> be the error.
>>
>> Use ".NOTPARALLEL: _gen/gtypes.h _gen/gtypes-body.h" will fix the
>> problem.
>
> This is going to be a game of whack-a-mole as the root cause is the
> bad temporary filename assignment in libtpcodegen.py.  Is there an
> upstream bug for this?

Had a quick look - there's a trivial race in the utility code that a
bit of tempfile can fix, but the more correct fix for
telepathy-mission-control is to do the same fix that telepathy-glib
did here:

http://cgit.freedesktop.org/telepathy/telepathy-glib/commit/?id=36c2a545c9c1d1cc6db205bfc33d980d29b0a0f6

https://bugs.freedesktop.org/show_bug.cgi?id=69446 is the upstream bug
I just filed for this.

Ross

Re: [PATCH 1/1] telepathy-mission-control: do_compile failed (race issue)

[Robert Yang]

On 09/17/2013 07:49 AM, Burton, Ross wrote:
> On 17 September 2013 00:08, Burton, Ross <ross.burton@intel.com> wrote:
>>> This is a race issue, the _gen/gtypes.h and _gen/gtypes-body.h may
>>> write(remove/rename) _gen/gtypes.tmp at the same time, then there would
>>> be the error.
>>>
>>> Use ".NOTPARALLEL: _gen/gtypes.h _gen/gtypes-body.h" will fix the
>>> problem.
>>
>> This is going to be a game of whack-a-mole as the root cause is the
>> bad temporary filename assignment in libtpcodegen.py.  Is there an
>> upstream bug for this?
>
> Had a quick look - there's a trivial race in the utility code that a
> bit of tempfile can fix, but the more correct fix for
> telepathy-mission-control is to do the same fix that telepathy-glib
> did here:
>
> http://cgit.freedesktop.org/telepathy/telepathy-glib/commit/?id=36c2a545c9c1d1cc6db205bfc33d980d29b0a0f6

I don't think that the commit you mentioned is a correct fix, he changed:

gen/gtypes.h _gen/gtypes-body.h: _gen/stable-spec.xml

To:

gen/gtypes.h: _gen/gtypes-body.h
	@:
gen/gtypes-body.h: _gen/stable-spec.xml

Now if we run "make gen/gtypes.h gen/gtypes-body.h", the gen/gtypes-body.h
will only run once, this is incorrect. And this fix has been dropped in the
up to date source.

// Robert

>
> https://bugs.freedesktop.org/show_bug.cgi?id=69446 is the upstream bug
> I just filed for this.
>
> Ross
>
>

Re: [PATCH 1/1] telepathy-mission-control: do_compile failed (race issue)

[Burton, Ross]

On 17 September 2013 03:03, Robert Yang <liezhi.yang@windriver.com> wrote:
> I don't think that the commit you mentioned is a correct fix, he changed:
>
> gen/gtypes.h _gen/gtypes-body.h: _gen/stable-spec.xml
>
> To:
>
> gen/gtypes.h: _gen/gtypes-body.h
>         @:
> gen/gtypes-body.h: _gen/stable-spec.xml
>
> Now if we run "make gen/gtypes.h gen/gtypes-body.h", the gen/gtypes-body.h
> will only run once, this is incorrect.

But the Python script used in the second rule generates *both* files,
which is why the current rules in telepathy-mission-control are
incorrect:

~/Local/mess/37/telepathy-mission-control/src
$ ls _gen/gtypes*
_gen/gtypes-body.h  _gen/gtypes.h

~/Local/mess/37/telepathy-mission-control/src
$ rm _gen/gtypes* -f

~/Local/mess/37/telepathy-mission-control/src
$ make _gen/gtypes.h
/usr/bin/python ../tools/glib-gtypes-generator.py \
_gen/mcd.xml _gen/gtypes mc

~/Local/mess/37/telepathy-mission-control/src
$ ls _gen/gtypes*
_gen/gtypes-body.h  _gen/gtypes.h

The anti-pattern of:

foo bar:
  this-script-generates-both-files.sh

is wrong in parallel builds.

Ross

Re: [PATCH 1/1] telepathy-mission-control: do_compile failed (race issue)

[Robert Yang]

On 09/17/2013 07:12 PM, Burton, Ross wrote:
> On 17 September 2013 03:03, Robert Yang <liezhi.yang@windriver.com> wrote:
>> I don't think that the commit you mentioned is a correct fix, he changed:
>>
>> gen/gtypes.h _gen/gtypes-body.h: _gen/stable-spec.xml
>>
>> To:
>>
>> gen/gtypes.h: _gen/gtypes-body.h
>>          @:
>> gen/gtypes-body.h: _gen/stable-spec.xml
>>
>> Now if we run "make gen/gtypes.h gen/gtypes-body.h", the gen/gtypes-body.h
>> will only run once, this is incorrect.
>
> But the Python script used in the second rule generates *both* files,

Ah, yes, thanks, what's your opinion on my patch ? Is it OK or should it be
re-worked ? I think that this fix is simple and easy to understand.

// Robert

> which is why the current rules in telepathy-mission-control are
> incorrect:
>
> ~/Local/mess/37/telepathy-mission-control/src
> $ ls _gen/gtypes*
> _gen/gtypes-body.h  _gen/gtypes.h
>
> ~/Local/mess/37/telepathy-mission-control/src
> $ rm _gen/gtypes* -f
>
> ~/Local/mess/37/telepathy-mission-control/src
> $ make _gen/gtypes.h
> /usr/bin/python ../tools/glib-gtypes-generator.py \
> _gen/mcd.xml _gen/gtypes mc
>
> ~/Local/mess/37/telepathy-mission-control/src
> $ ls _gen/gtypes*
> _gen/gtypes-body.h  _gen/gtypes.h
>
> The anti-pattern of:
>
> foo bar:
>    this-script-generates-both-files.sh
>
> is wrong in parallel builds.
>
> Ross
>
>

Re: [PATCH 1/1] telepathy-mission-control: do_compile failed (race issue)

[Burton, Ross]

On 18 September 2013 03:45, Robert Yang <liezhi.yang@windriver.com> wrote:
> Ah, yes, thanks, what's your opinion on my patch ? Is it OK or should it be
> re-worked ? I think that this fix is simple and easy to understand.

It should be re-worked to use the pattern in the commit I mentioned -
using .NOTPARALLEL is working around the problem that the rule
dependencies are incorrect.

Ross