* Race issues (lzma-native)
@ 2010-10-24 8:59 Andrea Adami
2010-10-25 3:59 ` Graham Gower
0 siblings, 1 reply; 10+ messages in thread
From: Andrea Adami @ 2010-10-24 8:59 UTC (permalink / raw)
To: openembedded-devel
Hello,
during first build of console image I notice a strange fault:
subsequent builds are ok.
My scripts builds console-image and opie-image.
ERROR: Task 2411
(virtual:native:/oe/openembedded/recipes/lzma/lzma_4.65.bb,
do_distribute_sources) failed with exit code '1'
NOTE: Unpacking ../sources/lzma465.tar.bz2 to
tmp/work/i686-linux/lzma-native-4.65-r4.1/
NOTE: package lzma-native-4.65-r4.1: task do_distribute_sources: Started
NOTE: package lzma-native-4.65-r4.1: task SRC_DISTRIBUTECOMMAND: Failed
ERROR: Function 'SRC_DISTRIBUTECOMMAND' failed
Looking at the
run.SRC_DISTRIBUTECOMMAND.18268 16384 (failing)
run.SRC_DISTRIBUTECOMMAND.9570 34334
it looks like the file is not complete / malformed:
package_generate_ipkg_conf() {
<------>mkdir -p /oe/build/tmp/sysroots/i686-linux/etc/
<------>echo "src oe file:/oe/build/tmp/deploy/glibc/ipk" >
/oe/build/tmp/sysroots/i686-linux/etc/opkg.conf
<------>echo "src oe file:/
EOF
run.do_distribute_sources.18268 is empty, while second time
run.do_distribute_sources.9570 1051 contains def do_distribute_sources(d):
This seems indeed a race issue...
Andrea
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: Race issues (lzma-native) 2010-10-24 8:59 Race issues (lzma-native) Andrea Adami @ 2010-10-25 3:59 ` Graham Gower 2010-10-26 7:58 ` Andrea Adami 0 siblings, 1 reply; 10+ messages in thread From: Graham Gower @ 2010-10-25 3:59 UTC (permalink / raw) To: openembedded-devel On 10/24/2010 07:29 PM, Andrea Adami wrote: > Hello, > > during first build of console image I notice a strange fault: > subsequent builds are ok. > My scripts builds console-image and opie-image. > > ERROR: Task 2411 > (virtual:native:/oe/openembedded/recipes/lzma/lzma_4.65.bb, > do_distribute_sources) failed with exit code '1' > NOTE: Unpacking ../sources/lzma465.tar.bz2 to > tmp/work/i686-linux/lzma-native-4.65-r4.1/ > NOTE: package lzma-native-4.65-r4.1: task do_distribute_sources: Started > NOTE: package lzma-native-4.65-r4.1: task SRC_DISTRIBUTECOMMAND: Failed > ERROR: Function 'SRC_DISTRIBUTECOMMAND' failed > > > Looking at the > run.SRC_DISTRIBUTECOMMAND.18268 16384 (failing) > run.SRC_DISTRIBUTECOMMAND.9570 34334 > > it looks like the file is not complete / malformed: > > package_generate_ipkg_conf() { > <------>mkdir -p /oe/build/tmp/sysroots/i686-linux/etc/ > <------>echo "src oe file:/oe/build/tmp/deploy/glibc/ipk" > > /oe/build/tmp/sysroots/i686-linux/etc/opkg.conf > <------>echo "src oe file:/ > EOF > > run.do_distribute_sources.18268 is empty, while second time > run.do_distribute_sources.9570 1051 contains def do_distribute_sources(d): > > This seems indeed a race issue... > > Andrea > > _______________________________________________ > Openembedded-devel mailing list > Openembedded-devel@lists.openembedded.org > http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel I can reproduce this one quite easily. Reverting a6a23f73 seems to make it disappear. Alternately the change below works, but I'm not convinced this is the right fix. Can someone with greater insight explain what causes this race? -Graham diff --git a/classes/src_distribute.bbclass b/classes/src_distribute.bbclass index 795a5cf..98d8d88 100644 --- a/classes/src_distribute.bbclass +++ b/classes/src_distribute.bbclass @@ -1,7 +1,7 @@ SRC_DISTRIBUTE_DLONLY ?= "0" SRC_DISTRIBUTECOMMAND[func] = "1" -addtask distribute_sources before do_build after do_fetch +addtask distribute_sources before do_build after do_unpack python do_distribute_sources () { bb.build.exec_func("do_fetch", d) ^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: Race issues (lzma-native) 2010-10-25 3:59 ` Graham Gower @ 2010-10-26 7:58 ` Andrea Adami 2010-11-04 12:27 ` Andrea Adami 0 siblings, 1 reply; 10+ messages in thread From: Andrea Adami @ 2010-10-26 7:58 UTC (permalink / raw) To: openembedded-devel Just to add this time the race appears building on Ubuntu 10.4 32bit. Last race-case (gtk+-native) was only on Gentoo and not on Ubuntu... Both distros are on the same Quad host and share an EXT3 OpenEmbedded partition, though. PARALLEL_MAKE = "-j5" BB_NUMBER_THREADS = "4" Perplexed... Andrea On Mon, Oct 25, 2010 at 5:59 AM, Graham Gower <graham.gower@gmail.com> wrote: > On 10/24/2010 07:29 PM, Andrea Adami wrote: >> Hello, >> >> during first build of console image I notice a strange fault: >> subsequent builds are ok. >> My scripts builds console-image and opie-image. >> >> ERROR: Task 2411 >> (virtual:native:/oe/openembedded/recipes/lzma/lzma_4.65.bb, >> do_distribute_sources) failed with exit code '1' >> NOTE: Unpacking ../sources/lzma465.tar.bz2 to >> tmp/work/i686-linux/lzma-native-4.65-r4.1/ >> NOTE: package lzma-native-4.65-r4.1: task do_distribute_sources: Started >> NOTE: package lzma-native-4.65-r4.1: task SRC_DISTRIBUTECOMMAND: Failed >> ERROR: Function 'SRC_DISTRIBUTECOMMAND' failed >> >> >> Looking at the >> run.SRC_DISTRIBUTECOMMAND.18268 16384 (failing) >> run.SRC_DISTRIBUTECOMMAND.9570 34334 >> >> it looks like the file is not complete / malformed: >> >> package_generate_ipkg_conf() { >> <------>mkdir -p /oe/build/tmp/sysroots/i686-linux/etc/ >> <------>echo "src oe file:/oe/build/tmp/deploy/glibc/ipk" > >> /oe/build/tmp/sysroots/i686-linux/etc/opkg.conf >> <------>echo "src oe file:/ >> EOF >> >> run.do_distribute_sources.18268 is empty, while second time >> run.do_distribute_sources.9570 1051 contains def do_distribute_sources(d): >> >> This seems indeed a race issue... >> >> Andrea >> >> _______________________________________________ >> Openembedded-devel mailing list >> Openembedded-devel@lists.openembedded.org >> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel > > I can reproduce this one quite easily. Reverting a6a23f73 seems to make it > disappear. Alternately the change below works, but I'm not convinced this is > the right fix. > > Can someone with greater insight explain what causes this race? > > -Graham > > > diff --git a/classes/src_distribute.bbclass b/classes/src_distribute.bbclass > index 795a5cf..98d8d88 100644 > --- a/classes/src_distribute.bbclass > +++ b/classes/src_distribute.bbclass > @@ -1,7 +1,7 @@ > SRC_DISTRIBUTE_DLONLY ?= "0" > SRC_DISTRIBUTECOMMAND[func] = "1" > > -addtask distribute_sources before do_build after do_fetch > +addtask distribute_sources before do_build after do_unpack > python do_distribute_sources () { > bb.build.exec_func("do_fetch", d) > > > _______________________________________________ > Openembedded-devel mailing list > Openembedded-devel@lists.openembedded.org > http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel > ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Race issues (lzma-native) 2010-10-26 7:58 ` Andrea Adami @ 2010-11-04 12:27 ` Andrea Adami 2010-11-06 23:21 ` Andrea Adami 0 siblings, 1 reply; 10+ messages in thread From: Andrea Adami @ 2010-11-04 12:27 UTC (permalink / raw) To: openembedded-devel FWIW Issue is still present. I'll try the workaround you posted. Andrea On Tue, Oct 26, 2010 at 9:58 AM, Andrea Adami <andrea.adami@gmail.com> wrote: > Just to add this time the race appears building on Ubuntu 10.4 32bit. > Last race-case (gtk+-native) was only on Gentoo and not on Ubuntu... > > Both distros are on the same Quad host and share an EXT3 OpenEmbedded > partition, though. > > PARALLEL_MAKE = "-j5" > BB_NUMBER_THREADS = "4" > > > Perplexed... > > Andrea > > On Mon, Oct 25, 2010 at 5:59 AM, Graham Gower <graham.gower@gmail.com> wrote: >> On 10/24/2010 07:29 PM, Andrea Adami wrote: >>> Hello, >>> >>> during first build of console image I notice a strange fault: >>> subsequent builds are ok. >>> My scripts builds console-image and opie-image. >>> >>> ERROR: Task 2411 >>> (virtual:native:/oe/openembedded/recipes/lzma/lzma_4.65.bb, >>> do_distribute_sources) failed with exit code '1' >>> NOTE: Unpacking ../sources/lzma465.tar.bz2 to >>> tmp/work/i686-linux/lzma-native-4.65-r4.1/ >>> NOTE: package lzma-native-4.65-r4.1: task do_distribute_sources: Started >>> NOTE: package lzma-native-4.65-r4.1: task SRC_DISTRIBUTECOMMAND: Failed >>> ERROR: Function 'SRC_DISTRIBUTECOMMAND' failed >>> >>> >>> Looking at the >>> run.SRC_DISTRIBUTECOMMAND.18268 16384 (failing) >>> run.SRC_DISTRIBUTECOMMAND.9570 34334 >>> >>> it looks like the file is not complete / malformed: >>> >>> package_generate_ipkg_conf() { >>> <------>mkdir -p /oe/build/tmp/sysroots/i686-linux/etc/ >>> <------>echo "src oe file:/oe/build/tmp/deploy/glibc/ipk" > >>> /oe/build/tmp/sysroots/i686-linux/etc/opkg.conf >>> <------>echo "src oe file:/ >>> EOF >>> >>> run.do_distribute_sources.18268 is empty, while second time >>> run.do_distribute_sources.9570 1051 contains def do_distribute_sources(d): >>> >>> This seems indeed a race issue... >>> >>> Andrea >>> >>> _______________________________________________ >>> Openembedded-devel mailing list >>> Openembedded-devel@lists.openembedded.org >>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel >> >> I can reproduce this one quite easily. Reverting a6a23f73 seems to make it >> disappear. Alternately the change below works, but I'm not convinced this is >> the right fix. >> >> Can someone with greater insight explain what causes this race? >> >> -Graham >> >> >> diff --git a/classes/src_distribute.bbclass b/classes/src_distribute.bbclass >> index 795a5cf..98d8d88 100644 >> --- a/classes/src_distribute.bbclass >> +++ b/classes/src_distribute.bbclass >> @@ -1,7 +1,7 @@ >> SRC_DISTRIBUTE_DLONLY ?= "0" >> SRC_DISTRIBUTECOMMAND[func] = "1" >> >> -addtask distribute_sources before do_build after do_fetch >> +addtask distribute_sources before do_build after do_unpack >> python do_distribute_sources () { >> bb.build.exec_func("do_fetch", d) >> >> >> _______________________________________________ >> Openembedded-devel mailing list >> Openembedded-devel@lists.openembedded.org >> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel >> > ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Race issues (lzma-native) 2010-11-04 12:27 ` Andrea Adami @ 2010-11-06 23:21 ` Andrea Adami 2010-11-22 0:12 ` Andrea Adami 0 siblings, 1 reply; 10+ messages in thread From: Andrea Adami @ 2010-11-06 23:21 UTC (permalink / raw) To: openembedded-devel Strange... Built today with Metadata rev fda42d9 and no issues. Tried on Ubuntu and Gentoo 32bit Regards Andrea On Thu, Nov 4, 2010 at 1:27 PM, Andrea Adami <andrea.adami@gmail.com> wrote: > FWIW Issue is still present. > > I'll try the workaround you posted. > > Andrea > > > > On Tue, Oct 26, 2010 at 9:58 AM, Andrea Adami <andrea.adami@gmail.com> wrote: >> Just to add this time the race appears building on Ubuntu 10.4 32bit. >> Last race-case (gtk+-native) was only on Gentoo and not on Ubuntu... >> >> Both distros are on the same Quad host and share an EXT3 OpenEmbedded >> partition, though. >> >> PARALLEL_MAKE = "-j5" >> BB_NUMBER_THREADS = "4" >> >> >> Perplexed... >> >> Andrea >> >> On Mon, Oct 25, 2010 at 5:59 AM, Graham Gower <graham.gower@gmail.com> wrote: >>> On 10/24/2010 07:29 PM, Andrea Adami wrote: >>>> Hello, >>>> >>>> during first build of console image I notice a strange fault: >>>> subsequent builds are ok. >>>> My scripts builds console-image and opie-image. >>>> >>>> ERROR: Task 2411 >>>> (virtual:native:/oe/openembedded/recipes/lzma/lzma_4.65.bb, >>>> do_distribute_sources) failed with exit code '1' >>>> NOTE: Unpacking ../sources/lzma465.tar.bz2 to >>>> tmp/work/i686-linux/lzma-native-4.65-r4.1/ >>>> NOTE: package lzma-native-4.65-r4.1: task do_distribute_sources: Started >>>> NOTE: package lzma-native-4.65-r4.1: task SRC_DISTRIBUTECOMMAND: Failed >>>> ERROR: Function 'SRC_DISTRIBUTECOMMAND' failed >>>> >>>> >>>> Looking at the >>>> run.SRC_DISTRIBUTECOMMAND.18268 16384 (failing) >>>> run.SRC_DISTRIBUTECOMMAND.9570 34334 >>>> >>>> it looks like the file is not complete / malformed: >>>> >>>> package_generate_ipkg_conf() { >>>> <------>mkdir -p /oe/build/tmp/sysroots/i686-linux/etc/ >>>> <------>echo "src oe file:/oe/build/tmp/deploy/glibc/ipk" > >>>> /oe/build/tmp/sysroots/i686-linux/etc/opkg.conf >>>> <------>echo "src oe file:/ >>>> EOF >>>> >>>> run.do_distribute_sources.18268 is empty, while second time >>>> run.do_distribute_sources.9570 1051 contains def do_distribute_sources(d): >>>> >>>> This seems indeed a race issue... >>>> >>>> Andrea >>>> >>>> _______________________________________________ >>>> Openembedded-devel mailing list >>>> Openembedded-devel@lists.openembedded.org >>>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel >>> >>> I can reproduce this one quite easily. Reverting a6a23f73 seems to make it >>> disappear. Alternately the change below works, but I'm not convinced this is >>> the right fix. >>> >>> Can someone with greater insight explain what causes this race? >>> >>> -Graham >>> >>> >>> diff --git a/classes/src_distribute.bbclass b/classes/src_distribute.bbclass >>> index 795a5cf..98d8d88 100644 >>> --- a/classes/src_distribute.bbclass >>> +++ b/classes/src_distribute.bbclass >>> @@ -1,7 +1,7 @@ >>> SRC_DISTRIBUTE_DLONLY ?= "0" >>> SRC_DISTRIBUTECOMMAND[func] = "1" >>> >>> -addtask distribute_sources before do_build after do_fetch >>> +addtask distribute_sources before do_build after do_unpack >>> python do_distribute_sources () { >>> bb.build.exec_func("do_fetch", d) >>> >>> >>> _______________________________________________ >>> Openembedded-devel mailing list >>> Openembedded-devel@lists.openembedded.org >>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel >>> >> > ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Race issues (lzma-native) 2010-11-06 23:21 ` Andrea Adami @ 2010-11-22 0:12 ` Andrea Adami 2010-11-22 0:28 ` Graham Gower 0 siblings, 1 reply; 10+ messages in thread From: Andrea Adami @ 2010-11-22 0:12 UTC (permalink / raw) To: openembedded-devel I though tthe issue had disappeared (in the meanwhile I remove an obsolete dependency on zlib) but yesterday again... run.SRC_DISTRIBUTECOMMAND was malformed, broken at line 122 (export docdir="/oe/bu ). On second build the size is much bigger (1102 lines). Retrying.. Andrea ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Race issues (lzma-native) 2010-11-22 0:12 ` Andrea Adami @ 2010-11-22 0:28 ` Graham Gower 2010-11-22 6:54 ` Frans Meulenbroeks 0 siblings, 1 reply; 10+ messages in thread From: Graham Gower @ 2010-11-22 0:28 UTC (permalink / raw) To: openembedded-devel On 22 November 2010 10:42, Andrea Adami <andrea.adami@gmail.com> wrote: > I though tthe issue had disappeared (in the meanwhile I remove an > obsolete dependency on zlib) > but yesterday again... > > run.SRC_DISTRIBUTECOMMAND was malformed, broken at line 122 (export > docdir="/oe/bu ). > > On second build the size is much bigger (1102 lines). > > Retrying.. > Andrea > > _______________________________________________ > Openembedded-devel mailing list > Openembedded-devel@lists.openembedded.org > http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel > With pstaging disabled, do_fetch is called twice. The second do_fetch races with do_unpack. Presumably we see a problem in lzma-native because it does oe.process.run() in do_unpack_append. I can't see exactly *what* is racing though. I previously spoke to Kergoth about it on IRC and he suggested making bb.build.exec_func("do_fetch", d) conditional on whether the do_fetch stamp file exists... I've not gotten around to doing that (bad me). Want to try it? -Graham ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Race issues (lzma-native) 2010-11-22 0:28 ` Graham Gower @ 2010-11-22 6:54 ` Frans Meulenbroeks 2010-11-22 20:40 ` Graham Gower 0 siblings, 1 reply; 10+ messages in thread From: Frans Meulenbroeks @ 2010-11-22 6:54 UTC (permalink / raw) To: openembedded-devel 2010/11/22 Graham Gower <graham.gower@gmail.com>: > On 22 November 2010 10:42, Andrea Adami <andrea.adami@gmail.com> wrote: >> I though tthe issue had disappeared (in the meanwhile I remove an >> obsolete dependency on zlib) >> but yesterday again... >> >> run.SRC_DISTRIBUTECOMMAND was malformed, broken at line 122 (export >> docdir="/oe/bu ). >> >> On second build the size is much bigger (1102 lines). >> >> Retrying.. >> Andrea >> >> _______________________________________________ >> Openembedded-devel mailing list >> Openembedded-devel@lists.openembedded.org >> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel >> > > With pstaging disabled, do_fetch is called twice. The second do_fetch > races with do_unpack. Presumably we see a problem in lzma-native > because it does oe.process.run() in do_unpack_append. I can't see > exactly *what* is racing though. > > I previously spoke to Kergoth about it on IRC and he suggested making > bb.build.exec_func("do_fetch", d) conditional on whether the do_fetch > stamp file exists... I've not gotten around to doing that (bad me). > Want to try it? Hm. I don't know the exact details, but this looks like a bandage, not like a fix for the root problem. Why is do_fetch called twice in the first place. I would expect this not to happen. Frans ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Race issues (lzma-native) 2010-11-22 6:54 ` Frans Meulenbroeks @ 2010-11-22 20:40 ` Graham Gower 2010-11-22 21:23 ` Frans Meulenbroeks 0 siblings, 1 reply; 10+ messages in thread From: Graham Gower @ 2010-11-22 20:40 UTC (permalink / raw) To: openembedded-devel On 22 November 2010 17:24, Frans Meulenbroeks <fransmeulenbroeks@gmail.com> wrote: > 2010/11/22 Graham Gower <graham.gower@gmail.com>: >> On 22 November 2010 10:42, Andrea Adami <andrea.adami@gmail.com> wrote: >>> I though tthe issue had disappeared (in the meanwhile I remove an >>> obsolete dependency on zlib) >>> but yesterday again... >>> >>> run.SRC_DISTRIBUTECOMMAND was malformed, broken at line 122 (export >>> docdir="/oe/bu ). >>> >>> On second build the size is much bigger (1102 lines). >>> >>> Retrying.. >>> Andrea >>> >>> _______________________________________________ >>> Openembedded-devel mailing list >>> Openembedded-devel@lists.openembedded.org >>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel >>> >> >> With pstaging disabled, do_fetch is called twice. The second do_fetch >> races with do_unpack. Presumably we see a problem in lzma-native >> because it does oe.process.run() in do_unpack_append. I can't see >> exactly *what* is racing though. >> >> I previously spoke to Kergoth about it on IRC and he suggested making >> bb.build.exec_func("do_fetch", d) conditional on whether the do_fetch >> stamp file exists... I've not gotten around to doing that (bad me). >> Want to try it? > > Hm. > I don't know the exact details, but this looks like a bandage, not > like a fix for the root problem. The conditional call of do_fetch is probably what we want, but I'd prefer to see this race fixed before we do that to ensure we aren't masking any other related races. > Why is do_fetch called twice in the first place. I would expect this > not to happen. See src_distribute.bbclass and refer to do_distribute_sources(). do_fetch is only called twice if pstaging is disabled (which is not the default). -Graham ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Race issues (lzma-native) 2010-11-22 20:40 ` Graham Gower @ 2010-11-22 21:23 ` Frans Meulenbroeks 0 siblings, 0 replies; 10+ messages in thread From: Frans Meulenbroeks @ 2010-11-22 21:23 UTC (permalink / raw) To: openembedded-devel 2010/11/22 Graham Gower <graham.gower@gmail.com>: > > The conditional call of do_fetch is probably what we want, but I'd > prefer to see this race fixed before we do that to ensure we aren't > masking any other related races. There should definitely be a dependency on the stamp and if the stamp is newer the fetch should not be called (or generated) > >> Why is do_fetch called twice in the first place. I would expect this >> not to happen. > > See src_distribute.bbclass and refer to do_distribute_sources(). > do_fetch is only called twice if pstaging is disabled (which is not > the default). Without too much knowledge on the internals, I'd say the 2nd fetch should be conditional on pstaging then. Did a quick peek at that bbclass file: addtask distribute_sources before do_build after do_fetch python do_distribute_sources () { bb.build.exec_func("do_fetch", d) This greatly puzzles me. distribute_sources comes after do_fetch and it starts doing a do_fetch ??? Isn't it possible to just eliminate this 2nd fetch ??? Puzzled. Frans ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2010-11-22 21:25 UTC | newest] Thread overview: 10+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2010-10-24 8:59 Race issues (lzma-native) Andrea Adami 2010-10-25 3:59 ` Graham Gower 2010-10-26 7:58 ` Andrea Adami 2010-11-04 12:27 ` Andrea Adami 2010-11-06 23:21 ` Andrea Adami 2010-11-22 0:12 ` Andrea Adami 2010-11-22 0:28 ` Graham Gower 2010-11-22 6:54 ` Frans Meulenbroeks 2010-11-22 20:40 ` Graham Gower 2010-11-22 21:23 ` Frans Meulenbroeks
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.