* [PATCH] use multiple processes to dump signatures. @ 2016-12-21 20:27 Jianxun Zhang 2016-12-22 8:59 ` Richard Purdie 0 siblings, 1 reply; 7+ messages in thread From: Jianxun Zhang @ 2016-12-21 20:27 UTC (permalink / raw) To: bitbake-devel This change significantly shortens the time on reparsing stage of '-S' option. Each file is reparsed and then dumped within a dedicated process. The maximum number of the running processes is not greater than the value of BB_NUMBER_PARSE_THREADS if it is set. The dump_sigs() in class SignatureGeneratorBasic is _replaced_ by a new dump_sigfn() interface, so calls from the outside and subclasses are dispatched to the implementation in the base class of SignatureGeneratorBasic. Fixes [YOCTO #10352] Signed-off-by: Jianxun Zhang <jianxun.zhang@linux.intel.com> --- bitbake/lib/bb/runqueue.py | 32 +++++++++++++++++++++++++++----- bitbake/lib/bb/siggen.py | 4 ++-- 2 files changed, 29 insertions(+), 7 deletions(-) diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py index 2ad8aad..c7d8d53 100644 --- a/bitbake/lib/bb/runqueue.py +++ b/bitbake/lib/bb/runqueue.py @@ -36,6 +36,7 @@ from bb import msg, data, event from bb import monitordisk import subprocess import pickle +from multiprocessing import Process bblogger = logging.getLogger("BitBake") logger = logging.getLogger("BitBake.RunQueue") @@ -1302,15 +1303,36 @@ class RunQueue: else: self.rqexe.finish() + def rq_dump_sigfn(self, fn, options): + bb_cache = bb.cache.NoCache(self.cooker.databuilder) + the_data = bb_cache.loadDataFull(fn, self.cooker.collection.get_file_appends(fn)) + siggen = bb.parse.siggen + dataCaches = self.rqdata.dataCaches + siggen.dump_sigfn(fn, dataCaches, options) + def dump_signatures(self, options): - done = set() + fns = set() bb.note("Reparsing files to collect dependency data") - bb_cache = bb.cache.NoCache(self.cooker.databuilder) + for tid in self.rqdata.runtaskentries: fn = fn_from_tid(tid) - if fn not in done: - the_data = bb_cache.loadDataFull(fn, self.cooker.collection.get_file_appends(fn)) - done.add(fn) + fns.add(fn) + + max_process = int(self.cfgData.getVar("BB_NUMBER_PARSE_THREADS") or os.cpu_count() or 1) + # We cannot use the real multiprocessing.Pool easily due to some local data + # that can't be pickled. This is a cheap multi-process solution. + launched = [] + while fns: + if len(launched) < max_process: + p = Process(target=self.rq_dump_sigfn, args=(fns.pop(), options)) + p.start() + launched.append(p) + for q in launched: + # The finished processes are joined when calling is_alive() + if not q.is_alive(): + launched.remove(q) + for p in launched: + p.join() bb.parse.siggen.dump_sigs(self.rqdata.dataCaches, options) diff --git a/bitbake/lib/bb/siggen.py b/bitbake/lib/bb/siggen.py index b20b9cf..ae50a18 100644 --- a/bitbake/lib/bb/siggen.py +++ b/bitbake/lib/bb/siggen.py @@ -307,8 +307,8 @@ class SignatureGeneratorBasic(SignatureGenerator): pass raise err - def dump_sigs(self, dataCaches, options): - for fn in self.taskdeps: + def dump_sigfn(self, fn, dataCaches, options): + if fn in self.taskdeps: for task in self.taskdeps[fn]: tid = fn + ":" + task (mc, _, _) = bb.runqueue.split_tid(tid) -- 2.7.4 ^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH] use multiple processes to dump signatures. 2016-12-21 20:27 [PATCH] use multiple processes to dump signatures Jianxun Zhang @ 2016-12-22 8:59 ` Richard Purdie 2016-12-22 18:39 ` Jianxun Zhang 0 siblings, 1 reply; 7+ messages in thread From: Richard Purdie @ 2016-12-22 8:59 UTC (permalink / raw) To: Jianxun Zhang, bitbake-devel On Wed, 2016-12-21 at 12:27 -0800, Jianxun Zhang wrote: > This change significantly shortens the time on reparsing stage > of '-S' option. > > Each file is reparsed and then dumped within a dedicated > process. The maximum number of the running processes is not > greater than the value of BB_NUMBER_PARSE_THREADS if it is set. > > The dump_sigs() in class SignatureGeneratorBasic is _replaced_ > by a new dump_sigfn() interface, so calls from the outside and > subclasses are dispatched to the implementation in the base > class of SignatureGeneratorBasic. > > Fixes [YOCTO #10352] Thanks, I think this is heading in the right direction. I am a little bit worried that this leaves OE's sstatesig.py with a dump_sigs() function which isn't used/connected into everything else though? Does this still write out a locked sigs file after this change? Cheers, Richard > Signed-off-by: Jianxun Zhang <jianxun.zhang@linux.intel.com> > --- > bitbake/lib/bb/runqueue.py | 32 +++++++++++++++++++++++++++----- > bitbake/lib/bb/siggen.py | 4 ++-- > 2 files changed, 29 insertions(+), 7 deletions(-) > > diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py > index 2ad8aad..c7d8d53 100644 > --- a/bitbake/lib/bb/runqueue.py > +++ b/bitbake/lib/bb/runqueue.py > @@ -36,6 +36,7 @@ from bb import msg, data, event > from bb import monitordisk > import subprocess > import pickle > +from multiprocessing import Process > > bblogger = logging.getLogger("BitBake") > logger = logging.getLogger("BitBake.RunQueue") > @@ -1302,15 +1303,36 @@ class RunQueue: > else: > self.rqexe.finish() > > + def rq_dump_sigfn(self, fn, options): > + bb_cache = bb.cache.NoCache(self.cooker.databuilder) > + the_data = bb_cache.loadDataFull(fn, > self.cooker.collection.get_file_appends(fn)) > + siggen = bb.parse.siggen > + dataCaches = self.rqdata.dataCaches > + siggen.dump_sigfn(fn, dataCaches, options) > + > def dump_signatures(self, options): > - done = set() > + fns = set() > bb.note("Reparsing files to collect dependency data") > - bb_cache = bb.cache.NoCache(self.cooker.databuilder) > + > for tid in self.rqdata.runtaskentries: > fn = fn_from_tid(tid) > - if fn not in done: > - the_data = bb_cache.loadDataFull(fn, > self.cooker.collection.get_file_appends(fn)) > - done.add(fn) > + fns.add(fn) > + > + max_process = > int(self.cfgData.getVar("BB_NUMBER_PARSE_THREADS") or os.cpu_count() > or 1) > + # We cannot use the real multiprocessing.Pool easily due to > some local data > + # that can't be pickled. This is a cheap multi-process > solution. > + launched = [] > + while fns: > + if len(launched) < max_process: > + p = Process(target=self.rq_dump_sigfn, > args=(fns.pop(), options)) > + p.start() > + launched.append(p) > + for q in launched: > + # The finished processes are joined when calling > is_alive() > + if not q.is_alive(): > + launched.remove(q) > + for p in launched: > + p.join() > > bb.parse.siggen.dump_sigs(self.rqdata.dataCaches, options) > > diff --git a/bitbake/lib/bb/siggen.py b/bitbake/lib/bb/siggen.py > index b20b9cf..ae50a18 100644 > --- a/bitbake/lib/bb/siggen.py > +++ b/bitbake/lib/bb/siggen.py > @@ -307,8 +307,8 @@ class > SignatureGeneratorBasic(SignatureGenerator): > pass > raise err > > - def dump_sigs(self, dataCaches, options): > - for fn in self.taskdeps: > + def dump_sigfn(self, fn, dataCaches, options): > + if fn in self.taskdeps: > for task in self.taskdeps[fn]: > tid = fn + ":" + task > (mc, _, _) = bb.runqueue.split_tid(tid) > -- > 2.7.4 > ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] use multiple processes to dump signatures. 2016-12-22 8:59 ` Richard Purdie @ 2016-12-22 18:39 ` Jianxun Zhang 2017-01-10 22:54 ` Jianxun Zhang 0 siblings, 1 reply; 7+ messages in thread From: Jianxun Zhang @ 2016-12-22 18:39 UTC (permalink / raw) To: Richard Purdie; +Cc: bitbake-devel > On Dec 22, 2016, at 12:59 AM, Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > > On Wed, 2016-12-21 at 12:27 -0800, Jianxun Zhang wrote: >> This change significantly shortens the time on reparsing stage >> of '-S' option. >> >> Each file is reparsed and then dumped within a dedicated >> process. The maximum number of the running processes is not >> greater than the value of BB_NUMBER_PARSE_THREADS if it is set. >> >> The dump_sigs() in class SignatureGeneratorBasic is _replaced_ >> by a new dump_sigfn() interface, so calls from the outside and >> subclasses are dispatched to the implementation in the base >> class of SignatureGeneratorBasic. >> >> Fixes [YOCTO #10352] > > Thanks, I think this is heading in the right direction. > > I am a little bit worried that this leaves OE's sstatesig.py with a > dump_sigs() function which isn't used/connected into everything else > though? Does this still write out a locked sigs file after this change? I am not the expert in this area, so just share the result and my understanding here. OE’s dump_sigs is still connected and used. The lock-sigs.inc is in output and looks okay with a manual diff against one from master tip. This is because I still keep the calling dump_sigs() in dump_signatures() in runqueue.py. That line triggers OE to dump lock-sigs.inc. The instance “siggen” in dump_signatures() should be the type of subclass in OE’s sstatesig.py. The dump_sigs() of OE class, in return, writes out the additional lock-sigs.inc and also calls its base class (BB) SignatureGeneratorBasic’s dump_sigs(), which is nothing now. This change removes dump_sigs() in SignatureGeneratorBasic’s to say "let’s fall back to our base class (SignatureGenerator) dump_sigs() since the inside of the removed dump_sigs() is replaced with parallel dumping". Another motivation of the removal is for code clarity. The loops in old dump_sigs() won’t work without a full data store. Keeping a loop in it to call new API doesn’t seem effective. What we miss by removing SignatureGeneratorBasic’s dump_sigs() is any existing logic/class directly calls dump_sigs() in this class outside of -S option path. But I don’t see such a case after some searching in BB and OE. (Another reason to remove it). In a short, it is a struggle between a parallelized stamp dumping and leaving lock-sigs.inc in a single main process. I don’t go further to parallelize writing lock-sigs.inc in OE because I think synchronization could be another deep hole, and this part doesn’t contribute much to perf either. I also ran test module sstatetest in oe-selftest and passed it. Feel free to let me know any better idea and improvement I should make. > > Cheers, > > Richard > > >> Signed-off-by: Jianxun Zhang <jianxun.zhang@linux.intel.com> >> --- >> bitbake/lib/bb/runqueue.py | 32 +++++++++++++++++++++++++++----- >> bitbake/lib/bb/siggen.py | 4 ++-- >> 2 files changed, 29 insertions(+), 7 deletions(-) >> >> diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py >> index 2ad8aad..c7d8d53 100644 >> --- a/bitbake/lib/bb/runqueue.py >> +++ b/bitbake/lib/bb/runqueue.py >> @@ -36,6 +36,7 @@ from bb import msg, data, event >> from bb import monitordisk >> import subprocess >> import pickle >> +from multiprocessing import Process >> >> bblogger = logging.getLogger("BitBake") >> logger = logging.getLogger("BitBake.RunQueue") >> @@ -1302,15 +1303,36 @@ class RunQueue: >> else: >> self.rqexe.finish() >> >> + def rq_dump_sigfn(self, fn, options): >> + bb_cache = bb.cache.NoCache(self.cooker.databuilder) >> + the_data = bb_cache.loadDataFull(fn, >> self.cooker.collection.get_file_appends(fn)) >> + siggen = bb.parse.siggen >> + dataCaches = self.rqdata.dataCaches >> + siggen.dump_sigfn(fn, dataCaches, options) >> + >> def dump_signatures(self, options): >> - done = set() >> + fns = set() >> bb.note("Reparsing files to collect dependency data") >> - bb_cache = bb.cache.NoCache(self.cooker.databuilder) >> + >> for tid in self.rqdata.runtaskentries: >> fn = fn_from_tid(tid) >> - if fn not in done: >> - the_data = bb_cache.loadDataFull(fn, >> self.cooker.collection.get_file_appends(fn)) >> - done.add(fn) >> + fns.add(fn) >> + >> + max_process = >> int(self.cfgData.getVar("BB_NUMBER_PARSE_THREADS") or os.cpu_count() >> or 1) >> + # We cannot use the real multiprocessing.Pool easily due to >> some local data >> + # that can't be pickled. This is a cheap multi-process >> solution. >> + launched = [] >> + while fns: >> + if len(launched) < max_process: >> + p = Process(target=self.rq_dump_sigfn, >> args=(fns.pop(), options)) >> + p.start() >> + launched.append(p) >> + for q in launched: >> + # The finished processes are joined when calling >> is_alive() >> + if not q.is_alive(): >> + launched.remove(q) >> + for p in launched: >> + p.join() >> >> bb.parse.siggen.dump_sigs(self.rqdata.dataCaches, options) >> >> diff --git a/bitbake/lib/bb/siggen.py b/bitbake/lib/bb/siggen.py >> index b20b9cf..ae50a18 100644 >> --- a/bitbake/lib/bb/siggen.py >> +++ b/bitbake/lib/bb/siggen.py >> @@ -307,8 +307,8 @@ class >> SignatureGeneratorBasic(SignatureGenerator): >> pass >> raise err >> >> - def dump_sigs(self, dataCaches, options): >> - for fn in self.taskdeps: >> + def dump_sigfn(self, fn, dataCaches, options): >> + if fn in self.taskdeps: >> for task in self.taskdeps[fn]: >> tid = fn + ":" + task >> (mc, _, _) = bb.runqueue.split_tid(tid) >> -- >> 2.7.4 >> ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] use multiple processes to dump signatures. 2016-12-22 18:39 ` Jianxun Zhang @ 2017-01-10 22:54 ` Jianxun Zhang 2017-01-12 17:41 ` Richard Purdie 0 siblings, 1 reply; 7+ messages in thread From: Jianxun Zhang @ 2017-01-10 22:54 UTC (permalink / raw) To: Richard Purdie; +Cc: bitbake-devel Hi RP, Ping~ Do you have further suggestion based on my explanation? https://bugzilla.yoctoproject.org/show_bug.cgi?id=10352 Thanks > On Dec 22, 2016, at 10:39 AM, Jianxun Zhang <jianxun.zhang@linux.intel.com> wrote: > >> >> On Dec 22, 2016, at 12:59 AM, Richard Purdie <richard.purdie@linuxfoundation.org> wrote: >> >> On Wed, 2016-12-21 at 12:27 -0800, Jianxun Zhang wrote: >>> This change significantly shortens the time on reparsing stage >>> of '-S' option. >>> >>> Each file is reparsed and then dumped within a dedicated >>> process. The maximum number of the running processes is not >>> greater than the value of BB_NUMBER_PARSE_THREADS if it is set. >>> >>> The dump_sigs() in class SignatureGeneratorBasic is _replaced_ >>> by a new dump_sigfn() interface, so calls from the outside and >>> subclasses are dispatched to the implementation in the base >>> class of SignatureGeneratorBasic. >>> >>> Fixes [YOCTO #10352] >> >> Thanks, I think this is heading in the right direction. >> >> I am a little bit worried that this leaves OE's sstatesig.py with a >> dump_sigs() function which isn't used/connected into everything else >> though? Does this still write out a locked sigs file after this change? > I am not the expert in this area, so just share the result and my understanding here. > > OE’s dump_sigs is still connected and used. The lock-sigs.inc is in output and looks okay with a manual diff against one from master tip. This is because I still keep the calling dump_sigs() in dump_signatures() in runqueue.py. That line triggers OE to dump lock-sigs.inc. > > The instance “siggen” in dump_signatures() should be the type of subclass in OE’s sstatesig.py. The dump_sigs() of OE class, in return, writes out the additional lock-sigs.inc and also calls its base class (BB) SignatureGeneratorBasic’s dump_sigs(), which is nothing now. > > This change removes dump_sigs() in SignatureGeneratorBasic’s to say "let’s fall back to our base class (SignatureGenerator) dump_sigs() since the inside of the removed dump_sigs() is replaced with parallel dumping". Another motivation of the removal is for code clarity. The loops in old dump_sigs() won’t work without a full data store. Keeping a loop in it to call new API doesn’t seem effective. > > What we miss by removing SignatureGeneratorBasic’s dump_sigs() is any existing logic/class directly calls dump_sigs() in this class outside of -S option path. But I don’t see such a case after some searching in BB and OE. (Another reason to remove it). > > In a short, it is a struggle between a parallelized stamp dumping and leaving lock-sigs.inc in a single main process. I don’t go further to parallelize writing lock-sigs.inc in OE because I think synchronization could be another deep hole, and this part doesn’t contribute much to perf either. > > I also ran test module sstatetest in oe-selftest and passed it. > > Feel free to let me know any better idea and improvement I should make. > > >> >> Cheers, >> >> Richard >> >> >>> Signed-off-by: Jianxun Zhang <jianxun.zhang@linux.intel.com> >>> --- >>> bitbake/lib/bb/runqueue.py | 32 +++++++++++++++++++++++++++----- >>> bitbake/lib/bb/siggen.py | 4 ++-- >>> 2 files changed, 29 insertions(+), 7 deletions(-) >>> >>> diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py >>> index 2ad8aad..c7d8d53 100644 >>> --- a/bitbake/lib/bb/runqueue.py >>> +++ b/bitbake/lib/bb/runqueue.py >>> @@ -36,6 +36,7 @@ from bb import msg, data, event >>> from bb import monitordisk >>> import subprocess >>> import pickle >>> +from multiprocessing import Process >>> >>> bblogger = logging.getLogger("BitBake") >>> logger = logging.getLogger("BitBake.RunQueue") >>> @@ -1302,15 +1303,36 @@ class RunQueue: >>> else: >>> self.rqexe.finish() >>> >>> + def rq_dump_sigfn(self, fn, options): >>> + bb_cache = bb.cache.NoCache(self.cooker.databuilder) >>> + the_data = bb_cache.loadDataFull(fn, >>> self.cooker.collection.get_file_appends(fn)) >>> + siggen = bb.parse.siggen >>> + dataCaches = self.rqdata.dataCaches >>> + siggen.dump_sigfn(fn, dataCaches, options) >>> + >>> def dump_signatures(self, options): >>> - done = set() >>> + fns = set() >>> bb.note("Reparsing files to collect dependency data") >>> - bb_cache = bb.cache.NoCache(self.cooker.databuilder) >>> + >>> for tid in self.rqdata.runtaskentries: >>> fn = fn_from_tid(tid) >>> - if fn not in done: >>> - the_data = bb_cache.loadDataFull(fn, >>> self.cooker.collection.get_file_appends(fn)) >>> - done.add(fn) >>> + fns.add(fn) >>> + >>> + max_process = >>> int(self.cfgData.getVar("BB_NUMBER_PARSE_THREADS") or os.cpu_count() >>> or 1) >>> + # We cannot use the real multiprocessing.Pool easily due to >>> some local data >>> + # that can't be pickled. This is a cheap multi-process >>> solution. >>> + launched = [] >>> + while fns: >>> + if len(launched) < max_process: >>> + p = Process(target=self.rq_dump_sigfn, >>> args=(fns.pop(), options)) >>> + p.start() >>> + launched.append(p) >>> + for q in launched: >>> + # The finished processes are joined when calling >>> is_alive() >>> + if not q.is_alive(): >>> + launched.remove(q) >>> + for p in launched: >>> + p.join() >>> >>> bb.parse.siggen.dump_sigs(self.rqdata.dataCaches, options) >>> >>> diff --git a/bitbake/lib/bb/siggen.py b/bitbake/lib/bb/siggen.py >>> index b20b9cf..ae50a18 100644 >>> --- a/bitbake/lib/bb/siggen.py >>> +++ b/bitbake/lib/bb/siggen.py >>> @@ -307,8 +307,8 @@ class >>> SignatureGeneratorBasic(SignatureGenerator): >>> pass >>> raise err >>> >>> - def dump_sigs(self, dataCaches, options): >>> - for fn in self.taskdeps: >>> + def dump_sigfn(self, fn, dataCaches, options): >>> + if fn in self.taskdeps: >>> for task in self.taskdeps[fn]: >>> tid = fn + ":" + task >>> (mc, _, _) = bb.runqueue.split_tid(tid) >>> -- >>> 2.7.4 >>> > > -- > _______________________________________________ > bitbake-devel mailing list > bitbake-devel@lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/bitbake-devel ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] use multiple processes to dump signatures. 2017-01-10 22:54 ` Jianxun Zhang @ 2017-01-12 17:41 ` Richard Purdie 2017-01-12 19:33 ` Jianxun Zhang 0 siblings, 1 reply; 7+ messages in thread From: Richard Purdie @ 2017-01-12 17:41 UTC (permalink / raw) To: Jianxun Zhang; +Cc: bitbake-devel Hi, On Tue, 2017-01-10 at 14:54 -0800, Jianxun Zhang wrote: > Do you have further suggestion based on my explanation? > > https://bugzilla.yoctoproject.org/show_bug.cgi?id=10352 I needed to sit and look at that patch for a while to check a few things but I think you have it right. I've been doing some local testing and its now queued in -next, thanks! Just out of interest, before: oe-selftest -r sstatetests.SStateTests.test_sstate_allarch_samesigs_multilib 2017-01-12 17:06:18 - test_sstate_allarch_samesigs_multilib (oeqa.selftest.sstatetests.SStateTests) ... OK (325.084s) and after: 2017-01-12 16:58:59 - test_sstate_allarch_samesigs_multilib (oeqa.selftest.sstatetests.SStateTests) ... OK (93.926s) and this change should help multiple tests so its a nice improvement! Cheers, Richard ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] use multiple processes to dump signatures. 2017-01-12 17:41 ` Richard Purdie @ 2017-01-12 19:33 ` Jianxun Zhang 2017-01-12 21:16 ` Richard Purdie 0 siblings, 1 reply; 7+ messages in thread From: Jianxun Zhang @ 2017-01-12 19:33 UTC (permalink / raw) To: Richard Purdie; +Cc: bitbake-devel RP, I appreciate your multiple hints on the road for this issue! I do _feel_ this patch needs some improvement or tests, but I cannot identify them in my sight. Well, things shouldn’t be such easy with a V1 due to the complexity. That’s why I have to hand over the burden to you. :-) Feel free to let me know any new possibilities. I am actually interested in the data you shared. I improved my sig-dumping over 10 times (14.4 sec to 1 sec). Your test should covers other steps, so the improvement on sig dumping is less significant, 325.08 to 93.93 sec. Maybe we should find some time to investigate any room to speed up the rest in test suite... Thanks > On Jan 12, 2017, at 9:41 AM, Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > > Hi, > > On Tue, 2017-01-10 at 14:54 -0800, Jianxun Zhang wrote: >> Do you have further suggestion based on my explanation? >> >> https://bugzilla.yoctoproject.org/show_bug.cgi?id=10352 > > I needed to sit and look at that patch for a while to check a few > things but I think you have it right. I've been doing some local > testing and its now queued in -next, thanks! > > Just out of interest, before: > > oe-selftest -r sstatetests.SStateTests.test_sstate_allarch_samesigs_multilib > > 2017-01-12 17:06:18 - test_sstate_allarch_samesigs_multilib (oeqa.selftest.sstatetests.SStateTests) ... OK (325.084s) > > and after: > > 2017-01-12 16:58:59 - test_sstate_allarch_samesigs_multilib (oeqa.selftest.sstatetests.SStateTests) ... OK (93.926s) > > and this change should help multiple tests so its a nice improvement! > > Cheers, > > Richard ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] use multiple processes to dump signatures. 2017-01-12 19:33 ` Jianxun Zhang @ 2017-01-12 21:16 ` Richard Purdie 0 siblings, 0 replies; 7+ messages in thread From: Richard Purdie @ 2017-01-12 21:16 UTC (permalink / raw) To: Jianxun Zhang; +Cc: bitbake-devel On Thu, 2017-01-12 at 11:33 -0800, Jianxun Zhang wrote: > I appreciate your multiple hints on the road for this issue! I do > _feel_ this patch needs some improvement or tests, but I cannot > identify them in my sight. Well, things shouldn’t be such easy with a > V1 due to the complexity. I think your testing was fine, I just wanted to double check a few things with the patch which I've now done and things seem fine. > That’s why I have to hand over the burden to you. :-) Feel free to > let me know any new possibilities. > > I am actually interested in the data you shared. I improved my sig- > dumping over 10 times (14.4 sec to 1 sec). Your test should covers > other steps, so the improvement on sig dumping is less significant, > 325.08 to 93.93 sec. > > Maybe we should find some time to investigate any room to speed up > the rest in test suite... I knew that improvement would help this set of tests significantly and 230s or ~4 minutes is a good saving to have per test. oe-selftest currently takes up to around 8 hours and if we can knock 4 minutes off say 10 tests that is a good saving! :) I very much agree that we need to look at other ways of speeding up oe- selftest. I actually found another problem Ross sent a patch for earlier today which improve the test load time significantly and should take another ~10mins off the 8 hours and makes running this manually less annoying. We also have plans to use memory resident bitbake to help. There are lots of places in oe-selftest it runs "bitbake -e XXX" to get variables an the time taken to do this mounts up quickly. If bitbake is memory resident, we should be able to get answers much faster. There is also the option of parallelising oe-selftest too, currently the tests run in series but running in parallel would help too. So plenty more work to do but this is a good start (and there are several other use cases for this codepath I'm happy to see speeded up too!). Cheers, Richard ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2017-01-12 21:19 UTC | newest] Thread overview: 7+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2016-12-21 20:27 [PATCH] use multiple processes to dump signatures Jianxun Zhang 2016-12-22 8:59 ` Richard Purdie 2016-12-22 18:39 ` Jianxun Zhang 2017-01-10 22:54 ` Jianxun Zhang 2017-01-12 17:41 ` Richard Purdie 2017-01-12 19:33 ` Jianxun Zhang 2017-01-12 21:16 ` Richard Purdie
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.