BitBake parallel parsing testing

BitBake parallel parsing testing

[Chris Larson]

Greetings all,
This is a request for further testers for the parallel parsing branch
of bitbake, as well as for review of the code.  I'm fairly certain
that the last real issue with it was resolved yesterday, and I haven't
gotten any reports of problems from those testing it since last week,
so I'm opening it up to more public testing, so we can be absolutely
certain that we can merge it to master in the near future without
risking problems.

For code review:
Either follow the below steps, then git log master..parallel-parsing
or git diff master..parallel-parsing, or use the web interface at
https://github.com/kergoth/bitbake/commits/parallel-parsing.

For testing from scratch:

git clone git://github.com/kergoth/bitbake
cd bitbake
git checkout -b parallel-parsing origin/parallel-parsing

From an existing bitbake repository:

git remote add kergoth git://github.com/kergoth/bitbake
git remote update
git checkout -b parallel-parsing kergoth/parallel-parsing

Thanks for your time, and do let me know what you think, either of the
code, performance in general, or the new progress bar (shows an ETA),
or anything else :)

-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

Re: BitBake parallel parsing testing

[Koen Kooi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 23-11-10 15:29, Chris Larson wrote:
> Greetings all,
> This is a request for further testers for the parallel parsing branch
> of bitbake, as well as for review of the code.  I'm fairly certain
> that the last real issue with it was resolved yesterday, and I haven't
> gotten any reports of problems from those testing it since last week,
> so I'm opening it up to more public testing, so we can be absolutely
> certain that we can merge it to master in the near future without
> risking problems.
> 
> For code review:
> Either follow the below steps, then git log master..parallel-parsing
> or git diff master..parallel-parsing, or use the web interface at
> https://github.com/kergoth/bitbake/commits/parallel-parsing.
> 
> For testing from scratch:
> 
> git clone git://github.com/kergoth/bitbake
> cd bitbake
> git checkout -b parallel-parsing origin/parallel-parsing
> 
> From an existing bitbake repository:
> 
> git remote add kergoth git://github.com/kergoth/bitbake
> git remote update
> git checkout -b parallel-parsing kergoth/parallel-parsing
> 
> Thanks for your time, and do let me know what you think, either of the
> code, performance in general, or the new progress bar (shows an ETA),
> or anything else :)

And don't forget to play with BB_NUMBER_PARSE_THREADS, it defaults to
the number of CPUs :)

regards,

Koen
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFM69DhMkyGM64RGpERAmxfAJ9jp1HgN3KqDzjdTaMqyuQmaarpMwCgiC9z
dj+92mj+UZJawJ1mhVJapKU=
=ZFS0
-----END PGP SIGNATURE-----

Re: BitBake parallel parsing testing

[Chris Larson]

On Tue, Nov 23, 2010 at 7:34 AM, Koen Kooi <k.kooi@student.utwente.nl> wrote:
>> Thanks for your time, and do let me know what you think, either of the
>> code, performance in general, or the new progress bar (shows an ETA),
>> or anything else :)
>
> And don't forget to play with BB_NUMBER_PARSE_THREADS, it defaults to
> the number of CPUs :)

Indeed, thanks, Koen.  Don't know how I forgot to mention that :)
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

Re: BitBake parallel parsing testing

[Robert Foerster]

Bitbake 1.10: 3m2.185s
parallel-parsing (BB_NUM_PARSE_THREADS=num_cpu): 1m48.232s
parallel-parsing (BB_NUM_PARSE_THREADS=2*num_cpu): 1m1.869s

Parallel-parsing of recipes is excellent.  Well done, Chris.


On Tue, Nov 23, 2010 at 9:38 AM, Chris Larson <clarson@kergoth.com> wrote:

> On Tue, Nov 23, 2010 at 7:34 AM, Koen Kooi <k.kooi@student.utwente.nl>
> wrote:
> >> Thanks for your time, and do let me know what you think, either of the
> >> code, performance in general, or the new progress bar (shows an ETA),
> >> or anything else :)
> >
> > And don't forget to play with BB_NUMBER_PARSE_THREADS, it defaults to
> > the number of CPUs :)
>
> Indeed, thanks, Koen.  Don't know how I forgot to mention that :)
> --
> Christopher Larson
> clarson at kergoth dot com
> Founder - BitBake, OpenEmbedded, OpenZaurus
> Maintainer - Tslib
> Senior Software Engineer, Mentor Graphics
>
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
>

Re: BitBake parallel parsing testing

[Chris Larson]

On Tue, Nov 23, 2010 at 8:02 AM, Robert Foerster <robert@erafx.com> wrote:
> Bitbake 1.10: 3m2.185s
> parallel-parsing (BB_NUM_PARSE_THREADS=num_cpu): 1m48.232s
> parallel-parsing (BB_NUM_PARSE_THREADS=2*num_cpu): 1m1.869s
>
> Parallel-parsing of recipes is excellent.  Well done, Chris.

Hmm, maybe I should revert the change that made it num_cpu instead of
2.5*num_cpu by default :)  Figured better to be conservative and not
kill anyone's machine, but..
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

Re: BitBake parallel parsing testing

[Michael Smith]

Chris Larson wrote:
> Thanks for your time, and do let me know what you think, either of the
> code, performance in general, or the new progress bar (shows an ETA),
> or anything else :)

On a Xen VM with 3 vcpus, this is just brilliant. I used the default 
number of parse threads (i.e. 3). The CPUs are pretty much pegged.

Mike

Re: BitBake parallel parsing testing

[Robert Foerster]

Yea, in testing again, I am getting the same results now with num_cpu or
2*num_cpu.  All 4 cores are pegged at 100% either way.

Not sure what causes the hiccup in the number before, must have been
something else consuming CPU.

On Tue, Nov 23, 2010 at 10:38 AM, Michael Smith <msmith@cbnco.com> wrote:

> Chris Larson wrote:
>
>> Thanks for your time, and do let me know what you think, either of the
>> code, performance in general, or the new progress bar (shows an ETA),
>> or anything else :)
>>
>
> On a Xen VM with 3 vcpus, this is just brilliant. I used the default number
> of parse threads (i.e. 3). The CPUs are pretty much pegged.
>
> Mike
>
>
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
>

Re: BitBake parallel parsing testing

[Khem Raj]

On Tue, Nov 23, 2010 at 7:41 AM, Robert Foerster <robert@erafx.com> wrote:
> Yea, in testing again, I am getting the same results now with num_cpu or
> 2*num_cpu.  All 4 cores are pegged at 100% either way.
>
> Not sure what causes the hiccup in the number before, must have been
> something else consuming CPU.
>
> On Tue, Nov 23, 2010 at 10:38 AM, Michael Smith <msmith@cbnco.com> wrote:
>
>> Chris Larson wrote:
>>
>>> Thanks for your time, and do let me know what you think, either of the
>>> code, performance in general, or the new progress bar (shows an ETA),
>>> or anything else :)
>>>
>>
FWIW I have been using it full time for past few days
and it has been delivering the results. So I am quite happy with
it. my parsing on a dual core p4 went down from 12min to 5mins


>> On a Xen VM with 3 vcpus, this is just brilliant. I used the default number
>> of parse threads (i.e. 3). The CPUs are pretty much pegged.
>>
>> Mike
>>
>>
>> _______________________________________________
>> Openembedded-devel mailing list
>> Openembedded-devel@lists.openembedded.org
>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
>>
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
>

Re: BitBake parallel parsing testing

[Koen Kooi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 23-11-10 15:29, Chris Larson wrote:

> Thanks for your time, and do let me know what you think, either of the
> code, performance in general, or the new progress bar (shows an ETA),

The heuristic for the progress needs some work, it currently seems to
take in account all the parsed recipes instead of the last N recipes.
Try ctrl-c'ing it at 75% and restarting, you'll see the ETA taking weird
values. In my case it stayed on "10 seconds" for more than a minute.

regards,

Koen
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFM7AdfMkyGM64RGpERAgITAJ98HG4eBtVW0WLPjfLnneli8xU6uwCfVvfv
vAnlvL5m7ZfOqNhGwKqJK/w=
=a6qT
-----END PGP SIGNATURE-----

Re: BitBake parallel parsing testing

[Chris Larson]

On Tue, Nov 23, 2010 at 11:26 AM, Koen Kooi <k.kooi@student.utwente.nl> wrote:
> On 23-11-10 15:29, Chris Larson wrote:
>
>> Thanks for your time, and do let me know what you think, either of the
>> code, performance in general, or the new progress bar (shows an ETA),
>
> The heuristic for the progress needs some work, it currently seems to
> take in account all the parsed recipes instead of the last N recipes.

That's what our progress display always did, it's just more obvious now.

> Try ctrl-c'ing it at 75% and restarting, you'll see the ETA taking weird
> values. In my case it stayed on "10 seconds" for more than a minute.

The issue is one I've discussed with foerster a fair bit already.
When you're fully cached, the majority of the time isn't in the
parsing, it's in the loading of the cache file from disk, which isn't
part of our progress at all right now.  Either we stall for seconds
before the progress bar comes up, or we don't appear to stall, but the
ETA is a bit off.

Long term the intent is to properly integrate cache loading into this
progress bar, or into its own -- see
https://github.com/foerster/bitbake/commit/320b114d92181b3f944776b7e8c8db4b868ee3f1
for foerster's initial work on this.

After thinking about what you mentioned about the recipe count being
parsed, and given the fact that cached recipes aren't "parsed",
they're loaded from disk and then added to our CacheData, perhaps we
should change the parsing progress display to truly only show the
recipes being *parsed*, which is what you implied, and add a "Loading
cache..." message before the progress bar comes up, so we don't appear
to stall, then add a proper progress bar for the cache file loading as
a second step.

Alternatively, we change the progress display to truly indicate what's
being done.. if we consider all recipes, it's not *parsing* that we're
doing, its .. perhaps.. processing?

*ponders*
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

Re: BitBake parallel parsing testing

[Chris Larson]

On Tue, Nov 23, 2010 at 11:42 AM, Chris Larson <clarson@kergoth.com> wrote:
> On Tue, Nov 23, 2010 at 11:26 AM, Koen Kooi <k.kooi@student.utwente.nl> wrote:
>> On 23-11-10 15:29, Chris Larson wrote:
>>
>>> Thanks for your time, and do let me know what you think, either of the
>>> code, performance in general, or the new progress bar (shows an ETA),
>>
>> The heuristic for the progress needs some work, it currently seems to
>> take in account all the parsed recipes instead of the last N recipes.
>
> That's what our progress display always did, it's just more obvious now.
>
>> Try ctrl-c'ing it at 75% and restarting, you'll see the ETA taking weird
>> values. In my case it stayed on "10 seconds" for more than a minute.

Okay, I disabled the ETA display for the time being.  It's useful, but
less than reliable, so we'll revisit it after revamping the progress
handling and incorporating cache file loading into the progress.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

Re: BitBake parallel parsing testing

[Khem Raj]

On Tue, Nov 23, 2010 at 11:05 AM, Chris Larson <clarson@kergoth.com> wrote:
> On Tue, Nov 23, 2010 at 11:42 AM, Chris Larson <clarson@kergoth.com> wrote:
>> On Tue, Nov 23, 2010 at 11:26 AM, Koen Kooi <k.kooi@student.utwente.nl> wrote:
>>> On 23-11-10 15:29, Chris Larson wrote:
>>>
>>>> Thanks for your time, and do let me know what you think, either of the
>>>> code, performance in general, or the new progress bar (shows an ETA),
>>>
>>> The heuristic for the progress needs some work, it currently seems to
>>> take in account all the parsed recipes instead of the last N recipes.
>>
>> That's what our progress display always did, it's just more obvious now.
>>
>>> Try ctrl-c'ing it at 75% and restarting, you'll see the ETA taking weird
>>> values. In my case it stayed on "10 seconds" for more than a minute.
>
> Okay, I disabled the ETA display for the time being.  It's useful, but
> less than reliable, so we'll revisit it after revamping the progress
> handling and incorporating cache file loading into the progress.

btw my timing mesasrurements were against the wall clock and not ETA
displayed by bitbake

> --
> Christopher Larson
> clarson at kergoth dot com
> Founder - BitBake, OpenEmbedded, OpenZaurus
> Maintainer - Tslib
> Senior Software Engineer, Mentor Graphics
>
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
>

Re: BitBake parallel parsing testing

[Koen Kooi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 23-11-10 19:42, Chris Larson wrote:
> On Tue, Nov 23, 2010 at 11:26 AM, Koen Kooi <k.kooi@student.utwente.nl> wrote:
>> On 23-11-10 15:29, Chris Larson wrote:
>>
>>> Thanks for your time, and do let me know what you think, either of the
>>> code, performance in general, or the new progress bar (shows an ETA),
>>
>> The heuristic for the progress needs some work, it currently seems to
>> take in account all the parsed recipes instead of the last N recipes.
> 
> That's what our progress display always did, it's just more obvious now.

That's not what I meant :) I meant that the progressbar will jump to 75%
since that was already parsed and the ETA heuristic gets thrown off by
the "75% in 1 second".

regards,

Koen

> 
>> Try ctrl-c'ing it at 75% and restarting, you'll see the ETA taking weird
>> values. In my case it stayed on "10 seconds" for more than a minute.
> 
> The issue is one I've discussed with foerster a fair bit already.
> When you're fully cached, the majority of the time isn't in the
> parsing, it's in the loading of the cache file from disk, which isn't
> part of our progress at all right now.  Either we stall for seconds
> before the progress bar comes up, or we don't appear to stall, but the
> ETA is a bit off.
> 
> Long term the intent is to properly integrate cache loading into this
> progress bar, or into its own -- see
> https://github.com/foerster/bitbake/commit/320b114d92181b3f944776b7e8c8db4b868ee3f1
> for foerster's initial work on this.
> 
> After thinking about what you mentioned about the recipe count being
> parsed, and given the fact that cached recipes aren't "parsed",
> they're loaded from disk and then added to our CacheData, perhaps we
> should change the parsing progress display to truly only show the
> recipes being *parsed*, which is what you implied, and add a "Loading
> cache..." message before the progress bar comes up, so we don't appear
> to stall, then add a proper progress bar for the cache file loading as
> a second step.
> 
> Alternatively, we change the progress display to truly indicate what's
> being done.. if we consider all recipes, it's not *parsing* that we're
> doing, its .. perhaps.. processing?
> 
> *ponders*

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFM7NePMkyGM64RGpERAqWGAJoDVbAtMbIibKGGzLw03IBQMipa3QCghnTK
wq00iIWQlV+R+SKupneaC7o=
=IUFS
-----END PGP SIGNATURE-----

Re: BitBake parallel parsing testing

[Chris Larson]

On Wed, Nov 24, 2010 at 2:14 AM, Koen Kooi <k.kooi@student.utwente.nl> wrote:
> On 23-11-10 19:42, Chris Larson wrote:
>> On Tue, Nov 23, 2010 at 11:26 AM, Koen Kooi <k.kooi@student.utwente.nl> wrote:
>>> On 23-11-10 15:29, Chris Larson wrote:
>>>
>>>> Thanks for your time, and do let me know what you think, either of the
>>>> code, performance in general, or the new progress bar (shows an ETA),
>>>
>>> The heuristic for the progress needs some work, it currently seems to
>>> take in account all the parsed recipes instead of the last N recipes.
>>
>> That's what our progress display always did, it's just more obvious now.
>
> That's not what I meant :) I meant that the progressbar will jump to 75%
> since that was already parsed and the ETA heuristic gets thrown off by
> the "75% in 1 second".

Ah, yes, well, that's because we're using the same progress bar / ETA
for cached recipes as for actually parsing recipes, and obviously the
amount of time for those are quite different.  The ETA is gone until
we give thought to if we want to revamp our progress, and how best to
do so.  Ideally we'd probably use one progress bar for cache loading
and one progress bar for *actual* parsing (as opposed to processing).
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

Re: BitBake parallel parsing testing

[Petr Štetiar]

Chris Larson <kergoth@gmail.com> [2010-11-23 07:29:00]:

> Greetings all,

Hi,

> Thanks for your time, and do let me know what you think, either of the
> code, performance in general, or the new progress bar (shows an ETA),
> or anything else :)

I'm using it on two different systems for a few days now and it works fine, no
issues at all. Well, I don't mind those eye candy progress bar glitches, speed
is more important for me. Thanks a lot, nice Christmas gift in November :-)

-- ynezz

Re: BitBake parallel parsing testing

[Reinhard Thies]

Hi Chris,

On Tuesday 23 November 2010 15:29:00 Chris Larson wrote:
> Greetings all,
> This is a request for further testers for the parallel parsing branch
> of bitbake, as well as for review of the code.  I'm fairly certain
> that the last real issue with it was resolved yesterday, and I haven't
> gotten any reports of problems from those testing it since last week,
> so I'm opening it up to more public testing, so we can be absolutely
> certain that we can merge it to master in the near future without
> risking problems.
> 
It's a great improvement :-))

I had it running on an older branch based on stable/2009.
I had to add the following workaround to get it running:

diff --git a/lib/bb/utils.py b/lib/bb/utils.py
index 19ca45d..11f9a01 100644
--- a/lib/bb/utils.py
+++ b/lib/bb/utils.py
@@ -93,6 +93,10 @@ def vercmp(ta, tb):
     (ea, va, ra) = ta
     (eb, vb, rb) = tb

+    if ea == '':
+        ea = '0'
+    if eb == '':
+        eb = '0'
     r = int(ea)-int(eb)
     if (r == 0):
         r = vercmp_part(va, vb)


With that fix it worked fine at all.

Now my CoreI7 makes fun ;-)

thx,
Reinhard 

Re: BitBake parallel parsing testing

[Chris Larson]

On Tue, Nov 23, 2010 at 7:29 AM, Chris Larson <kergoth@gmail.com> wrote:
> This is a request for further testers for the parallel parsing branch
> of bitbake, as well as for review of the code.  I'm fairly certain
> that the last real issue with it was resolved yesterday, and I haven't
> gotten any reports of problems from those testing it since last week,
> so I'm opening it up to more public testing, so we can be absolutely
> certain that we can merge it to master in the near future without
> risking problems.
>
> For code review:
> Either follow the below steps, then git log master..parallel-parsing
> or git diff master..parallel-parsing, or use the web interface at
> https://github.com/kergoth/bitbake/commits/parallel-parsing
>
> .
>
> For testing from scratch:
>
> git clone git://github.com/kergoth/bitbake
>
> cd bitbake
> git checkout -b parallel-parsing origin/parallel-parsing
>
> From an existing bitbake repository:
>
> git remote add kergoth git://github.com/kergoth/bitbake
>
> git remote update
> git checkout -b parallel-parsing kergoth/parallel-parsing
>
> Thanks for your time, and do let me know what you think, either of the
> code, performance in general, or the new progress bar (shows an ETA),
> or anything else :)

This has now been merged to the bitbake master branch, as it's been
tested for a couple weeks now without problems, and the benefits are
clearly worthwhile.  If any bugs crop up with it, we'll fix them as we
go.  Let me know if anyone has any issues with it.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics