PARALLEL_MAKE

PARALLEL_MAKE

[Ulf Samuelsson]

Is there any activity to make this more useful?
Just managed to install a Core i7 machine, which speeded
up my Buildroot build from about 4 hours to less than 1 hour.
(There was an error in the build right at the end,
so it could have been downto 40 minutes or so)
Building the cross compiler took 10 minutes with "make -j 16"
compared to 1-2 hours on a P4 @ 3GHz.

Openembedded does not seem to use PARALLEL_MAKE extensively.
I have the frequency meter activated for all 8 CPUs in the Core i7,
and most of the time the CPUs seems to idle at 1.6 GHz,
with one CPU (or rarely 2) are running at the full 2.67 GHz.

Last time I built OE on a P4@ 3 GHz it took ~40 hours 
for the SAM9263, so some speedup would be nice.
Maybe we should make an effort?

Best Regards
Ulf Samuelsson

Re: PARALLEL_MAKE

[Mike (mwester)]

Ulf Samuelsson wrote:
> Is there any activity to make this more useful?
> Just managed to install a Core i7 machine, which speeded
> up my Buildroot build from about 4 hours to less than 1 hour.
> (There was an error in the build right at the end,
> so it could have been downto 40 minutes or so)
> Building the cross compiler took 10 minutes with "make -j 16"
> compared to 1-2 hours on a P4 @ 3GHz.
> 
> Openembedded does not seem to use PARALLEL_MAKE extensively.
> I have the frequency meter activated for all 8 CPUs in the Core i7,
> and most of the time the CPUs seems to idle at 1.6 GHz,
> with one CPU (or rarely 2) are running at the full 2.67 GHz.
> 
> Last time I built OE on a P4@ 3 GHz it took ~40 hours 
> for the SAM9263, so some speedup would be nice.
> Maybe we should make an effort?
> 
> Best Regards
> Ulf Samuelsson

I build often with PARALLEL_MAKE set, and have been rather regularly
flushing out packages that break due to improper Makefile dependencies.
 The latest offender is thttpd; I'll be committing the fix to force that
to always run without PARALLEL_MAKE shortly.

My new quad-core does a dandy job of keeping all four cores busy almost
100% of the time -- but you need to specify BB_NUMBER_THREADS to get it
to do that.  Ever since somebody fixed the fetcher problem, that works
very, very nicely indeed.

-Mike (mwester)

Re: PARALLEL_MAKE

[Khem Raj]

[-- Attachment #1: Type: text/plain, Size: 1003 bytes --]

On Thursday 12 March 2009 16:32:16 Ulf Samuelsson wrote:
> Is there any activity to make this more useful?
> Just managed to install a Core i7 machine, which speeded
> up my Buildroot build from about 4 hours to less than 1 hour.
> (There was an error in the build right at the end,
> so it could have been downto 40 minutes or so)
> Building the cross compiler took 10 minutes with "make -j 16"
> compared to 1-2 hours on a P4 @ 3GHz.
>
> Openembedded does not seem to use PARALLEL_MAKE extensively.
> I have the frequency meter activated for all 8 CPUs in the Core i7,
> and most of the time the CPUs seems to idle at 1.6 GHz,
> with one CPU (or rarely 2) are running at the full 2.67 GHz.
>
> Last time I built OE on a P4@ 3 GHz it took ~40 hours
> for the SAM9263, so some speedup would be nice.
> Maybe we should make an effort?

I use 

PARALLEL_MAKE = "-j4" 
BB_NUMBER_THREADS = "2"

in my local.conf. I see my core 2 duo machine spawning parallel tasks.

-- 
Khem Raj

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 204 bytes --]

Re: PARALLEL_MAKE

[Denys Dmytriyenko]

On Thu, Mar 12, 2009 at 06:38:54PM -0700, Khem Raj wrote:
> On Thursday 12 March 2009 16:32:16 Ulf Samuelsson wrote:
> > Is there any activity to make this more useful?
> > Just managed to install a Core i7 machine, which speeded
> > up my Buildroot build from about 4 hours to less than 1 hour.
> > (There was an error in the build right at the end,
> > so it could have been downto 40 minutes or so)
> > Building the cross compiler took 10 minutes with "make -j 16"
> > compared to 1-2 hours on a P4 @ 3GHz.
> >
> > Openembedded does not seem to use PARALLEL_MAKE extensively.
> > I have the frequency meter activated for all 8 CPUs in the Core i7,
> > and most of the time the CPUs seems to idle at 1.6 GHz,
> > with one CPU (or rarely 2) are running at the full 2.67 GHz.
> >
> > Last time I built OE on a P4@ 3 GHz it took ~40 hours
> > for the SAM9263, so some speedup would be nice.
> > Maybe we should make an effort?
> 
> I use 
> 
> PARALLEL_MAKE = "-j4" 
> BB_NUMBER_THREADS = "2"
> 
> in my local.conf. I see my core 2 duo machine spawning parallel tasks.

I'm setting 4/4 to the above two variables on my Quad Core and it loads all 4 
cores at 100% almost all the time. Sometimes it slows down due to the 
bottleneck in the harddrive access...

-- 
Denys

Re: PARALLEL_MAKE

[Tom Rini]

On Thu, Mar 12, 2009 at 11:49:30PM -0400, Denys Dmytriyenko wrote:
> On Thu, Mar 12, 2009 at 06:38:54PM -0700, Khem Raj wrote:
> > On Thursday 12 March 2009 16:32:16 Ulf Samuelsson wrote:
> > > Is there any activity to make this more useful?
> > > Just managed to install a Core i7 machine, which speeded
> > > up my Buildroot build from about 4 hours to less than 1 hour.
> > > (There was an error in the build right at the end,
> > > so it could have been downto 40 minutes or so)
> > > Building the cross compiler took 10 minutes with "make -j 16"
> > > compared to 1-2 hours on a P4 @ 3GHz.
> > >
> > > Openembedded does not seem to use PARALLEL_MAKE extensively.
> > > I have the frequency meter activated for all 8 CPUs in the Core i7,
> > > and most of the time the CPUs seems to idle at 1.6 GHz,
> > > with one CPU (or rarely 2) are running at the full 2.67 GHz.
> > >
> > > Last time I built OE on a P4@ 3 GHz it took ~40 hours
> > > for the SAM9263, so some speedup would be nice.
> > > Maybe we should make an effort?
> > 
> > I use 
> > 
> > PARALLEL_MAKE = "-j4" 
> > BB_NUMBER_THREADS = "2"
> > 
> > in my local.conf. I see my core 2 duo machine spawning parallel tasks.
> 
> I'm setting 4/4 to the above two variables on my Quad Core and it loads all 4 
> cores at 100% almost all the time. Sometimes it slows down due to the 
> bottleneck in the harddrive access...

On the dual dual-core (4 total) boxes I have, I do -j6/4 threads.  On my
new quad-core box I haven't yet had a chance to figure out what's best.
I also use -j6/4 threads on an older dual-core amd64 box we've got
around and that seems best for my tasks at least (I tried a few
combinations of more jobs/threads and less and both were slower).

-- 
Tom Rini

Re: PARALLEL_MAKE

[Marcin Juszkiewicz]

Dnia piątek, 13 marca 2009 o 00:32:16 Ulf Samuelsson napisał(a):

> Is there any activity to make this more useful?
> Just managed to install a Core i7 machine, which speeded
> up my Buildroot build from about 4 hours to less than 1 hour.
> (There was an error in the build right at the end,
> so it could have been downto 40 minutes or so)
> Building the cross compiler took 10 minutes with "make -j 16"
> compared to 1-2 hours on a P4 @ 3GHz.

I use core2quad at home and dual quadcore Xeon at one client office. 
Both are 100% busy during my builds. On my quad I use 
PARALLEL_MAKE="-j4" with 4 BB threads and on 8core I have 10 BBthreads 
with -j12 but thats for nightly builds when no one other use this 
machine. Using ccache also speeds next compilations.

> Last time I built OE on a P4@ 3 GHz it took ~40 hours
> for the SAM9263, so some speedup would be nice.
> Maybe we should make an effort?

You mean 'time bitbake console-image'? I can test on my local machine 
how much time it takes.

Regards, 
-- 
JID:      hrw@jabber.org
Website:  http://marcin.juszkiewicz.com.pl/
LinkedIn: http://www.linkedin.com/in/marcinjuszkiewicz

Re: PARALLEL_MAKE

[Philip Balister]

[-- Attachment #1: Type: text/plain, Size: 961 bytes --]

Marcin Juszkiewicz wrote:
> Dnia piątek, 13 marca 2009 o 00:32:16 Ulf Samuelsson napisał(a):
> 
>> Is there any activity to make this more useful?
>> Just managed to install a Core i7 machine, which speeded
>> up my Buildroot build from about 4 hours to less than 1 hour.
>> (There was an error in the build right at the end,
>> so it could have been downto 40 minutes or so)
>> Building the cross compiler took 10 minutes with "make -j 16"
>> compared to 1-2 hours on a P4 @ 3GHz.
> 
> I use core2quad at home and dual quadcore Xeon at one client office. 
> Both are 100% busy during my builds. On my quad I use 
> PARALLEL_MAKE="-j4" with 4 BB threads and on 8core I have 10 BBthreads 
> with -j12 but thats for nightly builds when no one other use this 
> machine. Using ccache also speeds next compilations.

Marcin,


A while back you mentioned a program that displayed the per core CPU 
utilization. What was it again?

Philip


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/x-pkcs7-signature, Size: 3303 bytes --]

Re: PARALLEL_MAKE

[Sledz, Steffen]

> A while back you mentioned a program that displayed the per core CPU 
> utilization. What was it again?

Try htop.

Re: PARALLEL_MAKE

[Ulf Samuelsson]

>> A while back you mentioned a program that displayed the per core CPU 
>> utilization. What was it again?
> 
> Try htop.

On OpenSuSE you just right click on the bottom bar, and add the 
CPU Frequency meter. I added it 8 times, and configured
for each Virtual CPU in the Core i7.

It does not give you utilization, only frequency.
When not used, it is humming along quietly at 1.6 GHz,
and when used, then it ups the frequency to 2.67 GHz,

Best Regards
Ulf Samuelsson 

Re: PARALLEL_MAKE

[Ulf Samuelsson]

> On Thu, Mar 12, 2009 at 11:49:30PM -0400, Denys Dmytriyenko wrote:
>> On Thu, Mar 12, 2009 at 06:38:54PM -0700, Khem Raj wrote:
>> > On Thursday 12 March 2009 16:32:16 Ulf Samuelsson wrote:
>> > > Is there any activity to make this more useful?
>> > > Just managed to install a Core i7 machine, which speeded
>> > > up my Buildroot build from about 4 hours to less than 1 hour.
>> > > (There was an error in the build right at the end,
>> > > so it could have been downto 40 minutes or so)
>> > > Building the cross compiler took 10 minutes with "make -j 16"
>> > > compared to 1-2 hours on a P4 @ 3GHz.
>> > >
>> > > Openembedded does not seem to use PARALLEL_MAKE extensively.
>> > > I have the frequency meter activated for all 8 CPUs in the Core i7,
>> > > and most of the time the CPUs seems to idle at 1.6 GHz,
>> > > with one CPU (or rarely 2) are running at the full 2.67 GHz.
>> > >
>> > > Last time I built OE on a P4@ 3 GHz it took ~40 hours
>> > > for the SAM9263, so some speedup would be nice.
>> > > Maybe we should make an effort?
>> >
>> > I use
>> >
>> > PARALLEL_MAKE = "-j4"
>> > BB_NUMBER_THREADS = "2"
>> >
>> > in my local.conf. I see my core 2 duo machine spawning parallel tasks.
>>
>> I'm setting 4/4 to the above two variables on my Quad Core and it loads 
>> all 4
>> cores at 100% almost all the time. Sometimes it slows down due to the
>> bottleneck in the harddrive access...
>
> On the dual dual-core (4 total) boxes I have, I do -j6/4 threads.  On my
> new quad-core box I haven't yet had a chance to figure out what's best.
> I also use -j6/4 threads on an older dual-core amd64 box we've got
> around and that seems best for my tasks at least (I tried a few
> combinations of more jobs/threads and less and both were slower).
>

I looked at a few recipies which by a stanrge coincidence had 
PARALLEL_MAKE="",

When, I rechecked, I had
PARALLEL_MAKE ="-j 16"
as well and this should be
PARALLEL_MAKE ="-j16"

Changed to
PARALLEL_MAKE ="-j8"
BB_NUMBER_THREADS = "2"

I see some more cores beeing used.
The initial setup after you changed local.conf seems to run on a single core 
though.

Thanks for the help.
Ulf Samuelsson


> -- 
> Tom Rini
>
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
> 

Re: PARALLEL_MAKE

[Denys Dmytriyenko]

On Fri, Mar 13, 2009 at 03:19:37PM +0100, Ulf Samuelsson wrote:
>
>> On Thu, Mar 12, 2009 at 11:49:30PM -0400, Denys Dmytriyenko wrote:
>>> On Thu, Mar 12, 2009 at 06:38:54PM -0700, Khem Raj wrote:
>>> > On Thursday 12 March 2009 16:32:16 Ulf Samuelsson wrote:
>>> > > Is there any activity to make this more useful?
>>> > > Just managed to install a Core i7 machine, which speeded
>>> > > up my Buildroot build from about 4 hours to less than 1 hour.
>>> > > (There was an error in the build right at the end,
>>> > > so it could have been downto 40 minutes or so)
>>> > > Building the cross compiler took 10 minutes with "make -j 16"
>>> > > compared to 1-2 hours on a P4 @ 3GHz.
>>> > >
>>> > > Openembedded does not seem to use PARALLEL_MAKE extensively.
>>> > > I have the frequency meter activated for all 8 CPUs in the Core i7,
>>> > > and most of the time the CPUs seems to idle at 1.6 GHz,
>>> > > with one CPU (or rarely 2) are running at the full 2.67 GHz.
>>> > >
>>> > > Last time I built OE on a P4@ 3 GHz it took ~40 hours
>>> > > for the SAM9263, so some speedup would be nice.
>>> > > Maybe we should make an effort?
>>> >
>>> > I use
>>> >
>>> > PARALLEL_MAKE = "-j4"
>>> > BB_NUMBER_THREADS = "2"
>>> >
>>> > in my local.conf. I see my core 2 duo machine spawning parallel tasks.
>>>
>>> I'm setting 4/4 to the above two variables on my Quad Core and it loads 
>>> all 4
>>> cores at 100% almost all the time. Sometimes it slows down due to the
>>> bottleneck in the harddrive access...
>>
>> On the dual dual-core (4 total) boxes I have, I do -j6/4 threads.  On my
>> new quad-core box I haven't yet had a chance to figure out what's best.
>> I also use -j6/4 threads on an older dual-core amd64 box we've got
>> around and that seems best for my tasks at least (I tried a few
>> combinations of more jobs/threads and less and both were slower).
>>
>
> I looked at a few recipies which by a stanrge coincidence had 
> PARALLEL_MAKE="",

That's not a strange coincidence, that means a specific package has issues 
building in parallel - usually some dependencies go out of order.

> When, I rechecked, I had
> PARALLEL_MAKE ="-j 16"
> as well and this should be
> PARALLEL_MAKE ="-j16"

Having only 4 cores and setting number of parallel makes to 16 kills your 
performance, as system spends more time switching contexts instead of actually 
doing something useful - i.e. compiling.

> Changed to
> PARALLEL_MAKE ="-j8"
> BB_NUMBER_THREADS = "2"
>
> I see some more cores beeing used.
> The initial setup after you changed local.conf seems to run on a single 
> core though.

After you change your configuration, bitbake rebuilds the cache - that is a 
single threaded operation, hence uses only one core.

-- 
Denys

Re: PARALLEL_MAKE

[Denys Dmytriyenko]

On Fri, Mar 13, 2009 at 03:19:17PM +0100, Ulf Samuelsson wrote:
>
>>> A while back you mentioned a program that displayed the per core CPU 
>>> utilization. What was it again?
>> Try htop.
>
> On OpenSuSE you just right click on the bottom bar, and add the CPU 
> Frequency meter. I added it 8 times, and configured
> for each Virtual CPU in the Core i7.

You are probably talking about some sort of window manager or desktop 
environment for X Window - luckily we are talking about Linux and it can mean 
pretty much any of the available ones. So, saying things like "right click on 
the bar and add something" w/o specifying your environment doesn't mean much...

> It does not give you utilization, only frequency.
> When not used, it is humming along quietly at 1.6 GHz,
> and when used, then it ups the frequency to 2.67 GHz,

I doubt it is the only meter available in your environment. CPU 
load/utilization is very common and is available on pretty much any WM/DE.

-- 
Denys

Re: PARALLEL_MAKE

[Ulf Samuelsson]

>>> On the dual dual-core (4 total) boxes I have, I do -j6/4 threads.  On my
>>> new quad-core box I haven't yet had a chance to figure out what's best.
>>> I also use -j6/4 threads on an older dual-core amd64 box we've got
>>> around and that seems best for my tasks at least (I tried a few
>>> combinations of more jobs/threads and less and both were slower).
>>>
>>
>> I looked at a few recipies which by a stanrge coincidence had
>> PARALLEL_MAKE="",
>
> That's not a strange coincidence, that means a specific package has issues
> building in parallel - usually some dependencies go out of order.
>

The strange coincidence was that *ALL*  the packages I *looked* at
had PARALLEL_MAKE="".
I made the erroneous assumption that most packages were this way.

>> When, I rechecked, I had
>> PARALLEL_MAKE ="-j 16"
>> as well and this should be
>> PARALLEL_MAKE ="-j16"
>
> Having only 4 cores and setting number of parallel makes to 16 kills your
> performance, as system spends more time switching contexts instead of 
> actually
> doing something useful - i.e. compiling.

The Core i7 has 4 cores and each core has two virtual CPUs so the system 
sees 8 CPUs.
I saw some commone from someone claiming that if
you have n CPUs, then Buildroot should use make -j`2*n`.

I thought about this, and came up with the following explanation.
If you have one job per CPU, then  whenever you have to wait for the 
harddisk, you stall this CPU.
If you have several jobs per CPU, then you can gain some time
by running the second job while the first job was waiting for the hard disk.

Whether this works in practice, I cannot say, but it would be interesting
to have some real proof.

>
>> Changed to
>> PARALLEL_MAKE ="-j8"
>> BB_NUMBER_THREADS = "2"
>>
>> I see some more cores beeing used.
>> The initial setup after you changed local.conf seems to run on a single
>> core though.
>
> After you change your configuration, bitbake rebuilds the cache - that is 
> a
> single threaded operation, hence uses only one core.
>
> -- 
> Denys


Best Regards
Ulf Samuelsson

Re: PARALLEL_MAKE

[Denys Dmytriyenko]

On Fri, Mar 13, 2009 at 04:27:08PM +0100, Ulf Samuelsson wrote:
>>>> On the dual dual-core (4 total) boxes I have, I do -j6/4 threads.  On my
>>>> new quad-core box I haven't yet had a chance to figure out what's best.
>>>> I also use -j6/4 threads on an older dual-core amd64 box we've got
>>>> around and that seems best for my tasks at least (I tried a few
>>>> combinations of more jobs/threads and less and both were slower).
>>>>
>>>
>>> I looked at a few recipies which by a stanrge coincidence had
>>> PARALLEL_MAKE="",
>>
>> That's not a strange coincidence, that means a specific package has issues
>> building in parallel - usually some dependencies go out of order.
>>
>
> The strange coincidence was that *ALL*  the packages I *looked* at
> had PARALLEL_MAKE="".
> I made the erroneous assumption that most packages were this way.

Ah, I see. I'm not sure what is the percentage of "broken" packages...

>>> When, I rechecked, I had
>>> PARALLEL_MAKE ="-j 16"
>>> as well and this should be
>>> PARALLEL_MAKE ="-j16"
>>
>> Having only 4 cores and setting number of parallel makes to 16 kills your
>> performance, as system spends more time switching contexts instead of 
>> actually
>> doing something useful - i.e. compiling.
>
> The Core i7 has 4 cores and each core has two virtual CPUs so the system 
> sees 8 CPUs.
> I saw some commone from someone claiming that if
> you have n CPUs, then Buildroot should use make -j`2*n`.
>
> I thought about this, and came up with the following explanation.
> If you have one job per CPU, then  whenever you have to wait for the 
> harddisk, you stall this CPU.
> If you have several jobs per CPU, then you can gain some time
> by running the second job while the first job was waiting for the hard 
> disk.
>
> Whether this works in practice, I cannot say, but it would be interesting
> to have some real proof.

First of all, I believe -j`2*n` is very agressive. I've seen other suggestions 
of using -j`n+1` or -j`n+2`.
Second, your Core i7 still has 4 real cores. Even though HyperThreading 
presents each core as 2 virtual ones, those are not real 8 cores. And 
according to [1], sometimes using HyperThreading may degrade the overall 
performance...

So, you should experiment with your settings and find a sweet spot, which 
maybe somewhere between 4 and 16 threads :)

[1] http://en.wikipedia.org/wiki/Hyper-threading

-- 
Denys

Re: PARALLEL_MAKE

[Frans Meulenbroeks]

I'm not sure how PARALLEL_MAKE is supposed to work, but I noticed that
often there is a problem because with it a package on which there is a
dependency is build in parallel. Since there is a dependency in a lot
of cases the underlying package fails.

A typical scenario:
a package X depends on perl-native, typically they start to build in
parallel but perl-native does take much longer than X. X requires
perl-native to complete and fails.

Parallelism within a task (e.g. running two compiles in parallel would work).
Parallelism between tasks is tricky although some of them can always
be parallelized (e.g. downloading, unpacking and patching).

Frans

Re: PARALLEL_MAKE

[Tom Rini]

On Fri, Mar 13, 2009 at 05:20:51PM +0100, Frans Meulenbroeks wrote:

> I'm not sure how PARALLEL_MAKE is supposed to work, but I noticed that
> often there is a problem because with it a package on which there is a
> dependency is build in parallel. Since there is a dependency in a lot
> of cases the underlying package fails.

PARALLEL_MAKE is just a wrapper around make ( / cmake?) -j N.
BB_NUMBER_THREADS is how you get different packages going in parallel.

> A typical scenario:
> a package X depends on perl-native, typically they start to build in
> parallel but perl-native does take much longer than X. X requires
> perl-native to complete and fails.

Right.  This is a bug on package X not listing perl-native as a
dependancy as with BB_NUMBER_THREADS >1 perl-native will be worked until
do_populate_staging before X do_configure is started.

-- 
Tom Rini

Re: PARALLEL_MAKE

[Richard Purdie]

On Fri, 2009-03-13 at 15:19 +0100, Ulf Samuelsson wrote:
> I looked at a few recipies which by a stanrge coincidence had 
> PARALLEL_MAKE="",

That line means the recipe has some kind of Makefile dependency breakage
issues with "make -j" and therefore for that recipe only, parallel make
is disabled.

> Changed to
> PARALLEL_MAKE ="-j8"
> BB_NUMBER_THREADS = "2"

BB_NUMBER_THREADS is the one for real performance and can scale to a lot
of cores and use them all. As others have commented, disk bandwidth
becomes a real problem.

> I see some more cores beeing used.
> The initial setup after you changed local.conf seems to run on a single core 
> though.

When we added parallel core support the condition was it defaulted to
single core (backwards compatible). That has continued to this day and
perhaps the documentation needs to be improved. Poky's sample local.conf
has commented out entries with an explaination of why you'd set those
variables. These days I would accept an argument for:

PARALLEL_MAKE ="-j2"
BB_NUMBER_THREADS = "2"

in OE's sample local.conf as things have been well tested in this area
for a long time now and more cores are becoming the norm.

Cheers,

Richard

Re: PARALLEL_MAKE

[Richard Purdie]

On Fri, 2009-03-13 at 10:56 -0400, Denys Dmytriyenko wrote:
> After you change your configuration, bitbake rebuilds the cache - that is a 
> single threaded operation, hence uses only one core.

Some brave person could make the parsing process multi-threaded. Its
certainly possible and not that hard...

Cheers,

Richard

Re: PARALLEL_MAKE

[Richard Purdie]

On Fri, 2009-03-13 at 17:20 +0100, Frans Meulenbroeks wrote:
> I'm not sure how PARALLEL_MAKE is supposed to work, but I noticed that
> often there is a problem because with it a package on which there is a
> dependency is build in parallel. Since there is a dependency in a lot
> of cases the underlying package fails.
> 
> A typical scenario:
> a package X depends on perl-native, typically they start to build in
> parallel but perl-native does take much longer than X. X requires
> perl-native to complete and fails.
> 
> Parallelism within a task (e.g. running two compiles in parallel would work).
> Parallelism between tasks is tricky although some of them can always
> be parallelized (e.g. downloading, unpacking and patching).

Bitbake is aware of dependencies and will not build a task until all the
dependent tasks are complete. If something breaks due to something being
missing, its most likely a bug in the metadata and DEPENDS are probably
missing from a recipe.

Regards,

Richard

Re: PARALLEL_MAKE

[Ulf Samuelsson]

>> The Core i7 has 4 cores and each core has two virtual CPUs so the system
>> sees 8 CPUs.
>> I saw some commone from someone claiming that if
>> you have n CPUs, then Buildroot should use make -j`2*n`.
>>
>> I thought about this, and came up with the following explanation.
>> If you have one job per CPU, then  whenever you have to wait for the
>> harddisk, you stall this CPU.
>> If you have several jobs per CPU, then you can gain some time
>> by running the second job while the first job was waiting for the hard
>> disk.
>>
>> Whether this works in practice, I cannot say, but it would be interesting
>> to have some real proof.
>
> First of all, I believe -j`2*n` is very agressive. I've seen other 
> suggestions
> of using -j`n+1` or -j`n+2`.
> Second, your Core i7 still has 4 real cores. Even though HyperThreading
> presents each core as 2 virtual ones, those are not real 8 cores. And
> according to [1], sometimes using HyperThreading may degrade the overall
> performance...
>
> So, you should experiment with your settings and find a sweet spot, which
> maybe somewhere between 4 and 16 threads :)
>

Yes, I know,
Maybe it is a good idea if the result of a build is published somewhere,
so that others do not have to go through this process.

Right now, openembedded does not even build for me,
since the native perl build fails :-(

> [1] http://en.wikipedia.org/wiki/Hyper-threading
>


Best Regards
Ulf Samuelsson 

Re: PARALLEL_MAKE

[Frans Meulenbroeks]

Ah,

Thanks alot Tom, I was unaware of this. Mostly I repaired thse
breakages by disabling PARALLEL_MAKE. Apparently most of the time not
the optimal solution....
Will do things better in the future (and will try to repair things
where things go wrong.
One small Q though: The fact that a package is build during the build
of another pacakge, doesn't that implicitly say that there is a
dependency between these?
(actually thinking of it as I write this: guess most packages should
depend on gcc, glibc etc but these are not made explicit)

Frans.

2009/3/13 Tom Rini <trini@kernel.crashing.org>:
> On Fri, Mar 13, 2009 at 05:20:51PM +0100, Frans Meulenbroeks wrote:
>
>> I'm not sure how PARALLEL_MAKE is supposed to work, but I noticed that
>> often there is a problem because with it a package on which there is a
>> dependency is build in parallel. Since there is a dependency in a lot
>> of cases the underlying package fails.
>
> PARALLEL_MAKE is just a wrapper around make ( / cmake?) -j N.
> BB_NUMBER_THREADS is how you get different packages going in parallel.
>
>> A typical scenario:
>> a package X depends on perl-native, typically they start to build in
>> parallel but perl-native does take much longer than X. X requires
>> perl-native to complete and fails.
>
> Right.  This is a bug on package X not listing perl-native as a
> dependancy as with BB_NUMBER_THREADS >1 perl-native will be worked until
> do_populate_staging before X do_configure is started.
>
> --
> Tom Rini
>
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
>

Re: PARALLEL_MAKE

[Tom Rini]

On Sat, Mar 14, 2009 at 11:29:28AM +0100, Frans Meulenbroeks wrote:
> Ah,
> 
> Thanks alot Tom, I was unaware of this. Mostly I repaired thse
> breakages by disabling PARALLEL_MAKE. Apparently most of the time not
> the optimal solution....
> Will do things better in the future (and will try to repair things
> where things go wrong.
> One small Q though: The fact that a package is build during the build
> of another pacakge, doesn't that implicitly say that there is a
> dependency between these?

Nope, in fact it's the opposite.  BB_NUMBER_THREADS is a max, not a min
that can be going.  So with 4 threads, up to 4 tasks that have had all
pre-requisite tasks completed, may be running.

> (actually thinking of it as I write this: guess most packages should
> depend on gcc, glibc etc but these are not made explicit)

Actually, I think those deps are explicit in the DEFAULT_DEPS or so
variable, somewhere (which is why glibc and a few others have
INHIBIT_DEFAULT_DEPS = 1).

-- 
Tom Rini

Re: PARALLEL_MAKE

[Richard Purdie]

On Sat, 2009-03-14 at 10:42 -0700, Tom Rini wrote:
> On Sat, Mar 14, 2009 at 11:29:28AM +0100, Frans Meulenbroeks wrote:
> > (actually thinking of it as I write this: guess most packages should
> > depend on gcc, glibc etc but these are not made explicit)
> 
> Actually, I think those deps are explicit in the DEFAULT_DEPS or so
> variable, somewhere (which is why glibc and a few others have
> INHIBIT_DEFAULT_DEPS = 1).

Close. They get added to DEPENDS by base.bbclass unless
INHIBIT_DEFAULT_DEPS = 1.

Cheers,

Richard

Re: PARALLEL_MAKE

[Ulf Samuelsson]

>>> The Core i7 has 4 cores and each core has two virtual CPUs so the system
>>> sees 8 CPUs.
>>> I saw some commone from someone claiming that if
>>> you have n CPUs, then Buildroot should use make -j`2*n`.
>>>
>>> I thought about this, and came up with the following explanation.
>>> If you have one job per CPU, then  whenever you have to wait for the
>>> harddisk, you stall this CPU.
>>> If you have several jobs per CPU, then you can gain some time
>>> by running the second job while the first job was waiting for the hard
>>> disk.
>>>
>>> Whether this works in practice, I cannot say, but it would be 
>>> interesting
>>> to have some real proof.
>>
>> First of all, I believe -j`2*n` is very agressive. I've seen other 
>> suggestions
>> of using -j`n+1` or -j`n+2`.
>> Second, your Core i7 still has 4 real cores. Even though HyperThreading
>> presents each core as 2 virtual ones, those are not real 8 cores. And
>> according to [1], sometimes using HyperThreading may degrade the overall
>> performance...
>>
>> So, you should experiment with your settings and find a sweet spot, which
>> maybe somewhere between 4 and 16 threads :)
>>
>
> Yes, I know,
> Maybe it is a good idea if the result of a build is published somewhere,
> so that others do not have to go through this process.
>
> Right now, openembedded does not even build for me,
> since the native perl build fails :-(
>
>> [1] http://en.wikipedia.org/wiki/Hyper-threading
>>
>


I now managed to build the Angstrom image for the AT91SAM9263 by applying
a few fixes present in the ulf/linux branch.
(Did not load on H/W yet though)

perl-native was "fixed" by not building the library which crashed.
libtheora was fixd, not to use docs/examples.
busybox updated to 1.13.3
u-boot, gpsd was set to PARALLEL_MAKE = ""
u-boot updated to 2009.01
linux-2.6.28
at91bootstrap-2.10

Building with PARALLEL_MAKE & Multiple threads was not smooth.
The build crashed several times, due to erros
caused by parallellism. "gettext", "gpsd" and "u-boot" were culprits.

u-boot is NOT built cleanly, even after PARALLEL_MAKE was set to "".
The actual compile completes and generates an u-boot.bin
but this is not deployed to the result directory.
There are logs for different stages, but after the compile, they exist, but 
are empty.

If I remove all the stamps and "bitbake -b <u-boot>" afterwards
then the build completes correctly, and there is an u-boot-at91sam9263ek.bin
in the result directory.

Can it be so, that a stage is started, before the previous stage has been 
completed???



---------------------
BTW: The total buildtime was around 2 hours. There is some overhead
since the build was restarted 3-4 times, and local.conf was changed once.
I started the build with PARALLEL_MAKE = "-j2" and BB_NUMBER_THREADS=8.

I noticed that having a lot of threads helps most of the time,
since the CPUs were executing at max frequency.

When the cross compiler, was built,it was different,
most of the cores were not used with this configuration.
2-3 max while compiling gcc.

Maybe the best approach is to build the Angstrom in several stages
with different settings for parallelism.



Best Regards
Ulf Samuelsson 

Re: PARALLEL_MAKE

[Denys Dmytriyenko]

On Tue, Mar 17, 2009 at 05:28:27PM +0100, Ulf Samuelsson wrote:
>> Yes, I know,
>> Maybe it is a good idea if the result of a build is published somewhere,
>> so that others do not have to go through this process.
>>
>> Right now, openembedded does not even build for me,
>> since the native perl build fails :-(
>>
>>> [1] http://en.wikipedia.org/wiki/Hyper-threading
>
> I now managed to build the Angstrom image for the AT91SAM9263 by applying
> a few fixes present in the ulf/linux branch.
> (Did not load on H/W yet though)
>
> perl-native was "fixed" by not building the library which crashed.
> libtheora was fixd, not to use docs/examples.
> busybox updated to 1.13.3

Any specific changes between 1.13.2 and 1.13.3 you need? Would be nice to 
cherry-pick into .dev branch...

> u-boot, gpsd was set to PARALLEL_MAKE = ""
> u-boot updated to 2009.01
> linux-2.6.28
> at91bootstrap-2.10
>
> Building with PARALLEL_MAKE & Multiple threads was not smooth.
> The build crashed several times, due to erros
> caused by parallellism. "gettext", "gpsd" and "u-boot" were culprits.

Would be nice to fix those. Tom Rini was working on fixing some of those 
issues.

> u-boot is NOT built cleanly, even after PARALLEL_MAKE was set to "".
> The actual compile completes and generates an u-boot.bin
> but this is not deployed to the result directory.
> There are logs for different stages, but after the compile, they exist, but 
> are empty.
>
> If I remove all the stamps and "bitbake -b <u-boot>" afterwards
> then the build completes correctly, and there is an 
> u-boot-at91sam9263ek.bin
> in the result directory.
>
> Can it be so, that a stage is started, before the previous stage has been 
> completed???

If there are no dependencies between packages, their stage tasks can run in 
parallel.

> ---------------------
> BTW: The total buildtime was around 2 hours. There is some overhead
> since the build was restarted 3-4 times, and local.conf was changed once.
> I started the build with PARALLEL_MAKE = "-j2" and BB_NUMBER_THREADS=8.
>
> I noticed that having a lot of threads helps most of the time,
> since the CPUs were executing at max frequency.
>
> When the cross compiler, was built,it was different,
> most of the cores were not used with this configuration.
> 2-3 max while compiling gcc.

Since gcc/cross is quite a big package and lots of other packages depend on 
it, it is being built for you only using 2 cores, as specified by 
PARALLEL_MAKE. Everything else is waiting for it to finish.

> Maybe the best approach is to build the Angstrom in several stages
> with different settings for parallelism.

-- 
Denys

Re: PARALLEL_MAKE

[Tom Rini]

On Tue, Mar 17, 2009 at 05:28:27PM +0100, Ulf Samuelsson wrote:
[snip]
> BTW: The total buildtime was around 2 hours. There is some overhead
> since the build was restarted 3-4 times, and local.conf was changed once.
> I started the build with PARALLEL_MAKE = "-j2" and BB_NUMBER_THREADS=8.

Hmm.  I wonder if the gettext problem was related to having so many
threads.  For my builds I had been getting around 2 hours (once sources
are fetched), but as I've said, I do -j8/4 threads.  I haven't yet
expirimented with my new box, so I'll give a j2/8 threads one a shot and
see where gettext is falling down.

> I noticed that having a lot of threads helps most of the time,
> since the CPUs were executing at max frequency.
>
> When the cross compiler, was built,it was different,
> most of the cores were not used with this configuration.
> 2-3 max while compiling gcc.

Yeah.  I almost wonder if it's worth doing some python string magic to
try and manually bump up PARALLEL_MAKE in recipes that we know are
bottlenecks.

-- 
Tom Rini

Re: PARALLEL_MAKE

[Ulf Samuelsson]

Best Regards
Ulf Samuelsson                ulf@atmel.com
Atmel Nordic AB
Mail:  Box 2033, 174 02 Sundbyberg, Sweden
Visit:  Kavallerivägen 24, 174 58 Sundbyberg, Sweden
Phone +46 (8) 441 54 22     Fax +46 (8) 441 54 29
GSM    +46 (706) 22 44 57



> On Tue, Mar 17, 2009 at 05:28:27PM +0100, Ulf Samuelsson wrote:
>>> Yes, I know,
>>> Maybe it is a good idea if the result of a build is published somewhere,
>>> so that others do not have to go through this process.
>>>
>>> Right now, openembedded does not even build for me,
>>> since the native perl build fails :-(
>>>
>>>> [1] http://en.wikipedia.org/wiki/Hyper-threading
>>
>> I now managed to build the Angstrom image for the AT91SAM9263 by applying
>> a few fixes present in the ulf/linux branch.
>> (Did not load on H/W yet though)
>>
>> perl-native was "fixed" by not building the library which crashed.
>> libtheora was fixd, not to use docs/examples.
>> busybox updated to 1.13.3
>
> Any specific changes between 1.13.2 and 1.13.3 you need? Would be nice to
> cherry-pick into .dev branch...

The reason I did this was that I had an error on downloading one
of the patches for 1.13.2. (md5 Checksum was not the same as in recipe)
I did not investigate further.
Just generally thought it to be a good idea to use the latest.

>
>> u-boot, gpsd was set to PARALLEL_MAKE = ""
>> u-boot updated to 2009.01
>> linux-2.6.28
>> at91bootstrap-2.10
>>
>> Building with PARALLEL_MAKE & Multiple threads was not smooth.
>> The build crashed several times, due to erros
>> caused by parallellism. "gettext", "gpsd" and "u-boot" were culprits.
>
> Would be nice to fix those. Tom Rini was working on fixing some of those
> issues.
>
>> u-boot is NOT built cleanly, even after PARALLEL_MAKE was set to "".
>> The actual compile completes and generates an u-boot.bin
>> but this is not deployed to the result directory.
>> There are logs for different stages, but after the compile, they exist, 
>> but
>> are empty.
>>
>> If I remove all the stamps and "bitbake -b <u-boot>" afterwards
>> then the build completes correctly, and there is an
>> u-boot-at91sam9263ek.bin
>> in the result directory.
>>
>> Can it be so, that a stage is started, before the previous stage has been
>> completed???
>
> If there are no dependencies between packages, their stage tasks can run 
> in
> parallel.

I know that, but the u-boot problem seems to be that a stage
is started before an earlier stage is completed.
I have not verified this, but doing bitbake gpe-image
will cause the u-boot-2009.01 (in my branch) to fail.
The u-boot.bin is not copied to the result.
while doing "bitbake -b <u-boot-2009.01>.bb" will work.

The remainder of the stages will run, when I do bitbake gpe-image
but the log is empty in the first case.


>
>> ---------------------
>> BTW: The total buildtime was around 2 hours. There is some overhead
>> since the build was restarted 3-4 times, and local.conf was changed once.
>> I started the build with PARALLEL_MAKE = "-j2" and BB_NUMBER_THREADS=8.
>>
>> I noticed that having a lot of threads helps most of the time,
>> since the CPUs were executing at max frequency.
>>
>> When the cross compiler, was built,it was different,
>> most of the cores were not used with this configuration.
>> 2-3 max while compiling gcc.
>
> Since gcc/cross is quite a big package and lots of other packages depend 
> on
> it, it is being built for you only using 2 cores, as specified by
> PARALLEL_MAKE. Everything else is waiting for it to finish.

Yes, I realize that so I have to increase PARALLEL_MAKE
to speed up the  build of the tools,
but I would also like to have a large number of threads otherwise.
But I do not want to have a large number of threads and
and a large number of make jobs at the same time
if I want to run at optimum speed throughout the build.



>
>> Maybe the best approach is to build the Angstrom in several stages
>> with different settings for parallelism.
>
> -- 
> Denys
>
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
> 

Re: PARALLEL_MAKE

[Ulf Samuelsson]

> On Tue, Mar 17, 2009 at 05:28:27PM +0100, Ulf Samuelsson wrote:
> [snip]
>> BTW: The total buildtime was around 2 hours. There is some overhead
>> since the build was restarted 3-4 times, and local.conf was changed once.
>> I started the build with PARALLEL_MAKE = "-j2" and BB_NUMBER_THREADS=8.
>
> Hmm.  I wonder if the gettext problem was related to having so many
> threads.  For my builds I had been getting around 2 hours (once sources
> are fetched), but as I've said, I do -j8/4 threads.  I haven't yet
> expirimented with my new box, so I'll give a j2/8 threads one a shot and
> see where gettext is falling down.
>
>> I noticed that having a lot of threads helps most of the time,
>> since the CPUs were executing at max frequency.
>>
>> When the cross compiler, was built,it was different,
>> most of the cores were not used with this configuration.
>> 2-3 max while compiling gcc.
>
> Yeah.  I almost wonder if it's worth doing some python string magic to
> try and manually bump up PARALLEL_MAKE in recipes that we know are
> bottlenecks.
>
> -- 

I think that we should have a way to support parallelism
in a file OUTSIDE the recipes.

I think you would want to have a host configuration file
which can contains features to support parallelism.
It should also contain location of downloads
and other things which is host specific.

Maybe we should define the MAXIMUM number of make jobs
we will allow on top of the normal using PARALLEL_MAKE)
and then indicate in specific packages that we
want to use the MAX, rather than the "normal" number of jobs.

Best Regards
Ulf Samuelsson