for a quad-core machine, shouldn't parallelism be set to 8?

for a quad-core machine, shouldn't parallelism be set to 8?

[Robert P. J. Day]

  just noticed this in local.conf.sample:

# For a quad-core machine, BB_NUMBER_THREADS = "4", PARALLEL_MAKE = "-j 4" would
# be appropriate for example.

  i thought the general rule was that parallelism should be *twice*
the number of cores.  that's certainly what's suggested in the yocto
QS guide:

"Another couple of variables of interest are the BB_NUMBER_THREADS and
the PARALLEL_MAKE variables. By default, these variables are commented
out. However, if you have a multi-core CPU you might want to uncomment
the lines and set both variables equal to twice the number of your
host's processor cores. "

  thoughts?

rday

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

Re: for a quad-core machine, shouldn't parallelism be set to 8?

[Elvis Dowson]

Hi Robert,
On my i7 quad core machine, I've set both values to 6. Setting it to 6 appears to be nearly the same as setting it to 8, I dare say a marginally faster by a minute. I can build core image console in around 22 minutes.

Elvis
On Jan 9, 2013, at 4:55 PM, "Robert P. J. Day" <rpjday@crashcourse.ca> wrote:

> 
>  just noticed this in local.conf.sample:
> 
> # For a quad-core machine, BB_NUMBER_THREADS = "4", PARALLEL_MAKE = "-j 4" would
> # be appropriate for example.
> 
>  i thought the general rule was that parallelism should be *twice*
> the number of cores.  that's certainly what's suggested in the yocto
> QS guide:
> 
> "Another couple of variables of interest are the BB_NUMBER_THREADS and
> the PARALLEL_MAKE variables. By default, these variables are commented
> out. However, if you have a multi-core CPU you might want to uncomment
> the lines and set both variables equal to twice the number of your
> host's processor cores. "
> 
>  thoughts?
> 
> rday
> 
> -- 
> 
> ========================================================================
> Robert P. J. Day                                 Ottawa, Ontario, CANADA
>                        http://crashcourse.ca
> 
> Twitter:                                       http://twitter.com/rpjday
> LinkedIn:                               http://ca.linkedin.com/in/rpjday
> ========================================================================
> 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: for a quad-core machine, shouldn't parallelism be set to 8?

[Robert P. J. Day]

On Wed, 9 Jan 2013, Elvis Dowson wrote:

> Hi Robert,

> On my i7 quad core machine, I've set both values to 6. Setting it to
> 6 appears to be nearly the same as setting it to 8, I dare say a
> marginally faster by a minute. I can build core image console in
> around 22 minutes.
>
> Elvis

  i recall someone (richard purdie?) once benchmarked to check the
benefits of increasing parallelism and while it's obvious you'll hit
the wall eventually, it would still be nice for the documentation to
be consistent.

rday

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

Re: for a quad-core machine, shouldn't parallelism be set to 8?

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 820 bytes --]

On Wed, Jan 09, 2013 at 09:26:10AM -0500, Robert P. J. Day wrote:
> On Wed, 9 Jan 2013, Elvis Dowson wrote:
> 
> > Hi Robert,
> 
> > On my i7 quad core machine, I've set both values to 6. Setting it to
> > 6 appears to be nearly the same as setting it to 8, I dare say a
> > marginally faster by a minute. I can build core image console in
> > around 22 minutes.
> >
> > Elvis
> 
>   i recall someone (richard purdie?) once benchmarked to check the
> benefits of increasing parallelism and while it's obvious you'll hit
> the wall eventually, it would still be nice for the documentation to
> be consistent.

You can use openembedded-core/scripts/contrib/bb-perf/bb-matrix.sh to
find best value for your HW and bitbake target.

Cheers,

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: for a quad-core machine, shouldn't parallelism be set to 8?

[Scott Garman]

On 01/09/2013 06:26 AM, Robert P. J. Day wrote:
> On Wed, 9 Jan 2013, Elvis Dowson wrote:
>
>> Hi Robert,
>
>> On my i7 quad core machine, I've set both values to 6. Setting it to
>> 6 appears to be nearly the same as setting it to 8, I dare say a
>> marginally faster by a minute. I can build core image console in
>> around 22 minutes.
>>
>> Elvis
>
>    i recall someone (richard purdie?) once benchmarked to check the
> benefits of increasing parallelism and while it's obvious you'll hit
> the wall eventually, it would still be nice for the documentation to
> be consistent.

FWIW Robert, I think in the cases like this where the patch is fairly 
trivial, I'd recommend just submitting it and getting feedback on it. In 
general I've found you've raised some legitimate issues, but the fixes 
aren't happening because patches aren't coming in, and things like 
documentation consistency on this level is not going to compete with 
other issues the developers are munching on.

Just a suggestion,

Scott

-- 
Scott Garman
Embedded Linux Engineer - Yocto Project
Intel Open Source Technology Center

Re: for a quad-core machine, shouldn't parallelism be set to 8?

[Bjørn Forsman]

On 9 January 2013 13:55, Robert P. J. Day <rpjday@crashcourse.ca> wrote:
>
>   just noticed this in local.conf.sample:
>
> # For a quad-core machine, BB_NUMBER_THREADS = "4", PARALLEL_MAKE = "-j 4" would
> # be appropriate for example.
>
>   i thought the general rule was that parallelism should be *twice*
> the number of cores.  that's certainly what's suggested in the yocto
> QS guide:

Some time ago I wrote a script to build the linux kernel 3 times at
each -jN level from 1 to 50. Then I averaged the build time on each
-jN level. I got the fastest build with -jNCPUS. Going above NCPUS
just added overhead, although very little. The overhead seemed to
build up quite linearly from above -jNCPUS. Because the overhead is so
little, having -jNCPUS*2 may not be noticeably slower than -jNCPUS.

I ran the same benchmark another machine and it showed the same trend
curve. Note that NCPUS must include the hyperthread cores.

Best regards,
Bjørn Forsman

Re: for a quad-core machine, shouldn't parallelism be set to 8?

[Ross Burton]

On Wednesday, 9 January 2013 at 18:42, Bjørn Forsman wrote:
> > # For a quad-core machine, BB_NUMBER_THREADS = "4", PARALLEL_MAKE = "-j 4" would
> > # be appropriate for example.
> >  
> > i thought the general rule was that parallelism should be *twice*
> > the number of cores. that's certainly what's suggested in the yocto
> > QS guide:
>  
>  
>  
> Some time ago I wrote a script to build the linux kernel 3 times at
> each -jN level from 1 to 50. Then I averaged the build time on each
> -jN level. I got the fastest build with -jNCPUS. Going above NCPUS
> just added overhead, although very little. The overhead seemed to
> build up quite linearly from above -jNCPUS. Because the overhead is so
> little, having -jNCPUS*2 may not be noticeably slower than -jNCPUS.
>  
> I ran the same benchmark another machine and it showed the same trend
> curve. Note that NCPUS must include the hyperthread cores.

Of course if you set BB_NUM_THREADS and PARALLEL_MAKE to NCPUS, then you're actually potentially running NCPUS*NCPUS compile threads.  I'm not convinced that NCPUS*2 is useful with that considered.

Ross