* Procedure to setup icecc for performing a distributed build
@ 2012-08-15 20:30 Elvis Dowson
2012-09-06 10:17 ` Paul Eggleton
0 siblings, 1 reply; 8+ messages in thread
From: Elvis Dowson @ 2012-08-15 20:30 UTC (permalink / raw)
To: Yocto Discussion Mailing List
[-- Attachment #1: Type: text/plain, Size: 2748 bytes --]
Hi,
I followed these steps to setup and configure icecc and yocto, for performing a distributed build between two Ubuntu 12.04 64-bit machines, with one machine configured as the scheduler and daemon, and the other with only the icecc daemon.
I run a bitbake core-image-minimal, but get no tasks running on the second machine. Processor and CPU utilization is nearly zero, and both machines iceccd.log and icecc_scheduler.log are zero bytes.
Nothing happens when I run the yocto build, and no compilation tasks get distributed to the second machine in the build cluster. I am able to ping them both, however.
Step 01.00: Install required packages for performing a distributed build.
Step 01.01: Install icecc on all the machines.
$ sudo apt-get install icecc
On the scheduler machine, type the following commands:
$ sudo gedit /etc/default/icecc
# Defaults for icecc initscript
# sourced by /etc/init.d/icecc
START_ICECC="true"
START_ICECC_SCHEDULER="true"
Configure the build network name:
$ sudo /etc/init.d/icecc stop
$ sudo gedit /etc/init.d/icecc.conf
#
# Identification for the network the scheduler and daemon run on.
# You can have several distinct icecc networks in the same LAN
# for whatever reason.
#
# ICECC_NETNAME=""
ICECC_NETNAME="YoctoBuildNetwork"
#
# If the daemon can't find the scheduler by broadcast (e.g. because
# of a firewall) you can specify it.
#
# ICECC_SCHEDULER_HOST=""
ICECC_SCHEDULER_HOST="eos"
Start the scheduler with:
$ sudo /etc/init.d/icecc start
Make sure the service starts at boot time:
$ sudo update-rc.d icecc defaults
Step 02.00: Configure yocto for distributed compilation.
Step 02.01: Modify local.conf
#
# Parallelism Options
#
# These two options control how much parallelism BitBake should use. The first
# option determines how many tasks bitbake should run in parallel:
#
BB_NUMBER_THREADS = "8"
#
# The second option controls how many processes make should run in parallel when
# running compile tasks:
#
PARALLEL_MAKE = "-j 14"
# Setup yocto to use icecc for distributed compilation.
INHERIT += "icecc"
ICECC_PATH = "/usr/bin/icecc"
ICECC_ENV_EXEC = "/tool/yocto/poky/build/tmp/work/x86_64-linux/icecc-create-env-native-0.1-r2/icecc-create-env"
ICECC_USER_CLASS_BL = " native"
Copy the icecc-create-env script to the build output staging folder, by running the icecc-create-env-native.bb recipe.
$ cd /tool/yocto/poky;source oe-init-build-env build
$ bitbake icecc-create-env-native
The icecc-create-env script will automatically be copied to the build output staging folder.
Step 02.02: Start a bit bake build.
$ bitbake core-image-minimal
Best regards,
Elvis Dowson
[-- Attachment #2: Type: text/html, Size: 14832 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: Procedure to setup icecc for performing a distributed build
2012-08-15 20:30 Procedure to setup icecc for performing a distributed build Elvis Dowson
@ 2012-09-06 10:17 ` Paul Eggleton
2012-09-06 15:23 ` Elvis Dowson
2012-09-06 21:46 ` Dmitry Eremin-Solenikov
0 siblings, 2 replies; 8+ messages in thread
From: Paul Eggleton @ 2012-09-06 10:17 UTC (permalink / raw)
To: Elvis Dowson, Dmitry Eremin-Solenikov; +Cc: yocto
On Thursday 16 August 2012 00:30:15 Elvis Dowson wrote:
> Hi,
> I followed these steps to setup and configure icecc and yocto, for
> performing a distributed build between two Ubuntu 12.04 64-bit machines,
> with one machine configured as the scheduler and daemon, and the other with
> only the icecc daemon.
>
> I run a bitbake core-image-minimal, but get no tasks running on the second
> machine. Processor and CPU utilization is nearly zero, and both machines
> iceccd.log and icecc_scheduler.log are zero bytes.
>
> Nothing happens when I run the yocto build, and no compilation tasks get
> distributed to the second machine in the build cluster. I am able to ping
> them both, however.
>
> Step 01.00: Install required packages for performing a distributed build.
>
> Step 01.01: Install icecc on all the machines.
>
> $ sudo apt-get install icecc
>
> On the scheduler machine, type the following commands:
>
> $ sudo gedit /etc/default/icecc
>
> # Defaults for icecc initscript
> # sourced by /etc/init.d/icecc
> START_ICECC="true"
> START_ICECC_SCHEDULER="true"
>
> Configure the build network name:
>
> $ sudo /etc/init.d/icecc stop
>
> $ sudo gedit /etc/init.d/icecc.conf
>
> #
> # Identification for the network the scheduler and daemon run on.
> # You can have several distinct icecc networks in the same LAN
> # for whatever reason.
> #
> # ICECC_NETNAME=""
> ICECC_NETNAME="YoctoBuildNetwork"
>
> #
> # If the daemon can't find the scheduler by broadcast (e.g. because
> # of a firewall) you can specify it.
> #
> # ICECC_SCHEDULER_HOST=""
> ICECC_SCHEDULER_HOST="eos"
>
>
> Start the scheduler with:
>
> $ sudo /etc/init.d/icecc start
>
> Make sure the service starts at boot time:
>
> $ sudo update-rc.d icecc defaults
>
> Step 02.00: Configure yocto for distributed compilation.
>
> Step 02.01: Modify local.conf
>
> #
> # Parallelism Options
> #
> # These two options control how much parallelism BitBake should use. The
> first # option determines how many tasks bitbake should run in parallel: #
> BB_NUMBER_THREADS = "8"
> #
> # The second option controls how many processes make should run in parallel
> when # running compile tasks:
> #
> PARALLEL_MAKE = "-j 14"
>
> # Setup yocto to use icecc for distributed compilation.
> INHERIT += "icecc"
> ICECC_PATH = "/usr/bin/icecc"
> ICECC_ENV_EXEC =
> "/tool/yocto/poky/build/tmp/work/x86_64-linux/icecc-create-env-native-0.1-r
> 2/icecc-create-env" ICECC_USER_CLASS_BL = " native"
>
> Copy the icecc-create-env script to the build output staging folder, by
> running the icecc-create-env-native.bb recipe.
>
> $ cd /tool/yocto/poky;source oe-init-build-env build
> $ bitbake icecc-create-env-native
>
> The icecc-create-env script will automatically be copied to the build output
> staging folder.
>
> Step 02.02: Start a bit bake build.
>
> $ bitbake core-image-minimal
>
Elvis, did you have any further luck with this?
Otherwise, Dmitry, any suggestions? I'm assuming you made use of icecc.bbclass
since you made some changes to it a while ago...
Cheers,
Paul
--
Paul Eggleton
Intel Open Source Technology Centre
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Procedure to setup icecc for performing a distributed build
2012-09-06 10:17 ` Paul Eggleton
@ 2012-09-06 15:23 ` Elvis Dowson
2012-09-06 15:29 ` Paul Eggleton
[not found] ` <5048C1C2.1010903@mlbassoc.com>
2012-09-06 21:46 ` Dmitry Eremin-Solenikov
1 sibling, 2 replies; 8+ messages in thread
From: Elvis Dowson @ 2012-09-06 15:23 UTC (permalink / raw)
To: Paul Eggleton, Dmitry Eremin-Solenikov; +Cc: Yocto Discussion Mailing List
Hi Paul,
On Sep 6, 2012, at 2:17 PM, Paul Eggleton wrote:
> Elvis, did you have any further luck with this?
Unfortunately no. I've got two machines, both with quad-core intel i7 processors, but I just couldn't
get icecc to work with yocto. I end up regularly perform fresh builds at least 5 to 6 times a day,
and it takes me 2 hours to build core-image-minimal.
> Otherwise, Dmitry, any suggestions? I'm assuming you made use of icecc.bbclass
> since you made some changes to it a while ago...
Best regards,
Elvis Dowson
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: Procedure to setup icecc for performing a distributed build
2012-09-06 15:23 ` Elvis Dowson
@ 2012-09-06 15:29 ` Paul Eggleton
2012-09-06 15:33 ` Gary Thomas
[not found] ` <5048C1C2.1010903@mlbassoc.com>
1 sibling, 1 reply; 8+ messages in thread
From: Paul Eggleton @ 2012-09-06 15:29 UTC (permalink / raw)
To: Elvis Dowson; +Cc: Dmitry Eremin-Solenikov, Yocto Discussion Mailing List
On Thursday 06 September 2012 19:23:53 Elvis Dowson wrote:
> Unfortunately no. I've got two machines, both with quad-core intel i7
> processors, but I just couldn't get icecc to work with yocto. I end up
> regularly perform fresh builds at least 5 to 6 times a day, and it takes me
> 2 hours to build core-image-minimal.
At its core, icecc.bbclass just sets things up so that icecc is called instead
of the normal compiler. It would be worth verifying that that is actually
taking effect by looking at the commands being used as reported in
log.do_configure for specific recipes.
Cheers,
Paul
--
Paul Eggleton
Intel Open Source Technology Centre
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: Procedure to setup icecc for performing a distributed build
2012-09-06 15:29 ` Paul Eggleton
@ 2012-09-06 15:33 ` Gary Thomas
2012-09-06 15:38 ` Paul Eggleton
0 siblings, 1 reply; 8+ messages in thread
From: Gary Thomas @ 2012-09-06 15:33 UTC (permalink / raw)
To: yocto
On 2012-09-06 09:29, Paul Eggleton wrote:
> On Thursday 06 September 2012 19:23:53 Elvis Dowson wrote:
>> Unfortunately no. I've got two machines, both with quad-core intel i7
>> processors, but I just couldn't get icecc to work with yocto. I end up
>> regularly perform fresh builds at least 5 to 6 times a day, and it takes me
>> 2 hours to build core-image-minimal.
>
> At its core, icecc.bbclass just sets things up so that icecc is called instead
> of the normal compiler. It would be worth verifying that that is actually
> taking effect by looking at the commands being used as reported in
> log.do_configure for specific recipes.
Is this only for the host compiler or the target (cross) compilers as well?
If it's only the host compiler being used for "native" builds, how much is
there to be gained from distributing the work?
--
------------------------------------------------------------
Gary Thomas | Consulting for the
MLB Associates | Embedded world
------------------------------------------------------------
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Procedure to setup icecc for performing a distributed build
2012-09-06 15:33 ` Gary Thomas
@ 2012-09-06 15:38 ` Paul Eggleton
0 siblings, 0 replies; 8+ messages in thread
From: Paul Eggleton @ 2012-09-06 15:38 UTC (permalink / raw)
To: Gary Thomas; +Cc: yocto
On Thursday 06 September 2012 09:33:26 Gary Thomas wrote:
> On 2012-09-06 09:29, Paul Eggleton wrote:
> > On Thursday 06 September 2012 19:23:53 Elvis Dowson wrote:
> >> Unfortunately no. I've got two machines, both with quad-core intel i7
> >> processors, but I just couldn't get icecc to work with yocto. I end up
> >> regularly perform fresh builds at least 5 to 6 times a day, and it takes
> >> me
> >> 2 hours to build core-image-minimal.
> >
> > At its core, icecc.bbclass just sets things up so that icecc is called
> > instead of the normal compiler. It would be worth verifying that that is
> > actually taking effect by looking at the commands being used as reported
> > in log.do_configure for specific recipes.
>
> Is this only for the host compiler or the target (cross) compilers as well?
> If it's only the host compiler being used for "native" builds, how much is
> there to be gained from distributing the work?
It covers all according to the comments at the top of the file.
I have to be honest and say I've never used icecc.bbclass; I've been told that
the overhead mostly wipes out any advantage of individual compile jobs being
farmed out to different machines, but I've never actually verified that first
hand.
Cheers,
Paul
--
Paul Eggleton
Intel Open Source Technology Centre
^ permalink raw reply [flat|nested] 8+ messages in thread
[parent not found: <5048C1C2.1010903@mlbassoc.com>]
* Re: Procedure to setup icecc for performing a distributed build
[not found] ` <5048C1C2.1010903@mlbassoc.com>
@ 2012-09-06 17:52 ` Elvis Dowson
0 siblings, 0 replies; 8+ messages in thread
From: Elvis Dowson @ 2012-09-06 17:52 UTC (permalink / raw)
To: Gary Thomas; +Cc: Yocto Discussion Mailing List
Hi Gary,
On Sep 6, 2012, at 7:31 PM, Gary Thomas wrote:
> On 2012-09-06 09:23, Elvis Dowson wrote:
>>
>> Unfortunately no. I've got two machines, both with quad-core intel i7 processors, but I just couldn't
>> get icecc to work with yocto. I end up regularly perform fresh builds at least 5 to 6 times a day,
>> and it takes me 2 hours to build core-image-minimal.
>
> Just wondering the exact details of your system and why your performance is
> so very different from mine.
>
> I have a HP box with 8-way Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz & 12GB RAM
> It will build a Yocto image from scratch in around 40 minutes.
I'm running Ubuntu 12.04 LTS virtualized on a 2011 Apple MacBook Pro 15", with
VMware Fusion 5.0.1. The VMware disk image resides on a SSD drive with a 3G
SATA interface.
The specs are 2.3GHz Intel quad-core i7-2820QM with 8MB L3 cache,
8GB 1333 MHz RAM for the host.
The Ubuntu 12.04 guest is configured with 4GB RAM, 8 virtual CPU cores.
During core-image-minimal builds, the memory usage doesn't exceed 2 to 3GB.
A linux kernel 3.3.0 build for a powepc440 target takes 1 min 40 seconds, as
an additional data point.
Maybe next year, when the newer Mac Pros come out, I'll probably invest in
a 8-core or 10-core xeon configuration. The current Mac Pros are due for a refresh
then, so I'm waiting for the next hardware update.
I'm also considered buying an IBM Blade server, but haven't made up my mind yet.
Best regards,
Elvis Dowson
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Procedure to setup icecc for performing a distributed build
2012-09-06 10:17 ` Paul Eggleton
2012-09-06 15:23 ` Elvis Dowson
@ 2012-09-06 21:46 ` Dmitry Eremin-Solenikov
1 sibling, 0 replies; 8+ messages in thread
From: Dmitry Eremin-Solenikov @ 2012-09-06 21:46 UTC (permalink / raw)
To: Paul Eggleton; +Cc: yocto
On Thu, Sep 6, 2012 at 2:17 PM, Paul Eggleton
<paul.eggleton@linux.intel.com> wrote:
> On Thursday 16 August 2012 00:30:15 Elvis Dowson wrote:
>
> Otherwise, Dmitry, any suggestions? I'm assuming you made use of icecc.bbclass
> since you made some changes to it a while ago...
I'm sorry, I had not tried icecc.bbclass lately. I should give it a
try. Probably either
on Weekend, or next week.
>
> Cheers,
> Paul
>
> --
>
> Paul Eggleton
> Intel Open Source Technology Centre
--
With best wishes
Dmitry
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2012-09-06 21:46 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-08-15 20:30 Procedure to setup icecc for performing a distributed build Elvis Dowson
2012-09-06 10:17 ` Paul Eggleton
2012-09-06 15:23 ` Elvis Dowson
2012-09-06 15:29 ` Paul Eggleton
2012-09-06 15:33 ` Gary Thomas
2012-09-06 15:38 ` Paul Eggleton
[not found] ` <5048C1C2.1010903@mlbassoc.com>
2012-09-06 17:52 ` Elvis Dowson
2012-09-06 21:46 ` Dmitry Eremin-Solenikov
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.