Procedure to setup icecc for performing a distributed build

Procedure to setup icecc for performing a distributed build

[Elvis Dowson]

[-- Attachment #1: Type: text/plain, Size: 2748 bytes --]

Hi,
      I followed these steps to setup and configure icecc and yocto, for performing a distributed build between two Ubuntu 12.04 64-bit machines, with one machine configured as the scheduler and daemon, and the other with only the icecc daemon.

I run a bitbake core-image-minimal, but get no tasks running on the second machine. Processor and CPU utilization is nearly zero, and both machines iceccd.log and icecc_scheduler.log are zero bytes.

Nothing happens when I run the yocto build, and no compilation tasks get distributed to the second machine in the build cluster. I am able to ping them both, however.

Step 01.00: Install required packages for performing a distributed build.

Step 01.01: Install icecc on all the machines.

$ sudo apt-get install icecc

On the scheduler machine, type the following commands:

$ sudo gedit /etc/default/icecc

# Defaults for icecc initscript
# sourced by /etc/init.d/icecc
START_ICECC="true"
START_ICECC_SCHEDULER="true"

Configure the build network name:

$ sudo /etc/init.d/icecc stop

$ sudo gedit /etc/init.d/icecc.conf

#
# Identification for the network the scheduler and daemon run on.
# You can have several distinct icecc networks in the same LAN
# for whatever reason.
#
# ICECC_NETNAME=""
ICECC_NETNAME="YoctoBuildNetwork"

#
# If the daemon can't find the scheduler by broadcast (e.g. because
# of a firewall) you can specify it.
#
# ICECC_SCHEDULER_HOST=""
ICECC_SCHEDULER_HOST="eos"


Start the scheduler with:

$ sudo /etc/init.d/icecc start

Make sure the service starts at boot time:

$ sudo update-rc.d icecc defaults

Step 02.00: Configure yocto for distributed compilation.

Step 02.01: Modify local.conf

#
# Parallelism Options
#
# These two options control how much parallelism BitBake should use. The first 
# option determines how many tasks bitbake should run in parallel:
#
BB_NUMBER_THREADS = "8"
# 
# The second option controls how many processes make should run in parallel when
# running compile tasks:
#
PARALLEL_MAKE = "-j 14"

# Setup yocto to use icecc for distributed compilation.
INHERIT += "icecc"
ICECC_PATH = "/usr/bin/icecc"
ICECC_ENV_EXEC = "/tool/yocto/poky/build/tmp/work/x86_64-linux/icecc-create-env-native-0.1-r2/icecc-create-env"
ICECC_USER_CLASS_BL = " native"

Copy the icecc-create-env script to the build output staging folder, by running the  icecc-create-env-native.bb recipe.

$ cd /tool/yocto/poky;source oe-init-build-env build
$ bitbake icecc-create-env-native

The icecc-create-env script will automatically be copied to the build output staging folder.

Step 02.02: Start a bit bake build.

$ bitbake core-image-minimal


Best regards,

Elvis Dowson

[-- Attachment #2: Type: text/html, Size: 14832 bytes --]

Re: Procedure to setup icecc for performing a distributed build

[Paul Eggleton]

On Thursday 16 August 2012 00:30:15 Elvis Dowson wrote:
> Hi,
>       I followed these steps to setup and configure icecc and yocto, for
> performing a distributed build between two Ubuntu 12.04 64-bit machines,
> with one machine configured as the scheduler and daemon, and the other with
> only the icecc daemon.
> 
> I run a bitbake core-image-minimal, but get no tasks running on the second
> machine. Processor and CPU utilization is nearly zero, and both machines
> iceccd.log and icecc_scheduler.log are zero bytes.
> 
> Nothing happens when I run the yocto build, and no compilation tasks get
> distributed to the second machine in the build cluster. I am able to ping
> them both, however.
> 
> Step 01.00: Install required packages for performing a distributed build.
> 
> Step 01.01: Install icecc on all the machines.
> 
> $ sudo apt-get install icecc
> 
> On the scheduler machine, type the following commands:
> 
> $ sudo gedit /etc/default/icecc
> 
> # Defaults for icecc initscript
> # sourced by /etc/init.d/icecc
> START_ICECC="true"
> START_ICECC_SCHEDULER="true"
> 
> Configure the build network name:
> 
> $ sudo /etc/init.d/icecc stop
> 
> $ sudo gedit /etc/init.d/icecc.conf
> 
> #
> # Identification for the network the scheduler and daemon run on.
> # You can have several distinct icecc networks in the same LAN
> # for whatever reason.
> #
> # ICECC_NETNAME=""
> ICECC_NETNAME="YoctoBuildNetwork"
> 
> #
> # If the daemon can't find the scheduler by broadcast (e.g. because
> # of a firewall) you can specify it.
> #
> # ICECC_SCHEDULER_HOST=""
> ICECC_SCHEDULER_HOST="eos"
> 
> 
> Start the scheduler with:
> 
> $ sudo /etc/init.d/icecc start
> 
> Make sure the service starts at boot time:
> 
> $ sudo update-rc.d icecc defaults
> 
> Step 02.00: Configure yocto for distributed compilation.
> 
> Step 02.01: Modify local.conf
> 
> #
> # Parallelism Options
> #
> # These two options control how much parallelism BitBake should use. The
> first # option determines how many tasks bitbake should run in parallel: #
> BB_NUMBER_THREADS = "8"
> #
> # The second option controls how many processes make should run in parallel
> when # running compile tasks:
> #
> PARALLEL_MAKE = "-j 14"
> 
> # Setup yocto to use icecc for distributed compilation.
> INHERIT += "icecc"
> ICECC_PATH = "/usr/bin/icecc"
> ICECC_ENV_EXEC =
> "/tool/yocto/poky/build/tmp/work/x86_64-linux/icecc-create-env-native-0.1-r
> 2/icecc-create-env" ICECC_USER_CLASS_BL = " native"
> 
> Copy the icecc-create-env script to the build output staging folder, by
> running the  icecc-create-env-native.bb recipe.
> 
> $ cd /tool/yocto/poky;source oe-init-build-env build
> $ bitbake icecc-create-env-native
> 
> The icecc-create-env script will automatically be copied to the build output
> staging folder.
> 
> Step 02.02: Start a bit bake build.
> 
> $ bitbake core-image-minimal
> 

Elvis, did you have any further luck with this?

Otherwise, Dmitry, any suggestions? I'm assuming you made use of icecc.bbclass 
since you made some changes to it a while ago...

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: Procedure to setup icecc for performing a distributed build

[Elvis Dowson]

Hi Paul,

On Sep 6, 2012, at 2:17 PM, Paul Eggleton wrote:

> Elvis, did you have any further luck with this?

Unfortunately no. I've got two machines, both with quad-core intel i7 processors, but I just couldn't
get icecc to work with yocto. I end up regularly perform fresh builds at least 5 to 6 times a day,
and it takes me 2 hours to build core-image-minimal.

> Otherwise, Dmitry, any suggestions? I'm assuming you made use of icecc.bbclass 
> since you made some changes to it a while ago...

Best regards,

Elvis Dowson

Re: Procedure to setup icecc for performing a distributed build

[Paul Eggleton]

On Thursday 06 September 2012 19:23:53 Elvis Dowson wrote:
> Unfortunately no. I've got two machines, both with quad-core intel i7
> processors, but I just couldn't get icecc to work with yocto. I end up
> regularly perform fresh builds at least 5 to 6 times a day, and it takes me
> 2 hours to build core-image-minimal.

At its core, icecc.bbclass just sets things up so that icecc is called instead 
of the normal compiler. It would be worth verifying that that is actually 
taking effect by looking at the commands being used as reported in 
log.do_configure for specific recipes.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: Procedure to setup icecc for performing a distributed build

[Gary Thomas]

On 2012-09-06 09:29, Paul Eggleton wrote:
> On Thursday 06 September 2012 19:23:53 Elvis Dowson wrote:
>> Unfortunately no. I've got two machines, both with quad-core intel i7
>> processors, but I just couldn't get icecc to work with yocto. I end up
>> regularly perform fresh builds at least 5 to 6 times a day, and it takes me
>> 2 hours to build core-image-minimal.
>
> At its core, icecc.bbclass just sets things up so that icecc is called instead
> of the normal compiler. It would be worth verifying that that is actually
> taking effect by looking at the commands being used as reported in
> log.do_configure for specific recipes.

Is this only for the host compiler or the target (cross) compilers as well?
If it's only the host compiler being used for "native" builds, how much is
there to be gained from distributing the work?

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: Procedure to setup icecc for performing a distributed build

[Paul Eggleton]

On Thursday 06 September 2012 09:33:26 Gary Thomas wrote:
> On 2012-09-06 09:29, Paul Eggleton wrote:
> > On Thursday 06 September 2012 19:23:53 Elvis Dowson wrote:
> >> Unfortunately no. I've got two machines, both with quad-core intel i7
> >> processors, but I just couldn't get icecc to work with yocto. I end up
> >> regularly perform fresh builds at least 5 to 6 times a day, and it takes
> >> me
> >> 2 hours to build core-image-minimal.
> > 
> > At its core, icecc.bbclass just sets things up so that icecc is called
> > instead of the normal compiler. It would be worth verifying that that is
> > actually taking effect by looking at the commands being used as reported
> > in log.do_configure for specific recipes.
> 
> Is this only for the host compiler or the target (cross) compilers as well?
> If it's only the host compiler being used for "native" builds, how much is
> there to be gained from distributing the work?

It covers all according to the comments at the top of the file.

I have to be honest and say I've never used icecc.bbclass; I've been told that 
the overhead mostly wipes out any advantage of individual compile jobs being 
farmed out to different machines, but I've never actually verified that first 
hand.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: Procedure to setup icecc for performing a distributed build

[Elvis Dowson]

Hi Gary,

On Sep 6, 2012, at 7:31 PM, Gary Thomas wrote:

> On 2012-09-06 09:23, Elvis Dowson wrote:
>> 
>> Unfortunately no. I've got two machines, both with quad-core intel i7 processors, but I just couldn't
>> get icecc to work with yocto. I end up regularly perform fresh builds at least 5 to 6 times a day,
>> and it takes me 2 hours to build core-image-minimal.
> 
> Just wondering the exact details of your system and why your performance is
> so very different from mine.
> 
> I have a HP box with 8-way Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz & 12GB RAM
> It will build a Yocto image from scratch in around 40 minutes.

I'm running Ubuntu 12.04 LTS virtualized on a 2011 Apple MacBook Pro 15", with
VMware Fusion 5.0.1. The VMware disk image resides on a SSD drive with a 3G
SATA interface.

The specs are 2.3GHz Intel quad-core i7-2820QM with 8MB L3 cache, 
8GB 1333 MHz RAM for the host.

The Ubuntu 12.04 guest is configured with 4GB RAM, 8 virtual CPU cores.
During core-image-minimal builds, the memory usage doesn't exceed 2 to 3GB.

A linux kernel 3.3.0 build for a powepc440 target takes 1 min 40 seconds, as
an additional data point.

Maybe next year, when the newer Mac Pros come out, I'll probably invest in
a 8-core or 10-core xeon configuration. The current Mac Pros are due for a refresh
then, so I'm waiting for the next hardware update.

I'm also considered buying an IBM Blade server, but haven't made up my mind yet.

Best regards,

Elvis Dowson

Re: Procedure to setup icecc for performing a distributed build

[Dmitry Eremin-Solenikov]

On Thu, Sep 6, 2012 at 2:17 PM, Paul Eggleton
<paul.eggleton@linux.intel.com> wrote:
> On Thursday 16 August 2012 00:30:15 Elvis Dowson wrote:
>
> Otherwise, Dmitry, any suggestions? I'm assuming you made use of icecc.bbclass
> since you made some changes to it a while ago...

I'm sorry, I had not tried icecc.bbclass lately. I should give it a
try. Probably either
on Weekend, or next week.

>
> Cheers,
> Paul
>
> --
>
> Paul Eggleton
> Intel Open Source Technology Centre



-- 
With best wishes
Dmitry