Accelerate do_package

Accelerate do_package

[Samuel Stirtzel]

Hi,
after re-building Qt the third time today, I was curious why
do_package takes this long.

From my investigation it looks like the library stripping is the culprit,
it lets my system idle with 5% CPU usage over a long time.

Would it be possible to accelerate do_package with parallelisation?
Likely I could look into this sometime soon, but I'd rather hear your
thoughts on this first.


PS:
Also sstate creation and rootfs packing could be made faster with
pbzip2 [1] / pigz [2] (we have a recipe for pigz), or is anthing
preventing this?

[1] http://compression.ca/pbzip2/
[2] http://zlib.net/pigz/


-- 
Regards
Samuel

Re: Accelerate do_package

[Jack Mitchell]

On 12/10/12 12:43, Samuel Stirtzel wrote:
> Hi,
> after re-building Qt the third time today, I was curious why
> do_package takes this long.
>
>  From my investigation it looks like the library stripping is the culprit,
> it lets my system idle with 5% CPU usage over a long time.
>
> Would it be possible to accelerate do_package with parallelisation?
> Likely I could look into this sometime soon, but I'd rather hear your
> thoughts on this first.
>
>
> PS:
> Also sstate creation and rootfs packing could be made faster with
> pbzip2 [1] / pigz [2] (we have a recipe for pigz), or is anthing
> preventing this?
>
> [1] http://compression.ca/pbzip2/
> [2] http://zlib.net/pigz/
>
>

Samuel,

You being up an interesting point here as I have recently been mulling 
over ways of improving performance. I believe performance improvements 
are one of the targets of the 1.4 yocto/oe-core release so it will be 
properly discussed then.

In the meantime a couple of points I was thinking about related to 
improving performance was by deeming some tasks as "non active" such as 
they didn't take up a thread of your total specified. This could be 
applied to low-cpu intensive tasks such as do_fetch and do_package which 
I believe would greatly speed up the build and make more efficient use 
of the CPU.

As for using pbzip2/pigz I believe there was a discussion on it and 
something was said regarding the compatibility of the created archives. 
For example an archive created by pigz could sometimes not be 
uncompressed by gzip, I am just remembering this from memory though, I'm 
sure a search of the list would be more enlightening.

Regards,

-- 

   Jack Mitchell (jack@embed.me.uk)
   Embedded Systems Engineer
   http://www.embed.me.uk

--

Re: Accelerate do_package

[Richard Purdie]

On Fri, 2012-10-12 at 13:43 +0200, Samuel Stirtzel wrote:
> Hi,
> after re-building Qt the third time today, I was curious why
> do_package takes this long.

Try:

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t1&id=e355769c1dc2dd5c464f2fcfe0991f797613b43e

The trouble is this "breaks" certain rpm per file dependency pieces and
I've not found time to dive into finding an alternative way to do that.

> From my investigation it looks like the library stripping is the culprit,
> it lets my system idle with 5% CPU usage over a long time.

Its the fork/exec overhead from the above that does this.

> Would it be possible to accelerate do_package with parallelisation?
> Likely I could look into this sometime soon, but I'd rather hear your
> thoughts on this first.

Parallelisation is good but there is a better piece of low hanging fruit
here.

> PS:
> Also sstate creation and rootfs packing could be made faster with
> pbzip2 [1] / pigz [2] (we have a recipe for pigz), or is anthing
> preventing this?

The trouble is circular dependencies and being consistent about which
util gets used where without races and dependency loops.

Its also not the big time sync people think it is, I tested a patch
which "offlined" the whole sstate archive creation and it doesn't make
much difference to overall build time.

Cheers,

Richard (not around much over the next few days)