linux-yocto task performance numbers

linux-yocto task performance numbers

[Richard Purdie]

As people know, I've been looking at performance a little, one of the
benchmarks is how long the kernel takes to build. I dumped out the task
performance data from buildstats for a linux-yocto build (nothing else
running): 

do_fetch:                 Elapsed time: 0.04 seconds 
do_unpack:                Elapsed time: 1.61 seconds 
do_kernel_checkout:       Elapsed time: 4.39 seconds 
do_validate_branches:     Elapsed time: 0.47 seconds 
do_patch:                 Elapsed time: 57.77 seconds 
do_kernel_configme:       Elapsed time: 44.92 seconds 
do_kernel_configcheck:    Elapsed time: 8.94 seconds 
do_configure:             Elapsed time: 0.60 seconds 
do_compile:               Elapsed time: 72.95 seconds 
do_compile_kernelmodules: Elapsed time: 34.84 seconds 
do_populate_lic:          Elapsed time: 0.14 seconds 
do_strip:                 Elapsed time: 0.03 seconds 
do_uboot_mkimage:         Elapsed time: 0.03 seconds 
do_install:               Elapsed time: 1.87 seconds 
do_populate_sysroot:      Elapsed time: 0.16 seconds 
do_shared_workdir:        Elapsed time: 0.05 seconds 
do_sizecheck:             Elapsed time: 0.03 seconds 
do_bundle_initramfs:      Elapsed time: 0.03 seconds 
do_kernel_link_vmlinux:   Elapsed time: 0.03 seconds 
do_deploy:                Elapsed time: 13.17 seconds 
do_package:               Elapsed time: 31.54 seconds 
do_packagedata:           Elapsed time: 0.59 seconds 
do_package_qa:            Elapsed time: 5.30 seconds 
do_package_write_ipk:     Elapsed time: 83.24 seconds 
do_package_write_rpm:     Elapsed time: 44.58 seconds 

The "core" was getting blamed for a lot of the build time. As can be
seen, the "core" isn't taking that much time now, apart from the fact
that ipk packaging seems to be taking twice the time of rpm which needs
looking into.

Some tasks like the compile tasks are understandable and likely
minimised by upstream work already.

The other tasks which as consuming a disproportionate amount of time are
kernel_configme and patch, I believe we need to look into those a little
further too. To put it into context, should the kernel compile be at the
same order of magnitude as the patch and configure?

Cheers,

Richard

Re: linux-yocto task performance numbers

[Bruce Ashfield]

On Sun, Feb 22, 2015 at 8:04 AM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> As people know, I've been looking at performance a little, one of the
> benchmarks is how long the kernel takes to build. I dumped out the task
> performance data from buildstats for a linux-yocto build (nothing else
> running):
>
> do_fetch:                 Elapsed time: 0.04 seconds
> do_unpack:                Elapsed time: 1.61 seconds
> do_kernel_checkout:       Elapsed time: 4.39 seconds
> do_validate_branches:     Elapsed time: 0.47 seconds
> do_patch:                 Elapsed time: 57.77 seconds
> do_kernel_configme:       Elapsed time: 44.92 seconds
> do_kernel_configcheck:    Elapsed time: 8.94 seconds
> do_configure:             Elapsed time: 0.60 seconds
> do_compile:               Elapsed time: 72.95 seconds
> do_compile_kernelmodules: Elapsed time: 34.84 seconds
> do_populate_lic:          Elapsed time: 0.14 seconds
> do_strip:                 Elapsed time: 0.03 seconds
> do_uboot_mkimage:         Elapsed time: 0.03 seconds
> do_install:               Elapsed time: 1.87 seconds
> do_populate_sysroot:      Elapsed time: 0.16 seconds
> do_shared_workdir:        Elapsed time: 0.05 seconds
> do_sizecheck:             Elapsed time: 0.03 seconds
> do_bundle_initramfs:      Elapsed time: 0.03 seconds
> do_kernel_link_vmlinux:   Elapsed time: 0.03 seconds
> do_deploy:                Elapsed time: 13.17 seconds
> do_package:               Elapsed time: 31.54 seconds
> do_packagedata:           Elapsed time: 0.59 seconds
> do_package_qa:            Elapsed time: 5.30 seconds
> do_package_write_ipk:     Elapsed time: 83.24 seconds
> do_package_write_rpm:     Elapsed time: 44.58 seconds
>
> The "core" was getting blamed for a lot of the build time. As can be
> seen, the "core" isn't taking that much time now, apart from the fact
> that ipk packaging seems to be taking twice the time of rpm which needs
> looking into.
>
> Some tasks like the compile tasks are understandable and likely
> minimised by upstream work already.
>
> The other tasks which as consuming a disproportionate amount of time are
> kernel_configme and patch, I believe we need to look into those a little
> further too. To put it into context, should the kernel compile be at the
> same order of magnitude as the patch and configure?

I have some more changes that I didn't get done in M3, since finalizing
the 3.19 kernel took all my cycles.

But in general validation takes time, those pre-cursor steps are what allow
the kernel configuration audit to be performed, and also for the patches to
be pushed with proper attribution (and to confirm that it has been done).

But again, I do have some more changes that bring the time down, but
there's a floor to performance without breaking use cases.

Bruce

>
> Cheers,
>
> Richard
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core



-- 
"Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end"

Re: linux-yocto task performance numbers

[Bruce Ashfield]

On 2015-02-22 8:04 AM, Richard Purdie wrote:
> As people know, I've been looking at performance a little, one of the
> benchmarks is how long the kernel takes to build. I dumped out the task
> performance data from buildstats for a linux-yocto build (nothing else
> running):
>
> do_fetch:                 Elapsed time: 0.04 seconds
> do_unpack:                Elapsed time: 1.61 seconds
> do_kernel_checkout:       Elapsed time: 4.39 seconds
> do_validate_branches:     Elapsed time: 0.47 seconds
> do_patch:                 Elapsed time: 57.77 seconds
> do_kernel_configme:       Elapsed time: 44.92 seconds
> do_kernel_configcheck:    Elapsed time: 8.94 seconds
> do_configure:             Elapsed time: 0.60 seconds
> do_compile:               Elapsed time: 72.95 seconds
> do_compile_kernelmodules: Elapsed time: 34.84 seconds
> do_populate_lic:          Elapsed time: 0.14 seconds
> do_strip:                 Elapsed time: 0.03 seconds
> do_uboot_mkimage:         Elapsed time: 0.03 seconds
> do_install:               Elapsed time: 1.87 seconds
> do_populate_sysroot:      Elapsed time: 0.16 seconds
> do_shared_workdir:        Elapsed time: 0.05 seconds
> do_sizecheck:             Elapsed time: 0.03 seconds
> do_bundle_initramfs:      Elapsed time: 0.03 seconds
> do_kernel_link_vmlinux:   Elapsed time: 0.03 seconds
> do_deploy:                Elapsed time: 13.17 seconds
> do_package:               Elapsed time: 31.54 seconds
> do_packagedata:           Elapsed time: 0.59 seconds
> do_package_qa:            Elapsed time: 5.30 seconds
> do_package_write_ipk:     Elapsed time: 83.24 seconds
> do_package_write_rpm:     Elapsed time: 44.58 seconds
>
> The "core" was getting blamed for a lot of the build time. As can be
> seen, the "core" isn't taking that much time now, apart from the fact
> that ipk packaging seems to be taking twice the time of rpm which needs
> looking into.
>
> Some tasks like the compile tasks are understandable and likely
> minimised by upstream work already.
>
> The other tasks which as consuming a disproportionate amount of time are
> kernel_configme and patch, I believe we need to look into those a little
> further too. To put it into context, should the kernel compile be at the
> same order of magnitude as the patch and configure?

Was this the 3.19 kernel you used for these stats ? I just ran a quick
test (i.e. not with buildstats) with my latest kernel tree (on my
crappy I/O laptop) and get the following:

% time bitbake -f -c kernel_configme linux-yocto

real    0m7.378s
user    0m4.356s
sys     0m5.553s


So I'm showing 7 seconds, versus 45 in your run .. and I'm showing:

real    0m26.371s
user    0m12.561s
sys     0m19.534s

For do_patch (and that is before the already in progress changes that
I mentioned earlier).

Just with those differences alone, sounds like we should log this in
bugzilla and put the details of the configuration and build machines in
there .. since I'll need to isolate what is causing those extra long
runs, and ensure that everything is pushed to the various trees.

Cheers,

Bruce


>
> Cheers,
>
> Richard
>

Re: linux-yocto task performance numbers

[Richard Purdie]

On Mon, 2015-02-23 at 00:13 -0500, Bruce Ashfield wrote:
> Was this the 3.19 kernel you used for these stats ? I just ran a quick
> test (i.e. not with buildstats) with my latest kernel tree (on my
> crappy I/O laptop) and get the following:
> 
> % time bitbake -f -c kernel_configme linux-yocto
> 
> real    0m7.378s
> user    0m4.356s
> sys     0m5.553s
> 
> 
> So I'm showing 7 seconds, versus 45 in your run .. and I'm showing:
> 
> real    0m26.371s
> user    0m12.561s
> sys     0m19.534s
> 
> For do_patch (and that is before the already in progress changes that
> I mentioned earlier).
> 
> Just with those differences alone, sounds like we should log this in
> bugzilla and put the details of the configuration and build machines in
> there .. since I'll need to isolate what is causing those extra long
> runs, and ensure that everything is pushed to the various trees.

I think I messed up and ran against a slightly different version. The
numbers confirmed with 3.19 are:

do_bundle_initramfs: Elapsed time: 0.03 seconds 
do_compile: Elapsed time: 78.07 seconds 
do_compile_kernelmodules: Elapsed time: 35.23 seconds 
do_configure: Elapsed time: 0.60 seconds 
do_deploy: Elapsed time: 14.39 seconds 
do_fetch: Elapsed time: 0.04 seconds 
do_install: Elapsed time: 2.02 seconds 
do_kernel_checkout: Elapsed time: 10.67 seconds 
do_kernel_configcheck: Elapsed time: 8.03 seconds 
do_kernel_configme: Elapsed time: 70.41 seconds 
do_kernel_link_vmlinux: Elapsed time: 0.03 seconds 
do_package: Elapsed time: 33.48 seconds 
do_packagedata: Elapsed time: 2.30 seconds 
do_package_qa: Elapsed time: 4.94 seconds 
do_package_write_ipk: Elapsed time: 68.72 seconds 
do_package_write_rpm: Elapsed time: 44.33 seconds 
do_patch: Elapsed time: 56.43 seconds 
do_populate_lic: Elapsed time: 0.14 seconds 
do_populate_sysroot: Elapsed time: 0.16 seconds 
do_shared_workdir: Elapsed time: 0.04 seconds 
do_sizecheck: Elapsed time: 0.03 seconds 
do_strip: Elapsed time: 0.03 seconds 
do_uboot_mkimage: Elapsed time: 0.02 seconds 
do_unpack: Elapsed time: 5.06 seconds 
do_validate_branches: Elapsed time: 0.69 seconds 

To get the numbers:

bitbake linux-yocto -c cleansstate
bitbake linux-yocto
$ (cd tmp/buildstats/linux-yocto-qemux86-64/201502230954/linux-yocto-3.19+gitAUTOINC+8897ef68b3_43b9eced9b-r0/; grep El * | sed s/linux-yocto-3.19+gitAUTOINC+8897ef68b3_43b9eced9b-r0// | sed s/^.*:://)

So configcheck is better but the other times still seem high.

Cheers,

Richard

Re: linux-yocto task performance numbers

[Richard Purdie]

Also, for reference/interest, the recent kernel populate_sysroot changes
took the kernel build time from over 11 mins to 8 mins. With the move to
3.19:

fedora19,master:5db6a27f4847e51c6434a968f81d3ceffcc54952 7:58.74
fedora19,master:5db6a27f4847e51c6434a968f81d3ceffcc54952 7:57.59
fedora19,master:5db6a27f4847e51c6434a968f81d3ceffcc54952 7:59.18
fedora19,master:5db6a27f4847e51c6434a968f81d3ceffcc54952 8:00.81
fedora19,master:5db6a27f4847e51c6434a968f81d3ceffcc54952 8:01.44
fedora19,master:5db6a27f4847e51c6434a968f81d3ceffcc54952 7:59.12
fedora19,master:172606338757d0b788e3382bf413102647307e1d 8:29.71
fedora19,master:172606338757d0b788e3382bf413102647307e1d 8:37.66
fedora19,master:172606338757d0b788e3382bf413102647307e1d 8:36.41
fedora19,master:172606338757d0b788e3382bf413102647307e1d 8:34.24
fedora19,master:172606338757d0b788e3382bf413102647307e1d 8:36.19

You can see 3.19 takes around 30-35s longer to build. These numbers are
extracted from our benchmark machine which monitors things like this.

Cheers,

Richard

Re: linux-yocto task performance numbers

[Bruce Ashfield]

On 2015-02-23 5:08 AM, Richard Purdie wrote:
> On Mon, 2015-02-23 at 00:13 -0500, Bruce Ashfield wrote:
>> Was this the 3.19 kernel you used for these stats ? I just ran a quick
>> test (i.e. not with buildstats) with my latest kernel tree (on my
>> crappy I/O laptop) and get the following:
>>
>> % time bitbake -f -c kernel_configme linux-yocto
>>
>> real    0m7.378s
>> user    0m4.356s
>> sys     0m5.553s
>>
>>
>> So I'm showing 7 seconds, versus 45 in your run .. and I'm showing:
>>
>> real    0m26.371s
>> user    0m12.561s
>> sys     0m19.534s
>>
>> For do_patch (and that is before the already in progress changes that
>> I mentioned earlier).
>>
>> Just with those differences alone, sounds like we should log this in
>> bugzilla and put the details of the configuration and build machines in
>> there .. since I'll need to isolate what is causing those extra long
>> runs, and ensure that everything is pushed to the various trees.
>
> I think I messed up and ran against a slightly different version. The
> numbers confirmed with 3.19 are:
>
> do_bundle_initramfs: Elapsed time: 0.03 seconds
> do_compile: Elapsed time: 78.07 seconds
> do_compile_kernelmodules: Elapsed time: 35.23 seconds
> do_configure: Elapsed time: 0.60 seconds
> do_deploy: Elapsed time: 14.39 seconds
> do_fetch: Elapsed time: 0.04 seconds
> do_install: Elapsed time: 2.02 seconds
> do_kernel_checkout: Elapsed time: 10.67 seconds
> do_kernel_configcheck: Elapsed time: 8.03 seconds
> do_kernel_configme: Elapsed time: 70.41 seconds
> do_kernel_link_vmlinux: Elapsed time: 0.03 seconds
> do_package: Elapsed time: 33.48 seconds
> do_packagedata: Elapsed time: 2.30 seconds
> do_package_qa: Elapsed time: 4.94 seconds
> do_package_write_ipk: Elapsed time: 68.72 seconds
> do_package_write_rpm: Elapsed time: 44.33 seconds
> do_patch: Elapsed time: 56.43 seconds
> do_populate_lic: Elapsed time: 0.14 seconds
> do_populate_sysroot: Elapsed time: 0.16 seconds
> do_shared_workdir: Elapsed time: 0.04 seconds
> do_sizecheck: Elapsed time: 0.03 seconds
> do_strip: Elapsed time: 0.03 seconds
> do_uboot_mkimage: Elapsed time: 0.02 seconds
> do_unpack: Elapsed time: 5.06 seconds
> do_validate_branches: Elapsed time: 0.69 seconds
>
> To get the numbers:
>
> bitbake linux-yocto -c cleansstate
> bitbake linux-yocto
> $ (cd tmp/buildstats/linux-yocto-qemux86-64/201502230954/linux-yocto-3.19+gitAUTOINC+8897ef68b3_43b9eced9b-r0/; grep El * | sed s/linux-yocto-3.19+gitAUTOINC+8897ef68b3_43b9eced9b-r0// | sed s/^.*:://)
>
> So configcheck is better but the other times still seem high.

Yep. configme should never take that long, so something different
is happening in your qemu build than mine. I'll dig up my other
speedup changes and run them against the same machine build and
see what I get.

I just did a run for qemumips, and patching took 9 seconds :)

Bruce

>
> Cheers,
>
> Richard
>

Re: linux-yocto task performance numbers

[Hart, Darren]

On 2/22/15, 5:04 AM, "Richard Purdie" <richard.purdie@linuxfoundation.org>
wrote:

>As people know, I've been looking at performance a little, one of the
>benchmarks is how long the kernel takes to build. I dumped out the task
>performance data from buildstats for a linux-yocto build (nothing else
>running): 
>
>do_fetch:                 Elapsed time: 0.04 seconds
>do_unpack:                Elapsed time: 1.61 seconds
>do_kernel_checkout:       Elapsed time: 4.39 seconds
>do_validate_branches:     Elapsed time: 0.47 seconds
>do_patch:                 Elapsed time: 57.77 seconds
>do_kernel_configme:       Elapsed time: 44.92 seconds
>do_kernel_configcheck:    Elapsed time: 8.94 seconds
>do_configure:             Elapsed time: 0.60 seconds
>do_compile:               Elapsed time: 72.95 seconds
>do_compile_kernelmodules: Elapsed time: 34.84 seconds
>do_populate_lic:          Elapsed time: 0.14 seconds
>do_strip:                 Elapsed time: 0.03 seconds
>do_uboot_mkimage:         Elapsed time: 0.03 seconds
>do_install:               Elapsed time: 1.87 seconds
>do_populate_sysroot:      Elapsed time: 0.16 seconds
>do_shared_workdir:        Elapsed time: 0.05 seconds
>do_sizecheck:             Elapsed time: 0.03 seconds
>do_bundle_initramfs:      Elapsed time: 0.03 seconds
>do_kernel_link_vmlinux:   Elapsed time: 0.03 seconds
>do_deploy:                Elapsed time: 13.17 seconds
>do_package:               Elapsed time: 31.54 seconds
>do_packagedata:           Elapsed time: 0.59 seconds
>do_package_qa:            Elapsed time: 5.30 seconds
>do_package_write_ipk:     Elapsed time: 83.24 seconds
>do_package_write_rpm:     Elapsed time: 44.58 seconds
>
>The "core" was getting blamed for a lot of the build time. As can be
>seen, the "core" isn't taking that much time now, apart from the fact
>that ipk packaging seems to be taking twice the time of rpm which needs
>looking into.
>
>Some tasks like the compile tasks are understandable and likely
>minimised by upstream work already.
>
>The other tasks which as consuming a disproportionate amount of time are
>kernel_configme and patch, I believe we need to look into those a little
>further too. To put it into context, should the kernel compile be at the
>same order of magnitude as the patch and configure?

Agreed, that was may take away from the above.

-- 
Darren Hart
Intel Open Source Technology Center