Re: [OE-core] Add package managers as a plugin

["Fredrik Gustafsson" <fredrik.gustafsson@axis.com>]

[-- Attachment #1: Type: text/plain, Size: 4741 bytes --]

Hi Alex,

now I see what you mean. I got a script from Alex Stewart, here's the results for core-image-sato-sdk-ptest:

out.apk.1: 1:13.35
out.apk.2: 1:13.51
out.apk.3: 1:13.23
out.apk.4: 1:14.07
out.apk.5: 1:13.00
out.deb.1: 3:49.37
out.deb.2: 3:50.77
out.deb.3: 3:51.39
out.deb.4: 3:53.40
out.deb.5: 3:53.99
out.ipk.1: 2:38.99
out.ipk.2: 2:39.07
out.ipk.3: 2:35.34
out.ipk.4: 2:36.15
out.ipk.5: 2:34.55
out.rpm.1: 1:58.61
out.rpm.2: 1:59.42
out.rpm.3: 1:59.70
out.rpm.4: 1:58.96
out.rpm.5: 1:58.11


BR

Fredrik

________________________________
From: Alexander Kanavin <alex.kanavin@gmail.com>
Sent: Tuesday, June 30, 2020 5:43 PM
To: Fredrik Gustafsson
Cc: OE-core; tools-cfpbuild-internal; Hugo Cedervall
Subject: Re: [OE-core] Add package managers as a plugin

What I meant is that 15s vs 20s on a small image could have been 5 min vs 5min5s on a bigger image, so I wanted to see that the difference does scale up with image size.

Can you to provide sdk-ptest numbers for deb and ipk as well?

Alex

On Tue, 30 Jun 2020 at 17:04, Fredrik Gustafsson <Fredrik.Gustafsson@axis.com<mailto:Fredrik.Gustafsson@axis.com>> wrote:

Hi Alexander,

what are you finding strange with the numbers? Too fast or to big of a difference? Note that it's only the do_rootfs task that i measured.

I've tried the core-image-sato-sdk-ptest as you suggested. I run the following command multiple times:
bitbake core-image-sato-sdk-ptest -c cleansstate &&

bitbake core-image-sato-sdk-ptest -c clean &&
time bitbake core-image-sato-sdk-ptest -f -c do_rootfs

APK
====
0,85s user 0,16s system 0% cpu 1:13,66 total
0,87s user 0,14s system 0% cpu 1:13,28 total
0,89s user 0,11s system 0% cpu 1:13,10 total

RPM
===

0,67s user 0,12s system 0% cpu 2:02,48 total
0,68s user 0,11s system 0% cpu 2:03,39 total
0,73s user 0,09s system 0% cpu 1:59,87 total

5 seconds doesn't sound that much but it's 25% speedup. Since building is mostly cached but generating rootfs isn't this is a huge time consumption for our use case and our image takes around 4 minutes to build. Saving one minutes times the many builds we do is a huge win.

(Excuse my late reply, running core-image-sato-sdk-ptest uncovered a few bugs in the implementation of apk in OE. So thanks for having me testing this image)


/Fredrik


________________________________
From: Alexander Kanavin <alex.kanavin@gmail.com<mailto:alex.kanavin@gmail.com>>
Sent: Thursday, June 25, 2020 12:24 PM
To: Fredrik Gustafsson
Cc: OE-core; tools-cfpbuild-internal; Hugo Cedervall
Subject: Re: [OE-core] Add package managers as a plugin

Apologies, but these metrics do not look particularly convincing - 15s vs 20s? Can you try it with an image several gigabytes in size? Core-image-sato-sdk-ptest is a good option.

Alex

On Thu, 25 Jun 2020 at 12:14, Fredrik Gustafsson <fredrik.gustafsson@axis.com<mailto:fredrik.gustafsson@axis.com>> wrote:
Poky today has three different package managers, the well-known formats deb
and rpm is supported as well as ipkg that is good for embedded devices.

When building and having a good cache hit, a significant amount of time is
spent in the phase of generating a rootfs, which is really about the
performance of the package manager. ipkg is way slower than deb or rpm. To
save build time and also get a package manager that is suitanle for use on
targets where flash memory is a concern, support for apk is suggested.

However, it might or might not be what's wanted for Poky since it increases
the test matrix. Therefore this patch series refactors the package
management code so that it's possible to add more package managers in other
own layer. I will send another patch serie that will add apk.

Perfomance metrics below, (note that this includes build times).

APK
===
bitbake core-image-minimal
15.84s user 2.60s system 0% cpu 1:26:19.21 total
16.01s user 2.58s system 0% cpu 1:26:03.72 total
15.69s user 2.61s system 0% cpu 1:26:45.45 total

bitbake core-image-minimal -c cleansstate
bitbake core-image-minimal -c clean
bitbake core-image-minimal -f -c do_rootfs
0.55s user 0.06s system 4% cpu 14.236 total
0.54s user 0.08s system 4% cpu 15.247 total
0.52s user 0.15s system 4% cpu 15.143 total

RPM
===
bitbake core-image-minimal
18.57s user 3.09s system 0% cpu 1:31:29.09 total
18.58s user 3.08s system 0% cpu 1:30:53.80 total
18.20s user 3.31s system 0% cpu 1:31:06.69 total

bitbake core-image-minimal -c cleansstate
bitbake core-image-minimal -c clean
bitbake core-image-minimal -c do_rootfs -f
0.58s user 0.10s system 3% cpu 19.470 total
0.57s user 0.09s system 3% cpu 19.480 total
0.60s user 0.07s system 3% cpu 20.381 total


[-- Attachment #2: Type: text/html, Size: 9576 bytes --]