From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp2.axis.com (smtp2.axis.com [195.60.68.18]) by mx.groups.io with SMTP id smtpd.web11.4444.1593542801581953855 for ; Tue, 30 Jun 2020 11:46:42 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@axis.com header.s=axis-central1 header.b=Vd3rzbgk; spf=pass (domain: axis.com, ip: 195.60.68.18, mailfrom: fredrik.gustafsson@axis.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=axis.com; l=16614; q=dns/txt; s=axis-central1; t=1593542802; x=1625078802; h=from:to:cc:subject:date:message-id:references: in-reply-to:mime-version; bh=J8UlTr1gPOc10nwrL+lCWNTzq06VLXh63rRTEXAUFgM=; b=Vd3rzbgkNSE/VH4wJ6J9CWV1ogh57X06tATQlzmu0gh+U+zB9i2DeZFt mMwuXsCmSqCaXbhIxlo00Rp8G7TVD7wt/006RhYLninuZZMCQ1S0soTJC Ty0le51pnKbMT4p7nY4JS1+f6658Xj64IvG2Loy6A69vFLJ0HWwcCJ8mJ SDaCR9bGwvQGqApn4UjclsMVwRwddJ4rmcyP7ud1sBQTJX5YzUVjKAY8l XxhM6bSENxdOABLqyPbbSMWVHJedFHPX2xmh2FQRuhcqR6XXE4lm4dQGz /uNUwXVmU+a2D0cOd5pVHCvalKr1PF95l6AdV4a0Wflf1MvcHU2Fd97mZ g==; IronPort-SDR: 7aYV6SXEXuVgS2B0C2soZqTxNASTudBWyZeeyqqbIi0VCSC7t8vsonWRHQqmRFgFzD4GR03cS7 r/Oaq94nvIJPqhQz4aoueLkmMRSkiXRMUUhNG6H0m2TUfcXNev+j19ANnBICb4zSnwyD0n8w7E 7J/VHZBGma8O3lJo5q+n0P/zSi4bXFbAAnV/+txbSSshx0rTjAAO+Qk5N7btiyQTiTZhZV+9Qq DShrZsnTT+JrOnWm4tgzC40Wp/rgHZdPUkCFRBbfK0XHYylxxhyEuvMYFskFiOMlWolFEaQsmB SNw= X-IronPort-AV: E=Sophos;i="5.75,298,1589234400"; d="scan'208,217";a="10060545" From: "Fredrik Gustafsson" To: Alexander Kanavin CC: OE-core , tools-cfpbuild-internal Subject: Re: [OE-core] Add package managers as a plugin Thread-Topic: [OE-core] Add package managers as a plugin Thread-Index: AQHWStlhmROMe50x8EKtBnMpvXTtHajo/muAgAAzAiCACAHYAIAAU5Zs Date: Tue, 30 Jun 2020 18:46:37 +0000 Message-ID: <1593542797508.95041@axis.com> References: <20200625101413.2783-1-fredrigu@axis.com> <1593529480451.52800@axis.com>, In-Reply-To: Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [10.20.13.13] MIME-Version: 1.0 Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_159354279750895041axiscom_" --_000_159354279750895041axiscom_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hi Alex, now I see what you mean. I got a script from Alex Stewart, here's the resu= lts for core-image-sato-sdk-ptest: out.apk.1: 1:13.35 out.apk.2: 1:13.51 out.apk.3: 1:13.23 out.apk.4: 1:14.07 out.apk.5: 1:13.00 out.deb.1: 3:49.37 out.deb.2: 3:50.77 out.deb.3: 3:51.39 out.deb.4: 3:53.40 out.deb.5: 3:53.99 out.ipk.1: 2:38.99 out.ipk.2: 2:39.07 out.ipk.3: 2:35.34 out.ipk.4: 2:36.15 out.ipk.5: 2:34.55 out.rpm.1: 1:58.61 out.rpm.2: 1:59.42 out.rpm.3: 1:59.70 out.rpm.4: 1:58.96 out.rpm.5: 1:58.11 BR Fredrik ________________________________ From: Alexander Kanavin Sent: Tuesday, June 30, 2020 5:43 PM To: Fredrik Gustafsson Cc: OE-core; tools-cfpbuild-internal; Hugo Cedervall Subject: Re: [OE-core] Add package managers as a plugin What I meant is that 15s vs 20s on a small image could have been 5 min vs = 5min5s on a bigger image, so I wanted to see that the difference does scale= up with image size. Can you to provide sdk-ptest numbers for deb and ipk as well? Alex On Tue, 30 Jun 2020 at 17:04, Fredrik Gustafsson > wrote: Hi Alexander, what are you finding strange with the numbers? Too fast or to big of a dif= ference? Note that it's only the do_rootfs task that i measured. I've tried the core-image-sato-sdk-ptest as you suggested. I run the follo= wing command multiple times: bitbake core-image-sato-sdk-ptest -c cleansstate && bitbake core-image-sato-sdk-ptest -c clean && time bitbake core-image-sato-sdk-ptest -f -c do_rootfs APK =3D=3D=3D=3D 0,85s user 0,16s system 0% cpu 1:13,66 total 0,87s user 0,14s system 0% cpu 1:13,28 total 0,89s user 0,11s system 0% cpu 1:13,10 total RPM =3D=3D=3D 0,67s user 0,12s system 0% cpu 2:02,48 total 0,68s user 0,11s system 0% cpu 2:03,39 total 0,73s user 0,09s system 0% cpu 1:59,87 total 5 seconds doesn't sound that much but it's 25% speedup. Since building is = mostly cached but generating rootfs isn't this is a huge time consumption f= or our use case and our image takes around 4 minutes to build. Saving one m= inutes times the many builds we do is a huge win. (Excuse my late reply, running core-image-sato-sdk-ptest uncovered a few b= ugs in the implementation of apk in OE. So thanks for having me testing thi= s image) /Fredrik ________________________________ From: Alexander Kanavin > Sent: Thursday, June 25, 2020 12:24 PM To: Fredrik Gustafsson Cc: OE-core; tools-cfpbuild-internal; Hugo Cedervall Subject: Re: [OE-core] Add package managers as a plugin Apologies, but these metrics do not look particularly convincing - 15s vs = 20s? Can you try it with an image several gigabytes in size? Core-image-sat= o-sdk-ptest is a good option. Alex On Thu, 25 Jun 2020 at 12:14, Fredrik Gustafsson > wrote: Poky today has three different package managers, the well-known formats de= b and rpm is supported as well as ipkg that is good for embedded devices. When building and having a good cache hit, a significant amount of time is spent in the phase of generating a rootfs, which is really about the performance of the package manager. ipkg is way slower than deb or rpm. To save build time and also get a package manager that is suitanle for use on targets where flash memory is a concern, support for apk is suggested. However, it might or might not be what's wanted for Poky since it increase= s the test matrix. Therefore this patch series refactors the package management code so that it's possible to add more package managers in othe= r own layer. I will send another patch serie that will add apk. Perfomance metrics below, (note that this includes build times). APK =3D=3D=3D bitbake core-image-minimal 15.84s user 2.60s system 0% cpu 1:26:19.21 total 16.01s user 2.58s system 0% cpu 1:26:03.72 total 15.69s user 2.61s system 0% cpu 1:26:45.45 total bitbake core-image-minimal -c cleansstate bitbake core-image-minimal -c clean bitbake core-image-minimal -f -c do_rootfs 0.55s user 0.06s system 4% cpu 14.236 total 0.54s user 0.08s system 4% cpu 15.247 total 0.52s user 0.15s system 4% cpu 15.143 total RPM =3D=3D=3D bitbake core-image-minimal 18.57s user 3.09s system 0% cpu 1:31:29.09 total 18.58s user 3.08s system 0% cpu 1:30:53.80 total 18.20s user 3.31s system 0% cpu 1:31:06.69 total bitbake core-image-minimal -c cleansstate bitbake core-image-minimal -c clean bitbake core-image-minimal -c do_rootfs -f 0.58s user 0.10s system 3% cpu 19.470 total 0.57s user 0.09s system 3% cpu 19.480 total 0.60s user 0.07s system 3% cpu 20.381 total --_000_159354279750895041axiscom_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable

Hi Alex,

now I see what you mean. I got a script from Alex Stewart, here's the r= esults for core-image-sato-sdk-ptest:

out.apk.1: 1:13.35
out.apk.2: 1:13.51
out.apk.3: 1:13.23
out.apk.4: 1:14.07
out.apk.5: 1:13.00
out.deb.1: 3:49.37
out.deb.2: 3:50.77
out.deb.3: 3:51.39
out.deb.4: 3:53.40
out.deb.5: 3:53.99
out.ipk.1: 2:38.99
out.ipk.2: 2:39.07
out.ipk.3: 2:35.34
out.ipk.4: 2:36.15
out.ipk.5: 2:34.55
out.rpm.1: 1:58.61
out.rpm.2: 1:59.42
out.rpm.3: 1:59.70
out.rpm.4: 1:58.96
out.rpm.5: 1:58.11


BR

Fredrik

From: Alexander Kanavin = <alex.kanavin@gmail.com>
Sent: Tuesday, June 30, 2020 5:43 PM
To: Fredrik Gustafsson
Cc: OE-core; tools-cfpbuild-internal; Hugo Cedervall
Subject: Re: [OE-core] Add package managers as a plugin
 
What I meant is that 15s vs 20s on a small image could have been 5 mi= n vs 5min5s on a bigger image, so I wanted to see that the difference does = scale up with image size.

Can you to provide sdk-ptest numbers for deb and ipk as well?

Alex

On Tue, 30 Jun 2020 at 17:04, Fredri= k Gustafsson <Fredrik.Gus= tafsson@axis.com> wrote:

Hi Alexander,

what are you finding strange with the numbers? Too fast or to big of a = difference? Note that it's only the do_rootfs task that i measured.

I've tried the core-image-sato-sdk-ptest as you suggested. I run the follo= wing command multiple times:
bitbake core-i= mage-sato-sdk-ptest -c cleansstate &&

bitbake core-image-sato-sdk-ptest -c clean &&
time bitbake c= ore-image-sato-sdk-ptest -f -c do_rootfs


APK
=3D=3D=3D=3D
0,85s user 0,1= 6s system 0% cpu 1:13,66 total
0,87s user 0,1= 4s system 0% cpu 1:13,28 total
0,89s user 0,1= 1s system 0% cpu 1:13,10 total

RPM
=3D=3D=3D

0,67s user = 0,12s system 0% cpu 2:02,48 total
0,68s user 0,1= 1s system 0% cpu 2:03,39 total
0,73s user 0,0= 9s system 0% cpu 1:59,87 total

5 seconds doesn't sound that much but it's 25% speedup. Since building is = mostly cached but generating rootfs isn't this is a huge time consumption f= or our use case and our image takes around 4 minutes to build. Saving one m= inutes times the many builds we do is a huge win.

(Excuse my late reply, running core-image-sato-sdk-ptest uncovered a few b= ugs in the implementation of apk in OE. So thanks for having me testing thi= s image)


/Fredrik


Fro= m: Alexander Kanavin <alex.kanavin@gmail.com>
Sent: Thursday, June 25, 2020 12:24 PM
To: Fredrik Gustafsson
Cc: OE-core; tools-cfpbuild-internal; Hugo Cedervall
Subject: Re: [OE-core] Add package managers as a plugin
 
Apologies, but these metrics do not look particularly convincing - 15= s vs 20s? Can you try it with an image several gigabytes in size? Core-imag= e-sato-sdk-ptest is a good option.

Alex

On Thu, 25 Jun 2020 at 12:14, Fredri= k Gustafsson <fredrik.gustafsson@axis.com> wrote:
Poky today has three different package managers, the well-known formats de= b
and rpm is supported as well as ipkg that is good for embedded devices.
When building and having a good cache hit, a significant amount of time is=
spent in the phase of generating a rootfs, which is really about the
performance of the package manager. ipkg is way slower than deb or rpm. To=
save build time and also get a package manager that is suitanle for use on=
targets where flash memory is a concern, support for apk is suggested.

However, it might or might not be what's wanted for Poky since it increase= s
the test matrix. Therefore this patch series refactors the package
management code so that it's possible to add more package managers in othe= r
own layer. I will send another patch serie that will add apk.

Perfomance metrics below, (note that this includes build times).

APK
=3D=3D=3D
bitbake core-image-minimal
15.84s user 2.60s system 0% cpu 1:26:19.21 total
16.01s user 2.58s system 0% cpu 1:26:03.72 total
15.69s user 2.61s system 0% cpu 1:26:45.45 total

bitbake core-image-minimal -c cleansstate
bitbake core-image-minimal -c clean
bitbake core-image-minimal -f -c do_rootfs
0.55s user 0.06s system 4% cpu 14.236 total
0.54s user 0.08s system 4% cpu 15.247 total
0.52s user 0.15s system 4% cpu 15.143 total

RPM
=3D=3D=3D
bitbake core-image-minimal
18.57s user 3.09s system 0% cpu 1:31:29.09 total
18.58s user 3.08s system 0% cpu 1:30:53.80 total
18.20s user 3.31s system 0% cpu 1:31:06.69 total

bitbake core-image-minimal -c cleansstate
bitbake core-image-minimal -c clean
bitbake core-image-minimal -c do_rootfs -f
0.58s user 0.10s system 3% cpu 19.470 total
0.57s user 0.09s system 3% cpu 19.480 total
0.60s user 0.07s system 3% cpu 20.381 total

--_000_159354279750895041axiscom_--