Rough timing of rpm vs opkg rootfs builds

Rough timing of rpm vs opkg rootfs builds

[Saul Wold]

Mark, Qing, Dongxiao:

Richard and I were talking the other day and I started a little 
experiment with checking the timing of RPM rootfs build vs OPKG. For the 
minimal and SDK images:

Minimal - OPKG
real	1m39.456s
user	1m18.693s
sys	0m4.188s

SDK - OPKG
real	10m50.784s
user	6m8.059s
sys	0m51.523s

Minimal - RPM
real	4m25.166s
user	6m14.503s
sys	0m27.534s

SDK - RPM
real	24m40.979s
user	7m29.856s
sys	4m25.561s

Clearly there is some work we can do with RPM, which takes more than 
double the time, yes, I know it's doing more work, but maybe there are 
some optimizations that can be done to improve the speed.

For the autobuilder, we build 7 SDK images, which is about 3 hours using 
RPM vs about 75 minutes for OPKG, the Sato and LSB images are not much 
faster so this is another place we should be looking to help our build time.

This is just another place for us to open dialog and figure out what's 
going on.


Sau!

Re: Rough timing of rpm vs opkg rootfs builds

[Richard Purdie]

On Thu, 2010-11-11 at 14:30 -0800, Saul Wold wrote:
> Mark, Qing, Dongxiao:
> 
> Richard and I were talking the other day and I started a little 
> experiment with checking the timing of RPM rootfs build vs OPKG. For the 
> minimal and SDK images:
> 
> Minimal - OPKG
> real	1m39.456s
> user	1m18.693s
> sys	0m4.188s
> 
> SDK - OPKG
> real	10m50.784s
> user	6m8.059s
> sys	0m51.523s
> 
> Minimal - RPM
> real	4m25.166s
> user	6m14.503s
> sys	0m27.534s
> 
> SDK - RPM
> real	24m40.979s
> user	7m29.856s
> sys	4m25.561s
> 
> Clearly there is some work we can do with RPM, which takes more than 
> double the time, yes, I know it's doing more work, but maybe there are 
> some optimizations that can be done to improve the speed.

Actually, I'm not sure RPM should be doing much more work here.

> For the autobuilder, we build 7 SDK images, which is about 3 hours using 
> RPM vs about 75 minutes for OPKG, the Sato and LSB images are not much 
> faster so this is another place we should be looking to help our build time.
> 
> This is just another place for us to open dialog and figure out what's 
> going on.

This is good, thanks for the info.

What would be more interesting again is a breakdown of this to see how
much time is spent indexing the packages and how much is actual rootfs
generation with rpm.

I've worked on optimising this time for opkg before. The trick was to
make the package indexing incremental rather than reparsing every
package, every time which is slow. I therefore taught the package
indexer just to look at the timestamps and size of the packages, if they
change it will reindex them, if they are the same, the previous result
will be used. You need to be careful to notice when packages are removed
or added.

I suspect there is an optimisation that can be added for the rpm
indexing to make this incremental updating possible.

We might also want to also not do the package indexing under pseudo as
another idea to get some speed.

Cheers,

Richard

Rough timing of rpm vs opkg rootfs builds

[R P Herrold]

On Fri, 12 Nov 2010, Richard Purdie wrote:

> I suspect there is an optimisation that can be added for the rpm
> indexing to make this incremental updating possible.

Usually naiive ('Rough') timing tests with rpm do not disable 
un-needed checksum cross checks, key verifications, and such, 
which are senseless in a closed or protected build environment

Publishing the testing harnesses would permit evaluation of 
such -- absent a second person checking the methodology, it is 
probably premature to start kinkering.  Publishing final 
numbers without a reproduceable methodology is just not good 
science

-- Russ herrold

Re: Rough timing of rpm vs opkg rootfs builds

[Richard Purdie]

On Fri, 2010-11-12 at 10:12 -0500, R P Herrold wrote:
> On Fri, 12 Nov 2010, Richard Purdie wrote:
> 
> > I suspect there is an optimisation that can be added for the rpm
> > indexing to make this incremental updating possible.
> 
> Usually naiive ('Rough') timing tests with rpm do not disable 
> un-needed checksum cross checks, key verifications, and such, 
> which are senseless in a closed or protected build environment
> 
> Publishing the testing harnesses would permit evaluation of 
> such -- absent a second person checking the methodology, it is 
> probably premature to start kinkering.  Publishing final 
> numbers without a reproduceable methodology is just not good 
> science

Whilst Saul didn't specifically mention it, these numbers come from a
build system which is designed for reproducible builds. Saul is probably
doing something like enabling the ipk or rpm package backends, then
simply running:

time bitbake poky-image-minimal

then running it again so only the rootfs generation happens. The logs
output to the console will make it clear if that is the case or not.

So the tests can easily be reproduced and I'd welcome anyone else to
look into what the rootfs task is spending its time doing. The ideas you
mention above about checksums and key verifications would be very
interesting to looking into, particularly as we're in a closed
environment.

Cheers,

Richard

Re: Rough timing of rpm vs opkg rootfs builds

[Mark Hatle]

On 11/12/10 9:12 AM, R P Herrold wrote:
> On Fri, 12 Nov 2010, Richard Purdie wrote:
> 
>> I suspect there is an optimisation that can be added for the rpm
>> indexing to make this incremental updating possible.
> 
> Usually naiive ('Rough') timing tests with rpm do not disable 
> un-needed checksum cross checks, key verifications, and such, 
> which are senseless in a closed or protected build environment

RPM resolver db and installation uses minimal options...

Resolver creation:

${RPM} -i --replacepkgs --replacefiles --oldpackage \
 -D "_dbpath $pkgdir/solvedb" --justdb \
 --noaid --nodeps --noorder --noscripts --notriggers --noparentdirs --nolinktos
--stats \
 --ignoresize --nosignature --nodigest \
 -D "_dbi_tags_3 Packages:Name:Basenames:Providename:Nvra" \
 -D "__dbi_cdb create mp_mmapsize=128Mb mp_size=1Mb nofsync" \
 $pkgdir/solvedb/manifest


Note the --stats, if you look at the log file, you'll see the timing stats for RPM.


ROOTFS creation:

(generating the resolver solution):
${RPM} -D "_dbpath ${IMAGE_ROOTFS}/install" -D "`cat
${DEPLOY_DIR_RPM}/solvedb.macro`" \
 -D "__dbi_cdb create mp_mmapsize=128Mb mp_size=1Mb nofsync" \
 -U --justdb --noscripts --notriggers --noparentdirs --nolinktos \
 ${IMAGE_ROOTFS}/install/install.manifest


Actually doing the install:

${RPM} -D "_dbpath ${IMAGE_ROOTFS}/install" -D "`cat
${DEPLOY_DIR_RPM}/solvedb.macro`" \
 -D "__dbi_cdb create mp_mmapsize=128Mb mp_size=1Mb nofsync private" \
 -U --justdb --noscripts --notriggers --noparentdirs --nolinktos \
 $pkg_name >> "${WORKDIR}/temp/log.do_rootfs_attemptonly.${PID}" || true

So we are doing package validation and related work during FS generation, but
NOT suring the resolver setup.  (Note, according to the above we are validating
packages for the resolver solution... which likely is a mistake and causing
extra time.)

I'm hesitant to add the --nosignature --nodigest to the actual install step, but
adding it to the resolver step might be beneficial.

> Publishing the testing harnesses would permit evaluation of 
> such -- absent a second person checking the methodology, it is 
> probably premature to start kinkering.  Publishing final 
> numbers without a reproduceable methodology is just not good 
> science
> 
> -- Russ herrold
> _______________________________________________
> poky mailing list
> poky@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/poky