Packaged Staging - 'Current' Status

Packaged Staging - 'Current' Status

[Richard Purdie]

As has been mentioned, I'm working on integrating packaged staging and
its about time I mentioned the current status. I say 'current' as it
applies to what I have offline, not what's in OE.dev. I aim to commit
some of it tomorrow but I also have some changes which should first be
discussed, more on that below.

Basically the current status is good. I've extended the class to:

* Cover -cross and -native packages apart from a list of core native
dependencies. This only applies if tmpdir didn't change between the
builds but the code takes care of checking that.
* Have a StampUpdate event handler which catches and corrects cases
where a user tries to -c compile -f some package and no workdir exists,
like the handler recently added for rm_work.

A lot of the work is actually needed outside the core class too:

I've fixed some bugs in stage-manager (it mishandled symlinks and didn't
notice files being removed).

Fixed stamp handling in bitbake so the stamps created by packages
staging are noticed. I had to rewrite the new stamp handling code in
bitbake again to get this right :/.

Also, recrdeptask handling in bitbake has some broken edge cases where
dependencies aren't pulled in. Fixing this slows down the runqueue
operations but I can't see any other way and the existing handling is
broken and badly affects packaged staging.

I've also noticed some bugs in things like the perl native package
install vs. stage functions which I'm working on fixes for.

With this code, a build of a standard image with a tmp directory empty
apart from staging packages took 4.5 minutes. That time includes
installing all the staging packages and building all the core native
dependencies. I've also been able to build new packages against a
populated staging directory although the perl native breakage shows up
since intltool-native needs perl working in staging.

What are the issues that need discussion?

First, some background. In order to build staging packages we need to
know whats in staging and from which package. Sounds simple enough until
you try :).

The current approach is to make do_populate_staging single threaded so
only on task can run at once. We take a snapshot of cross/ and staging/,
run do_stage and then take another snapshot. We can then see whats
changed. This has two problems:

1. It assumes everything going into staging goes there in do_stage
2. It makes do_populate_staging single threaded - a bottleneck

At present I plan to ignore the second issue, I have ideas about how we
can overcome that but making it work at all would be good before we
think about optimisations.

The first issue is more problematic, we also touch files in staging from
do_package, specifically the pkgdata and shlibs code. I haven't found
any other cases but I'm still fixing bugs which could be hiding things.
I'm also aware of mono.bbclass having shlibs like code.

Based on these being the main problems to fix, there are a few ways we
could make it work:

1. We add locking around these accesses increasing the bottleneck and
track which files were touched so we can add them to the staging
package. I have the latter bit working, adding the locking is trivial.
Its extremely ugly code though.
2. As per 1 but have the snapshot code ignore the pkgdata and shlibs
directories so locking isn't needed. I really don't fancy the special
cases this entails.
3. We move the pkgdata and shlibs directories to a directory alongside
staging, nothing actually mandates they should be in staging. Thanks to
ongoing efforts to abstract things, this amounts to changing the
definition of PKGDATADIR and SHLIBSDIR but that changes staging ABI. In
this case an automatic conversion is possible and our new ABI checking
code supports inline upgrades though so in theory there would be no
disruption.

I'm currently leaning in favour of option 3...

We also place parts of deploy under control of the package manager. This
means that badly versioned ipk files can be ejected with an uninstall of
the staging package, a problem most developers have probably hit at some
point and been frustrated by. The packages aren't a problem, we can
predict where they are and can package them, the problem is the
unpredictable things that are staged, specifically from custom
do_deploy() tasks.

Why are they a problem? Currently we have things like image generation
of installkits which depends on being able to find deployed data.

So we either make installkit generation optional, or put the output from
do_deploy tasks under control of the stage manager. If we do that, do we
do it with the snapshot approach or do we add a special function call
deploy tasks are mandated to use which makes sure the staging package
gets taken care of too?

Since snapshots require locking and multiple things can write to
deploy/images, I'm probably in favour of modifying do_deploy tasks to
use something like oe_libinstall (oe_deployfile?).

Sorry this is as long, hopefully this shows people things are moving
forward :) Feedback is welcome on the two changes proposed above.

Cheers,

Richard

Re: Packaged Staging - 'Current' Status

[Koen Kooi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Richard Purdie schreef:
| As has been mentioned, I'm working on integrating packaged staging and
| its about time I mentioned the current status. I say 'current' as it
| applies to what I have offline, not what's in OE.dev.

I have a few questions that I can't deduce an answer to from your
description:

~ * How does packaged-staging handle switching ANGSTROM_MODE?

If the packaged-staging-packages (p-s-ps) get seperated by target tuple
it should be ok, and if the p-s-ps dir can be overriden (e.g.
PS_STAGING_DIR = foo/bar/${ANGSTROM_MODE}) we should be safe as well.

~ * do we have 'one' staging dir as we have now, or can each package have
its own staging dir built from its DEPENDS?

I know that this largely depends on how fast/well we debug sysroot
stuff, but I'm curious how this is in your current design.

~ * will p-s.bbclass be package format agnostic, or are we going to
insist on opkg?

I know from experience that making it format agnostic isn't hard, only
tedious, but most of all subject to combinatorial explosion for
unit-testing. It also doesn't really matter, since p-s-ps are orthogonal
to the ones in deploy, but I can image OE-newbies freaking out over this.

regards,

Koen
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFH2aVoMkyGM64RGpERAor2AKC2XM/v+c6hQFsZTZdv5F2dtMRFawCgpgAD
ZT5JM2KOiqUyTLNY3A6n4Io=
=T4cR
-----END PGP SIGNATURE-----

Re: Packaged Staging - 'Current' Status

[Richard Purdie]

On Thu, 2008-03-13 at 23:06 +0100, Koen Kooi wrote:
> Richard Purdie schreef:
> | As has been mentioned, I'm working on integrating packaged staging and
> | its about time I mentioned the current status. I say 'current' as it
> | applies to what I have offline, not what's in OE.dev.
> 
> I have a few questions that I can't deduce an answer to from your
> description:
> 
> ~ * How does packaged-staging handle switching ANGSTROM_MODE?
> 
> If the packaged-staging-packages (p-s-ps) get seperated by target tuple
> it should be ok, and if the p-s-ps dir can be overriden (e.g.
> PS_STAGING_DIR = foo/bar/${ANGSTROM_MODE}) we should be safe as well.

To lift code from my copy of the .bbclass:

PSTAGE_PKGVERSION = "${PV}-${PR}"
PSTAGE_PKGARCH    = "${MULTIMACH_ARCH}"
PSTAGE_EXTRAPATH  = ""
PSTAGE_PKGPATH    = "${DISTRO}${PSTAGE_EXTRAPATH}"
PSTAGE_PKGNAME 	  = "staging-${PN}_${PSTAGE_PKGVERSION}_${PSTAGE_PKGARCH}.ipk"
PSTAGE_PKG        = "${DEPLOY_DIR_PSTAGE}/${PSTAGE_PKGPATH}/${PSTAGE_PKGNAME}"

I'm 100% sure this isn't right yet :). The intent of PSTAGE_EXTRAPATH is
extra things needed can be injected by the distros. -cross and -native
are handled by the class itself appending the tmpdir path into that
variable (with / changed to -).

So yes, overrides are available but if we can make it work out the box
for everyone I will do that.

> ~ * do we have 'one' staging dir as we have now, or can each package have
> its own staging dir built from its DEPENDS?
>
> I know that this largely depends on how fast/well we debug sysroot
> stuff, but I'm curious how this is in your current design.

One staging dir at the moment. Multiple staging directories are a
headache for another time :).

> ~ * will p-s.bbclass be package format agnostic, or are we going to
> insist on opkg?
> 
> I know from experience that making it format agnostic isn't hard, only
> tedious, but most of all subject to combinatorial explosion for
> unit-testing. It also doesn't really matter, since p-s-ps are orthogonal
> to the ones in deploy, but I can image OE-newbies freaking out over this.

Good question. We're actually using so little of the package managers
features I'm nearly tempted to write something custom in python. The
benefit of that would be that no -native packages were needed as special
cases.

The current plan is to use ipkg/opkg and get something working, then
consider optimisations from there. I might find ipkg.py and rip that
apart...

Cheers,

Richard

Re: Packaged Staging - 'Current' Status

[Koen Kooi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Richard Purdie schreef:
| On Thu, 2008-03-13 at 23:06 +0100, Koen Kooi wrote:
|> Richard Purdie schreef:
|> | As has been mentioned, I'm working on integrating packaged staging and
|> | its about time I mentioned the current status. I say 'current' as it
|> | applies to what I have offline, not what's in OE.dev.
|>
|> I have a few questions that I can't deduce an answer to from your
|> description:
|>
|> ~ * How does packaged-staging handle switching ANGSTROM_MODE?
|>
|> If the packaged-staging-packages (p-s-ps) get seperated by target tuple
|> it should be ok, and if the p-s-ps dir can be overriden (e.g.
|> PS_STAGING_DIR = foo/bar/${ANGSTROM_MODE}) we should be safe as well.
|
| To lift code from my copy of the .bbclass:
|
| PSTAGE_PKGVERSION = "${PV}-${PR}"
| PSTAGE_PKGARCH    = "${MULTIMACH_ARCH}"
| PSTAGE_EXTRAPATH  = ""
| PSTAGE_PKGPATH    = "${DISTRO}${PSTAGE_EXTRAPATH}"
| PSTAGE_PKGNAME 	  =
"staging-${PN}_${PSTAGE_PKGVERSION}_${PSTAGE_PKGARCH}.ipk"
| PSTAGE_PKG        =
"${DEPLOY_DIR_PSTAGE}/${PSTAGE_PKGPATH}/${PSTAGE_PKGNAME}"

I just changed that to:

PSTAGE_EXTRAPATH  ?= ""

regards,

Koen
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFH2mtWMkyGM64RGpERAuJ3AJ48bG9A7NXofb0Vz/+i2qVtaMmlaACfZErK
/XMX+HXRlZqt9O1pvNUjUfc=
=2mSP
-----END PGP SIGNATURE-----

Re: Packaged Staging - 'Current' Status

[Koen Kooi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Richard Purdie schreef:
| As has been mentioned, I'm working on integrating packaged staging and
| its about time I mentioned the current status. I say 'current' as it
| applies to what I have offline, not what's in OE.dev. I aim to commit
| some of it tomorrow but I also have some changes which should first be
| discussed, more on that below.
|
| Basically the current status is good.

There is a small problem with generating the ipkg arch file properly,
Richard and I are thinking about a solution.

regards,

Koen
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFH2rBMMkyGM64RGpERAqwqAJ4iGz6sfKZ0TU4Kon1ryeUP96k/3gCeMpqg
uxAhUi26/OsGBQYWGPgXJjE=
=pzoj
-----END PGP SIGNATURE-----

Re: Packaged Staging - 'Current' Status

[Koen Kooi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Koen Kooi schreef:
| Richard Purdie schreef:
| | As has been mentioned, I'm working on integrating packaged staging and
| | its about time I mentioned the current status. I say 'current' as it
| | applies to what I have offline, not what's in OE.dev. I aim to commit
| | some of it tomorrow but I also have some changes which should first be
| | discussed, more on that below.
| |
| | Basically the current status is good.
|
| There is a small problem with generating the ipkg arch file properly,
| Richard and I are thinking about a solution.

A workaround is to change the staging_helper() method to something like
this:

staging_helper () {
~        # Assemble appropriate ipkg.conf
~        conffile=${PSTAGE_MACHCONFIG}
~        mkdir -p ${DEPLOY_DIR_PSTAGE}/pstaging_lists
~        if [ ! -e $conffile ]; then
	  rm $conffile
	fi
~        ipkgarchs="${BUILD_ARCH} all any noarch ${TARGET_ARCH}
${PACKAGE_ARCHS} ${PACKAGE_EXTRA_ARCHS} ${MACHINE}"
~        priority=1
~        for arch in $ipkgarchs; do
~                echo "arch $arch $priority" >> $conffile
~                priority=$(expr $priority + 5)
~        done
}

Attentive readers will notice that ${TARGET_ARCH} doesn't always hold
the value we want.

regards,

Koen
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFH2vHFMkyGM64RGpERAu0KAKCyiD1o1JGKipSrvY7Dt6BPYqovxACgkxei
KbZn4Mb+OZXJAP84Ra6Uo+s=
=UR5q
-----END PGP SIGNATURE-----