about staging

about staging

[Frans Meulenbroeks]

Hi,

Yesterday I ended up in a discussion on staging in #oe (ok, ok, I
mainly caused the discussion :-) )
Also based upon that I looked at various recipes on how people
implement staging.

There does not seem to be really standard way.
Some recipes copy from ${S} which ihmo is not desirable; i think it
should be a file that is exported by the application).
Others use other mechanisms.

For my own recipe I ended up with something like:

do_stage() {
        install -d ${STAGING_INCDIR}
        install -d ${STAGING_INCDIR}/${PN}
        install -m 0644 ${D}/${includedir}/${PN}/*.h ${STAGING_INCDIR}/${PN}
}

as that seems the best. ${D} contans the exported/installed files from
the application. Those files are the ones the application wants to
make externally available and those should end up in staging. Not
something from ${S}.

There is something odd here though. Only a few recipes do this
1453 recipes have a do_stage or do_stage_append function.
646 of these do autotools_stage_all (and perhaps other things).
The rest? all kind of interesting variants. Some understandable, some
outright odd.

That has lead me to the following thoughts:
Why is not all of install/usr/include (including subdirs) copied to
staging inc?
Why is not all of install/usr/lib copiied to staging lib?
(and of course alternatives like /usr/loocal/lib, /usr/local/include, ...)

A solution could be to do this automatically (and have a way with a
var to add additional dirs). This is more or less identical to making
the _dev packages but now copying stuff to staging.
Alternately we could opt for a manual solution where do_stage()
invokes e.g. do_stage_inc() and do_stage_lib(), which (apart from the
obvious defaults) use a variable to decide what directories/files are
copied to staging (normally from ${D}).
After that we can clean the existing recipes and probably get rid of
most of the do_stage rules.

What do people feel about this?

Any volunteers to implement this? I'm more than happy to do part of
the recipe cleaning. My knowledge of the internal structure and do's
and don't's of the build system are pretty weak (and so is my pyhthon
knowledge) so I'd rather not make the build system changes myself.
(unless someone stands up who wants to guide and mentor me).

Frans

PS: I used install in my example above, which I know is the regular
way to do it.
As I also had some subdirs it was more complicated than given above.
Would a cp -r followed by a chmod -R have been ok instead? (it would
definitely be a lot simpler). FM

Re: about staging

[Koen Kooi]

On 30-10-09 08:40, Frans Meulenbroeks wrote:
> Hi,
>
> Yesterday I ended up in a discussion on staging in #oe (ok, ok, I
> mainly caused the discussion :-) )
> Also based upon that I looked at various recipes on how people
> implement staging.
>
> There does not seem to be really standard way.
> Some recipes copy from ${S} which ihmo is not desirable; i think it
> should be a file that is exported by the application).
> Others use other mechanisms.
>
> For my own recipe I ended up with something like:
>
> do_stage() {
>          install -d ${STAGING_INCDIR}
>          install -d ${STAGING_INCDIR}/${PN}
>          install -m 0644 ${D}/${includedir}/${PN}/*.h ${STAGING_INCDIR}/${PN}
> }
>
> as that seems the best. ${D} contans the exported/installed files from
> the application. Those files are the ones the application wants to
> make externally available and those should end up in staging. Not
> something from ${S}.

Is ${D} garanteed to be present at staging time? I don't think we have a 
current hard rule for that.

regards,

Koen

Re: about staging

[Frans Meulenbroeks]

2009/10/30 Koen Kooi <k.kooi@student.utwente.nl>:
> On 30-10-09 08:40, Frans Meulenbroeks wrote:
>>
>> Hi,
>>
>> Yesterday I ended up in a discussion on staging in #oe (ok, ok, I
>> mainly caused the discussion :-) )
>> Also based upon that I looked at various recipes on how people
>> implement staging.
>>
>> There does not seem to be really standard way.
>> Some recipes copy from ${S} which ihmo is not desirable; i think it
>> should be a file that is exported by the application).
>> Others use other mechanisms.
>>
>> For my own recipe I ended up with something like:
>>
>> do_stage() {
>>         install -d ${STAGING_INCDIR}
>>         install -d ${STAGING_INCDIR}/${PN}
>>         install -m 0644 ${D}/${includedir}/${PN}/*.h
>> ${STAGING_INCDIR}/${PN}
>> }
>>
>> as that seems the best. ${D} contans the exported/installed files from
>> the application. Those files are the ones the application wants to
>> make externally available and those should end up in staging. Not
>> something from ${S}.
>
> Is ${D} garanteed to be present at staging time? I don't think we have a
> current hard rule for that.

${D} = install subdir, so in my case it is definitely there.
but it might be absent in the generic case (e.g. in case of an empty package)
A solution which automates everything should take that into account
(if ${D} does not exist: no action))

Btw: as a starter I would already be happy with some scripting that
allows easy copying from ${D} to staging

(and I hope you agree with me that copying from ${S} to staging should be a no)

Frans

Re: about staging

[Denys Dmytriyenko]

On Fri, Oct 30, 2009 at 06:32:05PM +0100, Frans Meulenbroeks wrote:
> 2009/10/30 Koen Kooi <k.kooi@student.utwente.nl>:
> > On 30-10-09 08:40, Frans Meulenbroeks wrote:
> >>
> >> Hi,
> >>
> >> Yesterday I ended up in a discussion on staging in #oe (ok, ok, I
> >> mainly caused the discussion :-) )
> >> Also based upon that I looked at various recipes on how people
> >> implement staging.
> >>
> >> There does not seem to be really standard way.
> >> Some recipes copy from ${S} which ihmo is not desirable; i think it
> >> should be a file that is exported by the application).
> >> Others use other mechanisms.
> >>
> >> For my own recipe I ended up with something like:
> >>
> >> do_stage() {
> >>         install -d ${STAGING_INCDIR}
> >>         install -d ${STAGING_INCDIR}/${PN}
> >>         install -m 0644 ${D}/${includedir}/${PN}/*.h
> >> ${STAGING_INCDIR}/${PN}
> >> }
> >>
> >> as that seems the best. ${D} contans the exported/installed files from
> >> the application. Those files are the ones the application wants to
> >> make externally available and those should end up in staging. Not
> >> something from ${S}.
> >
> > Is ${D} garanteed to be present at staging time? I don't think we have a
> > current hard rule for that.
> 
> ${D} = install subdir, so in my case it is definitely there.
> but it might be absent in the generic case (e.g. in case of an empty package)
> A solution which automates everything should take that into account
> (if ${D} does not exist: no action))
> 
> Btw: as a starter I would already be happy with some scripting that
> allows easy copying from ${D} to staging
> 
> (and I hope you agree with me that copying from ${S} to staging should be a no)

Hmm, I guess I missed the discussion on irc, but I don't think the above 
statement is correct. For example, -native packages do have do_stage and 
ususally don't have do_install. Also, staging some files w/o installing them 
on the "target" (either the main or -dev package) may be desireable in some 
cases...

-- 
Denys

Re: about staging

[Frans Meulenbroeks]

2009/10/31 Denys Dmytriyenko <denis@denix.org>:
> On Fri, Oct 30, 2009 at 06:32:05PM +0100, Frans Meulenbroeks wrote:
>> 2009/10/30 Koen Kooi <k.kooi@student.utwente.nl>:
>> > On 30-10-09 08:40, Frans Meulenbroeks wrote:
>> >>
>> >> Hi,
>> >>
>> >> Yesterday I ended up in a discussion on staging in #oe (ok, ok, I
>> >> mainly caused the discussion :-) )
>> >> Also based upon that I looked at various recipes on how people
>> >> implement staging.
>> >>
>> >> There does not seem to be really standard way.
>> >> Some recipes copy from ${S} which ihmo is not desirable; i think it
>> >> should be a file that is exported by the application).
>> >> Others use other mechanisms.
>> >>
>> >> For my own recipe I ended up with something like:
>> >>
>> >> do_stage() {
>> >>         install -d ${STAGING_INCDIR}
>> >>         install -d ${STAGING_INCDIR}/${PN}
>> >>         install -m 0644 ${D}/${includedir}/${PN}/*.h
>> >> ${STAGING_INCDIR}/${PN}
>> >> }
>> >>
>> >> as that seems the best. ${D} contans the exported/installed files from
>> >> the application. Those files are the ones the application wants to
>> >> make externally available and those should end up in staging. Not
>> >> something from ${S}.
>> >
>> > Is ${D} garanteed to be present at staging time? I don't think we have a
>> > current hard rule for that.
>>
>> ${D} = install subdir, so in my case it is definitely there.
>> but it might be absent in the generic case (e.g. in case of an empty package)
>> A solution which automates everything should take that into account
>> (if ${D} does not exist: no action))
>>
>> Btw: as a starter I would already be happy with some scripting that
>> allows easy copying from ${D} to staging
>>
>> (and I hope you agree with me that copying from ${S} to staging should be a no)
>
> Hmm, I guess I missed the discussion on irc, but I don't think the above
> statement is correct. For example, -native packages do have do_stage and
> ususally don't have do_install. Also, staging some files w/o installing them
> on the "target" (either the main or -dev package) may be desireable in some
> cases...

Ok, ofc I am a n00b when it comes to the build process.
If things work differently and properly for native, then this should
be done for non-native pacakges only.
What I want to get clear is what the basic flow should be.
For me the staging dir is the place where the libs, includes, tools
etc go that are needed for building packages).
If a package X delivers a .h file for general usage, it is normally
exported through make install.
For OE I would expect install to put it in the install dir or perhaps
in staging.
Bypassing install and selectively copying stuff from ${S} does not
seem right to me.
Also if things are missing and stuff is missing, the install from the
package should be fixed

Doing all kind of things manually in  do_stage will cause
inconsistencies and is error prone.(and IMHO should be avoided)

Just my $ 0.02

Frans