distinguishing between bitbake and "higher" layers WRT fetching variables [LONG]

distinguishing between bitbake and "higher" layers WRT fetching variables [LONG]

[Robert P. J. Day]

  (note: more annoying pedantry regarding bitbake configuration and
documentation but, if i'm going to explain this to my students, i
really need to understand it completely.)

  some questions and observations about how bitbake defines some
fetch-related variables as opposed to how those variables are defined
in higher layers such as oe-core and poky. as richard purdie(?)
pointed out recently, the goal should be to document bitbake as
independently as possible from what happens in layers such as oe and
poky, but it seems this is not as clean as it could be.

  consider first bitbake's current bitbake.conf file. first, can we
agree that this set of variables:

FETCHCOMMAND = ""
FETCHCOMMAND_cvs = "/usr/bin/env cvs -d${CVSROOT} co ${CVSCOOPTS} ${CVSMODULE}"
FETCHCOMMAND_svn = "/usr/bin/env svn co ${SVNCOOPTS} ${SVNROOT} ${SVNMODULE}"
FETCHCOMMAND_wget = "/usr/bin/env wget -t 5 --passive-ftp -P ${DL_DIR} ${URI}"

is entirely superfluous? i see nothing that uses them -- everything
seems to use the alternative "FETCHCMD_*" naming. can the above just
be deleted?

  with respect to the "FETCHCMD_*" variables, this is a bit messier.
the bitbake user manual defines that variable thusly:

file:///home/rpjday/oe/dist/layers/bitbake/doc/bitbake-user-manual/bitbake-user-manual.html#var-FETCHCMD

tossing out examples of FETCHCMD_git and FETCHCMD_svn, and here's the
messy part.

  who is responsible for setting the default values for those fetcher
commands? the first possibility is that bitbake itself can do that in
the fetcher code. for instance, here's the test from git.py:

ud.basecmd = data.getVar("FETCHCMD_git", d, True) or "git -c core.fsyncobjectfiles=0"

what the above seems to be doing (based on my mediocre understanding
of python) is that either that fetcher variable is set, or it's set to
the subsequent git command, correct? in short, the fundamental bitbake
git fetcher code is responsible for setting the basic git fetch
command. perfectly reasonable, but there's another possibility.

  a number of other fetch variables appear to be defined in oe-core,
whose bitbake.conf file contains:

FETCHCMD_svn = "/usr/bin/env svn --non-interactive --trust-server-cert"
FETCHCMD_cvs = "/usr/bin/env cvs"
FETCHCMD_wget = "/usr/bin/env wget -t 2 -T 30 -nv --passive-ftp --no-check-certificate"
FETCHCMD_bzr = "/usr/bin/env bzr"
FETCHCMD_hg = "/usr/bin/env hg"

and, back in bitbake, the fetch command for, say, svn, is determined
in svn.py simply with:

  ud.basecmd = d.getVar('FETCHCMD_svn', True)

if i read this correctly, then, that code from bitbake simply
*assumes* that that variable for the svn fetcher is set -- there is no
error checking if it isn't. this suggests that the fetcher variables
for things like svn and hg and bzr are assumed by bitbake to come from
elsewhere -- in this case, oe-core or poky or what have you.

  this seems inconsistent -- in some cases, the bitbake fetch code
will guarantee that a fetcher variable is set (as with git or
perforce):

git.py:        ud.basecmd = data.getVar("FETCHCMD_git", d, True) or "git -c core.fsyncobjectfiles=0"
perforce.py:        p4cmd = d.getVar('FETCHCMD_p4', True) or "p4"

in other cases, bitbake seems to count on someone else setting the
appropriate fetcher variable (as in oe-core):

FETCHCMD_svn = "/usr/bin/env svn --non-interactive --trust-server-cert"
FETCHCMD_cvs = "/usr/bin/env cvs"
FETCHCMD_wget = "/usr/bin/env wget -t 2 -T 30 -nv --passive-ftp --no-check-certificate"
FETCHCMD_bzr = "/usr/bin/env bzr"
FETCHCMD_hg = "/usr/bin/env hg"

and in one case, both the bitbake layer and oe-core currently (and
redundantly) take responsibility for setting the correct fetcher
command for wget:

self.basecmd = d.getVar("FETCHCMD_wget", True) or "/usr/bin/env wget -t 2 -T 30 -nv --passive-ftp --no-check-certificate"
FETCHCMD_wget = "/usr/bin/env wget -t 2 -T 30 -nv --passive-ftp --no-check-certificate"

this just seems confusing, as there is no obvious basis for deciding
who gets to set those variables.

  oh, and one more (admittedly minor) observation related to the
above. in my courses, i really like to use chris larson's cool "bb"
utility to display environment and variables, and the above situation
causes the following glitch.

  if a fetcher variable is set explicitly (say, in oe-core), as in:

FETCHCMD_bzr = "/usr/bin/env bzr"
FETCHCMD_hg = "/usr/bin/env hg"

then the "bb" command will happily show me the value of that variable:

$ bb show FETCHCMD_bzr
FETCHCMD_bzr="/usr/bin/env bzr"
$ bb show FETCHCMD_hg
FETCHCMD_hg="/usr/bin/env hg"
$

but if the fetch variable is determined by the bitbake fetcher code at
run-time, as with git or perforce, well, i'm screwed:

$ bb show FETCHCMD_git
WARNING: Requested variable 'FETCHCMD_git' does not exist
$ bb show FETCHCMD_p4
WARNING: Requested variable 'FETCHCMD_p4' does not exist
$

  it just seems that there isn't a rigourously clean separation
between bitbake and oe-core, and it's entirely possible there *can't*
be, but some things seem unnecessarily confusing as to where certain
values or variables are defined.

  yes, there should be a bitbake user manual covering bitbake-only
content independent of higher layers like oe-core, but at the same
time, given that readers will be using bitbake *in the context* of
building with oe-core or poky, i think bitbake should avoid code or
documentation that is almost guaranteed to be different in the context
of those higher layers. case in point -- bitbake:

  GITDIR = "${DL_DIR}/git"

oe-core:

  GITDIR = "${CO_DIR}/git2"

anyway, i've rambled sufficiently. thoughts?

rday

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

Re: distinguishing between bitbake and "higher" layers WRT fetching variables [LONG]

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 1283 bytes --]

On Fri, Jun 20, 2014 at 8:47 AM, Robert P. J. Day <rpjday@crashcourse.ca>
wrote:

>   (note: more annoying pedantry regarding bitbake configuration and
> documentation but, if i'm going to explain this to my students, i
> really need to understand it completely.)
>
>   some questions and observations about how bitbake defines some
> fetch-related variables as opposed to how those variables are defined
> in higher layers such as oe-core and poky. as richard purdie(?)
> pointed out recently, the goal should be to document bitbake as
> independently as possible from what happens in layers such as oe and
> poky, but it seems this is not as clean as it could be.
>
>   consider first bitbake's current bitbake.conf file. first, can we
> agree that this set of variables:
>

Before I dive into the remainder of the email, we should note that the
bitbake.conf in bitbake is never used when you use oe, it's solely as an
example / starting point for other non-oe projects wanting to use bitbake.
Presumably it gets forgotten about when changes to bitbake itself are made,
and so it's fallen behind.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 1721 bytes --]

Re: distinguishing between bitbake and "higher" layers WRT fetching variables [LONG]

[Robert P. J. Day]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2052 bytes --]

On Fri, 20 Jun 2014, Christopher Larson wrote:

>
> On Fri, Jun 20, 2014 at 8:47 AM, Robert P. J. Day <rpjday@crashcourse.ca> wrote:
>         (note: more annoying pedantry regarding bitbake configuration and
>       documentation but, if i'm going to explain this to my students, i
>       really need to understand it completely.)
>
>         some questions and observations about how bitbake defines some
>       fetch-related variables as opposed to how those variables are defined
>       in higher layers such as oe-core and poky. as richard purdie(?)
>       pointed out recently, the goal should be to document bitbake as
>       independently as possible from what happens in layers such as oe and
>       poky, but it seems this is not as clean as it could be.
>
>         consider first bitbake's current bitbake.conf file. first, can we
>       agree that this set of variables:
>
>
> Before I dive into the remainder of the email, we should note that
> the bitbake.conf in bitbake is never used when you use oe, it's
> solely as an example / starting point for other non-oe projects
> wanting to use bitbake. Presumably it gets forgotten about when
> changes to bitbake itself are made, and so it's fallen behind.

  oh, i realize that but (i think) my main point was that it seems
almost impossible to document bitbake completely stand-alone when
almost everyone is going to use it in the context of a higher layer
like OE, and there just seem to be unnecessary inconsistencies between
bitbake and OE that make the bitbake user manual confusing.

  more in a bit ...

rday

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

Re: distinguishing between bitbake and "higher" layers WRT fetching variables [LONG]

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 2316 bytes --]

On Fri, Jun 20, 2014 at 8:55 AM, Robert P. J. Day <rpjday@crashcourse.ca>
wrote:

> On Fri, 20 Jun 2014, Christopher Larson wrote:
>
> >
> > On Fri, Jun 20, 2014 at 8:47 AM, Robert P. J. Day <rpjday@crashcourse.ca>
> wrote:
> >         (note: more annoying pedantry regarding bitbake configuration and
> >       documentation but, if i'm going to explain this to my students, i
> >       really need to understand it completely.)
> >
> >         some questions and observations about how bitbake defines some
> >       fetch-related variables as opposed to how those variables are
> defined
> >       in higher layers such as oe-core and poky. as richard purdie(?)
> >       pointed out recently, the goal should be to document bitbake as
> >       independently as possible from what happens in layers such as oe
> and
> >       poky, but it seems this is not as clean as it could be.
> >
> >         consider first bitbake's current bitbake.conf file. first, can we
> >       agree that this set of variables:
> >
> >
> > Before I dive into the remainder of the email, we should note that
> > the bitbake.conf in bitbake is never used when you use oe, it's
> > solely as an example / starting point for other non-oe projects
> > wanting to use bitbake. Presumably it gets forgotten about when
> > changes to bitbake itself are made, and so it's fallen behind.
>
>   oh, i realize that but (i think) my main point was that it seems
> almost impossible to document bitbake completely stand-alone when
> almost everyone is going to use it in the context of a higher layer
> like OE, and there just seem to be unnecessary inconsistencies between
> bitbake and OE that make the bitbake user manual confusing.


Fair enough. An argument could be made that oe-core *is* the example of how
to use bitbake, but the problem with that is, the bitbake.conf there
doesn't differentiate between what's required for bitbake / what vars are
obeyed specifically by bitbake and the vars that are obeyed by oe-core's
classes, so a non-oe project would have difficulty in paring it down to
what they need for their own projects.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 2900 bytes --]