question about order of processing SRC_URI items

question about order of processing SRC_URI items

[Robert P. J. Day]

  i'm currently building up a BSP layer which is going to involve
quite a bit of FILESEXTRAPATHS and OVERRIDES and conditionals in my
kernel bbappend files so i want to make sure i understand the
processing order (or what there is of it).

  first, if one simply appends items to SRC_URI, as in:

  SRC_URI += " \
	derp.scc \
	gorp.scc \
	...
  "

then regardless of how FILESOVERRIDES adjusts the search order in the
eventual FILESPATH, those items *must* be found somewhere, correct?
(this seems pretty obvious to me, but i've been burned by obvious
things before.)

  next, i realize there are two forms of appending to SRC_URI:

 * SRC_URI +=
 * SRC_URI_append =

first, is there an aesthetic preference for one over the other if the
end result is the same? (just curious about what people here prefer to
use.)

  more significantly, i realize the final order of SRC_URI items
depending on which one you use might be different, as "_append" leaves
the appending until the final stage of processing. so what this tells
me is that all of the items you list in SRC_URI should *not* be
order-dependent, is that a fair statement? or, IOW, if your layer
depends on the items in SRC_URI being processed in a specific order,
you probably need to rework your recipe file.

  (side note: i do realize that, in a single .scc file, the patches
listed will be applied in precisely that order, but that's a different
issue and not relevant here.)

  next, if there are truly conditional SRC_URI items, the only way to
identify those is:

  SRC_URI_append_<override> = " ... blah blah ..."

you can't do this, can you?

  SRC_URI_<override> += " more stuff "

i don't think i've ever seen an example of the latter, but maybe i
just didn't look closely enough.

  and, finally, given that you can have a combination of SRC_URI
assignments and appends in the same .bbappend file:

  SRC_URI += "..."
  SRC_URI_append = "..."
  SRC_URI_append_<override> = "..."

that tells me that you *really* don't want any of that to be
order-dependent, as i mentioned before.

  is that about it? am i missing anything about SRC_URI definition and
processing? thanks muchly.

rday

p.s.  oh, wait, two more curiosities. first, i still see the
occasional example of this in numerous layers, including oe-core:

    meta/recipes-support/attr/attr_2.4.47.bb:
        SRC_URI_append += "file://attr-Missing-configure.ac.patch \
        ^^^^^^^^^^^^^^^^^

pretty sure combining "SRC_URI_append" with "+=" is just as redundant
as ever, no? :-)

  also, if the list of items in SRC_URI should be order-independent,
then it would seem to make little sense to specifically use:

  SRC_URI_prepend = "..."

anywhere, right? IOW, there should be no situation in which
specifically *prepending* to SRC_URI is necessary. but i found one
example in oe-core:

  meta/recipes-devtools/qemu/qemu_2.5.0.bb:

SRC_URI += "file://configure-fix-Darwin-target-detection.patch \
            file://qemu-enlarge-env-entry-size.patch \
            file://Qemu-Arm-versatilepb-Add-memory-size-checking.patch \
            file://no-valgrind.patch \
            file://CVE-2016-1568.patch \
            file://CVE-2016-2197.patch \
            file://CVE-2016-2198.patch \
            file://pathlimit.patch \
           "
SRC_URI_prepend = "http://wiki.qemu-project.org/download/${BP}.tar.bz2"
SRC_URI[md5sum] = "f469f2330bbe76e3e39db10e9ac4f8db"
SRC_URI[sha256sum] = "3443887401619fe33bfa5d900a4f2d6a79425ae2b7e43d5b8c36eb7a683772d4"

  uh ... yeah, i get that the above does the right thing but, come on,
that just looks weird. is there a reason the qemu recipe deliberately
goes out of its way to do this?

  ok, back to work ...

Re: question about order of processing SRC_URI items

[Richard Purdie]

On Thu, 2016-04-21 at 07:19 -0400, Robert P. J. Day wrote:
>   i'm currently building up a BSP layer which is going to involve
> quite a bit of FILESEXTRAPATHS and OVERRIDES and conditionals in my
> kernel bbappend files so i want to make sure i understand the
> processing order (or what there is of it).
> 
>   first, if one simply appends items to SRC_URI, as in:
> 
>   SRC_URI += " \
> 	derp.scc \
> 	gorp.scc \
> 	...
>   "
> 
> then regardless of how FILESOVERRIDES adjusts the search order in the
> eventual FILESPATH, those items *must* be found somewhere, correct?
> (this seems pretty obvious to me, but i've been burned by obvious
> things before.)

They need to be found somewhere, yes.

>   next, i realize there are two forms of appending to SRC_URI:
> 
>  * SRC_URI +=
>  * SRC_URI_append =
> 
> first, is there an aesthetic preference for one over the other if the
> end result is the same? (just curious about what people here prefer
> to
> use.)

In general we should try and use += if possible as its simpler and
easier to understand what it does.

>   more significantly, i realize the final order of SRC_URI items
> depending on which one you use might be different, as "_append"
> leaves
> the appending until the final stage of processing. so what this tells
> me is that all of the items you list in SRC_URI should *not* be
> order-dependent, is that a fair statement? or, IOW, if your layer
> depends on the items in SRC_URI being processed in a specific order,
> you probably need to rework your recipe file.

Sometimes order is important, e.g. two patches which change the same
set of files would need to be applied in the correct order.

Some things don't matter though.

>   (side note: i do realize that, in a single .scc file, the patches
> listed will be applied in precisely that order, but that's a
> different
> issue and not relevant here.)
> 
>   next, if there are truly conditional SRC_URI items, the only way to
> identify those is:
> 
>   SRC_URI_append_<override> = " ... blah blah ..."
> 
> you can't do this, can you?
> 
>   SRC_URI_<override> += " more stuff "

You can do that but it won't do what you'd want (it would overwrite the
original SRC_URI).

> i don't think i've ever seen an example of the latter, but maybe i
> just didn't look closely enough.

That would be because it wouldn't do anything useful in most cases.

>   and, finally, given that you can have a combination of SRC_URI
> assignments and appends in the same .bbappend file:
> 
>   SRC_URI += "..."
>   SRC_URI_append = "..."
>   SRC_URI_append_<override> = "..."
> 
> that tells me that you *really* don't want any of that to be
> order-dependent, as i mentioned before.

Right, you'd have to be careful with this.

>   is that about it? am i missing anything about SRC_URI definition
> and
> processing? thanks muchly.
> 
> rday
> 
> p.s.  oh, wait, two more curiosities. first, i still see the
> occasional example of this in numerous layers, including oe-core:
> 
>     meta/recipes-support/attr/attr_2.4.47.bb:
>         SRC_URI_append += "file://attr-Missing-configure.ac.patch \
>         ^^^^^^^^^^^^^^^^^
> 
> pretty sure combining "SRC_URI_append" with "+=" is just as redundant
> as ever, no? :-)

+= means there is a space prepended so its equivalent to:

SRC_URI_append = " file://attr-Missing-configure.ac.patch \

but we should remove such things.

>   also, if the list of items in SRC_URI should be order-independent,
> then it would seem to make little sense to specifically use:
> 
>   SRC_URI_prepend = "..."
> 
> anywhere, right? IOW, there should be no situation in which
> specifically *prepending* to SRC_URI is necessary. but i found one
> example in oe-core:
> 
>   meta/recipes-devtools/qemu/qemu_2.5.0.bb:
> 
> SRC_URI += "file://configure-fix-Darwin-target-detection.patch \
>             file://qemu-enlarge-env-entry-size.patch \
>             file://Qemu-Arm-versatilepb-Add-memory-size
> -checking.patch \
>             file://no-valgrind.patch \
>             file://CVE-2016-1568.patch \
>             file://CVE-2016-2197.patch \
>             file://CVE-2016-2198.patch \
>             file://pathlimit.patch \
>            "
> SRC_URI_prepend = "
> http://wiki.qemu-project.org/download/${BP}.tar.bz2"
> SRC_URI[md5sum] = "f469f2330bbe76e3e39db10e9ac4f8db"
> SRC_URI[sha256sum] =
> "3443887401619fe33bfa5d900a4f2d6a79425ae2b7e43d5b8c36eb7a683772d4"
> 
>   uh ... yeah, i get that the above does the right thing but, come
> on,
> that just looks weird. is there a reason the qemu recipe deliberately
> goes out of its way to do this?

Its convention that the main source is at the top of SRC_URI. I
therefore don't really have an issue with what the qemu recipe does but
agree it could be cleaned up.

Cheers,

Richard

Re: question about order of processing SRC_URI items

[Robert P. J. Day]

... some stuff snipped ...

On Thu, 21 Apr 2016, Richard Purdie wrote:

> On Thu, 2016-04-21 at 07:19 -0400, Robert P. J. Day wrote:

> >   more significantly, i realize the final order of SRC_URI items
> > depending on which one you use might be different, as "_append"
> > leaves the appending until the final stage of processing. so what
> > this tells me is that all of the items you list in SRC_URI should
> > *not* be order-dependent, is that a fair statement? or, IOW, if
> > your layer depends on the items in SRC_URI being processed in a
> > specific order, you probably need to rework your recipe file.
>
> Sometimes order is important, e.g. two patches which change the same
> set of files would need to be applied in the correct order.

  if i had that situation, i would think the obvious approach is that
those two patches should be bundled into the same .scc file, where
patch application ordering is preserved. my point was that i wouldn't
want to count on the ordering of items in the SRC_URI list itself.

> >   and, finally, given that you can have a combination of SRC_URI
> > assignments and appends in the same .bbappend file:
> >
> >   SRC_URI += "..."
> >   SRC_URI_append = "..."
> >   SRC_URI_append_<override> = "..."
> >
> > that tells me that you *really* don't want any of that to be
> > order-dependent, as i mentioned before.
>
> Right, you'd have to be careful with this.

  and that's the point i was making ... if you're mixing conditional
and unconditional SRC_URI appending, you *really* don't want to count
on any particular order of processing of the items in that variable.

  oh, one more thing related to coding style. given that .bb recipe
files are the foundation on which further .bbappend customization is
built, it would seem that .bb files should consistently use just:

  SRC_URI = "..."

and not

  SRC_URI += "..."

*unless* they're first including some .inc file that would set that
variable to some common initial value.

  i know it would make no difference in the end, but for me, it's a
visual thing -- when i see any use of "+=", i normally assume that
there might be some earlier value that's been assigned, so i take a
little more care in what i'm doing. so my coding style for new recipe
files would be to *always* use:

  SRC_URI = "..."

unless there was a good reason not to. and, yes, i am that pedantic.

rday

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

Re: question about order of processing SRC_URI items

[Robert P. J. Day]

On Thu, 21 Apr 2016, Richard Purdie wrote:

> In general we should try and use += if possible as its simpler and
> easier to understand what it does.

  regarding "_append +=", there are only two files left in oe-core
that i will let someone else deal with if they want. first:

meta/recipes-extended/images/core-image-kernel-dev.bb:

  IMAGE_ROOTFS_EXTRA_SPACE_append += "+ 3000000"

and there's this weirdness in meta/lib/oeqa/selftest/layerappend.py:

... snip ...
addtask build
"""
    append = """
FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"

SRC_URI_append = " file://appendtest.txt"

sysroot_stage_all_append() {
        install -m 644 ${WORKDIR}/appendtest.txt ${SYSROOT_DESTDIR}/
}

"""

    append2 = """
FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"

SRC_URI_append += "file://appendtest.txt"



  not quite sure what's going on there with those two variations of
assignment to SRC_URI, so i'm just going to leave it alone.

rday

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================