confusion about DEPENDS vs RDEPENDS

confusion about DEPENDS vs RDEPENDS

[Hans Beckérus]

Hi, I am a little bit confused about how to handle these two and what
they are supposed to solve. I have so far never used RDEPENDS but only
DEPENDS.
But I am also having severe problems when building a rootfs image when
one of my user space libraries are changed from eg. libfoo.so.1 to
libfoo.so.3. Even though all my packages that have dependencies to it
includes it in a DEPENDS.
The error I get during rootfs build is:

| Computing transaction...error: Can't install
someapp-1.0-r0@cortexa9_vfp: no package provides libfoo.so.1

But there is no libfoo.so.1 in my sysroot, it has been replaced by libfoo.so.3.
I know for sure that 'someapp' was rebuilt, but still I got the error message.
What do seem to help is to force a fetch of 'someapp' and then rebuild
which sort of indicates that some garbage was left behind. But having
a package listed in DEPENDS will not force a new fetch if I am not
mistaken.

Have I been using the DEPENDS variable incorrectly? Would it make a
difference if I used RDEPENDS instead?

Thanks.
Hans

Re: confusion about DEPENDS vs RDEPENDS

[Paul Eggleton]

Hi Hans,

On Wednesday 28 August 2013 17:08:41 Hans Beckérus wrote:
> Hi, I am a little bit confused about how to handle these two and what
> they are supposed to solve. I have so far never used RDEPENDS but only
> DEPENDS.

DEPENDS means a build-time dependency i.e. between recipes, RDEPENDS means a 
runtime dependency i.e. between packages. It is worth noting though that an 
explicitly stated RDEPENDS will cause bitbake to actually build the recipe 
providing the package named in the RDEPENDS value, just at a different time. To 
explain exactly what each of these do:

* DEPENDS = "b" in recipe "a" will translate to a's do_configure task depending 
on recipe b's do_populate_sysroot task, so that anything recipe b puts into 
the sysroot is available for when a configures itself.

* RDEPENDS_${PN} = "b" in recipe "a" will translate to a's do_build task 
depending on recipe b's do_package_write task, so that the package file b is 
available when the output for a has been completely built (of course assuming 
that recipe b produces a package called "b", which it will with the default 
value of PACKAGES). More importantly it will also ensure that package "a" is 
marked as depending on "b" in a manner understood by the package manager being 
used e.g. rpm / opkg / dpkg.

> But I am also having severe problems when building a rootfs image when
> one of my user space libraries are changed from eg. libfoo.so.1 to
> libfoo.so.3. Even though all my packages that have dependencies to it
> includes it in a DEPENDS.
> 
> The error I get during rootfs build is:
> | Computing transaction...error: Can't install
> 
> someapp-1.0-r0@cortexa9_vfp: no package provides libfoo.so.1
> 
> But there is no libfoo.so.1 in my sysroot, it has been replaced by
> libfoo.so.3. I know for sure that 'someapp' was rebuilt, but still I got
> the error message. What do seem to help is to force a fetch of 'someapp'
> and then rebuild which sort of indicates that some garbage was left behind.
> But having a package listed in DEPENDS will not force a new fetch if I am
> not mistaken.

By default, if recipe "foo" changes and it is mentioned in the "someapp" 
recipe's DEPENDS, then someapp's do_configure and all tasks that depend upon it 
will be re-executed next time it is called for i.e. you explicitly build 
someapp or build an image that contains it or some other recipe that depends 
upon it. The fact that you are getting the behaviour described suggests that 
this is either not happening, or more likely it is not having the desired 
effect; e.g. if internally someapp's build system doesn't drop or invalidate 
all of its  build output when it is reconfigured then you will get this kind of 
behaviour. Setting up B (the directory in which a recipe's source code is 
built) separate to S (the directory in which the recipe's source code has been 
unpacked to) can help with this since if they are separate, our build system 
will know it can delete B before re-executing do_compile after do_configure and 
you'll never have stale build output. Being able to set this up however is 
highly dependent on the software being built by the individual recipe; some 
lend themselves to this and others don't.
 
> Have I been using the DEPENDS variable incorrectly? Would it make a
> difference if I used RDEPENDS instead?

RDEPENDS would not be appropriate in this situation, since we're talking about 
a build-time dependency.

Hope that helps.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: confusion about DEPENDS vs RDEPENDS

[Hans Beckerus]

On 2013-08-28 6:06, Paul Eggleton wrote:
> Hi Hans,
>
> On Wednesday 28 August 2013 17:08:41 Hans Beckérus wrote:
>> Hi, I am a little bit confused about how to handle these two and what
>> they are supposed to solve. I have so far never used RDEPENDS but only
>> DEPENDS.
> DEPENDS means a build-time dependency i.e. between recipes, RDEPENDS means a
> runtime dependency i.e. between packages. It is worth noting though that an
> explicitly stated RDEPENDS will cause bitbake to actually build the recipe
> providing the package named in the RDEPENDS value, just at a different time. To
> explain exactly what each of these do:
>
> * DEPENDS = "b" in recipe "a" will translate to a's do_configure task depending
> on recipe b's do_populate_sysroot task, so that anything recipe b puts into
> the sysroot is available for when a configures itself.
>
> * RDEPENDS_${PN} = "b" in recipe "a" will translate to a's do_build task
> depending on recipe b's do_package_write task, so that the package file b is
> available when the output for a has been completely built (of course assuming
> that recipe b produces a package called "b", which it will with the default
> value of PACKAGES). More importantly it will also ensure that package "a" is
> marked as depending on "b" in a manner understood by the package manager being
> used e.g. rpm / opkg / dpkg.
Thanks a lot! This was definitely more than I got from the description 
of DEPENDS and RDEPENDS in the manual.
But I probably just read the wrong one ;)
>> But I am also having severe problems when building a rootfs image when
>> one of my user space libraries are changed from eg. libfoo.so.1 to
>> libfoo.so.3. Even though all my packages that have dependencies to it
>> includes it in a DEPENDS.
>>
>> The error I get during rootfs build is:
>> | Computing transaction...error: Can't install
>>
>> someapp-1.0-r0@cortexa9_vfp: no package provides libfoo.so.1
>>
>> But there is no libfoo.so.1 in my sysroot, it has been replaced by
>> libfoo.so.3. I know for sure that 'someapp' was rebuilt, but still I got
>> the error message. What do seem to help is to force a fetch of 'someapp'
>> and then rebuild which sort of indicates that some garbage was left behind.
>> But having a package listed in DEPENDS will not force a new fetch if I am
>> not mistaken.
> By default, if recipe "foo" changes and it is mentioned in the "someapp"
> recipe's DEPENDS, then someapp's do_configure and all tasks that depend upon it
> will be re-executed next time it is called for i.e. you explicitly build
> someapp or build an image that contains it or some other recipe that depends
> upon it. The fact that you are getting the behaviour described suggests that
> this is either not happening, or more likely it is not having the desired
> effect; e.g. if internally someapp's build system doesn't drop or invalidate
> all of its  build output when it is reconfigured then you will get this kind of
> behaviour. Setting up B (the directory in which a recipe's source code is
> built) separate to S (the directory in which the recipe's source code has been
> unpacked to) can help with this since if they are separate, our build system
> will know it can delete B before re-executing do_compile after do_configure and
> you'll never have stale build output. Being able to set this up however is
> highly dependent on the software being built by the individual recipe; some
> lend themselves to this and others don't.
>   
Well, I have been struggling before with packages not properly 
supporting different build and source folders so I can definitely relate 
to what you are saying. But, does that mean I actually *have* to do it 
this way for build dependencies to work correctly? In my case we are 
talking two simple autotools enabled packages and I (naively?) assumed 
this was not something I had to take care of myself. What strikes me is 
that you say ""if recipe "foo" changes"", which is actually not the case 
here! What is changed is the actual source code only. Is that what is 
going wrong here? If I change my "foo" recipe version, would that be 
different than to simply fetch/unpack the "foo" package source code? Is 
"someapp" going to become purged differently in such a  scenario?

>> Have I been using the DEPENDS variable incorrectly? Would it make a
>> difference if I used RDEPENDS instead?
> RDEPENDS would not be appropriate in this situation, since we're talking about
> a build-time dependency.
>
> Hope that helps.
What is still somewhat unclear to me is the difference between DEPENDS 
and RDEPENDS in a simple case as this.
A simple application needing a dynamic library is obviously a subject 
for DEPENDS but to me that also sounds like a typical RDEPENDS?
However, when I build an image and include 'someapp', will 'libfoo.so.x' 
automatically be installed or is that what I need to tell it to do using 
RDEPENDS?

> Cheers,
> Paul
>

Re: confusion about DEPENDS vs RDEPENDS

[Nicolas Dechesne]

[-- Attachment #1: Type: text/plain, Size: 2166 bytes --]

On Wed, Aug 28, 2013 at 9:22 PM, Hans Beckerus <hans.beckerus@gmail.com>wrote:

> Well, I have been struggling before with packages not properly supporting
> different build and source folders so I can definitely relate to what you
> are saying. But, does that mean I actually *have* to do it this way for
> build dependencies to work correctly? In my case we are talking two simple
> autotools enabled packages and I (naively?) assumed this was not something
> I had to take care of myself. What strikes me is that you say ""if recipe
> "foo" changes"", which is actually not the case here! What is changed is
> the actual source code only. Is that what is going wrong here? If I change
> my "foo" recipe version, would that be different than to simply
> fetch/unpack the "foo" package source code? Is "someapp" going to become
> purged differently in such a  scenario?
>

if the source code changes, the version of the recipe needs to change too.
if you change the source code without bumping the version, the package
might not be rebuilt properly indeed. and that can explain the behavior you
are seeing. if 'someapp' does not change, it would be rebuilt only if one
of its dependencies was rebuilt.


>
>
>  Have I been using the DEPENDS variable incorrectly? Would it make a
>>> difference if I used RDEPENDS instead?
>>>
>> RDEPENDS would not be appropriate in this situation, since we're talking
>> about
>> a build-time dependency.
>>
>> Hope that helps.
>>
> What is still somewhat unclear to me is the difference between DEPENDS and
> RDEPENDS in a simple case as this.
> A simple application needing a dynamic library is obviously a subject for
> DEPENDS but to me that also sounds like a typical RDEPENDS?
> However, when I build an image and include 'someapp', will 'libfoo.so.x'
> automatically be installed or is that what I need to tell it to do using
> RDEPENDS?
>
some (most?) of the RDEPENDS are computed auto-magically. so bitbake will
figure out the dependency between 'someapp' and 'libfoo.so.x' and will
automatically update 'someapp' RDEPENDS, without the need to do it
explicitely in the recipe!

[-- Attachment #2: Type: text/html, Size: 3066 bytes --]

Re: confusion about DEPENDS vs RDEPENDS

[Paul D. DeRocco]

> From: Nicolas Dechesne
> 
> if the source code changes, the version of the recipe needs 
> to change too. if you change the source code without bumping 
> the version, the package might not be rebuilt properly 
> indeed. and that can explain the behavior you are seeing. if 
> 'someapp' does not change, it would be rebuilt only if one of 
> its dependencies was rebuilt. 

If you're making lots of changes in the course of debugging, isn't it
reasonable just to do a cleansstate on the recipe to force it to be
rebuilt?

-- 

Ciao,               Paul D. DeRocco
Paul                mailto:pderocco@ix.netcom.com 

Re: confusion about DEPENDS vs RDEPENDS

[Paul Eggleton]

Hi Paul / Nicolas,

On Wednesday 28 August 2013 15:15:27 Paul D. DeRocco wrote:
> From: Nicolas Dechesne
> > if the source code changes, the version of the recipe needs
> > to change too. if you change the source code without bumping
> > the version, the package might not be rebuilt properly
> > indeed. and that can explain the behavior you are seeing. if
> > 'someapp' does not change, it would be rebuilt only if one of
> > its dependencies was rebuilt.
> 
> If you're making lots of changes in the course of debugging, isn't it
> reasonable just to do a cleansstate on the recipe to force it to be
> rebuilt?

Current versions of the build system (denzil/1.2+, although refinements have 
been made since then) are task signature-based. That means as far as the build 
system is able to determine its inputs, if those change it should be able to 
rebuild all of the output to match. Known limitations:

* Upstream software that doesn't properly rebuild when reconfigured. In most 
cases this should be considered as a bug in the recipe. Separating S from B 
can help here as I mentioned earlier, and you can see in dylan/1.4+ in 
meta/conf/distro/include/seperatebuilddir.inc that we've been enabling 
separate recipe build dirs for a number of recipes to help with this.

* Editing the unpacked/patched source code in the recipe's work directory 
(i.e. tmp/work/...). Note that this is not something that is discouraged - in 
fact it can be a very useful development aid. However, you do need to be aware 
of the need to force the appropriate tasks to re-execute after you have made 
changes in there i.e. bitbake -c compile -f <recipe> or bitbake -C compile 
<recipe>, since the build system cannot detect these kinds of changes on its 
own.

* Items remaining in the sysroot when recipes are completely renamed (i.e. 
when PN changes) or when a recipe is removed. We saw this recently with the 
mesa-dri -> mesa rename. Currently there's no way for the system to know what 
replaces what when PN changes or what to do when a recipe completely vanishes, 
you just have to clean out the old recipe's files in the sysroot. This can of 
course happen if you add and remove layers without deleting TMPDIR, so care 
should be taken when doing that. This is an difficult issue to solve 
practically; there is discussion of this issue here for those interested: 
https://bugzilla.yoctoproject.org/show_bug.cgi?id=4102

* If you use runtime packaging ("package-management" in IMAGE_FEATURES) and 
you're putting the packages into a feed and expect on-target upgrades to work 
consistently from those feeds without manually bumping PR in recipes on every 
material change, you need to enable the PR service as described in the 
development manual so that PR increments automatically.

* Changes on the host system affecting native recipes - less likely to cause 
issues, but worth being aware of. It can happen that adding or removing 
packages on the host system changes the configuration of native recipes without 
automatically triggering a rebuild - a good example is how we allow qemu-
native to build on systems with and without X11; if you added SDL and X11 to a 
system on which you'd already built qemu-native beforehand, in the absence of 
other changes you'd have to cleansstate or otherwise force a rebuild of qemu-
native to have a native QEMU that supported graphical output.


However, with the caveats above, most of the time you can rely upon the build 
system to determine what to do when things change. Of course, yes, if you want 
to just force a recipe to rebuild you have the option of bitbake -c 
cleansstate <recipe> before building it again, but most of the time that's 
going to be more than is needed.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: confusion about DEPENDS vs RDEPENDS

[lothar]

Hello Paul,

I followed this very interesting thread, and have a question on the last 
statement, just for understanding:

Are the limitations you are explaining here, the same as mentioned in...
http://www.yoctoproject.org/docs/current/mega-manual/mega-manual.html
...under "Checksums (Signatures)", telling that:
"Consider a case with in-line Python, for example, where BitBake is not 
able to figure out dependencies."   ? So to speak, are those limitations 
the cases, where bitbake is not able to figure out dependencies 
correctly?

Thanks in advance,
Lothar


> Hi Paul / Nicolas,
> 
> On Wednesday 28 August 2013 15:15:27 Paul D. DeRocco wrote:
>> From: Nicolas Dechesne
>> > if the source code changes, the version of the recipe needs
>> > to change too. if you change the source code without bumping
>> > the version, the package might not be rebuilt properly
>> > indeed. and that can explain the behavior you are seeing. if
>> > 'someapp' does not change, it would be rebuilt only if one of
>> > its dependencies was rebuilt.
>> 
>> If you're making lots of changes in the course of debugging, isn't it
>> reasonable just to do a cleansstate on the recipe to force it to be
>> rebuilt?
> 
> Current versions of the build system (denzil/1.2+, although refinements 
> have
> been made since then) are task signature-based. That means as far as 
> the build
> system is able to determine its inputs, if those change it should be 
> able to
> rebuild all of the output to match. Known limitations:
> 
> * Upstream software that doesn't properly rebuild when reconfigured. In 
> most
> cases this should be considered as a bug in the recipe. Separating S 
> from B
> can help here as I mentioned earlier, and you can see in dylan/1.4+ in
> meta/conf/distro/include/seperatebuilddir.inc that we've been enabling
> separate recipe build dirs for a number of recipes to help with this.
> 
> * Editing the unpacked/patched source code in the recipe's work 
> directory
> (i.e. tmp/work/...). Note that this is not something that is 
> discouraged - in
> fact it can be a very useful development aid. However, you do need to 
> be aware
> of the need to force the appropriate tasks to re-execute after you have 
> made
> changes in there i.e. bitbake -c compile -f <recipe> or bitbake -C 
> compile
> <recipe>, since the build system cannot detect these kinds of changes 
> on its
> own.
> 
> * Items remaining in the sysroot when recipes are completely renamed 
> (i.e.
> when PN changes) or when a recipe is removed. We saw this recently with 
> the
> mesa-dri -> mesa rename. Currently there's no way for the system to 
> know what
> replaces what when PN changes or what to do when a recipe completely 
> vanishes,
> you just have to clean out the old recipe's files in the sysroot. This 
> can of
> course happen if you add and remove layers without deleting TMPDIR, so 
> care
> should be taken when doing that. This is an difficult issue to solve
> practically; there is discussion of this issue here for those 
> interested:
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=4102
> 
> * If you use runtime packaging ("package-management" in IMAGE_FEATURES) 
> and
> you're putting the packages into a feed and expect on-target upgrades 
> to work
> consistently from those feeds without manually bumping PR in recipes on 
> every
> material change, you need to enable the PR service as described in the
> development manual so that PR increments automatically.
> 
> * Changes on the host system affecting native recipes - less likely to 
> cause
> issues, but worth being aware of. It can happen that adding or removing
> packages on the host system changes the configuration of native recipes 
> without
> automatically triggering a rebuild - a good example is how we allow 
> qemu-
> native to build on systems with and without X11; if you added SDL and 
> X11 to a
> system on which you'd already built qemu-native beforehand, in the 
> absence of
> other changes you'd have to cleansstate or otherwise force a rebuild of 
> qemu-
> native to have a native QEMU that supported graphical output.
> 
> 
> However, with the caveats above, most of the time you can rely upon the 
> build
> system to determine what to do when things change. Of course, yes, if 
> you want
> to just force a recipe to rebuild you have the option of bitbake -c
> cleansstate <recipe> before building it again, but most of the time 
> that's
> going to be more than is needed.
> 
> Cheers,
> Paul

Re: confusion about DEPENDS vs RDEPENDS

[Paul Eggleton]

Hi Lothar,

On Monday 02 September 2013 15:17:28 lothar@denx.de wrote:
> I followed this very interesting thread, and have a question on the last
> statement, just for understanding:
> 
> Are the limitations you are explaining here, the same as mentioned in...
> http://www.yoctoproject.org/docs/current/mega-manual/mega-manual.html
> ...under "Checksums (Signatures)", telling that:
> "Consider a case with in-line Python, for example, where BitBake is not
> able to figure out dependencies."   ? So to speak, are those limitations
> the cases, where bitbake is not able to figure out dependencies
> correctly?

That is to do with variable dependencies so it's not exactly what is being 
discussed earlier in the thread; however if these are not able to be detected 
they can result in signatures not changing when you change the value of one of 
the variables upon which there is a missing dependency and thus items not 
being rebuilt as expected. However, there is some "magic" in 
bitbake/lib/bb/codeparser.py which picks up most of the common cases within 
python code (e.g. usage of d.getVar()), so most of the time this is not a 
concern.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: confusion about DEPENDS vs RDEPENDS

[Paul Eggleton]

Hi Hans,

On Wednesday 28 August 2013 21:22:36 Hans Beckerus wrote:
> On 2013-08-28 6:06, Paul Eggleton wrote:
> > On Wednesday 28 August 2013 17:08:41 Hans Beckérus wrote:
> >> Hi, I am a little bit confused about how to handle these two and what
> >> they are supposed to solve. I have so far never used RDEPENDS but only
> >> DEPENDS.
> > 
> > DEPENDS means a build-time dependency i.e. between recipes, RDEPENDS means
> > a runtime dependency i.e. between packages. It is worth noting though
> > that an explicitly stated RDEPENDS will cause bitbake to actually build
> > the recipe providing the package named in the RDEPENDS value, just at a
> > different time. To explain exactly what each of these do:
> > 
> > * DEPENDS = "b" in recipe "a" will translate to a's do_configure task
> > depending on recipe b's do_populate_sysroot task, so that anything recipe
> > b puts into the sysroot is available for when a configures itself.
> > 
> > * RDEPENDS_${PN} = "b" in recipe "a" will translate to a's do_build task
> > depending on recipe b's do_package_write task, so that the package file b
> > is available when the output for a has been completely built (of course
> > assuming that recipe b produces a package called "b", which it will with
> > the default value of PACKAGES). More importantly it will also ensure that
> > package "a" is marked as depending on "b" in a manner understood by the
> > package manager being used e.g. rpm / opkg / dpkg.
> 
> Thanks a lot! This was definitely more than I got from the description
> of DEPENDS and RDEPENDS in the manual.
> But I probably just read the wrong one ;)

We probably should explain things to that level, I'm fairly sure we don't at 
the moment. I'll talk to Scott to see if we can work something like the above 
into the manual.

> > By default, if recipe "foo" changes and it is mentioned in the "someapp"
> > recipe's DEPENDS, then someapp's do_configure and all tasks that depend
> > upon it will be re-executed next time it is called for i.e. you
> > explicitly build someapp or build an image that contains it or some other
> > recipe that depends upon it. The fact that you are getting the behaviour
> > described suggests that this is either not happening, or more likely it
> > is not having the desired effect; e.g. if internally someapp's build
> > system doesn't drop or invalidate all of its  build output when it is
> > reconfigured then you will get this kind of behaviour. Setting up B (the
> > directory in which a recipe's source code is built) separate to S (the
> > directory in which the recipe's source code has been unpacked to) can
> > help with this since if they are separate, our build system will know it
> > can delete B before re-executing do_compile after do_configure and you'll
> > never have stale build output. Being able to set this up however is
> > highly dependent on the software being built by the individual recipe;
> > some lend themselves to this and others don't.
> 
> Well, I have been struggling before with packages not properly
> supporting different build and source folders so I can definitely relate
> to what you are saying. But, does that mean I actually *have* to do it
> this way for build dependencies to work correctly? 

I wouldn't have thought so, to be honest I'm just positing one situation where 
I can see how this kind of thing might be able to happen. Either do_configure, 
do_compile etc. are not re-executing (and by default they should - *if* the 
build system has a means to know about the change you have made), or they are 
re-executing but the re-execution didn't materially change the output, 
certainly not to the point where it linked against the new library version. It 
would be pretty easy to figure out which of the two happened by looking at task 
logs - either you'd see no extra task logs or you'd see in the latest 
do_compile/do_configure log that nothing much was being done.

> In my case we are talking two simple autotools enabled packages and I
> (naively?) assumed this was not something I had to take care of myself. 

In the ideal case it should not be.

> What strikes me is that you say ""if recipe "foo" changes"", which is
> actually not the case here! What is changed is the actual source code only.
> Is that what is going wrong here? If I change my "foo" recipe version, would
> that be different than to simply fetch/unpack the "foo" package source code?
> Is "someapp" going to become purged differently in such a  scenario?

OK, so just to make sure I understand what changes you are making - are you 
changing configuration/recipes/files pointed to in SRC_URI at all, or just 
modifying the unpacked source in the recipe's work directory?

> What is still somewhat unclear to me is the difference between DEPENDS
> and RDEPENDS in a simple case as this. A simple application needing a
> dynamic library is obviously a subject for DEPENDS but to me that also
> sounds like a typical RDEPENDS?
>
> However, when I build an image and include 'someapp', will 'libfoo.so.x'
> automatically be installed or is that what I need to tell it to do using
> RDEPENDS?

For dynamically linked libraries there is one additional aspect that I didn't 
mention - we have some code in meta/classes/package.bbclass called during 
do_package that looks at the shared libraries that all binaries in a package 
link to, and automatically adds RDEPENDS for these so that the appropriate 
libraries always get installed. The subtlety though is that this happens too 
late for the build system to be able to ensure that the corresponding build-
time requirements of the recipe are always satisfied; thus, it is possible if 
one of the libraries is rebuilding due to some other change and its recipe is 
not in your application recipe's DEPENDS, problems can occur. Bottom line is, 
if your application needs to link to a library, the recipe that builds that 
library must be in your application recipe's DEPENDS. Because of the automatic 
RDEPENDS addition at packaging time you don't need to also add the library to 
RDEPENDS yourself though; the only time you need to add something to RDEPENDS 
is when you know that a runtime dependency exists that the build system can't 
detect - examples would be packages providing shell commands that your 
application relies upon, perl, python and other such module dependencies, etc.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: confusion about DEPENDS vs RDEPENDS

[Hans Beckérus]

Hi Paul,

On Thu, Aug 29, 2013 at 10:58 AM, Paul Eggleton
<paul.eggleton@linux.intel.com> wrote:
> Hi Hans,
>
> On Wednesday 28 August 2013 21:22:36 Hans Beckerus wrote:
>> On 2013-08-28 6:06, Paul Eggleton wrote:
>> > On Wednesday 28 August 2013 17:08:41 Hans Beckérus wrote:
>> >> Hi, I am a little bit confused about how to handle these two and what
>> >> they are supposed to solve. I have so far never used RDEPENDS but only
>> >> DEPENDS.
>> >
>> > DEPENDS means a build-time dependency i.e. between recipes, RDEPENDS means
>> > a runtime dependency i.e. between packages. It is worth noting though
>> > that an explicitly stated RDEPENDS will cause bitbake to actually build
>> > the recipe providing the package named in the RDEPENDS value, just at a
>> > different time. To explain exactly what each of these do:
>> >
>> > * DEPENDS = "b" in recipe "a" will translate to a's do_configure task
>> > depending on recipe b's do_populate_sysroot task, so that anything recipe
>> > b puts into the sysroot is available for when a configures itself.
>> >
>> > * RDEPENDS_${PN} = "b" in recipe "a" will translate to a's do_build task
>> > depending on recipe b's do_package_write task, so that the package file b
>> > is available when the output for a has been completely built (of course
>> > assuming that recipe b produces a package called "b", which it will with
>> > the default value of PACKAGES). More importantly it will also ensure that
>> > package "a" is marked as depending on "b" in a manner understood by the
>> > package manager being used e.g. rpm / opkg / dpkg.
>>
>> Thanks a lot! This was definitely more than I got from the description
>> of DEPENDS and RDEPENDS in the manual.
>> But I probably just read the wrong one ;)
>
> We probably should explain things to that level, I'm fairly sure we don't at
> the moment. I'll talk to Scott to see if we can work something like the above
> into the manual.
>
Great! Your input has been really helpful.

>> > By default, if recipe "foo" changes and it is mentioned in the "someapp"
>> > recipe's DEPENDS, then someapp's do_configure and all tasks that depend
>> > upon it will be re-executed next time it is called for i.e. you
>> > explicitly build someapp or build an image that contains it or some other
>> > recipe that depends upon it. The fact that you are getting the behaviour
>> > described suggests that this is either not happening, or more likely it
>> > is not having the desired effect; e.g. if internally someapp's build
>> > system doesn't drop or invalidate all of its  build output when it is
>> > reconfigured then you will get this kind of behaviour. Setting up B (the
>> > directory in which a recipe's source code is built) separate to S (the
>> > directory in which the recipe's source code has been unpacked to) can
>> > help with this since if they are separate, our build system will know it
>> > can delete B before re-executing do_compile after do_configure and you'll
>> > never have stale build output. Being able to set this up however is
>> > highly dependent on the software being built by the individual recipe;
>> > some lend themselves to this and others don't.
>>
>> Well, I have been struggling before with packages not properly
>> supporting different build and source folders so I can definitely relate
>> to what you are saying. But, does that mean I actually *have* to do it
>> this way for build dependencies to work correctly?
>
> I wouldn't have thought so, to be honest I'm just positing one situation where
> I can see how this kind of thing might be able to happen. Either do_configure,
> do_compile etc. are not re-executing (and by default they should - *if* the
> build system has a means to know about the change you have made), or they are
> re-executing but the re-execution didn't materially change the output,
> certainly not to the point where it linked against the new library version. It
> would be pretty easy to figure out which of the two happened by looking at task
> logs - either you'd see no extra task logs or you'd see in the latest
> do_compile/do_configure log that nothing much was being done.
>
>> In my case we are talking two simple autotools enabled packages and I
>> (naively?) assumed this was not something I had to take care of myself.
>
> In the ideal case it should not be.
>
>> What strikes me is that you say ""if recipe "foo" changes"", which is
>> actually not the case here! What is changed is the actual source code only.
>> Is that what is going wrong here? If I change my "foo" recipe version, would
>> that be different than to simply fetch/unpack the "foo" package source code?
>> Is "someapp" going to become purged differently in such a  scenario?
>
> OK, so just to make sure I understand what changes you are making - are you
> changing configuration/recipes/files pointed to in SRC_URI at all, or just
> modifying the unpacked source in the recipe's work directory?
>
It is the source code in the source repo that has changed. I.e. before
the change the repo for 'foo' contained version 1.0.0 of the library.
After the change the library revision was stepped to 3.0.0 due to it
not being compatible anymore and pushed. Common libtool version
handling that is.

>> What is still somewhat unclear to me is the difference between DEPENDS
>> and RDEPENDS in a simple case as this. A simple application needing a
>> dynamic library is obviously a subject for DEPENDS but to me that also
>> sounds like a typical RDEPENDS?
>>
>> However, when I build an image and include 'someapp', will 'libfoo.so.x'
>> automatically be installed or is that what I need to tell it to do using
>> RDEPENDS?
>
> For dynamically linked libraries there is one additional aspect that I didn't
> mention - we have some code in meta/classes/package.bbclass called during
> do_package that looks at the shared libraries that all binaries in a package
> link to, and automatically adds RDEPENDS for these so that the appropriate
> libraries always get installed. The subtlety though is that this happens too
> late for the build system to be able to ensure that the corresponding build-
> time requirements of the recipe are always satisfied; thus, it is possible if
> one of the libraries is rebuilding due to some other change and its recipe is
> not in your application recipe's DEPENDS, problems can occur. Bottom line is,
> if your application needs to link to a library, the recipe that builds that
> library must be in your application recipe's DEPENDS. Because of the automatic
> RDEPENDS addition at packaging time you don't need to also add the library to
> RDEPENDS yourself though; the only time you need to add something to RDEPENDS
> is when you know that a runtime dependency exists that the build system can't
> detect - examples would be packages providing shell commands that your
> application relies upon, perl, python and other such module dependencies, etc.
>
I see. The picture is getting clearer ;)
As stated before, of course I could simply do a complete clean and get
away with it. But in this case we are dealing with several developers
and it would simplify things a lot if changes like this got populated
to everyone automagically. At least now I know how it is supposed to
work, which is a giant leap forward :)

Thanks again!
Hans

> Cheers,
> Paul
>
> --
>
> Paul Eggleton
> Intel Open Source Technology Centre