RDEPENDS of packages created by PACKAGES_DYNAMIC not built before image rootfs

RDEPENDS of packages created by PACKAGES_DYNAMIC not built before image rootfs

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1295 bytes --]

Hi,

pulseaudio has
RDEPENDS_pulseaudio-module-console-kit =+ "consolekit"

and pulseaudio-module-console-kit is dynamic package
PACKAGES_DYNAMIC += "^pulseaudio-lib.* ^pulseaudio-module.* ^libpulse-lib.* ^libpulse-module.* "

If you have image where consolekit is pulled to image only by 
pulseaudio-module-console-kit (e.g. through pulseaudio-server). 
Then consolekit is not built before image do_rootfs is executed and do_rootfs fails.

Easy way to reproduce this is to build e.g. core-image-minimal with 
pulseaudio-module-console-kit or pulseaudio-server added to IMAGE_INSTALL. 
Will fail with:
  * satisfy_dependencies_for: Cannot satisfy the following dependencies for pulseaudio-server:
  *    consolekit *
  * opkg_install_cmd: Cannot install package pulseaudio-server.


Not sure what's proper fix, bitbake probably just checks that right provider for 
pulseaudio-module-console-kit is pulseaudio because of PACKAGES_DYNAMIC, but does not check 
RDEPENDS_pulseaudio-module-console-kit after that.

We can add consolekit to build time deps like it was before, but that's just 
work around and not correct fix.

Also filled as
https://bugzilla.yoctoproject.org/show_bug.cgi?id=3498

Cheers,

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: RDEPENDS of packages created by PACKAGES_DYNAMIC not built before image rootfs

[Otavio Salvador]

On Tue, Nov 27, 2012 at 1:41 PM, Martin Jansa <martin.jansa@gmail.com> wrote:
> pulseaudio has
> RDEPENDS_pulseaudio-module-console-kit =+ "consolekit"
>
> and pulseaudio-module-console-kit is dynamic package
> PACKAGES_DYNAMIC += "^pulseaudio-lib.* ^pulseaudio-module.* ^libpulse-lib.* ^libpulse-module.* "
>
> If you have image where consolekit is pulled to image only by
> pulseaudio-module-console-kit (e.g. through pulseaudio-server).
> Then consolekit is not built before image do_rootfs is executed and do_rootfs fails.
>
> Easy way to reproduce this is to build e.g. core-image-minimal with
> pulseaudio-module-console-kit or pulseaudio-server added to IMAGE_INSTALL.
> Will fail with:
>   * satisfy_dependencies_for: Cannot satisfy the following dependencies for pulseaudio-server:
>   *    consolekit *
>   * opkg_install_cmd: Cannot install package pulseaudio-server.
>
>
> Not sure what's proper fix, bitbake probably just checks that right provider for
> pulseaudio-module-console-kit is pulseaudio because of PACKAGES_DYNAMIC, but does not check
> RDEPENDS_pulseaudio-module-console-kit after that.
>
> We can add consolekit to build time deps like it was before, but that's just
> work around and not correct fix.
>
> Also filled as
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=3498

Yes; I have this issue in an internal recipe as well. I workarounded
it adding them in the depends.

-- 
Otavio Salvador                             O.S. Systems
E-mail: otavio@ossystems.com.br  http://www.ossystems.com.br
Mobile: +55 53 9981-7854              http://projetos.ossystems.com.br

Re: RDEPENDS of packages created by PACKAGES_DYNAMIC not built before image rootfs

[Richard Purdie]

On Tue, 2012-11-27 at 16:41 +0100, Martin Jansa wrote:
> Hi,
> 
> pulseaudio has
> RDEPENDS_pulseaudio-module-console-kit =+ "consolekit"
> 
> and pulseaudio-module-console-kit is dynamic package
> PACKAGES_DYNAMIC += "^pulseaudio-lib.* ^pulseaudio-module.* ^libpulse-lib.* ^libpulse-module.* "
> 
> If you have image where consolekit is pulled to image only by 
> pulseaudio-module-console-kit (e.g. through pulseaudio-server). 
> Then consolekit is not built before image do_rootfs is executed and do_rootfs fails.
> 
> Easy way to reproduce this is to build e.g. core-image-minimal with 
> pulseaudio-module-console-kit or pulseaudio-server added to IMAGE_INSTALL. 
> Will fail with:
>   * satisfy_dependencies_for: Cannot satisfy the following dependencies for pulseaudio-server:
>   *    consolekit *
>   * opkg_install_cmd: Cannot install package pulseaudio-server.
> 
> 
> Not sure what's proper fix, bitbake probably just checks that right provider for 
> pulseaudio-module-console-kit is pulseaudio because of PACKAGES_DYNAMIC, but does not check 
> RDEPENDS_pulseaudio-module-console-kit after that.
> 
> We can add consolekit to build time deps like it was before, but that's just 
> work around and not correct fix.
> 
> Also filled as
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=3498

Well, you can't have this both ways :)

Bitbake doesn't know which modules this package will produce since its
using PACKAGES_DYNAMIC. It does its best effort to try and find DEPENDS
but it can't take the regexps and come up with
"pulseaudio-module-console-kit", hence it doesn't see the RDEPENDS.
There isn't any generic way we can teach it to do that either. If it
did, it would simply always build consolekit which you say you don't
want so its not relevant anyway.

So basically, you need to decide whether you want consolekit enabled or
not and build accordingly, there is nothing you can do within bitbake to
"mind read" what the image is going to want...

Cheers,

Richard

Re: RDEPENDS of packages created by PACKAGES_DYNAMIC not built before image rootfs

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 2697 bytes --]

On Tue, Nov 27, 2012 at 08:11:42PM +0000, Richard Purdie wrote:
> On Tue, 2012-11-27 at 16:41 +0100, Martin Jansa wrote:
> > Hi,
> > 
> > pulseaudio has
> > RDEPENDS_pulseaudio-module-console-kit =+ "consolekit"
> > 
> > and pulseaudio-module-console-kit is dynamic package
> > PACKAGES_DYNAMIC += "^pulseaudio-lib.* ^pulseaudio-module.* ^libpulse-lib.* ^libpulse-module.* "
> > 
> > If you have image where consolekit is pulled to image only by 
> > pulseaudio-module-console-kit (e.g. through pulseaudio-server). 
> > Then consolekit is not built before image do_rootfs is executed and do_rootfs fails.
> > 
> > Easy way to reproduce this is to build e.g. core-image-minimal with 
> > pulseaudio-module-console-kit or pulseaudio-server added to IMAGE_INSTALL. 
> > Will fail with:
> >   * satisfy_dependencies_for: Cannot satisfy the following dependencies for pulseaudio-server:
> >   *    consolekit *
> >   * opkg_install_cmd: Cannot install package pulseaudio-server.
> > 
> > 
> > Not sure what's proper fix, bitbake probably just checks that right provider for 
> > pulseaudio-module-console-kit is pulseaudio because of PACKAGES_DYNAMIC, but does not check 
> > RDEPENDS_pulseaudio-module-console-kit after that.
> > 
> > We can add consolekit to build time deps like it was before, but that's just 
> > work around and not correct fix.
> > 
> > Also filled as
> > https://bugzilla.yoctoproject.org/show_bug.cgi?id=3498
> 
> Well, you can't have this both ways :)

IMHO Only one way is correct and that would be enough :).

> Bitbake doesn't know which modules this package will produce since its
> using PACKAGES_DYNAMIC. It does its best effort to try and find DEPENDS
> but it can't take the regexps and come up with
> "pulseaudio-module-console-kit", hence it doesn't see the RDEPENDS.
> There isn't any generic way we can teach it to do that either. If it
> did, it would simply always build consolekit which you say you don't
> want so its not relevant anyway.
> 
> So basically, you need to decide whether you want consolekit enabled or
> not and build accordingly, there is nothing you can do within bitbake to
> "mind read" what the image is going to want...

No, you know that you will need pulseaudio-module-console-kit because
pulseaudio-server RDEPENDs on it (not on any pulseaudio-module-* but
pulseaudio-module-console-kit exactly). So it's not really mind reading.

If it's not possible with current code-base then we should document it
and add some work around in pulseaudio (I'm fine with building
consolekit even when it's not used later).

Cheers,

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: RDEPENDS of packages created by PACKAGES_DYNAMIC not built before image rootfs

[Mark Hatle]

On 11/27/12 2:23 PM, Martin Jansa wrote:
> On Tue, Nov 27, 2012 at 08:11:42PM +0000, Richard Purdie wrote:
>> On Tue, 2012-11-27 at 16:41 +0100, Martin Jansa wrote:
>>> Hi,
>>>
>>> pulseaudio has
>>> RDEPENDS_pulseaudio-module-console-kit =+ "consolekit"
>>>
>>> and pulseaudio-module-console-kit is dynamic package
>>> PACKAGES_DYNAMIC += "^pulseaudio-lib.* ^pulseaudio-module.* ^libpulse-lib.* ^libpulse-module.* "
>>>
>>> If you have image where consolekit is pulled to image only by
>>> pulseaudio-module-console-kit (e.g. through pulseaudio-server).
>>> Then consolekit is not built before image do_rootfs is executed and do_rootfs fails.
>>>
>>> Easy way to reproduce this is to build e.g. core-image-minimal with
>>> pulseaudio-module-console-kit or pulseaudio-server added to IMAGE_INSTALL.
>>> Will fail with:
>>>    * satisfy_dependencies_for: Cannot satisfy the following dependencies for pulseaudio-server:
>>>    *    consolekit *
>>>    * opkg_install_cmd: Cannot install package pulseaudio-server.
>>>
>>>
>>> Not sure what's proper fix, bitbake probably just checks that right provider for
>>> pulseaudio-module-console-kit is pulseaudio because of PACKAGES_DYNAMIC, but does not check
>>> RDEPENDS_pulseaudio-module-console-kit after that.
>>>
>>> We can add consolekit to build time deps like it was before, but that's just
>>> work around and not correct fix.
>>>
>>> Also filled as
>>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=3498
>>
>> Well, you can't have this both ways :)
>
> IMHO Only one way is correct and that would be enough :).
>
>> Bitbake doesn't know which modules this package will produce since its
>> using PACKAGES_DYNAMIC. It does its best effort to try and find DEPENDS
>> but it can't take the regexps and come up with
>> "pulseaudio-module-console-kit", hence it doesn't see the RDEPENDS.
>> There isn't any generic way we can teach it to do that either. If it
>> did, it would simply always build consolekit which you say you don't
>> want so its not relevant anyway.
>>
>> So basically, you need to decide whether you want consolekit enabled or
>> not and build accordingly, there is nothing you can do within bitbake to
>> "mind read" what the image is going to want...

For the resolver, it has to use the RPROVIDES, PACKAGES and PACKAGES_DYNAMIC...

> No, you know that you will need pulseaudio-module-console-kit because
> pulseaudio-server RDEPENDs on it (not on any pulseaudio-module-* but
> pulseaudio-module-console-kit exactly). So it's not really mind reading.

But after the recipe has been built, the required item from PACKAGES_DYNAMIC 
should now be in PACKAGES... and if it doesn't end up in the PACKAGES list, it 
would be a good idea to at a minimum trigger a warning...  The catch is, is 
there any way to tell if something from PACKAGES_DYNAMIC was used..

--Mark

> If it's not possible with current code-base then we should document it
> and add some work around in pulseaudio (I'm fine with building
> consolekit even when it's not used later).
>
> Cheers,
>
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>

Re: RDEPENDS of packages created by PACKAGES_DYNAMIC not built before image rootfs

[Richard Purdie]

On Tue, 2012-11-27 at 15:07 -0600, Mark Hatle wrote:
> On 11/27/12 2:23 PM, Martin Jansa wrote:
> >> Bitbake doesn't know which modules this package will produce since its
> >> using PACKAGES_DYNAMIC. It does its best effort to try and find DEPENDS
> >> but it can't take the regexps and come up with
> >> "pulseaudio-module-console-kit", hence it doesn't see the RDEPENDS.
> >> There isn't any generic way we can teach it to do that either. If it
> >> did, it would simply always build consolekit which you say you don't
> >> want so its not relevant anyway.
> >>
> >> So basically, you need to decide whether you want consolekit enabled or
> >> not and build accordingly, there is nothing you can do within bitbake to
> >> "mind read" what the image is going to want...
> 
> For the resolver, it has to use the RPROVIDES, PACKAGES and PACKAGES_DYNAMIC...

and it does to the limits that are reasonable...

> > No, you know that you will need pulseaudio-module-console-kit because
> > pulseaudio-server RDEPENDs on it (not on any pulseaudio-module-* but
> > pulseaudio-module-console-kit exactly). So it's not really mind reading.
> 
> But after the recipe has been built, the required item from PACKAGES_DYNAMIC 
> should now be in PACKAGES... and if it doesn't end up in the PACKAGES list, it 
> would be a good idea to at a minimum trigger a warning...  The catch is, is 
> there any way to tell if something from PACKAGES_DYNAMIC was used..

"After it is built" is no good to bitbake. It has to know what its doing
without building things.

All Bitbake can see is that there is some regexp in PACKAGES_DYNAMIC.

Yes, Bitbake could take all the keys in the datastore, expand each one
(since it could be constructed using another variable name) and find the
ones starting with RDEPENDS, then apply the rexexp to each key and see
if it matches. We one tried something like this and it is extremely
slow, parsing would take a massive speed hit.

The rule is therefore in corner cases like this (which are rather
unusual), you add something to DEPENDS and be done with it.

It is not a reasonable expectation for bitbake to figure this kind of
thing out.

I agree that if it builds things and discovers some kind of missing
dependency, that would be good to tell the user about.

Cheers,

Richard

Re: RDEPENDS of packages created by PACKAGES_DYNAMIC not built before image rootfs

[Richard Purdie]

On Tue, 2012-11-27 at 21:23 +0100, Martin Jansa wrote:
> On Tue, Nov 27, 2012 at 08:11:42PM +0000, Richard Purdie wrote:
> > On Tue, 2012-11-27 at 16:41 +0100, Martin Jansa wrote:
> > So basically, you need to decide whether you want consolekit enabled or
> > not and build accordingly, there is nothing you can do within bitbake to
> > "mind read" what the image is going to want...
> 
> No, you know that you will need pulseaudio-module-console-kit because
> pulseaudio-server RDEPENDs on it (not on any pulseaudio-module-* but
> pulseaudio-module-console-kit exactly). So it's not really mind reading.

Bitbake sees "pulseaudio-module-console-kit" and is has to ask "Which
recipe provides this". To answer that question, it looks first at
PACKAGES and then at PACKAGES_DYNAMIC. In this case it will find a match
for PACKAGES_DYNAMIC but bitbake considers this a fuzzy match.

We only match dynamic packages right at the end of processing and they
are "fuzzy". Basically, they are just a sanity check in bitbake to
ensure there is some chance of the thing getting built.

I can see in this case that a dependency exists that BitBake could use
to enhance its knowledge, the trouble is that is would have to first
decide which provider is providing PACKAGES_DYNAMIC, and then search
that package for extra RDEPENDS. In this case its "easier" since its all
the same package but with our generic providers mechanism, this could
just as easily be split between two recipes which makes the problem at
lot more complex and the code has to be written for the general case.

> If it's not possible with current code-base then we should document it
> and add some work around in pulseaudio (I'm fine with building
> consolekit even when it's not used later).

I suspect we'll just have to add this. I agree it would be possible to
try and figure out this corner case but whether its worth the complexity
and extra parse time I'm not sure. The risk of adding further bugs
trying to add this functionality would worry me, as would the extra
complexity and the time taken to do it...

As such I'd probably mark the bug report "LATER" and suggest working
around this. If we were hitting this on many recipes, fine but we're not
as far as I know.

Cheers,

Richard

Re: RDEPENDS of packages created by PACKAGES_DYNAMIC not built before image rootfs

[Mark Hatle]

On 11/27/12 11:26 PM, Richard Purdie wrote:
> On Tue, 2012-11-27 at 15:07 -0600, Mark Hatle wrote:
>> On 11/27/12 2:23 PM, Martin Jansa wrote:
>>>> Bitbake doesn't know which modules this package will produce since its
>>>> using PACKAGES_DYNAMIC. It does its best effort to try and find DEPENDS
>>>> but it can't take the regexps and come up with
>>>> "pulseaudio-module-console-kit", hence it doesn't see the RDEPENDS.
>>>> There isn't any generic way we can teach it to do that either. If it
>>>> did, it would simply always build consolekit which you say you don't
>>>> want so its not relevant anyway.
>>>>
>>>> So basically, you need to decide whether you want consolekit enabled or
>>>> not and build accordingly, there is nothing you can do within bitbake to
>>>> "mind read" what the image is going to want...
>>
>> For the resolver, it has to use the RPROVIDES, PACKAGES and PACKAGES_DYNAMIC...
>
> and it does to the limits that are reasonable...
>
>>> No, you know that you will need pulseaudio-module-console-kit because
>>> pulseaudio-server RDEPENDs on it (not on any pulseaudio-module-* but
>>> pulseaudio-module-console-kit exactly). So it's not really mind reading.
>>
>> But after the recipe has been built, the required item from PACKAGES_DYNAMIC
>> should now be in PACKAGES... and if it doesn't end up in the PACKAGES list, it
>> would be a good idea to at a minimum trigger a warning...  The catch is, is
>> there any way to tell if something from PACKAGES_DYNAMIC was used..
>
> "After it is built" is no good to bitbake. It has to know what its doing
> without building things.
>
> All Bitbake can see is that there is some regexp in PACKAGES_DYNAMIC.
>
> Yes, Bitbake could take all the keys in the datastore, expand each one
> (since it could be constructed using another variable name) and find the
> ones starting with RDEPENDS, then apply the rexexp to each key and see
> if it matches. We one tried something like this and it is extremely
> slow, parsing would take a massive speed hit.

I was more thinking in the oe-core recipe QA stuff..  if there was a way to know 
that the following items were matched in PACKAGES_DYNAMIC for this item, and 
then check to see if they ended up in PACKAGES.  Then warn/error as appropriate 
if they don't match.

> The rule is therefore in corner cases like this (which are rather
> unusual), you add something to DEPENDS and be done with it.
>
> It is not a reasonable expectation for bitbake to figure this kind of
> thing out.

Ya, I wasn't advocating a change in bitbake for the detection, just a way to 
pass the data to the recipe and do it at that point.

> I agree that if it builds things and discovers some kind of missing
> dependency, that would be good to tell the user about.

--Mark

> Cheers,
>
> Richard
>
>
>