Another weird case of PACKAGES_DYNAMIC and task deps, it seems

Another weird case of PACKAGES_DYNAMIC and task deps, it seems

[Paul Sokolovsky]

Hello openembedded-devel,

  Situation (as reported by Thomas Kunze on IRC):
1. Build from scratch
2. Kernel has been built.
3. Bitbake of initramfs-image.
4. Failure with:

| ERROR: Cannot satisfy the following dependencies for initramfs-module-loop:
|        update-modules update-modules update-modules
| ERROR: Cannot satisfy the following dependencies for initramfs-module-nfs:
|        update-modules

initramfs-module-* doesn't Depends: on update-modules directly, but
they Suggest: some kernel-module-* which in turn Depends: on
update-modules.

5. Manual explicit build of update-modules clears those errors.

   My analysis:

kernel.bbclass has:

DEPENDS += "virtual/${TARGET_PREFIX}depmod-${@get_kernelmajorversion('${PV}')} virtual/${TARGET_PREFIX}gcc${KERNEL_CCSUFFIX} update-modules"

But as we now know, that doesn't mean there will be package written
for update-modules, only that it will be "built".

Then, do_spli_package() kinda should set RDEPENDS for each module, but
I remember we already discussed issue that this doesn't really work
for PACKAGES_DYNAMIC ;-(.

So, any ideas how to at least work this around? I recently hit similar
issue when package_ipk.bbclass tried to execute ipkg-build before it
was staged, and that was fixed in natural way:

do_package_write_ipk[depends] = "ipkg-utils-native:do_populate_staging"

Maybe we could use that as a workaround for kernel.bbclass, so we
don't go packing modules, until update-modules' package is written.
The issue, kernel.bbclass doesn't even define proper task for that,
only a prepender, so I have no idea what to put dependency on.


     Richard, any comments?


-- 
Best regards,
 Paul                          mailto:pmiscml@gmail.com

Re: Another weird case of PACKAGES_DYNAMIC and task deps, it seems

[Richard Purdie]

Hi Paul,

On Fri, 2008-01-04 at 03:17 +0200, Paul Sokolovsky wrote:
> Hello openembedded-devel,
> 
>   Situation (as reported by Thomas Kunze on IRC):
> 1. Build from scratch
> 2. Kernel has been built.
> 3. Bitbake of initramfs-image.
> 4. Failure with:
> 
> | ERROR: Cannot satisfy the following dependencies for initramfs-module-loop:
> |        update-modules update-modules update-modules
> | ERROR: Cannot satisfy the following dependencies for initramfs-module-nfs:
> |        update-modules
> 
> initramfs-module-* doesn't Depends: on update-modules directly, but
> they Suggest: some kernel-module-* which in turn Depends: on
> update-modules.

Hmm. Which kernel is this with?


> kernel.bbclass has:
> 
> DEPENDS += "virtual/${TARGET_PREFIX}depmod-${@get_kernelmajorversion('${PV}')} virtual/${TARGET_PREFIX}gcc${KERNEL_CCSUFFIX} update-modules"
> 
> But as we now know, that doesn't mean there will be package written
> for update-modules, only that it will be "built".

Nearly but not quite. rootfs_ipk.bbclass says:

do_rootfs[recrdeptask] += "do_package_write_ipk"

This means that every RDEPENDS, RRECOMMENDS and DEPENDS is followed and
the package_write_ipk for every package found must have run.

I just did a:

"bitbake poky-image-minimal -g; cat task-depends.dot | grep update-modules"

which gave me this list:

"poky-image-minimal.do_rootfs" -> "update-modules.do_populate_staging"
"poky-image-minimal.do_rootfs" -> "update-modules.do_package_write_ipk"
"linux.do_package" -> "update-modules.do_package"
"linux.do_configure" -> "update-modules.do_populate_staging"

and a load of other dependencies which we aren't interested in. The key
line is:

"poky-image-minimal.do_rootfs" -> "update-modules.do_package_write_ipk"

So the question is why doesn't your test case have this (I'm assuming it
doesn't). My best guess is that the kernel .bb file in question alters
DEPENDS and that update-modules isn't really in DEPENDS. Which
kernel .bb file serves the machine in question?

Cheers,

Richard

Re: Another weird case of PACKAGES_DYNAMIC and task deps, it seems

[Paul Sokolovsky]

Hello Richard,

Monday, January 7, 2008, 1:47:23 AM, you wrote:

> Hi Paul,

> On Fri, 2008-01-04 at 03:17 +0200, Paul Sokolovsky wrote:
>> Hello openembedded-devel,
>> 
>>   Situation (as reported by Thomas Kunze on IRC):
>> 1. Build from scratch
>> 2. Kernel has been built.
>> 3. Bitbake of initramfs-image.
>> 4. Failure with:
>> 
>> | ERROR: Cannot satisfy the following dependencies for initramfs-module-loop:
>> |        update-modules update-modules update-modules
>> | ERROR: Cannot satisfy the following dependencies for initramfs-module-nfs:
>> |        update-modules
>> 
>> initramfs-module-* doesn't Depends: on update-modules directly, but
>> they Suggest: some kernel-module-* which in turn Depends: on
>> update-modules.

> Hmm. Which kernel is this with?

  Thomas wanted to build adhoc bootloading initramfs for Collie, so I
assume he built for it.


>> kernel.bbclass has:
>> 
>> DEPENDS += "virtual/${TARGET_PREFIX}depmod-${@get_kernelmajorversion('${PV}')} virtual/${TARGET_PREFIX}gcc${KERNEL_CCSUFFIX} update-modules"
>> 
>> But as we now know, that doesn't mean there will be package written
>> for update-modules, only that it will be "built".

> Nearly but not quite. rootfs_ipk.bbclass says:

> do_rootfs[recrdeptask] += "do_package_write_ipk"

> This means that every RDEPENDS, RRECOMMENDS and DEPENDS is followed and
> the package_write_ipk for every package found must have run.

> I just did a:

> "bitbake poky-image-minimal -g; cat task-depends.dot | grep update-modules"

> which gave me this list:

> "poky-image-minimal.do_rootfs" -> "update-modules.do_populate_staging"
> "poky-image-minimal.do_rootfs" ->
> "update-modules.do_package_write_ipk"
> "linux.do_package" -> "update-modules.do_package"
> "linux.do_configure" -> "update-modules.do_populate_staging"

> and a load of other dependencies which we aren't interested in. The key
> line is:

> "poky-image-minimal.do_rootfs" ->
> "update-modules.do_package_write_ipk"

> So the question is why doesn't your test case have this (I'm assuming it
> doesn't). My best guess is that the kernel .bb file in question alters
> DEPENDS and that update-modules isn't really in DEPENDS. Which
> kernel .bb file serves the machine in question?

  Gotcha! I tried

 MACHINE=h4000 bitbake initramfs-image -g

and it gives the same matches as you quote below. But

 MACHINE=collie bitbake initramfs-image -g

Gives zero matches for update-modules. I don't try to investigate
further now, as you probably know better about it ;-).

> Cheers,

> Richard





-- 
Best regards,
 Paul                            mailto:pmiscml@gmail.com

Re: Another weird case of PACKAGES_DYNAMIC and task deps, it seems

[Richard Purdie]

On Mon, 2008-01-07 at 22:18 +0200, Paul Sokolovsky wrote:
> Monday, January 7, 2008, 1:47:23 AM, you wrote:
> > So the question is why doesn't your test case have this (I'm assuming it
> > doesn't). My best guess is that the kernel .bb file in question alters
> > DEPENDS and that update-modules isn't really in DEPENDS. Which
> > kernel .bb file serves the machine in question?
> 
>   Gotcha! I tried
> 
>  MACHINE=h4000 bitbake initramfs-image -g
> 
> and it gives the same matches as you quote below. But
> 
>  MACHINE=collie bitbake initramfs-image -g
> 
> Gives zero matches for update-modules. I don't try to investigate
> further now, as you probably know better about it ;-).

My money is on:

DEPENDS_collie += "bc-native"

in linux-rp.inc. Try removing that...

Cheers,

Richard

Re: Another weird case of PACKAGES_DYNAMIC and task deps, it seems

[Richard Purdie]

On Mon, 2008-01-07 at 20:44 +0000, Richard Purdie wrote:
> My money is on:
> 
> DEPENDS_collie += "bc-native"
> 
> in linux-rp.inc. Try removing that...

Confirmed and fixed in 00610b176af0e4f4563f78fb71dc1219c777d13e.

Let me stress once again, += and overrides do not do what you expect. It
adds "bc_native" to the variable named "DEPENDS_collie", not "DEPENDS".

DEPENDS_append_collie = " bc-native" 

is a very different thing and corrects this problem...

Cheers,

Richard

Metadata Q/A (was: Another weird case of PACKAGES_DYNAMIC and task deps, it seems)

[Michael 'Mickey' Lauer]

Richard Purdie wrote:
> On Mon, 2008-01-07 at 20:44 +0000, Richard Purdie wrote:
>> My money is on:
>> 
>> DEPENDS_collie += "bc-native"
>> 
>> in linux-rp.inc. Try removing that...

> Confirmed and fixed in 00610b176af0e4f4563f78fb71dc1219c777d13e.

> Let me stress once again, += and overrides do not do what you expect. It
> adds "bc_native" to the variable named "DEPENDS_collie", not "DEPENDS".

> DEPENDS_append_collie = " bc-native" 

> is a very different thing and corrects this problem...

Can we

a) forbid variable assignment to foo_bar in general (when bar is not
in {append, prepend, <override>} ?

b) have an extra Q/A class that scans the metadata for those kinds of
problems?

Regards,

:M:

-- 
Dr. Michael 'Mickey' Lauer | IT-Freelancer | http://www.vanille-media.de

Re: Metadata Q/A (was: Another weird case of PACKAGES_DYNAMIC and task deps, it seems)

[Richard Purdie]

On Mon, 2008-01-07 at 23:52 +0100, Michael 'Mickey' Lauer wrote:
> Richard Purdie wrote:
> > On Mon, 2008-01-07 at 20:44 +0000, Richard Purdie wrote:
> >> My money is on:
> >> 
> >> DEPENDS_collie += "bc-native"
> >> 
> >> in linux-rp.inc. Try removing that...
> 
> > Confirmed and fixed in 00610b176af0e4f4563f78fb71dc1219c777d13e.
> 
> > Let me stress once again, += and overrides do not do what you expect. It
> > adds "bc_native" to the variable named "DEPENDS_collie", not "DEPENDS".
> 
> > DEPENDS_append_collie = " bc-native" 
> 
> > is a very different thing and corrects this problem...
> 
> Can we
> 
> a) forbid variable assignment to foo_bar in general (when bar is not
> in {append, prepend, <override>} ?

Like SRC_URI = "" ?

We have a lot of variables with _ in the names since it generally looks
better than SRC-URI. I think we do need a policy on this though since it
will:

a) Bite us at some point
b) It slows bitbake down (although I've never confirmed that)

I actually have a idea rolling around the back of my head here. All our
overrides are lowercase (I think). A lot of our variable names are upper
case. I'm wondering if we'd get a speedup from allowing only lowercase
overrides since bitbake could tell from inspection that "URI" would
never be an override.

This lets us keep the large number of uppercase variables with _ in the
name and we can conceivably rename the lowercase ones since there are a
lot less of them.

> b) have an extra Q/A class that scans the metadata for those kinds of
> problems?

If and only if we have some rules that tell us what the valid cases are.
We'd probably need an exhaustive list of permitted overrides too.

Cheers,

Richard

Re: Metadata Q/A (was: Another weird case of PACKAGES_DYNAMIC and task deps, it seems)

[Paul Sokolovsky]

Hello Richard,

Tuesday, January 8, 2008, 1:14:16 PM, you wrote:

> On Mon, 2008-01-07 at 23:52 +0100, Michael 'Mickey' Lauer wrote:
>> Richard Purdie wrote:
>> > On Mon, 2008-01-07 at 20:44 +0000, Richard Purdie wrote:
>> >> My money is on:
>> >> 
>> >> DEPENDS_collie += "bc-native"
>> >> 
>> >> in linux-rp.inc. Try removing that...
>> 
>> > Confirmed and fixed in 00610b176af0e4f4563f78fb71dc1219c777d13e.
>> 
>> > Let me stress once again, += and overrides do not do what you expect. It
>> > adds "bc_native" to the variable named "DEPENDS_collie", not "DEPENDS".
>> 
>> > DEPENDS_append_collie = " bc-native" 
>> 
>> > is a very different thing and corrects this problem...
>> 
>> Can we
>> 
>> a) forbid variable assignment to foo_bar in general (when bar is not
>> in {append, prepend, <override>} ?

> Like SRC_URI = "" ?

> We have a lot of variables with _ in the names since it generally looks
> better than SRC-URI. I think we do need a policy on this though since it
> will:

> a) Bite us at some point
> b) It slows bitbake down (although I've never confirmed that)

> I actually have a idea rolling around the back of my head here. All our
> overrides are lowercase (I think). A lot of our variable names are upper
> case. I'm wondering if we'd get a speedup from allowing only lowercase
> overrides since bitbake could tell from inspection that "URI" would
> never be an override.

> This lets us keep the large number of uppercase variables with _ in the
> name and we can conceivably rename the lowercase ones since there are a
> lot less of them.

  Like module_autoload. Has been on my list for a long time. The
problem, it (ab)uses override-like syntax to do something completely
different. It actually tries to emulate dictionary with sh-like
syntax. So, when it says

module_autoload_evdev = "evdev"

, in developed language it would be module_autoload["evdev"] = "evdev".

  So, it's kinda special case, that's why I didn't get around to doing
something about it, but we probably should, because it's inconsistent.

>> b) have an extra Q/A class that scans the metadata for those kinds of
>> problems?

> If and only if we have some rules that tell us what the valid cases are.
> We'd probably need an exhaustive list of permitted overrides too.

  I also think that such tests are not really effective...

> Cheers,

> Richard




> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel


-- 
Best regards,
 Paul                            mailto:pmiscml@gmail.com

Re: Another weird case of PACKAGES_DYNAMIC and task deps, it seems

[Paul Sokolovsky]

Hello Richard,

Monday, January 7, 2008, 11:47:39 PM, you wrote:

> On Mon, 2008-01-07 at 20:44 +0000, Richard Purdie wrote:
>> My money is on:
>> 
>> DEPENDS_collie += "bc-native"
>> 
>> in linux-rp.inc. Try removing that...

> Confirmed and fixed in 00610b176af0e4f4563f78fb71dc1219c777d13e.

> Let me stress once again, += and overrides do not do what you expect. It
> adds "bc_native" to the variable named "DEPENDS_collie", not "DEPENDS".

> DEPENDS_append_collie = " bc-native" 

> is a very different thing and corrects this problem...

  Mea culpa, thanks for fixing it. I know it was it when you posted
that line, but didn't have time to test then.

  But maybe we indeed should do something about it on bitbake level.
If not teach to do the intuitive thing, then at least add adhoc test
with warning for += on UPPER_lower variable. Should be simple but
effective.

> Cheers,

> Richard



-- 
Best regards,
 Paul                            mailto:pmiscml@gmail.com