RFC: Maintain backwards compatibility or not for module-base.bbclass

RFC: Maintain backwards compatibility or not for module-base.bbclass

[Peter Kjellerstedt]

Background: Back in September, Richard made a commit to 
linux-libc-headers.inc describing why one should not fork the 
linux-libc-headers recipe:

http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=babbf7a46acaefd9b36031483cafce053f607e66

As a result I created a local bbclass for our layer called 
kernel-headers, which provides a recipe with kernel headers 
that match the actually used version of the Linux kernel.
This was needed for packages that need to access hardware 
specific features that are not present in the generic kernel 
headers provided by linux-libc-headers.

My intention for this class was that it should be generic 
enough to be able to upstream it to OE Core.

Now, the other day a colleague of mine had a build failure due 
to this class. It turned out that even though the class adds a 
dependency on virtual/kernel and then uses the files that are 
installed to ${STAGING_KERNEL_DIR} when running oe_runmake 
headers_install, the command could fail because the 
${STAGING_KERNEL_DIR}/scripts was not populated. After asking 
Richard about this, I learned that this is due to problems 
with the sstate cache and not knowing whether a 32 bit host or 
a 64 bit host was used to generate the files. Thus I also 
learned that the scripts are actually built as a result of 
building modules.

Since I did not want my class to depend on modules having been 
built, I looked into modules.bbclass and modules-base.bbclass. 
There I found the function do_make_scripts() which is 
responsible for building  the kernel scripts. However, the 
current setup doesn't lend itself very well to use the 
modules-base.bbclass for something other than modules.

My idea then was to break this part out into a separate class, 
kernel-scripts, which I did. You can find both the 
kernel-scripts and kernel-headers classes here: 

http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=pkj/kernel-headers

When I showed this to Richard he noted that my change was not 
backwards compatible (as I no longer provide the 
do_make_scripts() function from the module-base.bbclass). 
However, there is nothing besides module.bbclass in OE Core 
and meta-oe that use the module-base.bbclass.

Anyway, I made a modified version that does maintain backwards 
compatibility for module-base.bbclass here:

http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=pkj/kernel-headers-backwards-compatible

This time Richard complained about the extra class 
(kernel-scripts-base.bbclass), and noted that there was no 
way to win... He then suggested that I take the question of 
whether we need to maintain backwards compatibility for 
modules-base.bbclass to the mailing list.

So, here I am now. I do not know who else use the 
do_make_scripts() function from module-base.bbclass and in what 
way, and whether restructuring the functionality into the new 
kernel-scripts.bbclass without maintaining backwards 
compatibility would be a problem. If you know anything about 
this, please let me know.

Any comments appreciated.

//Peter

Re: RFC: Maintain backwards compatibility or not for module-base.bbclass

[Bruce Ashfield]

On Thu, Jan 16, 2014 at 8:58 AM, Peter Kjellerstedt
<peter.kjellerstedt@axis.com> wrote:
> Background: Back in September, Richard made a commit to
> linux-libc-headers.inc describing why one should not fork the
> linux-libc-headers recipe:
>
> http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=babbf7a46acaefd9b36031483cafce053f607e66
>
> As a result I created a local bbclass for our layer called
> kernel-headers, which provides a recipe with kernel headers
> that match the actually used version of the Linux kernel.
> This was needed for packages that need to access hardware
> specific features that are not present in the generic kernel
> headers provided by linux-libc-headers.
>
> My intention for this class was that it should be generic
> enough to be able to upstream it to OE Core.
>
> Now, the other day a colleague of mine had a build failure due
> to this class. It turned out that even though the class adds a
> dependency on virtual/kernel and then uses the files that are
> installed to ${STAGING_KERNEL_DIR} when running oe_runmake
> headers_install, the command could fail because the
> ${STAGING_KERNEL_DIR}/scripts was not populated. After asking
> Richard about this, I learned that this is due to problems
> with the sstate cache and not knowing whether a 32 bit host or
> a 64 bit host was used to generate the files. Thus I also
> learned that the scripts are actually built as a result of
> building modules.
>
> Since I did not want my class to depend on modules having been
> built, I looked into modules.bbclass and modules-base.bbclass.
> There I found the function do_make_scripts() which is
> responsible for building  the kernel scripts. However, the
> current setup doesn't lend itself very well to use the
> modules-base.bbclass for something other than modules.
>
> My idea then was to break this part out into a separate class,
> kernel-scripts, which I did. You can find both the
> kernel-scripts and kernel-headers classes here:
>
> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=pkj/kernel-headers
>
> When I showed this to Richard he noted that my change was not
> backwards compatible (as I no longer provide the
> do_make_scripts() function from the module-base.bbclass).
> However, there is nothing besides module.bbclass in OE Core
> and meta-oe that use the module-base.bbclass.
>
> Anyway, I made a modified version that does maintain backwards
> compatibility for module-base.bbclass here:
>
> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=pkj/kernel-headers-backwards-compatible
>
> This time Richard complained about the extra class
> (kernel-scripts-base.bbclass), and noted that there was no
> way to win... He then suggested that I take the question of
> whether we need to maintain backwards compatibility for
> modules-base.bbclass to the mailing list.

FWIW: I agree that have to many small, single purpose
kernel-*.bbclasses is a pain,
since it provides granularity, but more opportunity for varying behaviour during
kernel builds.

I have uses of the module-base.bbclass and an expectation that it will generate
the scripts, largely around the SDK and some custom kernel recipes. So they
only inherit module.bbclass, and would be impacted if that functionality was
changed to require another inherit.

Speaking of that, we through something like this late last year with
automatically
restoring the scripts into the sysroot, which ended up being reverted:

see b2c948d56241ff7cdea2e9e68b740f305c72f5ca in oe-core

At least the module (and your scripts) class avoid the sstate problems and
compiler dependencies that we hit with that solution.

What are the alternatives to more classes, isn't this something that could be
a .inc routine ? And modules simply includes it, and you can do the same ..
but I suppose a .inc versus a class inherit is largely semantics in
the difference.

bottom line, my rambling says that backwards compatibility matters here, and
that if we can avoid a new class, that would also be a good thing.

Cheers,

Bruce

>
> So, here I am now. I do not know who else use the
> do_make_scripts() function from module-base.bbclass and in what
> way, and whether restructuring the functionality into the new
> kernel-scripts.bbclass without maintaining backwards
> compatibility would be a problem. If you know anything about
> this, please let me know.



>
> Any comments appreciated.
>
> //Peter
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core



-- 
"Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end"

Re: RFC: Maintain backwards compatibility or not for module-base.bbclass

[Peter Kjellerstedt]

I will aggregate my responses to Koen, Bruce and Phil below.

> -----Original Message-----
> From: Koen Kooi [mailto:koen@dominion.thruhere.net]
> Sent: den 16 januari 2014 20:19
> To: Peter Kjellerstedt
> Cc: OE Core (openembedded-core@lists.openembedded.org); Phil Blundell
> Subject: Re: [OE-core] RFC: Maintain backwards compatibility or not for
> module-base.bbclass
> 
> Op 16 jan. 2014, om 14:58 heeft Peter Kjellerstedt
> <peter.kjellerstedt@axis.com> het volgende geschreven:
> 
> > Background: Back in September, Richard made a commit to
> > linux-libc-headers.inc describing why one should not fork the
> > linux-libc-headers recipe:
> >
> > http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=babbf7a46acaefd9b36031483cafce053f607e66
> >
> > As a result I created a local bbclass for our layer called
> > kernel-headers, which provides a recipe with kernel headers
> > that match the actually used version of the Linux kernel.
> > This was needed for packages that need to access hardware
> > specific features that are not present in the generic kernel
> > headers provided by linux-libc-headers.
> >
> > My intention for this class was that it should be generic
> > enough to be able to upstream it to OE Core.
> >
> > Now, the other day a colleague of mine had a build failure due
> > to this class. It turned out that even though the class adds a
> > dependency on virtual/kernel and then uses the files that are
> > installed to ${STAGING_KERNEL_DIR} when running oe_runmake
> > headers_install, the command could fail because the
> > ${STAGING_KERNEL_DIR}/scripts was not populated. After asking
> > Richard about this, I learned that this is due to problems
> > with the sstate cache and not knowing whether a 32 bit host or
> > a 64 bit host was used to generate the files. Thus I also
> > learned that the scripts are actually built as a result of
> > building modules.
> >
> > Since I did not want my class to depend on modules having been
> > built, I looked into modules.bbclass and modules-base.bbclass.
> > There I found the function do_make_scripts() which is
> > responsible for building  the kernel scripts. However, the
> > current setup doesn't lend itself very well to use the
> > modules-base.bbclass for something other than modules.
> >
> > My idea then was to break this part out into a separate class,
> > kernel-scripts, which I did. You can find both the
> > kernel-scripts and kernel-headers classes here:
> >
> > http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=pkj/kernel-headers
> >
> > When I showed this to Richard he noted that my change was not
> > backwards compatible (as I no longer provide the
> > do_make_scripts() function from the module-base.bbclass).
> > However, there is nothing besides module.bbclass in OE Core
> > and meta-oe that use the module-base.bbclass.
> 
> Can't do_make_script run as an sstate post function?

I think that is exactly what the reverted change Bruce mentions 
below did.

> regards,
> 
> Koen

> -----Original Message-----
> From: Bruce Ashfield [mailto:bruce.ashfield@gmail.com]
> Sent: den 16 januari 2014 19:41
> To: Peter Kjellerstedt
> Cc: OE Core (openembedded-core@lists.openembedded.org); Phil Blundell
> Subject: Re: [OE-core] RFC: Maintain backwards compatibility or not for
> module-base.bbclass
> 
> On Thu, Jan 16, 2014 at 8:58 AM, Peter Kjellerstedt
> <peter.kjellerstedt@axis.com> wrote:
> > Anyway, I made a modified version that does maintain backwards
> > compatibility for module-base.bbclass here:
> >
> > http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=pkj/kernel-headers-backwards-compatible
> >
> > This time Richard complained about the extra class
> > (kernel-scripts-base.bbclass), and noted that there was no
> > way to win... He then suggested that I take the question of
> > whether we need to maintain backwards compatibility for
> > modules-base.bbclass to the mailing list.
> 
> FWIW: I agree that have to many small, single purpose
> kernel-*.bbclasses is a pain, since it provides granularity, but 
> more opportunity for varying behaviour during kernel builds.
> 
> I have uses of the module-base.bbclass and an expectation that it 
> will generate the scripts, largely around the SDK and some custom 
> kernel recipes. So they only inherit module.bbclass, and would be 
> impacted if that functionality was changed to require another inherit.

As long as they inherit module (and not add task dependencies on 
do_make_scripts) they should not be affected. It is only if you 
inherit module-base directly and expect the do_make_scripts() 
function to exist that you are affected.

> Speaking of that, we through something like this late last year 
> with automatically restoring the scripts into the sysroot, which 
> ended up being reverted:
> 
> see b2c948d56241ff7cdea2e9e68b740f305c72f5ca in oe-core
> 
> At least the module (and your scripts) class avoid the sstate 
> problems and compiler dependencies that we hit with that solution.

Interesting. It looks like it would have been a good solution 
if it had worked... 

> What are the alternatives to more classes, isn't this something 
> that could be a .inc routine ? And modules simply includes it, and 
> you can do the same .. but I suppose a .inc versus a class inherit 
> is largely semantics in the difference.

Tbh, I think keeping it as a class makes it more straight forward.
And class or include file is merely semantics.

> bottom line, my rambling says that backwards compatibility matters 
> here, and that if we can avoid a new class, that would also be a 
> good thing.

But I think that it should be easy enough for any external party 
to adopt to the new class if they do indeed have a direct 
dependency on module-base and expecting to find do_make_scripts() 
there. It is not like we would change it in the middle of Dora 
anyway. Thus I advocate for the change above on the kernel-headers 
branch that does not maintain backwards compatibility.

> Cheers,
> 
> Bruce

> -----Original Message-----
> From: openembedded-core-bounces@lists.openembedded.org
> [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf Of
> Phil Blundell
> Sent: den 16 januari 2014 22:39
> To: Peter Kjellerstedt
> Cc: OE Core (openembedded-core@lists.openembedded.org)
> Subject: Re: [OE-core] RFC: Maintain backwards compatibility or not for
> module-base.bbclass
> 
> On Thu, 2014-01-16 at 14:58 +0100, Peter Kjellerstedt wrote:
> > So, here I am now. I do not know who else use the
> > do_make_scripts() function from module-base.bbclass and in what
> > way, and whether restructuring the functionality into the new
> > kernel-scripts.bbclass without maintaining backwards
> > compatibility would be a problem. If you know anything about
> > this, please let me know.
> 
> I'm not entirely clear why you couldn't maintain compatibility 
> by moving the task in question to a new class but keeping its 
> name the same (i.e. refrain from renaming "do_make_scripts" to 
> "do_kernel_scripts") and having module-base.bbclass simply 
> inherit the newly-added class.  That seems like it ought to be 
> fairly straightforward and uncontroversial.

Well, that is basically what my backwards compatible branch above 
does. However, it introduces two new classes since it needs to 
maintain the split between where the addtask is done and the 
actual function.

> That said, though, I don't think there is any pressing need to 
> maintain backwards compatibility around module-base.bbclass.  In 
> fact, I think it would probably be fine for module-base.bbclass 
> to simply go away altogether and have its functionality subsumed 
> into module.bbclass; the split between those two classes is 
> mostly a relic of yesteryear and I can't think of any good 
> purpose that it serves nowadays.  So I would be happy enough to 
> see that (and the tangly mess that is the kernel classes in 
> general) cleaned up irrespective of what happens with
> do_make_scripts.

I won't argue with that. However, I do not feel that I am 
comfortable enough about these classes to do it. I had a quick 
look around in OE core, and I believe you are right. The number 
of references to "module-base" were limited, and in most cases 
it should just be a matter of replacing them with "module".

> p.

//Peter

Re: RFC: Maintain backwards compatibility or not for module-base.bbclass

[Bruce Ashfield]

On Fri, Jan 17, 2014 at 8:53 AM, Peter Kjellerstedt
<peter.kjellerstedt@axis.com> wrote:
> I will aggregate my responses to Koen, Bruce and Phil below.
>
>> -----Original Message-----
>> From: Koen Kooi [mailto:koen@dominion.thruhere.net]
>> Sent: den 16 januari 2014 20:19
>> To: Peter Kjellerstedt

[snip]

>>
>> On Thu, Jan 16, 2014 at 8:58 AM, Peter Kjellerstedt
>> <peter.kjellerstedt@axis.com> wrote:
>> > Anyway, I made a modified version that does maintain backwards
>> > compatibility for module-base.bbclass here:
>> >
>> > http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=pkj/kernel-headers-backwards-compatible
>> >
>> > This time Richard complained about the extra class
>> > (kernel-scripts-base.bbclass), and noted that there was no
>> > way to win... He then suggested that I take the question of
>> > whether we need to maintain backwards compatibility for
>> > modules-base.bbclass to the mailing list.
>>
>> FWIW: I agree that have to many small, single purpose
>> kernel-*.bbclasses is a pain, since it provides granularity, but
>> more opportunity for varying behaviour during kernel builds.
>>
>> I have uses of the module-base.bbclass and an expectation that it
>> will generate the scripts, largely around the SDK and some custom
>> kernel recipes. So they only inherit module.bbclass, and would be
>> impacted if that functionality was changed to require another inherit.
>
> As long as they inherit module (and not add task dependencies on
> do_make_scripts) they should not be affected. It is only if you
> inherit module-base directly and expect the do_make_scripts()
> function to exist that you are affected.

I've seen some direct inherits of module-base, and even one that
referenced the task (to ensure that recordmcount and friends
were in place before using them).

>
>> Speaking of that, we through something like this late last year
>> with automatically restoring the scripts into the sysroot, which
>> ended up being reverted:
>>
>> see b2c948d56241ff7cdea2e9e68b740f305c72f5ca in oe-core
>>
>> At least the module (and your scripts) class avoid the sstate
>> problems and compiler dependencies that we hit with that solution.
>
> Interesting. It looks like it would have been a good solution
> if it had worked...

It worked :) But it introduced complexity when building from sstate,
since the compiler was required to restore them, which meant that
the sstate builds had to depend on gcc, which sort of defeats the
purpose of a quick sstate startup and build .. so we decided to abandon
it and educate people to include module-base.

>
>> What are the alternatives to more classes, isn't this something
>> that could be a .inc routine ? And modules simply includes it, and
>> you can do the same .. but I suppose a .inc versus a class inherit
>> is largely semantics in the difference.
>
> Tbh, I think keeping it as a class makes it more straight forward.
> And class or include file is merely semantics.

Agreed. It really wouldn't help.

>
>> bottom line, my rambling says that backwards compatibility matters
>> here, and that if we can avoid a new class, that would also be a
>> good thing.
>
> But I think that it should be easy enough for any external party
> to adopt to the new class if they do indeed have a direct
> dependency on module-base and expecting to find do_make_scripts()
> there. It is not like we would change it in the middle of Dora
> anyway. Thus I advocate for the change above on the kernel-headers
> branch that does not maintain backwards compatibility.

I think we all agree that it really isn't a hard thing to switch (just as in
the examples I sited, it wouldn't be that hard to fix them), it comes down
to more of a perception thing about how often a layer needs to be adapted
to a changing core structure. Developers understand changing interfaces,
customers and larger companies .. not so much (but I'm stating what we
all know).

So in the end, the question is the extra class "clutter" versus compatibility.
I lean towards the extra class with the compatibility maintained .. but I'm
not adamant about the issue :)

Cheers,

Bruce

>
>> Cheers,
>>
>> Bruce
>
>> -----Original Message-----
>> From: openembedded-core-bounces@lists.openembedded.org
>> [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf Of
>> Phil Blundell
>> Sent: den 16 januari 2014 22:39
>> To: Peter Kjellerstedt
>> Cc: OE Core (openembedded-core@lists.openembedded.org)
>> Subject: Re: [OE-core] RFC: Maintain backwards compatibility or not for
>> module-base.bbclass
>>
>> On Thu, 2014-01-16 at 14:58 +0100, Peter Kjellerstedt wrote:
>> > So, here I am now. I do not know who else use the
>> > do_make_scripts() function from module-base.bbclass and in what
>> > way, and whether restructuring the functionality into the new
>> > kernel-scripts.bbclass without maintaining backwards
>> > compatibility would be a problem. If you know anything about
>> > this, please let me know.
>>
>> I'm not entirely clear why you couldn't maintain compatibility
>> by moving the task in question to a new class but keeping its
>> name the same (i.e. refrain from renaming "do_make_scripts" to
>> "do_kernel_scripts") and having module-base.bbclass simply
>> inherit the newly-added class.  That seems like it ought to be
>> fairly straightforward and uncontroversial.
>
> Well, that is basically what my backwards compatible branch above
> does. However, it introduces two new classes since it needs to
> maintain the split between where the addtask is done and the
> actual function.
>
>> That said, though, I don't think there is any pressing need to
>> maintain backwards compatibility around module-base.bbclass.  In
>> fact, I think it would probably be fine for module-base.bbclass
>> to simply go away altogether and have its functionality subsumed
>> into module.bbclass; the split between those two classes is
>> mostly a relic of yesteryear and I can't think of any good
>> purpose that it serves nowadays.  So I would be happy enough to
>> see that (and the tangly mess that is the kernel classes in
>> general) cleaned up irrespective of what happens with
>> do_make_scripts.
>
> I won't argue with that. However, I do not feel that I am
> comfortable enough about these classes to do it. I had a quick
> look around in OE core, and I believe you are right. The number
> of references to "module-base" were limited, and in most cases
> it should just be a matter of replacing them with "module".
>
>> p.
>
> //Peter
>



-- 
"Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end"

Re: RFC: Maintain backwards compatibility or not for module-base.bbclass

[Koen Kooi]

Op 16 jan. 2014, om 14:58 heeft Peter Kjellerstedt <peter.kjellerstedt@axis.com> het volgende geschreven:

> Background: Back in September, Richard made a commit to 
> linux-libc-headers.inc describing why one should not fork the 
> linux-libc-headers recipe:
> 
> http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=babbf7a46acaefd9b36031483cafce053f607e66
> 
> As a result I created a local bbclass for our layer called 
> kernel-headers, which provides a recipe with kernel headers 
> that match the actually used version of the Linux kernel.
> This was needed for packages that need to access hardware 
> specific features that are not present in the generic kernel 
> headers provided by linux-libc-headers.
> 
> My intention for this class was that it should be generic 
> enough to be able to upstream it to OE Core.
> 
> Now, the other day a colleague of mine had a build failure due 
> to this class. It turned out that even though the class adds a 
> dependency on virtual/kernel and then uses the files that are 
> installed to ${STAGING_KERNEL_DIR} when running oe_runmake 
> headers_install, the command could fail because the 
> ${STAGING_KERNEL_DIR}/scripts was not populated. After asking 
> Richard about this, I learned that this is due to problems 
> with the sstate cache and not knowing whether a 32 bit host or 
> a 64 bit host was used to generate the files. Thus I also 
> learned that the scripts are actually built as a result of 
> building modules.
> 
> Since I did not want my class to depend on modules having been 
> built, I looked into modules.bbclass and modules-base.bbclass. 
> There I found the function do_make_scripts() which is 
> responsible for building  the kernel scripts. However, the 
> current setup doesn't lend itself very well to use the 
> modules-base.bbclass for something other than modules.
> 
> My idea then was to break this part out into a separate class, 
> kernel-scripts, which I did. You can find both the 
> kernel-scripts and kernel-headers classes here: 
> 
> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=pkj/kernel-headers
> 
> When I showed this to Richard he noted that my change was not 
> backwards compatible (as I no longer provide the 
> do_make_scripts() function from the module-base.bbclass). 
> However, there is nothing besides module.bbclass in OE Core 
> and meta-oe that use the module-base.bbclass.

Can't do_make_script run as an sstate post function? 

regards,

Koen

Re: RFC: Maintain backwards compatibility or not for module-base.bbclass

[Phil Blundell]

On Thu, 2014-01-16 at 14:58 +0100, Peter Kjellerstedt wrote:
> So, here I am now. I do not know who else use the 
> do_make_scripts() function from module-base.bbclass and in what 
> way, and whether restructuring the functionality into the new 
> kernel-scripts.bbclass without maintaining backwards 
> compatibility would be a problem. If you know anything about 
> this, please let me know.

I'm not entirely clear why you couldn't maintain compatibility by moving
the task in question to a new class but keeping its name the same (i.e.
refrain from renaming "do_make_scripts" to "do_kernel_scripts") and
having module-base.bbclass simply inherit the newly-added class.  That
seems like it ought to be fairly straightforward and uncontroversial.

That said, though, I don't think there is any pressing need to maintain
backwards compatibility around module-base.bbclass.  In fact, I think it
would probably be fine for module-base.bbclass to simply go away
altogether and have its functionality subsumed into module.bbclass; the
split between those two classes is mostly a relic of yesteryear and I
can't think of any good purpose that it serves nowadays.  So I would be
happy enough to see that (and the tangly mess that is the kernel classes
in general) cleaned up irrespective of what happens with
do_make_scripts.

p.