[PATCH] siteinfo: Add mechanism to extend siteinfo information from BSP layer

[PATCH] siteinfo: Add mechanism to extend siteinfo information from BSP layer

[Richard Purdie]

In order to add a new architecture or sub-architecture to OE, you currently
need to tweak the table in siteinfo.bbclass. This adds a mechanism so this
can be done from a BSP layer. It needs a function definition which needs
a class file but can then be done with something like:

def rp_testfunc2(archinfo, osinfo, targetinfo, d):
    archinfo['testarch'] = "little-endian bit-32"
    osinfo['testos'] = "common-linux"
    targetinfo['mymach-linux'] = "mymach-linux-common"

    return archinfo, osinfo, targetinfo

SITEINFO_EXTRA_DATAFUNCS = "rp_testfunc2"

[YOCTO #8554]

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

diff --git a/meta/classes/siteinfo.bbclass b/meta/classes/siteinfo.bbclass
index 50141a3..3c9ab2a 100644
--- a/meta/classes/siteinfo.bbclass
+++ b/meta/classes/siteinfo.bbclass
@@ -107,6 +107,14 @@ def siteinfo_data(d):
         "x86_64-mingw32": "bit-64",
     }
 
+    # Add in any extra user supplied data which may come from a BSP layer, removing the
+    # need to always change this class directly
+    extra_siteinfo = (d.getVar("SITEINFO_EXTRA_DATAFUNCS", True) or "").split()
+    for m in extra_siteinfo:
+        call = m + "(archinfo, osinfo, targetinfo, d)"
+        locs = { "archinfo" : archinfo, "osinfo" : osinfo, "targetinfo" : targetinfo, "d" : d}
+        archinfo, osinfo, targetinfo = bb.utils.better_eval(call, locs)
+
     hostarch = d.getVar("HOST_ARCH", True)
     hostos = d.getVar("HOST_OS", True)
     target = "%s-%s" % (hostarch, hostos)

Re: [PATCH] siteinfo: Add mechanism to extend siteinfo information from BSP layer

[Khem Raj]

[-- Attachment #1: Type: text/plain, Size: 2094 bytes --]

Whats the motivation here. This looks like a backdoor for bsps. To user it
will make inconsistent behaviour. Even now its quite hard to root cause
errors due to siteinfo

On Jul 22, 2016 7:26 AM, "Richard Purdie" <
richard.purdie@linuxfoundation.org> wrote:
>
> In order to add a new architecture or sub-architecture to OE, you
currently
> need to tweak the table in siteinfo.bbclass. This adds a mechanism so this
> can be done from a BSP layer. It needs a function definition which needs
> a class file but can then be done with something like:
>
> def rp_testfunc2(archinfo, osinfo, targetinfo, d):
>     archinfo['testarch'] = "little-endian bit-32"
>     osinfo['testos'] = "common-linux"
>     targetinfo['mymach-linux'] = "mymach-linux-common"
>
>     return archinfo, osinfo, targetinfo
>
> SITEINFO_EXTRA_DATAFUNCS = "rp_testfunc2"
>
> [YOCTO #8554]
>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
>
> diff --git a/meta/classes/siteinfo.bbclass b/meta/classes/siteinfo.bbclass
> index 50141a3..3c9ab2a 100644
> --- a/meta/classes/siteinfo.bbclass
> +++ b/meta/classes/siteinfo.bbclass
> @@ -107,6 +107,14 @@ def siteinfo_data(d):
>          "x86_64-mingw32": "bit-64",
>      }
>
> +    # Add in any extra user supplied data which may come from a BSP
layer, removing the
> +    # need to always change this class directly
> +    extra_siteinfo = (d.getVar("SITEINFO_EXTRA_DATAFUNCS", True) or
"").split()
> +    for m in extra_siteinfo:
> +        call = m + "(archinfo, osinfo, targetinfo, d)"
> +        locs = { "archinfo" : archinfo, "osinfo" : osinfo, "targetinfo"
: targetinfo, "d" : d}
> +        archinfo, osinfo, targetinfo = bb.utils.better_eval(call, locs)
> +
>      hostarch = d.getVar("HOST_ARCH", True)
>      hostos = d.getVar("HOST_OS", True)
>      target = "%s-%s" % (hostarch, hostos)
>
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

[-- Attachment #2: Type: text/html, Size: 2941 bytes --]

Re: [PATCH] siteinfo: Add mechanism to extend siteinfo information from BSP layer

[Richard Purdie]

On Fri, 2016-07-22 at 08:47 -0700, Khem Raj wrote:
> Whats the motivation here. This looks like a backdoor for bsps. To
> user it will make inconsistent behaviour. Even now its quite hard to
> root cause errors due to siteinfo

Right now if you have some new target, your only option is to either
patch OE-Core, or copy the class into your new BSP. This is
particularly problematic if you're working on hardware that isn't
supported by core OE or that is in active development.

Having hooks to allow you to write a new BSP without having to make
changes to the core would seem to be a good thing.

I'm not expecting users to regularly use this, just those writing more
exotic BSPs or doing baremetal work for example.

I can sympathise with the problem as I've run into it several times now
myself. We can't really demand that every time this happens, people
send a patch for OE-Core (although if something is commonly used we'd
obviously then encourage it).

Cheers,

Richard

Re: [PATCH] siteinfo: Add mechanism to extend siteinfo information from BSP layer

[Khem Raj]

[-- Attachment #1: Type: text/plain, Size: 1624 bytes --]


> On Jul 22, 2016, at 8:55 AM, Richard Purdie <richard.purdie@linuxfoundation.org> wrote:
> 
> On Fri, 2016-07-22 at 08:47 -0700, Khem Raj wrote:
>> Whats the motivation here. This looks like a backdoor for bsps. To
>> user it will make inconsistent behaviour. Even now its quite hard to
>> root cause errors due to siteinfo
> 
> Right now if you have some new target, your only option is to either
> patch OE-Core, or copy the class into your new BSP.

May be we can make an exception and accept patches for these changes into release branches
via master ofcourse

> This is
> particularly problematic if you're working on hardware that isn't
> supported by core OE or that is in active development.
> 
> Having hooks to allow you to write a new BSP without having to make
> changes to the core would seem to be a good thing.

This seems to be fair, however, they will surely keep these fragments
and supply their BSP to users. who will now have to chase different
places for finding siteinfo details.

> 
> I'm not expecting users to regularly use this, just those writing more
> exotic BSPs or doing baremetal work for example.
> 
> I can sympathise with the problem as I've run into it several times now
> myself. We can't really demand that every time this happens, people
> send a patch for OE-Core (although if something is commonly used we'd
> obviously then encourage it).

This could also mean that I can override a definition e.g. size of off_t
for arm in my BSP layer and inject it into a MultiBSP setups where others will
get it too ?

> 
> Cheers,
> 
> Richard
> 


[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 211 bytes --]

Re: [PATCH] siteinfo: Add mechanism to extend siteinfo information from BSP layer

[Richard Purdie]

On Fri, 2016-07-22 at 11:20 -0700, Khem Raj wrote:
> > On Jul 22, 2016, at 8:55 AM, Richard Purdie <
> > richard.purdie@linuxfoundation.org> wrote:
> > 
> > On Fri, 2016-07-22 at 08:47 -0700, Khem Raj wrote:
> > > Whats the motivation here. This looks like a backdoor for bsps.
> > > To
> > > user it will make inconsistent behaviour. Even now its quite hard
> > > to
> > > root cause errors due to siteinfo
> > 
> > Right now if you have some new target, your only option is to
> > either
> > patch OE-Core, or copy the class into your new BSP.
> 
> May be we can make an exception and accept patches for these changes
> into release branches
> via master ofcourse

That isn't really helping people doing development work though. Sure,
eventually these changes can be ready for master but I've worked on
things where master never really made sense for various reasons.

> > This is
> > particularly problematic if you're working on hardware that isn't
> > supported by core OE or that is in active development.
> > 
> > Having hooks to allow you to write a new BSP without having to make
> > changes to the core would seem to be a good thing.
> 
> This seems to be fair, however, they will surely keep these fragments
> and supply their BSP to users. who will now have to chase different
> places for finding siteinfo details.

The code is pretty good about printing a list of the siteinfo files its
using so it should be clear if a siteinfo file is coming from a
different layer and which siteinfo files are in use.

> > I'm not expecting users to regularly use this, just those writing
> > more
> > exotic BSPs or doing baremetal work for example.
> > 
> > I can sympathise with the problem as I've run into it several times
> > now
> > myself. We can't really demand that every time this happens, people
> > send a patch for OE-Core (although if something is commonly used
> > we'd
> > obviously then encourage it).
> 
> This could also mean that I can override a definition e.g. size of
> off_t
> for arm in my BSP layer and inject it into a MultiBSP setups where
> others will
> get it too ?

In theory you could. There are an awful lot of things you can do with
the build system which aren't really recommended though. I'm not sure
I'd single this one out as being something we should enforce as not
being good.

Let me put this another way.

The principle is that you can add new hardware support to OE in a BSP
layer. Juro and I tried that and we ran into problems where we could
not make this work without hacking OE-Core meaning the principle
failed.

We made a list and there were three places there were problems. I've
sent patches for two of them, I've proposed a different approach for
the third in the bug entry.

I do feel pretty strongly that OE should have good support of
independent BSP layers and if there are barriers to this, we should
have mechanisms to cope with that.

Yes, there is a risk of abuse but the whole system is so flexible we
always have that. We can document when these things should be used and
also name/shame anyone abusing them.

Cheers,

Richard

Re: [PATCH] siteinfo: Add mechanism to extend siteinfo information from BSP layer

[Khem Raj]

[-- Attachment #1: Type: text/plain, Size: 4161 bytes --]


> On Jul 22, 2016, at 3:16 PM, Richard Purdie <richard.purdie@linuxfoundation.org> wrote:
> 
> On Fri, 2016-07-22 at 11:20 -0700, Khem Raj wrote:
>>> On Jul 22, 2016, at 8:55 AM, Richard Purdie <
>>> richard.purdie@linuxfoundation.org> wrote:
>>> 
>>> On Fri, 2016-07-22 at 08:47 -0700, Khem Raj wrote:
>>>> Whats the motivation here. This looks like a backdoor for bsps.
>>>> To
>>>> user it will make inconsistent behaviour. Even now its quite hard
>>>> to
>>>> root cause errors due to siteinfo
>>> 
>>> Right now if you have some new target, your only option is to
>>> either
>>> patch OE-Core, or copy the class into your new BSP.
>> 
>> May be we can make an exception and accept patches for these changes
>> into release branches
>> via master ofcourse
> 
> That isn't really helping people doing development work though. Sure,
> eventually these changes can be ready for master but I've worked on
> things where master never really made sense for various reasons.
> 
>>> This is
>>> particularly problematic if you're working on hardware that isn't
>>> supported by core OE or that is in active development.
>>> 
>>> Having hooks to allow you to write a new BSP without having to make
>>> changes to the core would seem to be a good thing.
>> 
>> This seems to be fair, however, they will surely keep these fragments
>> and supply their BSP to users. who will now have to chase different
>> places for finding siteinfo details.
> 
> The code is pretty good about printing a list of the siteinfo files its
> using so it should be clear if a siteinfo file is coming from a
> different layer and which siteinfo files are in use.
> 
>>> I'm not expecting users to regularly use this, just those writing
>>> more
>>> exotic BSPs or doing baremetal work for example.
>>> 
>>> I can sympathise with the problem as I've run into it several times
>>> now
>>> myself. We can't really demand that every time this happens, people
>>> send a patch for OE-Core (although if something is commonly used
>>> we'd
>>> obviously then encourage it).
>> 
>> This could also mean that I can override a definition e.g. size of
>> off_t
>> for arm in my BSP layer and inject it into a MultiBSP setups where
>> others will
>> get it too ?
> 
> In theory you could. There are an awful lot of things you can do with
> the build system which aren't really recommended though. I'm not sure
> I'd single this one out as being something we should enforce as not
> being good.
> 
> Let me put this another way.
> 
> The principle is that you can add new hardware support to OE in a BSP
> layer. Juro and I tried that and we ran into problems where we could
> not make this work without hacking OE-Core meaning the principle
> failed.
> 
> We made a list and there were three places there were problems. I've
> sent patches for two of them, I've proposed a different approach for
> the third in the bug entry.
> 
> I do feel pretty strongly that OE should have good support of
> independent BSP layers and if there are barriers to this, we should
> have mechanisms to cope with that.

That seems a good in principle, but then we have been hard on some
things e.g. linux-libc-headers, citing that we want interfaces
from kernel to userspace to be consistent across all. But this
falls apart when you insert different type system underneath
this recommendation also makes no sense.


> 
> Yes, there is a risk of abuse but the whole system is so flexible we
> always have that. We can document when these things should be used and
> also name/shame anyone abusing them.
> 

I think implications on usability are huge. I just get this constant
feedback that its complex to debug issues with yocto based systems.
a real world example is where folks spend 4 weeks a problem which turns
out to be a wrong assumption on OE’s part for cashing size of off_t for
one architecture and curl turning on heels over it.

anyhow, I just wanted to present a perspective from a non BSP writers
point of view, in the end these BSPs has to be used by someone and
that should be more important IMO




[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 211 bytes --]