[Buildroot] auto-detecting toolchain metadata?

[Buildroot] auto-detecting toolchain metadata?

[Hollis Blanchard]

I'm running into a problem where I'm distributing a Buildroot-based tree 
(config + scripts) to some users. They really only use Linux, 
bootloader, and busybox, and since they've been building those pieces 
for a while, they already have toolchains laying around.

I'd like to let them use whatever toolchain they've been using, and 
ideally that would be as simple as "change BR2_TOOLCHAIN_EXTERNAL_PATH 
in defconfig". Of course, it's not that simple; Buildroot wants more 
information: gcc version, kernel headers, target tuple, etc. I suspect 
the reason Buildroot needs that extra metadata is to build userland 
packages that are more complex than busybox.

Providing another toolchain to them, with known settings for the 
defconfig, is one option. However, it's hard to justify throwing away 
the toolchain they've been using without problem, especially since they 
don't really care about userland. (Some of them may want to get really 
fancy and add an SSH server, but I've only seen that once.)

Another option I'm considering is a script to try to detect the 
necessary metadata from a given toolchain path. As far as I can see, 
every piece of it can be detected from external observation. It could be 
invoked outside Buildroot (modifying defconfig), but of course it could 
be invoked from within Buildroot's toolchain recipes too. Is there a 
reason this is a bad idea or hasn't already been done?

Thanks!

-- 
Hollis Blanchard <hollis_blanchard@mentor.com>
Mentor Graphics Emulation Division

[Buildroot] auto-detecting toolchain metadata?

[Yann E. MORIN]

Hollis, All,

On 2016-11-01 11:39 -0700, Hollis Blanchard spake thusly:
[--SNIP--]
> Another option I'm considering is a script to try to detect the necessary
> metadata from a given toolchain path. As far as I can see, every piece of it
> can be detected from external observation. It could be invoked outside
> Buildroot (modifying defconfig), but of course it could be invoked from
> within Buildroot's toolchain recipes too. Is there a reason this is a bad
> idea or hasn't already been done?

Such a script would be ideed very usefull.

The problem is that we need to know the characteristics of the toolchain
inside the menuconfig, so we can hide/show packages that have strict
requirements (e.g. on gcc version for C++11, on kernel headers, on the C
library...) but we only know the toolchain to use from inside the
menuconfig, so it is too late to run the script.

So what Buildroot currently does is check the settings after the fact,
because there is no way we can do otherwise.

But if there was a script that would look at a toolchain and spit out
the settings, like:

    ./support/scripts/scan-ext-toolchain /path/to/toolchain
    BR2_USES_MUSL=y
    BR2_GCC_ATLEAST_4_9=y
    [and so on]

with that output to be used as a base defconfig, then that would be
tremendously useful, indeed.

And probably not very dificult to do either...

Bonus would be if we could also use that script to check the current
configuration against the configured toolchain (that would mean
extracting a lot of logic out of the Makefiles, which is not necessarily
a bad idea either).

Waiting for your patch... ;-]

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] auto-detecting toolchain metadata?

[Thomas Petazzoni]

Hello,

On Tue, 1 Nov 2016 19:46:26 +0100, Yann E. MORIN wrote:

> On 2016-11-01 11:39 -0700, Hollis Blanchard spake thusly:
> [--SNIP--]
> > Another option I'm considering is a script to try to detect the necessary
> > metadata from a given toolchain path. As far as I can see, every piece of it
> > can be detected from external observation. It could be invoked outside
> > Buildroot (modifying defconfig), but of course it could be invoked from
> > within Buildroot's toolchain recipes too. Is there a reason this is a bad
> > idea or hasn't already been done?  
> 
> Such a script would be ideed very usefull.

Agreed.

> The problem is that we need to know the characteristics of the toolchain
> inside the menuconfig, so we can hide/show packages that have strict
> requirements (e.g. on gcc version for C++11, on kernel headers, on the C
> library...) but we only know the toolchain to use from inside the
> menuconfig, so it is too late to run the script.

Absolutely. In your e-mail Hollis you said:

"""
but of course it could 
be invoked from within Buildroot's toolchain recipes too
"""

but as Yann explained, this is simply not possible. We need to know, at
the Kconfig level, the properties of the toolchain. Except that the
toolchain location/choice itself is defined at the Kconfig level. So
there's no way we can have Kconfig automatically "know" the properties
of the toolchain.

> So what Buildroot currently does is check the settings after the fact,
> because there is no way we can do otherwise.
> 
> But if there was a script that would look at a toolchain and spit out
> the settings, like:
> 
>     ./support/scripts/scan-ext-toolchain /path/to/toolchain
>     BR2_USES_MUSL=y
>     BR2_GCC_ATLEAST_4_9=y
>     [and so on]

Yes, this woudl be great.

> with that output to be used as a base defconfig, then that would be
> tremendously useful, indeed.
> 
> And probably not very dificult to do either...
> 
> Bonus would be if we could also use that script to check the current
> configuration against the configured toolchain (that would mean
> extracting a lot of logic out of the Makefiles, which is not necessarily
> a bad idea either).

I didn't think about this idea, but maybe it makes sense indeed. At
least it's worth trying and see what it looks like.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] auto-detecting toolchain metadata?

[Hollis Blanchard]

On 11/01/2016 11:46 AM, Yann E. MORIN wrote:
> But if there was a script that would look at a toolchain and spit out
> the settings, like:
>
>      ./support/scripts/scan-ext-toolchain /path/to/toolchain
>      BR2_USES_MUSL=y
>      BR2_GCC_ATLEAST_4_9=y
>      [and so on]
>
> with that output to be used as a base defconfig, then that would be
> tremendously useful, indeed.
My use case is slightly different; I already have a defconfig; I just 
want this script to replace the toolchain-related bits of it. Would 
editing defconfig "in place" (e.g. create .defconfig.new, copy it over 
the original) be acceptable?
> And probably not very dificult to do either...
Here's one complexity: once we know the version, which GCC_AT_LEAST_X_Y 
should be emitted? (Probably the same question for the other "AT_LEAST" 
variables too.)

We could make a new list that matches toolchain/toolchain-common.in, but 
that will undoubtedly lead to mismatches in the future...

We could just auto generate every major/minor version from 4.0 up to 
<toolchain version>, but then how many minors for each major? 10? That 
would yield 4.0 -> 4.10, 5.0 -> 5.10, 6.0 -> 6.10, ...

-- 
Hollis Blanchard <hollis_blanchard@mentor.com>
Mentor Graphics Emulation Division

[Buildroot] auto-detecting toolchain metadata?

[Thomas Petazzoni]

Hello,

On Mon, 21 Nov 2016 17:04:57 -0800, Hollis Blanchard wrote:

> > with that output to be used as a base defconfig, then that would be
> > tremendously useful, indeed.  
> My use case is slightly different; I already have a defconfig; I just 
> want this script to replace the toolchain-related bits of it. Would 
> editing defconfig "in place" (e.g. create .defconfig.new, copy it over 
> the original) be acceptable?

How are you going to filter out all the toolchain-related options from
the original defconfig? It's not like all of them are prefixed by
"BR2_TOOLCHAIN" for example. So I hardly see how you can achieve what
you want in a reliable way.

> > And probably not very dificult to do either...  
> Here's one complexity: once we know the version, which GCC_AT_LEAST_X_Y 
> should be emitted? (Probably the same question for the other "AT_LEAST" 
> variables too.)
> 
> We could make a new list that matches toolchain/toolchain-common.in, but 
> that will undoubtedly lead to mismatches in the future...
> 
> We could just auto generate every major/minor version from 4.0 up to 
> <toolchain version>, but then how many minors for each major? 10? That 
> would yield 4.0 -> 4.10, 5.0 -> 5.10, 6.0 -> 6.10, ...

For gcc versions >= 5, we don't have an option for each minor version.
So it's actually pretty easy:

 if major < 5:
   option = BR2_TOOLCHAIN_GCC_AT_LEAST_<major>_<minor>
 else
   option = BR2_TOOLCHAIN_GCC_AT_LEAST_<major>

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] auto-detecting toolchain metadata?

[Hollis Blanchard]

On 11/22/2016 12:29 AM, Thomas Petazzoni wrote:
> On Mon, 21 Nov 2016 17:04:57 -0800, Hollis Blanchard wrote:
>>> with that output to be used as a base defconfig, then that would be
>>> tremendously useful, indeed.
>> My use case is slightly different; I already have a defconfig; I just
>> want this script to replace the toolchain-related bits of it. Would
>> editing defconfig "in place" (e.g. create .defconfig.new, copy it over
>> the original) be acceptable?
> How are you going to filter out all the toolchain-related options from
> the original defconfig? It's not like all of them are prefixed by
> "BR2_TOOLCHAIN" for example. So I hardly see how you can achieve what
> you want in a reliable way.
My defconfig contains this:
BR2_TOOLCHAIN_EXTERNAL=y
BR2_TOOLCHAIN_EXTERNAL_CUSTOM=y
BR2_TOOLCHAIN_EXTERNAL_CUSTOM_PREFIX="$(ARCH)-linux-gnu"
BR2_TOOLCHAIN_EXTERNAL_GCC_4_9=y
BR2_TOOLCHAIN_EXTERNAL_HEADERS_3_7=y
BR2_TOOLCHAIN_EXTERNAL_CUSTOM_GLIBC=y
BR2_TOOLCHAIN_EXTERNAL_CXX=y

There are no other toolchain-related options (prefixed with 
BR2_TOOLCHAIN or otherwise). Can you elaborate on why I can't just 
replace these?
>> Here's one complexity: once we know the version, which GCC_AT_LEAST_X_Y
>> should be emitted? (Probably the same question for the other "AT_LEAST"
>> variables too.)
>>
>> We could make a new list that matches toolchain/toolchain-common.in, but
>> that will undoubtedly lead to mismatches in the future...
>>
>> We could just auto generate every major/minor version from 4.0 up to
>> <toolchain version>, but then how many minors for each major? 10? That
>> would yield 4.0 -> 4.10, 5.0 -> 5.10, 6.0 -> 6.10, ...
> For gcc versions >= 5, we don't have an option for each minor version.
> So it's actually pretty easy:
>
>   if major < 5:
>     option = BR2_TOOLCHAIN_GCC_AT_LEAST_<major>_<minor>
>   else
>     option = BR2_TOOLCHAIN_GCC_AT_LEAST_<major>
Oops, yeah, I only need to generate the options for defconfig, not 
.config. That will be much easier, thanks. :-)

Hollis Blanchard <hollis_blanchard@mentor.com>
Mentor Graphics Emulation Division

[Buildroot] auto-detecting toolchain metadata?

[Thomas Petazzoni]

Hello,

On Tue, 22 Nov 2016 07:44:57 -0800, Hollis Blanchard wrote:

> My defconfig contains this:
> BR2_TOOLCHAIN_EXTERNAL=y
> BR2_TOOLCHAIN_EXTERNAL_CUSTOM=y
> BR2_TOOLCHAIN_EXTERNAL_CUSTOM_PREFIX="$(ARCH)-linux-gnu"
> BR2_TOOLCHAIN_EXTERNAL_GCC_4_9=y
> BR2_TOOLCHAIN_EXTERNAL_HEADERS_3_7=y
> BR2_TOOLCHAIN_EXTERNAL_CUSTOM_GLIBC=y
> BR2_TOOLCHAIN_EXTERNAL_CXX=y
> 
> There are no other toolchain-related options (prefixed with 
> BR2_TOOLCHAIN or otherwise). Can you elaborate on why I can't just 
> replace these?

Hum, right we're talking only about external toolchains, and only about
a defconfig. So indeed, all options should start with
BR2_TOOLCHAIN_EXTERNAL_.

Still, I find a tool that replaces/edits stuff in an existing defconfig
to be somewhat confusing. Something that generates a defconfig fragment
seems a lot clearer to understand for new users.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] auto-detecting toolchain metadata?

[Hollis Blanchard]

On 11/22/2016 07:49 AM, Thomas Petazzoni wrote:
> Hello,
>
> On Tue, 22 Nov 2016 07:44:57 -0800, Hollis Blanchard wrote:
>
>> My defconfig contains this:
>> BR2_TOOLCHAIN_EXTERNAL=y
>> BR2_TOOLCHAIN_EXTERNAL_CUSTOM=y
>> BR2_TOOLCHAIN_EXTERNAL_CUSTOM_PREFIX="$(ARCH)-linux-gnu"
>> BR2_TOOLCHAIN_EXTERNAL_GCC_4_9=y
>> BR2_TOOLCHAIN_EXTERNAL_HEADERS_3_7=y
>> BR2_TOOLCHAIN_EXTERNAL_CUSTOM_GLIBC=y
>> BR2_TOOLCHAIN_EXTERNAL_CXX=y
>>
>> There are no other toolchain-related options (prefixed with
>> BR2_TOOLCHAIN or otherwise). Can you elaborate on why I can't just
>> replace these?
> Hum, right we're talking only about external toolchains, and only about
> a defconfig. So indeed, all options should start with
> BR2_TOOLCHAIN_EXTERNAL_.
>
> Still, I find a tool that replaces/edits stuff in an existing defconfig
> to be somewhat confusing. Something that generates a defconfig fragment
> seems a lot clearer to understand for new users.
OK, how can I use defconfig fragments? I can't find mention of it in the 
manual or the top-level Makefile.

I do see a little information about using config fragments for 
Kconfig-based packages, and that says it breaks savedefconfig. So if I 
can use fragments for the BR defconfig, how could I update the defconfig 
in the future?

Hollis Blanchard <hollis_blanchard@mentor.com>
Mentor Graphics Emulation Division

[Buildroot] auto-detecting toolchain metadata?

[Thomas Petazzoni]

Hello,

On Tue, 22 Nov 2016 08:12:34 -0800, Hollis Blanchard wrote:

> OK, how can I use defconfig fragments? I can't find mention of it in the 
> manual or the top-level Makefile.

Easy:

 cat frag1 frag2 frag3 > defconfig

And that's it. There's nothing fancy in fragments, it's just a fragment
of defconfig, and you create the real defconfig by concatenating
fragments.

> I do see a little information about using config fragments for 
> Kconfig-based packages, and that says it breaks savedefconfig. So if I 
> can use fragments for the BR defconfig, how could I update the defconfig 
> in the future?

Fragments are hand-written, by extracting data from a defconfig.
Typically, with the above example, you generate your defconfig from
frag1, frag2 and frag3. You then run "make menuconfig", which updates
your configuration, and run "make savedefconfig", which generates a new
defconfig.

It is impossible for Buildroot to know in which fragment should a given
change be propagated. So what I typically do is "diff -u defconfig.old
defconfig", and then propagate the change to the appropriate fragment.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com