[Buildroot] board-specific autotools env customization

[Buildroot] board-specific autotools env customization

[Lionel Orry]

Hi all,

I would like to use a specific XXX_CONF_ENV variable to send ac_cv_...
variables to configure, but they are specific to my compiler. For my tests,
I currently write the XXX_CONF_ENV directly into the package .mk file, but
I would like to know if there is a better way. I already use the
recommended board/<company>/<board>/patches/ infrastructure, but this is
not a patch so I don't know where to put it...

Any advice / best practice ?

Thank you very much,
Lionel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20150202/c22a2f5d/attachment.html>

[Buildroot] board-specific autotools env customization

[Thomas Petazzoni]

Dear Lionel Orry,

On Mon, 2 Feb 2015 15:05:06 +0100, Lionel Orry wrote:

> I would like to use a specific XXX_CONF_ENV variable to send ac_cv_...
> variables to configure, but they are specific to my compiler. For my tests,
> I currently write the XXX_CONF_ENV directly into the package .mk file, but
> I would like to know if there is a better way. I already use the
> recommended board/<company>/<board>/patches/ infrastructure, but this is
> not a patch so I don't know where to put it...
> 
> Any advice / best practice ?

Well, only pass this <pkg>_CONF_ENV if you're using your specific
compiler?

Maybe if you could be more specific and tell us which compiler / which
environment variable you need to pass, we could give more hints.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] board-specific autotools env customization

[Lionel Orry]

Dear Thomas,

On Mon, Feb 2, 2015 at 5:54 PM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
>
> Dear Lionel Orry,
>
> On Mon, 2 Feb 2015 15:05:06 +0100, Lionel Orry wrote:
>
> > I would like to use a specific XXX_CONF_ENV variable to send ac_cv_...
> > variables to configure, but they are specific to my compiler. For my tests,
> > I currently write the XXX_CONF_ENV directly into the package .mk file, but
> > I would like to know if there is a better way. I already use the
> > recommended board/<company>/<board>/patches/ infrastructure, but this is
> > not a patch so I don't know where to put it...
> >
> > Any advice / best practice ?
>
> Well, only pass this <pkg>_CONF_ENV if you're using your specific
> compiler?

Indeed, but I was expecting to achieve this without modifying the .mk
file provided by buildroot.

>
> Maybe if you could be more specific and tell us which compiler / which
> environment variable you need to pass, we could give more hints.

Of course.

First, when I say it is specific to my compiler, I mean that it is
specific because it assumes some cached results for some './configure'
tests which are supposed to run on the target, and thus, when
cross-compiling, the maintainers of some autotools-based packages take
some assumptions which may not be right. I will precise my case below,
but I first wanted to say that this kind of situation could happen in
several packages present in buildroot, and for various
cross-compilers. It is not specific to *me*, it is an issue for
everyone using buildroot.

My case :
* I am cross-compiling the package zeromq (v4.0.4) on a Super-H target
using a pre-built toolchain from STMicroelectronics (STLinux 2.4).
* I noticed the configure script attempts to check for SO_KEEPALIVE
and other similar options existence, and it does so by compiling a
snippet, running it on the target and checking the output value,
excepted when cross-compiling (of course), where it cannot
automatically run the code. So the assumption taken by zeromq devs is
that this flag is not supported. But it is in fact supported (I
manually cross-compiled and ran the snippet on the target to be 100%
sure) and I would like to pass the right assumption to the configure
script, for example by setting "libzmq_so_keepalive=yes" in
ZEROMQ_CONF_ENV.
You will agree with me that the SO_KEEPALIVE flag availability is
dependent on the compiler you use, right? I try to do as few
modifications to the official buildroot as possible, but still I would
like zeromq to make use of this flag on my platform.

Aside note : in fact, in zeromq, the macros written for these flags
detection are wrong because they don't check if the value is set in
environment and always override them; so it is a bad example, but the
general question still arises, that would be interesting to have a
similar infrastructure like for patches, to pass user-specific
environments without messing with the .mk files...

I hope I was clear enough.

>
>
> Best regards,
>
> Thomas
> --
> Thomas Petazzoni, CTO, Free Electrons
> Embedded Linux, Kernel and Android engineering
> http://free-electrons.com

[Buildroot] board-specific autotools env customization

[Arnout Vandecappelle]

 Hi Lionel,

On 02/02/15 18:19, Lionel Orry wrote:
[snip]
> First, when I say it is specific to my compiler, I mean that it is
> specific because it assumes some cached results for some './configure'
> tests which are supposed to run on the target, and thus, when
> cross-compiling, the maintainers of some autotools-based packages take
> some assumptions which may not be right. I will precise my case below,
> but I first wanted to say that this kind of situation could happen in
> several packages present in buildroot, and for various
> cross-compilers. It is not specific to *me*, it is an issue for
> everyone using buildroot.

 It is indeed not specific to you, but applies to all cross-compiling users of
zeromq.

> 
> My case :
> * I am cross-compiling the package zeromq (v4.0.4) on a Super-H target
> using a pre-built toolchain from STMicroelectronics (STLinux 2.4).
> * I noticed the configure script attempts to check for SO_KEEPALIVE
> and other similar options existence, and it does so by compiling a
> snippet, running it on the target and checking the output value,
> excepted when cross-compiling (of course), where it cannot
> automatically run the code. So the assumption taken by zeromq devs is
> that this flag is not supported. But it is in fact supported (I
> manually cross-compiled and ran the snippet on the target to be 100%
> sure) and I would like to pass the right assumption to the configure
> script, for example by setting "libzmq_so_keepalive=yes" in
> ZEROMQ_CONF_ENV.
> You will agree with me that the SO_KEEPALIVE flag availability is
> dependent on the compiler you use, right? I try to do as few
> modifications to the official buildroot as possible, but still I would
> like zeromq to make use of this flag on my platform.

 In fact, this flag is not dependent on the toolchain you use, but only on the
kernel version (and of course your toolchain should also define the flag).
That's why it runs the code, to find out if the kernel really supports it.
SO_KEEPALIVE was introduced a really long time ago so you're safe to set
libzmq_so_keepalive=yes unconditionally in ZEROMQ_CONF_ENV. And in fact, many of
the other flags that zeromq checks for are also always available in Linux.

 So we would really appreciate if you could check the various flags that zeromq
tries to run and set them to yes if they are available, and send a patch to the
mailing list. I've already checked libzmq_cv_sock_cloexec: SOCK_CLOEXEC was
introduced in 2.6.27 so we can also assume that it's always available. Which
kernel version do you run?



> Aside note : in fact, in zeromq, the macros written for these flags
> detection are wrong because they don't check if the value is set in
> environment and always override them; so it is a bad example, but the

 Yeah, indeed, you should also patch acinclude.m4 to correctly use a cache
variable with AC_CACHE_VAL and AC_CACHE_CHECK. And that patch should be sent
upstream of course.


> general question still arises, that would be interesting to have a
> similar infrastructure like for patches, to pass user-specific
> environments without messing with the .mk files...

 We will do that when there is a situation where this would be useful. But most
of our bases are covered already. We can check on the kernel version, the type
of C library, various components of the toolchain. We are indeed missing a check
for e.g. C/C++11 availability, but that will come at some point.


 Regards,
 Arnout



-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] board-specific autotools env customization

[Lionel Orry]

Hi Arnout,

On Tue, Feb 3, 2015 at 9:54 AM, Arnout Vandecappelle <arnout@mind.be> wrote:
>
>  Hi Lionel,
>
> On 02/02/15 18:19, Lionel Orry wrote:
> [snip]
> > First, when I say it is specific to my compiler, I mean that it is
> > specific because it assumes some cached results for some './configure'
> > tests which are supposed to run on the target, and thus, when
> > cross-compiling, the maintainers of some autotools-based packages take
> > some assumptions which may not be right. I will precise my case below,
> > but I first wanted to say that this kind of situation could happen in
> > several packages present in buildroot, and for various
> > cross-compilers. It is not specific to *me*, it is an issue for
> > everyone using buildroot.
>
>  It is indeed not specific to you, but applies to all cross-compiling users of
> zeromq.
>
> >
> > My case :
> > * I am cross-compiling the package zeromq (v4.0.4) on a Super-H target
> > using a pre-built toolchain from STMicroelectronics (STLinux 2.4).
> > * I noticed the configure script attempts to check for SO_KEEPALIVE
> > and other similar options existence, and it does so by compiling a
> > snippet, running it on the target and checking the output value,
> > excepted when cross-compiling (of course), where it cannot
> > automatically run the code. So the assumption taken by zeromq devs is
> > that this flag is not supported. But it is in fact supported (I
> > manually cross-compiled and ran the snippet on the target to be 100%
> > sure) and I would like to pass the right assumption to the configure
> > script, for example by setting "libzmq_so_keepalive=yes" in
> > ZEROMQ_CONF_ENV.
> > You will agree with me that the SO_KEEPALIVE flag availability is
> > dependent on the compiler you use, right? I try to do as few
> > modifications to the official buildroot as possible, but still I would
> > like zeromq to make use of this flag on my platform.
>
>  In fact, this flag is not dependent on the toolchain you use, but only on the
> kernel version (and of course your toolchain should also define the flag).
> That's why it runs the code, to find out if the kernel really supports it.
> SO_KEEPALIVE was introduced a really long time ago so you're safe to set
> libzmq_so_keepalive=yes unconditionally in ZEROMQ_CONF_ENV. And in fact, many of
> the other flags that zeromq checks for are also always available in Linux.
>
>  So we would really appreciate if you could check the various flags that zeromq
> tries to run and set them to yes if they are available, and send a patch to the
> mailing list. I've already checked libzmq_cv_sock_cloexec: SOCK_CLOEXEC was
> introduced in 2.6.27 so we can also assume that it's always available. Which
> kernel version do you run?
>

I have this list in here (there are 6 flags, 5 of them are available)
for my compiler, so I could make this patch. It won't do anything
useful until the acinclude.m4 is patched though, so I suppose the
first step is to fix acinclude.m4.

I use a 2.6.32 based kernel.

For reference, with my current compiler :
* compiler version : sh4-linux-gcc (GCC) 4.7.3 20130514
(STMicroelectronics/Linux Base 4.7.3-124)
* SOCK_CLOEXEC : available
* SO_KEEPALIVE : available
* TCP_KEEPCNT : available
* TCP_KEEPIDLE : available
* TCP_KEEPINTVL : available
* TCP_KEEPALIVE : not available

>
>
> > Aside note : in fact, in zeromq, the macros written for these flags
> > detection are wrong because they don't check if the value is set in
> > environment and always override them; so it is a bad example, but the
>
>  Yeah, indeed, you should also patch acinclude.m4 to correctly use a cache
> variable with AC_CACHE_VAL and AC_CACHE_CHECK. And that patch should be sent
> upstream of course.
>

I totally agree. But I'm still quite bad at autotools stuff and I
don't know how to use AC_CACHE_xxx macros
but I'll try to do it anyway. Any hints appreciated, the most clean
the patch is, the more chances it will have to hit upstream...

>
> > general question still arises, that would be interesting to have a
> > similar infrastructure like for patches, to pass user-specific
> > environments without messing with the .mk files...
>
>  We will do that when there is a situation where this would be useful. But most
> of our bases are covered already. We can check on the kernel version, the type
> of C library, various components of the toolchain. We are indeed missing a check
> for e.g. C/C++11 availability, but that will come at some point.

Thanks for the precisions.

>
>
>  Regards,
>  Arnout
>
>
>
> --
> Arnout Vandecappelle                          arnout at mind be
> Senior Embedded Software Architect            +32-16-286500
> Essensium/Mind                                http://www.mind.be
> G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
> LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
> GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

Cheers,
Lionel

[Buildroot] board-specific autotools env customization

[Arnout Vandecappelle]

On 04/02/15 08:52, Lionel Orry wrote:
> Hi Arnout,
> 
> On Tue, Feb 3, 2015 at 9:54 AM, Arnout Vandecappelle <arnout@mind.be> wrote:

[snip]
>>  So we would really appreciate if you could check the various flags that zeromq
>> tries to run and set them to yes if they are available, and send a patch to the
>> mailing list. I've already checked libzmq_cv_sock_cloexec: SOCK_CLOEXEC was
>> introduced in 2.6.27 so we can also assume that it's always available. Which
>> kernel version do you run?
>>
> 
> I have this list in here (there are 6 flags, 5 of them are available)
> for my compiler, so I could make this patch. It won't do anything
> useful until the acinclude.m4 is patched though, so I suppose the
> first step is to fix acinclude.m4.
> 
> I use a 2.6.32 based kernel.
> 
> For reference, with my current compiler :
> * compiler version : sh4-linux-gcc (GCC) 4.7.3 20130514
> (STMicroelectronics/Linux Base 4.7.3-124)
> * SOCK_CLOEXEC : available
> * SO_KEEPALIVE : available
> * TCP_KEEPCNT : available
> * TCP_KEEPIDLE : available
> * TCP_KEEPINTVL : available
> * TCP_KEEPALIVE : not available

 This indeed seems to correspond to the current situation (kernel > 2.6.30).


>>> Aside note : in fact, in zeromq, the macros written for these flags
>>> detection are wrong because they don't check if the value is set in
>>> environment and always override them; so it is a bad example, but the
>>
>>  Yeah, indeed, you should also patch acinclude.m4 to correctly use a cache
>> variable with AC_CACHE_VAL and AC_CACHE_CHECK. And that patch should be sent
>> upstream of course.
>>
> 
> I totally agree. But I'm still quite bad at autotools stuff and I
> don't know how to use AC_CACHE_xxx macros
> but I'll try to do it anyway. Any hints appreciated, the most clean
> the patch is, the more chances it will have to hit upstream...

The basic pattern is:

AC_CACHE_CHECK(whether SOCK_CLOEXEC is supported, libzmq_cv_sock_cloexec,
[AC_TRY_RUN(...)])

and then of course remove the AC_MSG_CHECKING that comes before it. The cache
variable should be set in the body of the AC_CACHE_CHECK, like it currently is.


 Regards,
 Arnout


[snip]

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] board-specific autotools env customization

[Lionel Orry]

Dear all,

On Wed, Feb 4, 2015 at 9:21 PM, Arnout Vandecappelle <arnout@mind.be> wrote:
> On 04/02/15 08:52, Lionel Orry wrote:
>> Hi Arnout,
>>
>> On Tue, Feb 3, 2015 at 9:54 AM, Arnout Vandecappelle <arnout@mind.be> wrote:
>
> [snip]
>>>  So we would really appreciate if you could check the various flags that zeromq
>>> tries to run and set them to yes if they are available, and send a patch to the
>>> mailing list. I've already checked libzmq_cv_sock_cloexec: SOCK_CLOEXEC was
>>> introduced in 2.6.27 so we can also assume that it's always available. Which
>>> kernel version do you run?
>>>
>>
>> I have this list in here (there are 6 flags, 5 of them are available)
>> for my compiler, so I could make this patch. It won't do anything
>> useful until the acinclude.m4 is patched though, so I suppose the
>> first step is to fix acinclude.m4.
>>
>> I use a 2.6.32 based kernel.
>>
>> For reference, with my current compiler :
>> * compiler version : sh4-linux-gcc (GCC) 4.7.3 20130514
>> (STMicroelectronics/Linux Base 4.7.3-124)
>> * SOCK_CLOEXEC : available
>> * SO_KEEPALIVE : available
>> * TCP_KEEPCNT : available
>> * TCP_KEEPIDLE : available
>> * TCP_KEEPINTVL : available
>> * TCP_KEEPALIVE : not available
>
>  This indeed seems to correspond to the current situation (kernel > 2.6.30).
>
>
>>>> Aside note : in fact, in zeromq, the macros written for these flags
>>>> detection are wrong because they don't check if the value is set in
>>>> environment and always override them; so it is a bad example, but the
>>>
>>>  Yeah, indeed, you should also patch acinclude.m4 to correctly use a cache
>>> variable with AC_CACHE_VAL and AC_CACHE_CHECK. And that patch should be sent
>>> upstream of course.
>>>
>>
>> I totally agree. But I'm still quite bad at autotools stuff and I
>> don't know how to use AC_CACHE_xxx macros
>> but I'll try to do it anyway. Any hints appreciated, the most clean
>> the patch is, the more chances it will have to hit upstream...
>
> The basic pattern is:
>
> AC_CACHE_CHECK(whether SOCK_CLOEXEC is supported, libzmq_cv_sock_cloexec,
> [AC_TRY_RUN(...)])
>
> and then of course remove the AC_MSG_CHECKING that comes before it. The cache
> variable should be set in the body of the AC_CACHE_CHECK, like it currently is.
>

Thank you Arnout for your valuable help. I sent a patch to the
mailing-list, and I am preparing the patch for upstream.
I have to adapt it for the current upstream master version (v.4.2.x)
because acinclude.m4 changed, so I can't send it as-is...
Should I try to send the v4.0.x based patch to their maintenance
branch ? They say it's for bugfixes only so I'm pretty sure it will be
rejected.

>
>  Regards,
>  Arnout
>
>
> [snip]
>
> --
> Arnout Vandecappelle                          arnout at mind be
> Senior Embedded Software Architect            +32-16-286500
> Essensium/Mind                                http://www.mind.be
> G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
> LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
> GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

Kind regards,
Lionel

[Buildroot] board-specific autotools env customization

[Arnout Vandecappelle]

On 06/02/15 10:31, Lionel Orry wrote:
[snip]
> Thank you Arnout for your valuable help. I sent a patch to the
> mailing-list, and I am preparing the patch for upstream.
> I have to adapt it for the current upstream master version (v.4.2.x)
> because acinclude.m4 changed, so I can't send it as-is...
> Should I try to send the v4.0.x based patch to their maintenance
> branch ? They say it's for bugfixes only so I'm pretty sure it will be
> rejected.

 Sending to the upstream maintenance branch isn't very useful.

 Sending an adapted patch to buildroot like you did is OK. But perhaps it's more
useful to bump the buildroot version. Yes, we're trying to squeeze as much
contribution out of you as we can :-)

 Regards,
 Arnout


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] board-specific autotools env customization

[Lionel Orry]

Hi,

On Fri, Feb 6, 2015 at 11:49 AM, Arnout Vandecappelle <arnout@mind.be> wrote:
> On 06/02/15 10:31, Lionel Orry wrote:
> [snip]
>> Thank you Arnout for your valuable help. I sent a patch to the
>> mailing-list, and I am preparing the patch for upstream.
>> I have to adapt it for the current upstream master version (v.4.2.x)
>> because acinclude.m4 changed, so I can't send it as-is...
>> Should I try to send the v4.0.x based patch to their maintenance
>> branch ? They say it's for bugfixes only so I'm pretty sure it will be
>> rejected.
>
>  Sending to the upstream maintenance branch isn't very useful.
>
>  Sending an adapted patch to buildroot like you did is OK. But perhaps it's more
> useful to bump the buildroot version. Yes, we're trying to squeeze as much
> contribution out of you as we can :-)

Well I created a pull request on upstream dev branch which was
accepted and merged a few minutes later. I'm glad they are so
responsive.

But they are in the process of stabilizing zeromq v4.1.0 (they are in
RC currently) and I'm afraid this code won't be merged there, so we
will have to wait for another late release before dropping the patch
from buildroot...

>
>  Regards,
>  Arnout
>
>
> --
> Arnout Vandecappelle                          arnout at mind be
> Senior Embedded Software Architect            +32-16-286500
> Essensium/Mind                                http://www.mind.be
> G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
> LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
> GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

Regards
Lionel

[Buildroot] board-specific autotools env customization

[Thomas Petazzoni]

Dear Lionel Orry,

On Fri, 6 Feb 2015 15:51:03 +0100, Lionel Orry wrote:

> But they are in the process of stabilizing zeromq v4.1.0 (they are in
> RC currently) and I'm afraid this code won't be merged there, so we
> will have to wait for another late release before dropping the patch
> from buildroot...

That's fine: as long as a "fixes" patch is already merged upstream, but
we don't really mind carrying it in Buildroot.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com