[PATCH] openssl: use nativeperl to build

[PATCH] openssl: use nativeperl to build

[Hongxu Jia]

If host perl is old version (such as perl v5.16.3 on CentOS-7.5),
the openssl-native failed at do_install, the reason is its Makefile
is not correct which is generated at do_configure by perl script.

Use nativeperl to assure latest perl on host for openssl build.

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 meta/recipes-connectivity/openssl/openssl_1.1.1.bb | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-connectivity/openssl/openssl_1.1.1.bb b/meta/recipes-connectivity/openssl/openssl_1.1.1.bb
index 5a43a15..52d4e60 100644
--- a/meta/recipes-connectivity/openssl/openssl_1.1.1.bb
+++ b/meta/recipes-connectivity/openssl/openssl_1.1.1.bb
@@ -23,7 +23,7 @@ SRC_URI_append_class-nativesdk = " \
 SRC_URI[md5sum] = "7079eb017429e0ffb9efb42bf80ccb21"
 SRC_URI[sha256sum] = "2836875a0f89c03d0fdf483941512613a50cfb421d6fd94b9f41d7279d586a3d"
 
-inherit lib_package multilib_header ptest
+inherit lib_package multilib_header ptest perlnative
 
 #| ./libcrypto.so: undefined reference to `getcontext'
 #| ./libcrypto.so: undefined reference to `setcontext'
@@ -105,7 +105,9 @@ do_configure () {
 	fi
 	# WARNING: do not set compiler/linker flags (-I/-D etc.) in EXTRA_OECONF, as they will fully replace the
 	# environment variables set by bitbake. Adjust the environment variables instead.
-	perl ./Configure ${EXTRA_OECONF} ${PACKAGECONFIG_CONFARGS} --prefix=$useprefix --openssldir=${libdir}/ssl-1.1 --libdir=${libdir} $target
+	export PERL="nativeperl"
+	/usr/bin/env nativeperl	./Configure ${EXTRA_OECONF} ${PACKAGECONFIG_CONFARGS} \
+	    --prefix=$useprefix --openssldir=${libdir}/ssl-1.1 --libdir=${libdir} $target
 }
 
 do_install () {
-- 
2.7.4

Re: [PATCH] openssl: use nativeperl to build

[richard.purdie]

On Thu, 2018-10-11 at 02:55 -0700, Hongxu Jia wrote:
> If host perl is old version (such as perl v5.16.3 on CentOS-7.5),
> the openssl-native failed at do_install, the reason is its Makefile
> is not correct which is generated at do_configure by perl script.
> 
> Use nativeperl to assure latest perl on host for openssl build.
> 
> Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
> ---
>  meta/recipes-connectivity/openssl/openssl_1.1.1.bb | 6 ++++--
>  1 file changed, 4 insertions(+), 2 deletions(-)

I think this will have a noticeable effect on built times unfortunately
since it means openssl-native would then depend upon perl-native...

Cheers,

Richard

Re: [PATCH] openssl: use nativeperl to build

[Hongxu Jia]

On 2018年10月11日 20:00, richard.purdie@linuxfoundation.org wrote:
> On Thu, 2018-10-11 at 02:55 -0700, Hongxu Jia wrote:
>> If host perl is old version (such as perl v5.16.3 on CentOS-7.5),
>> the openssl-native failed at do_install, the reason is its Makefile
>> is not correct which is generated at do_configure by perl script.
>>
>> Use nativeperl to assure latest perl on host for openssl build.
>>
>> Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
>> ---
>>   meta/recipes-connectivity/openssl/openssl_1.1.1.bb | 6 ++++--
>>   1 file changed, 4 insertions(+), 2 deletions(-)
> I think this will have a noticeable effect on built times unfortunately
> since it means openssl-native would then depend upon perl-native...
Hi RP,

Yes, I do the test to build openssl-native from scratch, it requires 
extra 4 minutes

1. Use host perl
18 task
real    1m33.978s
user    0m0.728s
sys     0m0.080s

2. Use nativeperl
114 tasks
real    5m49.911s
user    0m1.372s
sys     0m0.140s

But for a image build, the cost is not much
(To exclude network affect, we assure all tarball/git exist in downloads)
But build a image from scratch:

time bitbake core-image-minimal

1. Use host perl
NOTE: Tasks Summary: Attempted 1655 tasks of which 5 didn't need to be 
rerun and all succeeded.

real    28m29.625s
user    0m7.296s
sys     0m0.988s

2. Use native perl
NOTE: Tasks Summary: Attempted 1655 tasks of which 5 didn't need to be 
rerun and all succeeded.

real    28m42.619s
user    0m7.496s
sys     0m0.952s


As upstream said, it indeed require specific PERL version

`If you are building OpenSSL 1.1.0 and above, then you will also need 
PERL 5.10 or high'

https://wiki.openssl.org/index.php/Compilation_and_Installation

And I found at least PERL 5.16.3 not work for OpenSSL 1.1.1

Or add a perl version detecting in insane check? We should report a lowest
workable perl version number when detect failed. But we may update the 
version
number in insane check when openssl is upgraded.

//Hongxu

> Cheers,
>
> Richard
>

Re: [PATCH] openssl: use nativeperl to build

[Hongxu Jia]

On 2018年10月11日 22:58, Hongxu Jia wrote:
> On 2018年10月11日 20:00, richard.purdie@linuxfoundation.org wrote:
>> On Thu, 2018-10-11 at 02:55 -0700, Hongxu Jia wrote:
>>> If host perl is old version (such as perl v5.16.3 on CentOS-7.5),
>>> the openssl-native failed at do_install, the reason is its Makefile
>>> is not correct which is generated at do_configure by perl script.
>>>
>>> Use nativeperl to assure latest perl on host for openssl build.
>>>
>>> Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
>>> ---
>>>   meta/recipes-connectivity/openssl/openssl_1.1.1.bb | 6 ++++--
>>>   1 file changed, 4 insertions(+), 2 deletions(-)
>> I think this will have a noticeable effect on built times unfortunately
>> since it means openssl-native would then depend upon perl-native...
Hi RP,

With some digging on upstream, more than perl version,
the building also requires to install perl Text::Template module
which does not exist in core perl, I think we have to add the recipe

[snip]
  Notes on Perl modules we use
  ----------------------------
  We make increasing use of Perl modules, and do our best to limit
  ourselves to core Perl modules to keep the requirements down. There
  are just a few exceptions:
  Test::More         We require the minimum version to be 0.96, which
                     appeared in Perl 5.13.4, because that version was
                     the first to have all the features we're using.
                     This module is required for testing only!  If you
                     don't plan on running the tests, you don't need to
                     bother with this one.

  Text::Template     This module is not part of the core Perl modules.
                     As a matter of fact, the core Perl modules do not
                     include any templating module to date.
                     This module is absolutely needed, configuration
depends on it.
[snip]

https://github.com/openssl/openssl/blob/master/NOTES.PERL
https://github.com/openssl/openssl/commit/4e351ca92e3a1f447cef3d2e330f13941f9412c6

//Hongxu



> Hi RP,
>
> Yes, I do the test to build openssl-native from scratch, it requires 
> extra 4 minutes
>
> 1. Use host perl
> 18 task
> real    1m33.978s
> user    0m0.728s
> sys     0m0.080s
>
> 2. Use nativeperl
> 114 tasks
> real    5m49.911s
> user    0m1.372s
> sys     0m0.140s
>
> But for a image build, the cost is not much
> (To exclude network affect, we assure all tarball/git exist in downloads)
> But build a image from scratch:
>
> time bitbake core-image-minimal
>
> 1. Use host perl
> NOTE: Tasks Summary: Attempted 1655 tasks of which 5 didn't need to be 
> rerun and all succeeded.
>
> real    28m29.625s
> user    0m7.296s
> sys     0m0.988s
>
> 2. Use native perl
> NOTE: Tasks Summary: Attempted 1655 tasks of which 5 didn't need to be 
> rerun and all succeeded.
>
> real    28m42.619s
> user    0m7.496s
> sys     0m0.952s
>
>
> As upstream said, it indeed require specific PERL version
>
> `If you are building OpenSSL 1.1.0 and above, then you will also need 
> PERL 5.10 or high'
>
> https://wiki.openssl.org/index.php/Compilation_and_Installation
>
> And I found at least PERL 5.16.3 not work for OpenSSL 1.1.1
>
> Or add a perl version detecting in insane check? We should report a 
> lowest
> workable perl version number when detect failed. But we may update the 
> version
> number in insane check when openssl is upgraded.
>
> //Hongxu
>
>> Cheers,
>>
>> Richard
>>
>

Re: [PATCH] openssl: use nativeperl to build

[richard.purdie]

On Fri, 2018-10-12 at 15:54 +0800, Hongxu Jia wrote:
> On 2018年10月11日 22:58, Hongxu Jia wrote:
> > On 2018年10月11日 20:00, richard.purdie@linuxfoundation.org wrote:
> > > On Thu, 2018-10-11 at 02:55 -0700, Hongxu Jia wrote:
> > > > If host perl is old version (such as perl v5.16.3 on CentOS-
> > > > 7.5),
> > > > the openssl-native failed at do_install, the reason is its
> > > > Makefile
> > > > is not correct which is generated at do_configure by perl
> > > > script.
> > > > 
> > > > Use nativeperl to assure latest perl on host for openssl build.
> > > > 
> > > > Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
> > > > ---
> > > >   meta/recipes-connectivity/openssl/openssl_1.1.1.bb | 6 ++++--
> > > >   1 file changed, 4 insertions(+), 2 deletions(-)
> > > 
> > > I think this will have a noticeable effect on built times
> > > unfortunately
> > > since it means openssl-native would then depend upon perl-
> > > native...
> 
> Hi RP,
> 
> With some digging on upstream, more than perl version,
> the building also requires to install perl Text::Template module
> which does not exist in core perl, I think we have to add the recipe
> 
> [snip]
>   Notes on Perl modules we use
>   ----------------------------
>   We make increasing use of Perl modules, and do our best to limit
>   ourselves to core Perl modules to keep the requirements down. There
>   are just a few exceptions:
>   Test::More         We require the minimum version to be 0.96, which
>                      appeared in Perl 5.13.4, because that version
> was
>                      the first to have all the features we're using.
>                      This module is required for testing only!  If
> you
>                      don't plan on running the tests, you don't need
> to
>                      bother with this one.
> 
>   Text::Template     This module is not part of the core Perl
> modules.
>                      As a matter of fact, the core Perl modules do
> not
>                      include any templating module to date.
>                      This module is absolutely needed, configuration
> depends on it.
> [snip]
> 
> https://github.com/openssl/openssl/blob/master/NOTES.PERL
> 
https://github.com/openssl/openssl/commit/4e351ca92e3a1f447cef3d2e330f13941f9412c6

That is a good find. Perhaps we could check for that in sanity.bbclass?
Is it commonly available in distributions like centos? 

I am worried about the build time impact of a nativeperl dependency as
whilst your test doesn't show it in the image build, it will show up on
slower build machines with fewer threads.

Cheers,

Richard

Re: [PATCH] openssl: use nativeperl to build

[Hongxu Jia]

[-- Attachment #1: Type: text/plain, Size: 2226 bytes --]

On 2018年10月12日 16:57, richard.purdie@linuxfoundation.org wrote:
>> Hi RP,
>>
>> With some digging on upstream, more than perl version,
>> the building also requires to install perl Text::Template module
>> which does not exist in core perl, I think we have to add the recipe
>>
>> [snip]
>>    Notes on Perl modules we use
>>    ----------------------------
>>    We make increasing use of Perl modules, and do our best to limit
>>    ourselves to core Perl modules to keep the requirements down. There
>>    are just a few exceptions:
>>    Test::More         We require the minimum version to be 0.96, which
>>                       appeared in Perl 5.13.4, because that version
>> was
>>                       the first to have all the features we're using.
>>                       This module is required for testing only!  If
>> you
>>                       don't plan on running the tests, you don't need
>> to
>>                       bother with this one.
>>
>>    Text::Template     This module is not part of the core Perl
>> modules.
>>                       As a matter of fact, the core Perl modules do
>> not
>>                       include any templating module to date.
>>                       This module is absolutely needed, configuration
>> depends on it.
>> [snip]
>>
>> https://github.com/openssl/openssl/blob/master/NOTES.PERL
>>
> https://github.com/openssl/openssl/commit/4e351ca92e3a1f447cef3d2e330f13941f9412c6
>
> That is a good find. Perhaps we could check for that in sanity.bbclass?
> Is it commonly available in distributions like centos?
I am afraid the perl module (Text::Template) is not common,
available on most distro, since they do not need to build
openssl :(

But I am curious that without that perl module, the build seem
ok with newer perl (perl v5.22.1 in ubuntu 1804 and without Text::Template)

I am going on digging the root cause why the build works well
without the perl module

//Hongxu


> I am worried about the build time impact of a nativeperl dependency as
> whilst your test doesn't show it in the image build, it will show up on
> slower build machines with fewer threads.
>
> Cheers,
>
> Richard
>


[-- Attachment #2: Type: text/html, Size: 3231 bytes --]

Re: [PATCH] openssl: use nativeperl to build

[Hongxu Jia]

[-- Attachment #1: Type: text/plain, Size: 2061 bytes --]

On 2018年10月12日 17:03, Hongxu Jia wrote:
>> https://github.com/openssl/openssl/commit/4e351ca92e3a1f447cef3d2e330f13941f9412c6
>>
>> That is a good find. Perhaps we could check for that in sanity.bbclass?
>> Is it commonly available in distributions like centos?
> I am afraid the perl module (Text::Template) is not common,
> available on most distro, since they do not need to build
> openssl :(
>
> But I am curious that without that perl module, the build seem
> ok with newer perl (perl v5.22.1 in ubuntu 1804 and without 
> Text::Template)
>
> I am going on digging the root cause why the build works well
> without the perl module
>

I know the reason

It will try to use Text::Template, if failed it fallback to load the old 
one in perl source

[snip util/dofile.pl]
use with_fallback "Text::Template 1.46";
[snip util/dofile.pl]

[snip util/perl/with_fallback.pm]
sub import {
     shift;

     use File::Basename;
     use File::Spec::Functions;
     foreach (@_) {
         eval "use $_";
         if ($@) {
             unshift @INC, catdir(dirname(__FILE__),
                                  "..", "..", "external", "perl");
             my $transfer = "transfer::$_";
             eval "use $transfer";
             shift @INC;
             warn $@ if $@;
         }
     }
}
[snip util/perl/with_fallback.pm]

https://github.com/openssl/openssl/commit/8ff2af548303d311ce3591406111f77862875a60
[comments]
Bundle the non core Perl module Text::Template

Because we're using Text::Template and we know it's a non core Perl
module, we choose to bundle it into our source, for convenience.

external/perl/Downloaded.txt document what modules we choose to bundle
this way and exactly where we downloaded it from.

With this changes comes the transfer module for with_fallback.

Reviewed-by: Rich Salz <rsalz@openssl.org>
[comments]

//Hongxu

> //Hongxu



[-- Attachment #2: Type: text/html, Size: 3547 bytes --]