[Buildroot] perl XS modules

[Buildroot] perl XS modules

[Arne Caspari]

Hi all, 

I try to build the libxml-parser-perl package for an ARM target.
Unfortunately the compiled perl module fails to load with the following
error message: 

loadable library and perl binaries are mismatched

After some digging I found that the module needs to be compiled with
the same toolchain and compiler defines as the perl binary in order to
be ABI compatible. Forcing the defines to be exactly the same as the
ones used for the perl binary did not fix the issue so I believe it
must be a problem with the toolchain.

I get it right, buildroot builds the host-perl package with the GCC
installed on the build system. In my case this is GCC Version 6.1 while
the toolchain in buildroot uses 4.9.3. If I get the build procedure
right, the ABI for perl is then configured for the 6.1 toolchain while
module builds will use the toolchain provided by buildroot which is ABI
incompatible. 

Is there a way to make buildroot build an initial toolchain to build
the host packages with? Did anybody ever successfully build perlXS
modules?

Thanks, 

/Arne

[Buildroot] perl XS modules

[Thomas Petazzoni]

Hello,

Fran?ois, could you have a look at the below question, that is Perl and
cross-compilation related?

Thanks!

Thomas

On Fri, 8 Jul 2016 09:59:10 +0200, Arne Caspari wrote:
> Hi all, 
> 
> I try to build the libxml-parser-perl package for an ARM target.
> Unfortunately the compiled perl module fails to load with the following
> error message: 
> 
> loadable library and perl binaries are mismatched
> 
> After some digging I found that the module needs to be compiled with
> the same toolchain and compiler defines as the perl binary in order to
> be ABI compatible. Forcing the defines to be exactly the same as the
> ones used for the perl binary did not fix the issue so I believe it
> must be a problem with the toolchain.
> 
> I get it right, buildroot builds the host-perl package with the GCC
> installed on the build system. In my case this is GCC Version 6.1 while
> the toolchain in buildroot uses 4.9.3. If I get the build procedure
> right, the ABI for perl is then configured for the 6.1 toolchain while
> module builds will use the toolchain provided by buildroot which is ABI
> incompatible. 
> 
> Is there a way to make buildroot build an initial toolchain to build
> the host packages with? Did anybody ever successfully build perlXS
> modules?
> 
> Thanks, 
> 
> /Arne
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot



-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] perl XS modules

[François Perrad]

The BR package libxml-parser-perl is a very specific host package which
works only with the the system perl.
(The reason is : this packge is heavy used, and we don't want the cost|time
of the host-perl build)

All Perl modules (XS or Pure Perl) use the naming scheme :
perl-module-name.
For example, the wrapper for libxml2 is perl-xml-libxml.
All these packages use the BR infrastructure perl-package.

If you want a target package perl-xml-parser (XML expat wrapper), start by
running:
   $ support/script/scancpan XML-Parser
which creates a decent template.
After add manually, the specific part required by the expat C library.
(And obviously, read
https://buildroot.org/downloads/manual/manual.html#_infrastructure_for_perl_cpan_packages
)

Fran?ois

2016-07-08 17:40 GMT+02:00 Thomas Petazzoni <
thomas.petazzoni@free-electrons.com>:

> Hello,
>
> Fran?ois, could you have a look at the below question, that is Perl and
> cross-compilation related?
>
> Thanks!
>
> Thomas
>
> On Fri, 8 Jul 2016 09:59:10 +0200, Arne Caspari wrote:
> > Hi all,
> >
> > I try to build the libxml-parser-perl package for an ARM target.
> > Unfortunately the compiled perl module fails to load with the following
> > error message:
> >
> > loadable library and perl binaries are mismatched
> >
> > After some digging I found that the module needs to be compiled with
> > the same toolchain and compiler defines as the perl binary in order to
> > be ABI compatible. Forcing the defines to be exactly the same as the
> > ones used for the perl binary did not fix the issue so I believe it
> > must be a problem with the toolchain.
> >
> > I get it right, buildroot builds the host-perl package with the GCC
> > installed on the build system. In my case this is GCC Version 6.1 while
> > the toolchain in buildroot uses 4.9.3. If I get the build procedure
> > right, the ABI for perl is then configured for the 6.1 toolchain while
> > module builds will use the toolchain provided by buildroot which is ABI
> > incompatible.
> >
> > Is there a way to make buildroot build an initial toolchain to build
> > the host packages with? Did anybody ever successfully build perlXS
> > modules?
> >
> > Thanks,
> >
> > /Arne
> > _______________________________________________
> > buildroot mailing list
> > buildroot at busybox.net
> > http://lists.busybox.net/mailman/listinfo/buildroot
>
>
>
> --
> Thomas Petazzoni, CTO, Free Electrons
> Embedded Linux, Kernel and Android engineering
> http://free-electrons.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20160712/c5cd40bc/attachment.html>

[Buildroot] perl XS modules

[Thomas Petazzoni]

Hello,

On Tue, 12 Jul 2016 10:00:41 +0200, Fran?ois Perrad wrote:

> The BR package libxml-parser-perl is a very specific host package which
> works only with the the system perl.
> (The reason is : this packge is heavy used, and we don't want the cost|time
> of the host-perl build)

The question from Arne was not about the host variant of
libxml-parser-perl, but about he target variant.

> All Perl modules (XS or Pure Perl) use the naming scheme :
> perl-module-name.
> For example, the wrapper for libxml2 is perl-xml-libxml.
> All these packages use the BR infrastructure perl-package.

Right, but Arne seems to indicate that when you build a *target* Perl
modules with cross-compiled C code (i.e not a pure Perl module), there
is an ABI compatibility issue, because the host perl and the target
perl don't use the same ABI.

Arne: can you give a specific Buildroot configuration and commands to
test that exhibit the problem?

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] perl XS modules

[Arne Caspari]

Fran?ois, 

Thank you very much for coming back to me. It is highly appreciated. 

I did not know about the scancpan-Script - this would have saved me a
ton of work, literally since I just ported a package with very very
many module dependencies (the Logitech Media Server) to buildroot. Ah
well...

I finally found the issue with the perl XS packages though: The perl
MakeMaker module fails to generate a correct Makefile but instead
messes up the configuration with the host-perl one. I kind of fixed the
issue for me in a very hackish way by modifying the generated Makefile
with the following script: 

define PERL_XML_PARSER_BUILD_CMDS
	sed "s~/usr/bin/gcc~$(TARGET_CC) -Os~" -i $(@D)/Makefile
	sed "s~/usr/bin/gcc~$(TARGET_CC) -Os~" -i $(@D)/Expat/Makefile
	sed "s~x86_64~$(KERNEL_ARCH)~" -i $(@D)/Expat/Makefile
	sed "s~host/usr/lib~staging/usr/lib~" -i $(@D)/Expat/Makefile
	$(MAKE) $(@D)/Makefile -C $(@D)
endef

The module loads correctly afterwards. Unfortunately I do not have any
experience with perl or the MakeMaker module to come up with a less
insane fix.


/Arne

Am Tue, 12 Jul 2016 10:00:41 +0200
schrieb Fran?ois Perrad <fperrad@gmail.com>:

> The BR package libxml-parser-perl is a very specific host package
> which works only with the the system perl.
> (The reason is : this packge is heavy used, and we don't want the
> cost|time of the host-perl build)
> 
> All Perl modules (XS or Pure Perl) use the naming scheme :
> perl-module-name.
> For example, the wrapper for libxml2 is perl-xml-libxml.
> All these packages use the BR infrastructure perl-package.
> 
> If you want a target package perl-xml-parser (XML expat wrapper),
> start by running:
>    $ support/script/scancpan XML-Parser
> which creates a decent template.
> After add manually, the specific part required by the expat C library.
> (And obviously, read
> https://buildroot.org/downloads/manual/manual.html#_infrastructure_for_perl_cpan_packages
> )
> 
> Fran?ois
> 
> 2016-07-08 17:40 GMT+02:00 Thomas Petazzoni <
> thomas.petazzoni at free-electrons.com>:
> 
> > Hello,
> >
> > Fran?ois, could you have a look at the below question, that is Perl
> > and cross-compilation related?
> >
> > Thanks!
> >
> > Thomas
> >
> > On Fri, 8 Jul 2016 09:59:10 +0200, Arne Caspari wrote:  
> > > Hi all,
> > >
> > > I try to build the libxml-parser-perl package for an ARM target.
> > > Unfortunately the compiled perl module fails to load with the
> > > following error message:
> > >
> > > loadable library and perl binaries are mismatched
> > >
> > > After some digging I found that the module needs to be compiled
> > > with the same toolchain and compiler defines as the perl binary
> > > in order to be ABI compatible. Forcing the defines to be exactly
> > > the same as the ones used for the perl binary did not fix the
> > > issue so I believe it must be a problem with the toolchain.
> > >
> > > I get it right, buildroot builds the host-perl package with the
> > > GCC installed on the build system. In my case this is GCC Version
> > > 6.1 while the toolchain in buildroot uses 4.9.3. If I get the
> > > build procedure right, the ABI for perl is then configured for
> > > the 6.1 toolchain while module builds will use the toolchain
> > > provided by buildroot which is ABI incompatible.
> > >
> > > Is there a way to make buildroot build an initial toolchain to
> > > build the host packages with? Did anybody ever successfully build
> > > perlXS modules?
> > >
> > > Thanks,
> > >
> > > /Arne
> > > _______________________________________________
> > > buildroot mailing list
> > > buildroot at busybox.net
> > > http://lists.busybox.net/mailman/listinfo/buildroot  
> >
> >
> >
> > --
> > Thomas Petazzoni, CTO, Free Electrons
> > Embedded Linux, Kernel and Android engineering
> > http://free-electrons.com
> >  

[Buildroot] perl XS modules

[Thomas Petazzoni]

Hello,

On Tue, 12 Jul 2016 10:15:54 +0200, Arne Caspari wrote:

> Thank you very much for coming back to me. It is highly appreciated. 
> 
> I did not know about the scancpan-Script - this would have saved me a
> ton of work, literally since I just ported a package with very very
> many module dependencies (the Logitech Media Server) to buildroot. Ah
> well...

The scancpan script is mentioned in the Buildroot manual:

"""

 So, this file and the Config.in can be generated by running the script
 supports/scripts/scancpan Foo-Bar in the Buildroot directory (or in
 the BR2_EXTERNAL directory). This script creates a Config.in file and
 foo-bar.mk file for the requested package, and also recursively for
 all dependencies specified by CPAN. You should still manually edit the
 result. In particular, the following things should be checked.

"""

> I finally found the issue with the perl XS packages though: The perl
> MakeMaker module fails to generate a correct Makefile but instead
> messes up the configuration with the host-perl one. I kind of fixed the
> issue for me in a very hackish way by modifying the generated Makefile
> with the following script: 
> 
> define PERL_XML_PARSER_BUILD_CMDS
> 	sed "s~/usr/bin/gcc~$(TARGET_CC) -Os~" -i $(@D)/Makefile
> 	sed "s~/usr/bin/gcc~$(TARGET_CC) -Os~" -i $(@D)/Expat/Makefile
> 	sed "s~x86_64~$(KERNEL_ARCH)~" -i $(@D)/Expat/Makefile
> 	sed "s~host/usr/lib~staging/usr/lib~" -i $(@D)/Expat/Makefile
> 	$(MAKE) $(@D)/Makefile -C $(@D)
> endef
> 
> The module loads correctly afterwards. Unfortunately I do not have any
> experience with perl or the MakeMaker module to come up with a less
> insane fix.

Again, could you give a Buildroot configuration, and a test scenario
that allows us to reproduce the issue ?

Thanks,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] perl XS modules

[Arne Caspari]

Am Tue, 12 Jul 2016 10:56:24 +0200
schrieb Thomas Petazzoni <thomas.petazzoni@free-electrons.com>:
> 
> Again, could you give a Buildroot configuration, and a test scenario
> that allows us to reproduce the issue ?
> 

Thomas, 

for me the following steps reproduce the issue: 

support/scripts/scancpan XML-Parser
[Enable the perl-xml-parser module in the config]
make

Then on the target: 

perl -e "require XML::Parser"


The buildroot configuration should not matter as long as the target has
a different architecture (eg: ARM ). If it is the same architecture as
the host, it could produce binary compatible modules by accident. 

You could even verify the presence of this issue by just compiling the
perl-xml-parser module (any architecture): If you look at the build
output, you will see that the Makefile uses the configuration and
destination paths of the host perl. 

/Arne

[Buildroot] perl XS modules

[François Perrad]

Arne,

Could you try to build a package like perl-gd or perl-net-ssleay, in order
to verify your infrastructure ?
These packages are known to build on autobuild.buildroot.net .

Fran?ois


2016-07-12 11:04 GMT+02:00 Arne Caspari <arne.caspari@gmail.com>:

> Am Tue, 12 Jul 2016 10:56:24 +0200
> schrieb Thomas Petazzoni <thomas.petazzoni@free-electrons.com>:
> >
> > Again, could you give a Buildroot configuration, and a test scenario
> > that allows us to reproduce the issue ?
> >
>
> Thomas,
>
> for me the following steps reproduce the issue:
>
> support/scripts/scancpan XML-Parser
> [Enable the perl-xml-parser module in the config]
> make
>
> Then on the target:
>
> perl -e "require XML::Parser"
>
>
> The buildroot configuration should not matter as long as the target has
> a different architecture (eg: ARM ). If it is the same architecture as
> the host, it could produce binary compatible modules by accident.
>
> You could even verify the presence of this issue by just compiling the
> perl-xml-parser module (any architecture): If you look at the build
> output, you will see that the Makefile uses the configuration and
> destination paths of the host perl.
>
> /Arne
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20160712/ed41024a/attachment.html>

[Buildroot] perl XS modules

[Thomas Petazzoni]

Hello,

On Tue, 12 Jul 2016 11:43:49 +0200, Fran?ois Perrad wrote:

> Could you try to build a package like perl-gd or perl-net-ssleay, in order
> to verify your infrastructure ?
> These packages are known to build on autobuild.buildroot.net .

Arne is not complaining about build problems, but about runtime
problems. Runtime problems are not detected by autobuild.buildroot.net.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] perl XS modules

[François Perrad]

2016-07-12 12:16 GMT+02:00 Thomas Petazzoni <
thomas.petazzoni@free-electrons.com>:

> Hello,
>
> On Tue, 12 Jul 2016 11:43:49 +0200, Fran?ois Perrad wrote:
>
> > Could you try to build a package like perl-gd or perl-net-ssleay, in
> order
> > to verify your infrastructure ?
> > These packages are known to build on autobuild.buildroot.net .
>
> Arne is not complaining about build problems, but about runtime
> problems. Runtime problems are not detected by autobuild.buildroot.net.
>
>
After building, it's easy to test them:
    # perl -MGD -e 1
    ...

Fran?ois


> Thomas
> --
> Thomas Petazzoni, CTO, Free Electrons
> Embedded Linux, Kernel and Android engineering
> http://free-electrons.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20160712/dfc21e80/attachment.html>

[Buildroot] perl XS modules

[Thomas Petazzoni]

Hello,

On Tue, 12 Jul 2016 13:05:52 +0200, Fran?ois Perrad wrote:

> > Arne is not complaining about build problems, but about runtime
> > problems. Runtime problems are not detected by autobuild.buildroot.net.
> >
> After building, it's easy to test them:
>     # perl -MGD -e 1
>     ...

This involves booting the generated system on a target platform (or
Qemu), which is not currently supported by autobuild.b.o.

I did start an effort to add runtime tests for Buildroot, see
https://github.com/tpetazzoni/buildroot-runtime-test, but there's still
a lot of work that remains to be done.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com