* [Buildroot] [PATCH 3/3] python3: Compile host-python3 statically @ 2016-09-16 13:30 Frederik Aalund 2016-09-16 17:08 ` Thomas Petazzoni 0 siblings, 1 reply; 5+ messages in thread From: Frederik Aalund @ 2016-09-16 13:30 UTC (permalink / raw) To: buildroot The previous behaviour was to compile host-python3 with `--enable-shared` as is done for the target python installation. The problem is that if the host *already* has a python installation, then host-python3 will be called with the system `libpython3.5.so`. So even though the $(HOST_DIR) python executable is called, the system .so is used. On my system, I have python 3.5.1. I was trying to python 3.5.2 for buildroot. I noticed druing the build that the host-python thought it was 3.5.1 (as on the system). This was caused by the above. In combination with the `PYTHON3_PYC_ONLY`, this caused all the *.pyc files to be compiled for 3.5.1 even though it should have been 3.5.2. I've changed the `python3.mk` file so that host-python3 is built statically. This way, such errors will not occur. Alternatively, one could make sure to always call host-python3 with an `LD_LIBRARY_PATH` which points to the `$(HOST_DIR)`. However, this requires a much more invasive change than simply compiling python statically. Signed-off-by: Frederik Aalund <fpa@sbtaqua.com> --- package/python3/python3.mk | 1 + 1 file changed, 1 insertion(+) diff --git a/package/python3/python3.mk b/package/python3/python3.mk index 1b63f95..093f570 100644 --- a/package/python3/python3.mk +++ b/package/python3/python3.mk @@ -25,6 +25,7 @@ PYTHON3_LIBTOOL_PATCH = NO # third-party Python modules. HOST_PYTHON3_CONF_OPTS += \ + --disable-shared \ --without-ensurepip \ --without-cxx-main \ --disable-sqlite3 \ -- 2.5.0 ^ permalink raw reply related [flat|nested] 5+ messages in thread
* [Buildroot] [PATCH 3/3] python3: Compile host-python3 statically 2016-09-16 13:30 [Buildroot] [PATCH 3/3] python3: Compile host-python3 statically Frederik Aalund @ 2016-09-16 17:08 ` Thomas Petazzoni 2016-09-20 13:57 ` Frederik Peter Aalund 0 siblings, 1 reply; 5+ messages in thread From: Thomas Petazzoni @ 2016-09-16 17:08 UTC (permalink / raw) To: buildroot Hello, Please wrap the text in your commit messages to a reasonable line width (72 characters). On Fri, 16 Sep 2016 15:30:31 +0200, Frederik Aalund wrote: > The previous behaviour was to compile host-python3 with `--enable-shared` as is done for the target python installation. The problem is that if the host *already* has a python installation, then host-python3 will be called with the system `libpython3.5.so`. So even though the $(HOST_DIR) python executable is called, the system .so is used. This should not be the case. All host binaries we build are built with -Wl,-rpath,$(HOST_DIR)/usr/lib, which means that they should look up libraries in $(HOST_DIR)/usr/lib *before* looking at /usr/lib. Can you double check what "readelf -a output/host/usr/bin/python" says about this? readelf should show you this rpath encoded into the Python binary. If that's not the case, then this is the problem. Best regards, Thomas -- Thomas Petazzoni, CTO, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com ^ permalink raw reply [flat|nested] 5+ messages in thread
* [Buildroot] [PATCH 3/3] python3: Compile host-python3 statically 2016-09-16 17:08 ` Thomas Petazzoni @ 2016-09-20 13:57 ` Frederik Peter Aalund 2016-09-23 16:10 ` Frederik Peter Aalund 0 siblings, 1 reply; 5+ messages in thread From: Frederik Peter Aalund @ 2016-09-20 13:57 UTC (permalink / raw) To: buildroot Hi Thomas Thanks again for reviewing. It appears that the host-python3 executable doesn't have an rpath entry. This would indeed explain the weird behaviour. I looked at the build log and found that "-Wl,-rpath" is indeed correctly passed on: -Wl,-rpath,/projects/RedPitaya/OS/buildroot/buildroot/output/host/usr/lib This is reflected in the resulting Makefile: CONFIGURE_LDFLAGS= -L/projects/RedPitaya/OS/buildroot/buildroot/output/host/lib -L/projects/RedPitaya/OS/buildroot/buildroot/output/host/usr/lib -Wl,-rpath,/projects/RedPitaya/OS/buildroot/buildroot/output/host/usr/lib -Wl,--enable-new-dtags So I don't know how it can be that the rpath is not set in the final binary. Do you also see this problem? Thanks again. *Frederik Aalund* Chief Information Officer, Co-owner SBT Aqua Copenhagen, Denmark Mobile: +45 30340086 fpa at sbtaqua.com sbtaqua.com On Fri, 16 Sep 2016 at 19:08 Thomas Petazzoni < thomas.petazzoni@free-electrons.com> wrote: > Hello, > > Please wrap the text in your commit messages to a reasonable line width > (72 characters). > > On Fri, 16 Sep 2016 15:30:31 +0200, Frederik Aalund wrote: > > The previous behaviour was to compile host-python3 with > `--enable-shared` as is done for the target python installation. The > problem is that if the host *already* has a python installation, then > host-python3 will be called with the system `libpython3.5.so`. So even > though the $(HOST_DIR) python executable is called, the system .so is used. > > This should not be the case. All host binaries we build are built with > -Wl,-rpath,$(HOST_DIR)/usr/lib, which means that they should look up > libraries in $(HOST_DIR)/usr/lib *before* looking at /usr/lib. > > Can you double check what "readelf -a output/host/usr/bin/python" says > about this? readelf should show you this rpath encoded into the Python > binary. If that's not the case, then this is the problem. > > Best regards, > > Thomas > -- > Thomas Petazzoni, CTO, Free Electrons > Embedded Linux and Kernel engineering > http://free-electrons.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20160920/9859dd75/attachment.html> ^ permalink raw reply [flat|nested] 5+ messages in thread
* [Buildroot] [PATCH 3/3] python3: Compile host-python3 statically 2016-09-20 13:57 ` Frederik Peter Aalund @ 2016-09-23 16:10 ` Frederik Peter Aalund 2016-09-23 21:14 ` Thomas Petazzoni 0 siblings, 1 reply; 5+ messages in thread From: Frederik Peter Aalund @ 2016-09-23 16:10 UTC (permalink / raw) To: buildroot I noticed that while the host-python executable doesn't have an rpath value, it does have a runpath value: Tag Type Name/Value 0x0000000000000001 (NEEDED) Shared library: [libpython3.5m.so.1.0] 0x0000000000000001 (NEEDED) Shared library: [libpthread.so.0] 0x0000000000000001 (NEEDED) Shared library: [libc.so.6] 0x000000000000001d (RUNPATH) Library runpath: [/projects/RedPitaya/OS/buildroot/buildroot/output/host/usr/lib] 0x000000000000000c (INIT) 0x4008c0 Turns out that I had LD_LIBRARY_PATH defined which overrides runpath (though not rpath). After undefining LD_LIBRARY_PATH, everything worked. Sorry for the confusion. *Frederik Aalund* Chief Information Officer, Co-owner SBT Aqua Copenhagen, Denmark Mobile: +45 30340086 fpa at sbtaqua.com sbtaqua.com On Tue, 20 Sep 2016 at 15:57 Frederik Peter Aalund <fpa@sbtaqua.com> wrote: > Hi Thomas > > Thanks again for reviewing. It appears that the host-python3 executable > doesn't have an rpath entry. This would indeed explain the weird behaviour. > > I looked at the build log and found that "-Wl,-rpath" is indeed correctly > passed on: > -Wl,-rpath,/projects/RedPitaya/OS/buildroot/buildroot/output/host/usr/lib > > This is reflected in the resulting Makefile: > CONFIGURE_LDFLAGS= > -L/projects/RedPitaya/OS/buildroot/buildroot/output/host/lib > -L/projects/RedPitaya/OS/buildroot/buildroot/output/host/usr/lib > -Wl,-rpath,/projects/RedPitaya/OS/buildroot/buildroot/output/host/usr/lib > -Wl,--enable-new-dtags > > So I don't know how it can be that the rpath is not set in the final > binary. Do you also see this problem? > > Thanks again. > > *Frederik Aalund* > Chief Information Officer, Co-owner > > SBT Aqua > Copenhagen, Denmark > Mobile: +45 30340086 > fpa at sbtaqua.com > sbtaqua.com > > On Fri, 16 Sep 2016 at 19:08 Thomas Petazzoni < > thomas.petazzoni at free-electrons.com> wrote: > >> Hello, >> >> Please wrap the text in your commit messages to a reasonable line width >> (72 characters). >> >> On Fri, 16 Sep 2016 15:30:31 +0200, Frederik Aalund wrote: >> > The previous behaviour was to compile host-python3 with >> `--enable-shared` as is done for the target python installation. The >> problem is that if the host *already* has a python installation, then >> host-python3 will be called with the system `libpython3.5.so`. So even >> though the $(HOST_DIR) python executable is called, the system .so is used. >> >> This should not be the case. All host binaries we build are built with >> -Wl,-rpath,$(HOST_DIR)/usr/lib, which means that they should look up >> libraries in $(HOST_DIR)/usr/lib *before* looking at /usr/lib. >> >> Can you double check what "readelf -a output/host/usr/bin/python" says >> about this? readelf should show you this rpath encoded into the Python >> binary. If that's not the case, then this is the problem. >> >> Best regards, >> >> Thomas >> -- >> Thomas Petazzoni, CTO, Free Electrons >> Embedded Linux and Kernel engineering >> http://free-electrons.com >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20160923/9ad57756/attachment.html> ^ permalink raw reply [flat|nested] 5+ messages in thread
* [Buildroot] [PATCH 3/3] python3: Compile host-python3 statically 2016-09-23 16:10 ` Frederik Peter Aalund @ 2016-09-23 21:14 ` Thomas Petazzoni 0 siblings, 0 replies; 5+ messages in thread From: Thomas Petazzoni @ 2016-09-23 21:14 UTC (permalink / raw) To: buildroot Hello, On Fri, 23 Sep 2016 16:10:41 +0000, Frederik Peter Aalund wrote: > I noticed that while the host-python executable doesn't have an rpath > value, it does have a runpath value: > > Tag Type Name/Value > 0x0000000000000001 (NEEDED) Shared library: > [libpython3.5m.so.1.0] > 0x0000000000000001 (NEEDED) Shared library: [libpthread.so.0] > 0x0000000000000001 (NEEDED) Shared library: [libc.so.6] > 0x000000000000001d (RUNPATH) Library runpath: > [/projects/RedPitaya/OS/buildroot/buildroot/output/host/usr/lib] > 0x000000000000000c (INIT) 0x4008c0 Interesting. We are passing -Wl,-rpath, so it should be a rpath. Not sure why it gets turned into a runpath :/ > Turns out that I had LD_LIBRARY_PATH defined which overrides runpath > (though not rpath). After undefining LD_LIBRARY_PATH, everything worked. Ah, yes, indeed. But as said above, we're supposed to have a rpath in all binaries installed in output/host/usr/bin, so having a runpath here is not expected. However, it at least explains why you were seeing the problem, while many other people were using host-python3 without any problem. Thanks! Thomas -- Thomas Petazzoni, CTO, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2016-09-23 21:14 UTC | newest] Thread overview: 5+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2016-09-16 13:30 [Buildroot] [PATCH 3/3] python3: Compile host-python3 statically Frederik Aalund 2016-09-16 17:08 ` Thomas Petazzoni 2016-09-20 13:57 ` Frederik Peter Aalund 2016-09-23 16:10 ` Frederik Peter Aalund 2016-09-23 21:14 ` Thomas Petazzoni
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox