[Buildroot] Compiling Python packages with C extensions with per-package directories

[Buildroot] Compiling Python packages with C extensions with per-package directories

[Asaf Kahlon]

Hello,

As now some work on per-package directories has been merged for the
next release, I tried to play with it a little and encountered the
following problem:
When compiling a Python package with C sources, the compilation fails.
An example from python-cryptography (same for python-cffi, and others):
/home/asaf/repos/buildroot/output/per-package/python3/host/bin/i586-linux-gcc
-shared -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE
-D_FILE_OFFSET_BITS=64 -Os -pthread
build/temp.linux-x86_64-3.8/build/temp.linux-x86_64-3.8/_openssl.o
-L/home/asaf/repos/buildroot/output/per-package/python-cryptography/host/i586-buildroot-linux-gnu/sysroot/usr/lib
-lssl -lcrypto -o
build/lib.linux-x86_64-3.8/cryptography/hazmat/bindings/_openssl.abi3.so
/opt/host/bin/../lib/gcc/i586-buildroot-linux-gnu/8.3.0/../../../../i586-buildroot-linux-gnu/bin/ld:
cannot find /lib/libc.so.6
/opt/host/bin/../lib/gcc/i586-buildroot-linux-gnu/8.3.0/../../../../i586-buildroot-linux-gnu/bin/ld:
cannot find /usr/lib/libc_nonshared.a

As one can notice - we used gcc from the python3 directory and not the
one from python-cryptography. When trying to understand the problem, I
did 2 things:
* Replacing the compiler to the one from python-cryptography.
* Replacing the "-L" part to the correspondent one from python3 directory.
Each of the above (individually) fixes the problem, but I don't think
the second option is good because it uses a compiler from another
sysroot.
I think the problem occurs since the _sysconfigdata file is installed
when compiling Python and contains paths from the python3 directory,
although it's used by many other Python packages (I also verified that
by manually replacing relevant paths from the copy at
python-cryptography).

So, what do we want to do with this?
I don't think we want to tweak the _sysconfigdata file for every
python package (although it would lead to a more "correct" situation,
in which every sysroot contains _sysconfidata with proper paths).
In addition, I wonder why do we need to copy all the toolchain-related
files for every single package? (Maybe a symlink can help?)
Any other suggestions?

Thanks,
Asaf.

[Buildroot] Compiling Python packages with C extensions with per-package directories

[Thomas Petazzoni]

Hello Asaf,

On Sat, 30 Nov 2019 21:20:05 +0200
Asaf Kahlon <asafka7@gmail.com> wrote:

> As now some work on per-package directories has been merged for the
> next release, I tried to play with it a little and encountered the
> following problem:
> When compiling a Python package with C sources, the compilation fails.

I have this patch:

  https://github.com/tpetazzoni/buildroot/commit/f2a92b1321bd0d63e5fad81b0cfc9ace910d17a0

which I believe should fix the problems you mention. My patch series
for per-package was just the core support. I have a number of
additional patches at
https://github.com/tpetazzoni/buildroot/commits/ppsh-v8-work that fix
some issues with per-package directories (but not all, of course).


> I don't think we want to tweak the _sysconfigdata file for every
> python package (although it would lead to a more "correct" situation,
> in which every sysroot contains _sysconfidata with proper paths).
> In addition, I wonder why do we need to copy all the toolchain-related
> files for every single package? (Maybe a symlink can help?)

We don't copy things for every single package, we do hard links. All
the per-package logic is based on making copy using hard links, so it
is very cheap.

My patch at
https://github.com/tpetazzoni/buildroot/commit/f2a92b1321bd0d63e5fad81b0cfc9ace910d17a0
indeed consists in tweaking the sysconfigdata on a per package basis.
Do not hesitate to let me know if you see any other solution. Ideally,
all the paths in sysconfigdata should be defined relatively to the
location of the sysconfidata file itself. I think I tried to do that
back then, but didn't succeed.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] Compiling Python packages with C extensions with per-package directories

[Asaf Kahlon]

Hello,

On Sat, Nov 30, 2019 at 10:07 PM Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> Hello Asaf,
>
> On Sat, 30 Nov 2019 21:20:05 +0200
> Asaf Kahlon <asafka7@gmail.com> wrote:
>
> > As now some work on per-package directories has been merged for the
> > next release, I tried to play with it a little and encountered the
> > following problem:
> > When compiling a Python package with C sources, the compilation fails.
>
> I have this patch:
>
>   https://github.com/tpetazzoni/buildroot/commit/f2a92b1321bd0d63e5fad81b0cfc9ace910d17a0
>
> which I believe should fix the problems you mention. My patch series
> for per-package was just the core support. I have a number of
> additional patches at
> https://github.com/tpetazzoni/buildroot/commits/ppsh-v8-work that fix
> some issues with per-package directories (but not all, of course).

Yeah, it looks like some of your additional patches will solve those issues.

Just out of curiosity, do we have any roadmap for merging those patches or any
"known issues" regarding this feature that have to be resolved?

>
>
> > I don't think we want to tweak the _sysconfigdata file for every
> > python package (although it would lead to a more "correct" situation,
> > in which every sysroot contains _sysconfidata with proper paths).
> > In addition, I wonder why do we need to copy all the toolchain-related
> > files for every single package? (Maybe a symlink can help?)
>
> We don't copy things for every single package, we do hard links. All
> the per-package logic is based on making copy using hard links, so it
> is very cheap.
>
> My patch at
> https://github.com/tpetazzoni/buildroot/commit/f2a92b1321bd0d63e5fad81b0cfc9ace910d17a0
> indeed consists in tweaking the sysconfigdata on a per package basis.
> Do not hesitate to let me know if you see any other solution. Ideally,
> all the paths in sysconfigdata should be defined relatively to the
> location of the sysconfidata file itself. I think I tried to do that
> back then, but didn't succeed.
>
> Best regards,
>
> Thomas
> --
> Thomas Petazzoni, CTO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com

Thanks,
Asaf.

[Buildroot] Compiling Python packages with C extensions with per-package directories

[Thomas Petazzoni]

Hello Asaf,

On Sat, 30 Nov 2019 22:57:47 +0200
Asaf Kahlon <asafka7@gmail.com> wrote:

> Yeah, it looks like some of your additional patches will solve those issues.
> 
> Just out of curiosity, do we have any roadmap for merging those patches or any
> "known issues" regarding this feature that have to be resolved?

My plan was to send those additional patches once the core per-package
support would be merged, which happened just a few days ago.

I hope to be able to send those additional patches next week. Note that
they don't fix all problems. For example, it is known that Qt5 is also
broken with per-package support. Andreas had sent a patch series fixing
this, but it was not trivial.

Next week, I will also try to update
https://elinux.org/Buildroot:Top_Level_Parallel_Build to keep track of
the status: what is merged, what is fixed but not merged yet, and what
are the remaining known issues.

I will also send a patch to enable per-package support usage in the
autobuilders. Not for all builds, but for a small fraction of the
builds.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com