[Buildroot] [PATCH 1/2] target-finalize: generate all *.pyc

[Buildroot] [PATCH 1/2] target-finalize: generate all *.pyc

[Samuel Martin]

This patch generates all *.pyc files from *.py located in
<target>/usr/python*/site-packages, before stripping the rootfs.

This prevents modules from packages that do not compile the *.py
files from disappearing.

Reported-by: Yegor Yefremov <yegorslists@googlemail.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Samuel Martin <s.martin49@gmail.com>
---
 Makefile | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/Makefile b/Makefile
index af043a3..ec0b923 100644
--- a/Makefile
+++ b/Makefile
@@ -564,6 +564,11 @@ $(TARGETS_ROOTFS): target-finalize
 
 target-finalize: $(TARGETS)
 	@$(call MESSAGE,"Finalizing target directory")
+ifneq ($(BR2_PACKAGE_PYTHON)$(BR2_PACKAGE_PYTHON3),)
+	PYTHONPATH="$(PYTHON_PATH)" \
+		$(HOST_DIR)/usr/bin/python -c "import compileall; \
+			compileall.compile_dir('$(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/site-packages')"
+endif
 	$(foreach hook,$(TARGET_FINALIZE_HOOKS),$($(hook))$(sep))
 	rm -rf $(TARGET_DIR)/usr/include $(TARGET_DIR)/usr/share/aclocal \
 		$(TARGET_DIR)/usr/lib/pkgconfig $(TARGET_DIR)/usr/share/pkgconfig \
-- 
2.1.0

[Buildroot] [PATCH 2/2] package/samba4: remove SAMBA4_BUILD_PYC_FILES hook

[Samuel Martin]

This is now handled by target-finalize.

Signed-off-by: Samuel Martin <s.martin49@gmail.com>
---
 package/samba4/samba4.mk | 10 ----------
 1 file changed, 10 deletions(-)

diff --git a/package/samba4/samba4.mk b/package/samba4/samba4.mk
index 78d92ef..7b104fb 100644
--- a/package/samba4/samba4.mk
+++ b/package/samba4/samba4.mk
@@ -122,16 +122,6 @@ define SAMBA4_INSTALL_TARGET_CMDS
 	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D) DESTDIR=$(TARGET_DIR) install
 endef
 
-# Samba just installs .py files so the purge causes problems with some tools
-ifeq ($(BR2_PACKAGE_PYTHON_PYC_ONLY),y)
-define SAMBA4_BUILD_PYC_FILES
-	PYTHONPATH="$(PYTHON_PATH)" \
-		$(HOST_DIR)/usr/bin/python -c "import compileall; \
-		compileall.compile_dir('$(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/site-packages/samba')"
-endef
-SAMBA4_POST_INSTALL_TARGET_HOOKS += SAMBA4_BUILD_PYC_FILES
-endif
-
 ifeq ($(BR2_PACKAGE_SAMBA4_AD_DC),)
 SAMBA4_CONF_OPTS += --without-ad-dc
 endif
-- 
2.1.0

[Buildroot] [PATCH 1/2] target-finalize: generate all *.pyc

[Yegor Yefremov]

On Mon, Mar 9, 2015 at 5:24 PM, Samuel Martin <s.martin49@gmail.com> wrote:
> This patch generates all *.pyc files from *.py located in
> <target>/usr/python*/site-packages, before stripping the rootfs.
>
> This prevents modules from packages that do not compile the *.py
> files from disappearing.
>
> Reported-by: Yegor Yefremov <yegorslists@googlemail.com>
> Cc: Yegor Yefremov <yegorslists@googlemail.com>
> Signed-off-by: Samuel Martin <s.martin49@gmail.com>

Tested-by: Yegor Yefremov <yegorslists@googlemail.com>

> ---
>  Makefile | 5 +++++
>  1 file changed, 5 insertions(+)
>
> diff --git a/Makefile b/Makefile
> index af043a3..ec0b923 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -564,6 +564,11 @@ $(TARGETS_ROOTFS): target-finalize
>
>  target-finalize: $(TARGETS)
>         @$(call MESSAGE,"Finalizing target directory")
> +ifneq ($(BR2_PACKAGE_PYTHON)$(BR2_PACKAGE_PYTHON3),)
> +       PYTHONPATH="$(PYTHON_PATH)" \
> +               $(HOST_DIR)/usr/bin/python -c "import compileall; \
> +                       compileall.compile_dir('$(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/site-packages')"
> +endif
>         $(foreach hook,$(TARGET_FINALIZE_HOOKS),$($(hook))$(sep))
>         rm -rf $(TARGET_DIR)/usr/include $(TARGET_DIR)/usr/share/aclocal \
>                 $(TARGET_DIR)/usr/lib/pkgconfig $(TARGET_DIR)/usr/share/pkgconfig \
> --
> 2.1.0
>

[Buildroot] [PATCH 1/2] target-finalize: generate all *.pyc

[Thomas Petazzoni]

Dear Samuel Martin,

On Mon,  9 Mar 2015 17:24:23 +0100, Samuel Martin wrote:
> This patch generates all *.pyc files from *.py located in
> <target>/usr/python*/site-packages, before stripping the rootfs.
> 
> This prevents modules from packages that do not compile the *.py
> files from disappearing.
> 
> Reported-by: Yegor Yefremov <yegorslists@googlemail.com>
> Cc: Yegor Yefremov <yegorslists@googlemail.com>
> Signed-off-by: Samuel Martin <s.martin49@gmail.com>
> ---
>  Makefile | 5 +++++
>  1 file changed, 5 insertions(+)
> 
> diff --git a/Makefile b/Makefile
> index af043a3..ec0b923 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -564,6 +564,11 @@ $(TARGETS_ROOTFS): target-finalize
>  
>  target-finalize: $(TARGETS)
>  	@$(call MESSAGE,"Finalizing target directory")
> +ifneq ($(BR2_PACKAGE_PYTHON)$(BR2_PACKAGE_PYTHON3),)
> +	PYTHONPATH="$(PYTHON_PATH)" \
> +		$(HOST_DIR)/usr/bin/python -c "import compileall; \
> +			compileall.compile_dir('$(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/site-packages')"
> +endif

Why not, but then don't we want to do something in the Python package
infra, and in the Python 2/3 packages themselves to avoid spending time
biulding the .pyc files?

Also, using TARGET_FINALIZE_HOOKS would be nicer. But there is the
issue that the hook is the same between python and python3, so we
wouldn't know where to put it.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 1/2] target-finalize: generate all *.pyc

[Samuel Martin]

Hi Thomas, all,

On Mon, Mar 9, 2015 at 6:54 PM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Dear Samuel Martin,
>
> On Mon,  9 Mar 2015 17:24:23 +0100, Samuel Martin wrote:
>> This patch generates all *.pyc files from *.py located in
>> <target>/usr/python*/site-packages, before stripping the rootfs.
>>
>> This prevents modules from packages that do not compile the *.py
>> files from disappearing.
>>
>> Reported-by: Yegor Yefremov <yegorslists@googlemail.com>
>> Cc: Yegor Yefremov <yegorslists@googlemail.com>
>> Signed-off-by: Samuel Martin <s.martin49@gmail.com>
>> ---
>>  Makefile | 5 +++++
>>  1 file changed, 5 insertions(+)
>>
>> diff --git a/Makefile b/Makefile
>> index af043a3..ec0b923 100644
>> --- a/Makefile
>> +++ b/Makefile
>> @@ -564,6 +564,11 @@ $(TARGETS_ROOTFS): target-finalize
>>
>>  target-finalize: $(TARGETS)
>>       @$(call MESSAGE,"Finalizing target directory")
>> +ifneq ($(BR2_PACKAGE_PYTHON)$(BR2_PACKAGE_PYTHON3),)
>> +     PYTHONPATH="$(PYTHON_PATH)" \
>> +             $(HOST_DIR)/usr/bin/python -c "import compileall; \
>> +                     compileall.compile_dir('$(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/site-packages')"
>> +endif
>
> Why not, but then don't we want to do something in the Python package
> infra, and in the Python 2/3 packages themselves to avoid spending time
> biulding the .pyc files?

Indeed, it'd be better to avoid  wasting time building *.pyc during
per-package build, though it does not seem really easy to do at first
glance.
Well, after hacking around, it is not that hard to do it! :-)
I cleanup my hacks and will submit them.

>
> Also, using TARGET_FINALIZE_HOOKS would be nicer. But there is the
> issue that the hook is the same between python and python3, so we
> wouldn't know where to put it.

The hook is simple enough to be moved in pkg-python.mk as a helper,
then just being added to the PYTHON{,3}_TARGET_FINALIZE_HOOKS.

However, if we decide to not go using a global TARGET_FINALIZE_HOOKS,
but per-package target-install hook, the helper can be easily added to
the packages that need it; the biggest remaining task being
identifying and updating all packages not using the pkg-python infra
but providing a python module.


Regards,

-- 
Samuel

[Buildroot] [PATCH 1/2] target-finalize: generate all *.pyc

[Arnout Vandecappelle]

On 09/03/15 18:54, Thomas Petazzoni wrote:
> Dear Samuel Martin,
> 
> On Mon,  9 Mar 2015 17:24:23 +0100, Samuel Martin wrote:
>> This patch generates all *.pyc files from *.py located in
>> <target>/usr/python*/site-packages, before stripping the rootfs.
>>
>> This prevents modules from packages that do not compile the *.py
>> files from disappearing.
>>
>> Reported-by: Yegor Yefremov <yegorslists@googlemail.com>
>> Cc: Yegor Yefremov <yegorslists@googlemail.com>
>> Signed-off-by: Samuel Martin <s.martin49@gmail.com>
>> ---
>>  Makefile | 5 +++++
>>  1 file changed, 5 insertions(+)
>>
>> diff --git a/Makefile b/Makefile
>> index af043a3..ec0b923 100644
>> --- a/Makefile
>> +++ b/Makefile
>> @@ -564,6 +564,11 @@ $(TARGETS_ROOTFS): target-finalize
>>  
>>  target-finalize: $(TARGETS)
>>  	@$(call MESSAGE,"Finalizing target directory")
>> +ifneq ($(BR2_PACKAGE_PYTHON)$(BR2_PACKAGE_PYTHON3),)

 It should also be conditional on !BR2_PACKAGE_PYTHON_PY_ONLY

>> +	PYTHONPATH="$(PYTHON_PATH)" \
>> +		$(HOST_DIR)/usr/bin/python -c "import compileall; \
>> +			compileall.compile_dir('$(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/site-packages')"

 I believe the canonical way to call compileall from the command line is using
the -m option:

	$(HOST_DIR)/usr/bin/python -m compileall \
		$(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/site-packages

>> +endif
> 
> Why not, but then don't we want to do something in the Python package
> infra, and in the Python 2/3 packages themselves to avoid spending time
> biulding the .pyc files?

 compileall will check timestamps to avoid recompiling unnecessarily. So you
just have the tree walk to consider, which I think is a minor issue.


> Also, using TARGET_FINALIZE_HOOKS would be nicer. But there is the
> issue that the hook is the same between python and python3, so we
> wouldn't know where to put it.

 In fact, we already have a PYTHON_FINALIZE_TARGET (and similar for PYTHON3)
that does the removal of the .py files, so the compileall should be added there.
And that immediately handles the BR2_PACKAGE_PYTHON_PY_ONLY condition as well.


 Regards,
 Arnout


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [PATCH 1/2] target-finalize: generate all *.pyc

[Thomas Petazzoni]

Dear Samuel Martin,

On Mon, 9 Mar 2015 22:30:02 +0100, Samuel Martin wrote:

> Indeed, it'd be better to avoid  wasting time building *.pyc during
> per-package build, though it does not seem really easy to do at first
> glance.
> Well, after hacking around, it is not that hard to do it! :-)
> I cleanup my hacks and will submit them.

Ok, cool.

> The hook is simple enough to be moved in pkg-python.mk as a helper,
> then just being added to the PYTHON{,3}_TARGET_FINALIZE_HOOKS.

Indeed.

> However, if we decide to not go using a global TARGET_FINALIZE_HOOKS,
> but per-package target-install hook, the helper can be easily added to
> the packages that need it; the biggest remaining task being
> identifying and updating all packages not using the pkg-python infra
> but providing a python module.

Well, as it is currently implemented, the hook byte-compiles *all* .py
files. So if you call this hook multiple times, you're going to
byte-compile all .py files multiple times. Doesn't seem like the most
efficient thing to do.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 1/2] target-finalize: generate all *.pyc

[Thomas Petazzoni]

Dear Arnout Vandecappelle,

On Mon, 09 Mar 2015 22:42:04 +0100, Arnout Vandecappelle wrote:

> > Why not, but then don't we want to do something in the Python package
> > infra, and in the Python 2/3 packages themselves to avoid spending time
> > biulding the .pyc files?
> 
>  compileall will check timestamps to avoid recompiling unnecessarily. So you
> just have the tree walk to consider, which I think is a minor issue.

Ah, good to know. Then indeed it's useless to do any special trick:
let's just call compileall as a target finalize hook.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 1/2] target-finalize: generate all *.pyc

[Yegor Yefremov]

On Mon, Mar 9, 2015 at 10:46 PM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Dear Arnout Vandecappelle,
>
> On Mon, 09 Mar 2015 22:42:04 +0100, Arnout Vandecappelle wrote:
>
>> > Why not, but then don't we want to do something in the Python package
>> > infra, and in the Python 2/3 packages themselves to avoid spending time
>> > biulding the .pyc files?
>>
>>  compileall will check timestamps to avoid recompiling unnecessarily. So you
>> just have the tree walk to consider, which I think is a minor issue.
>
> Ah, good to know. Then indeed it's useless to do any special trick:
> let's just call compileall as a target finalize hook.

What's the current state of this issue? It would be great, if this
could be fixed for the upcoming release.

Yegor