[PATCH 0/3] distutils3: add egg files to path

[PATCH 0/3] distutils3: add egg files to path

[Jose Lamego]

Packages that use egg files or directories for installation may
not be found when imported at the python3 interpreter.
egg files/directories path must be included in a pth file to
be appropriately included in python path.

These changes handle the egg files from the distutils3 class to avoid
the need to perform it individually at each recipe.

[YOCTO #8673]

The following changes since commit 63f899a950daf1018999455bafa7a2be8b22f164:

  bitbake: toaster: bin/toaster whitelist TOASTER_DIR (2017-01-17 13:18:47 +0000)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib lamego/bug8673
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=lamego/bug8673

Jose Lamego (3):
  distutils3.bbclass: add egg files/directories to python path
  python3-pip: Remove pth file creation at recipe level
  python3-setuptools: Remove pth file creation at recipe level

 meta/classes/distutils3.bbclass                           | 10 ++++++++++
 meta/recipes-devtools/python/python3-pip_9.0.1.bb         |  3 ---
 meta/recipes-devtools/python/python3-setuptools_31.0.0.bb |  3 ---
 3 files changed, 10 insertions(+), 6 deletions(-)

-- 
1.8.3.1

[PATCH 1/3] distutils3.bbclass: add egg files/directories to python path

[Jose Lamego]

Packages that use .egg files or directories for installation may
not be found when imported at the python3 interpreter.
.egg files/directories path must be included in a .pth file to
be appropriately included in python path.

This change looks for .egg files/directories in sitepackages
and adds its path to a .pth file during package installation.
It ensures that any new package that uses .egg files/recipes
will be appropriately added to path by performing the check from
the distutils3 class.

[YOCTO #8673]

Signed-off-by: Jose Lamego <jose.a.lamego@linux.intel.com>
---
 meta/classes/distutils3.bbclass | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/meta/classes/distutils3.bbclass b/meta/classes/distutils3.bbclass
index a6720c5..f8e2e2e 100644
--- a/meta/classes/distutils3.bbclass
+++ b/meta/classes/distutils3.bbclass
@@ -68,6 +68,16 @@ distutils3_do_install() {
             mv -f ${D}${datadir}/share/* ${D}${datadir}/
             rmdir ${D}${datadir}/share
         fi
+
+        # detect if .egg files/directories were created and add their
+        # path to a .pth file
+        SHORT_PN=$(echo "${PN}" | sed 's/${PYTHON_PN}-//g')
+        if test -e ${D}${PYTHON_SITEPACKAGES_DIR}/${SHORT_PN}*.egg; then
+            EGG_NAME=$(basename $(find ${D}${PYTHON_SITEPACKAGES_DIR}/ \
+-name ${SHORT_PN}\*.egg))
+            echo "./${EGG_NAME}" > ${D}${PYTHON_SITEPACKAGES_DIR}/\
+${SHORT_PN}.pth
+        fi
 }
 distutils3_do_install[vardepsexclude] = "MACHINE"
 
-- 
1.8.3.1

Re: [PATCH 1/3] distutils3.bbclass: add egg files/directories to python path

[Jose Lamego]

[-- Attachment #1.1: Type: text/plain, Size: 1729 bytes --]

Ping.

On 01/17/2017 09:42 AM, Jose Lamego wrote:
> Packages that use .egg files or directories for installation may
> not be found when imported at the python3 interpreter.
> .egg files/directories path must be included in a .pth file to
> be appropriately included in python path.
> 
> This change looks for .egg files/directories in sitepackages
> and adds its path to a .pth file during package installation.
> It ensures that any new package that uses .egg files/recipes
> will be appropriately added to path by performing the check from
> the distutils3 class.
> 
> [YOCTO #8673]
> 
> Signed-off-by: Jose Lamego <jose.a.lamego@linux.intel.com>
> ---
>  meta/classes/distutils3.bbclass | 10 ++++++++++
>  1 file changed, 10 insertions(+)
> 
> diff --git a/meta/classes/distutils3.bbclass b/meta/classes/distutils3.bbclass
> index a6720c5..f8e2e2e 100644
> --- a/meta/classes/distutils3.bbclass
> +++ b/meta/classes/distutils3.bbclass
> @@ -68,6 +68,16 @@ distutils3_do_install() {
>              mv -f ${D}${datadir}/share/* ${D}${datadir}/
>              rmdir ${D}${datadir}/share
>          fi
> +
> +        # detect if .egg files/directories were created and add their
> +        # path to a .pth file
> +        SHORT_PN=$(echo "${PN}" | sed 's/${PYTHON_PN}-//g')
> +        if test -e ${D}${PYTHON_SITEPACKAGES_DIR}/${SHORT_PN}*.egg; then
> +            EGG_NAME=$(basename $(find ${D}${PYTHON_SITEPACKAGES_DIR}/ \
> +-name ${SHORT_PN}\*.egg))
> +            echo "./${EGG_NAME}" > ${D}${PYTHON_SITEPACKAGES_DIR}/\
> +${SHORT_PN}.pth
> +        fi
>  }
>  distutils3_do_install[vardepsexclude] = "MACHINE"
>  
> 

-- 
Jose Lamego | OTC Embedded Platforms & Tools | GDC


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 501 bytes --]

Re: [PATCH 1/3] distutils3.bbclass: add egg files/directories to python path

[Jose Lamego]

[-- Attachment #1.1: Type: text/plain, Size: 1834 bytes --]

Ping

On 02/08/2017 04:30 PM, Jose Lamego wrote:
> Ping.
> 
> On 01/17/2017 09:42 AM, Jose Lamego wrote:
>> Packages that use .egg files or directories for installation may
>> not be found when imported at the python3 interpreter.
>> .egg files/directories path must be included in a .pth file to
>> be appropriately included in python path.
>>
>> This change looks for .egg files/directories in sitepackages
>> and adds its path to a .pth file during package installation.
>> It ensures that any new package that uses .egg files/recipes
>> will be appropriately added to path by performing the check from
>> the distutils3 class.
>>
>> [YOCTO #8673]
>>
>> Signed-off-by: Jose Lamego <jose.a.lamego@linux.intel.com>
>> ---
>>  meta/classes/distutils3.bbclass | 10 ++++++++++
>>  1 file changed, 10 insertions(+)
>>
>> diff --git a/meta/classes/distutils3.bbclass b/meta/classes/distutils3.bbclass
>> index a6720c5..f8e2e2e 100644
>> --- a/meta/classes/distutils3.bbclass
>> +++ b/meta/classes/distutils3.bbclass
>> @@ -68,6 +68,16 @@ distutils3_do_install() {
>>              mv -f ${D}${datadir}/share/* ${D}${datadir}/
>>              rmdir ${D}${datadir}/share
>>          fi
>> +
>> +        # detect if .egg files/directories were created and add their
>> +        # path to a .pth file
>> +        SHORT_PN=$(echo "${PN}" | sed 's/${PYTHON_PN}-//g')
>> +        if test -e ${D}${PYTHON_SITEPACKAGES_DIR}/${SHORT_PN}*.egg; then
>> +            EGG_NAME=$(basename $(find ${D}${PYTHON_SITEPACKAGES_DIR}/ \
>> +-name ${SHORT_PN}\*.egg))
>> +            echo "./${EGG_NAME}" > ${D}${PYTHON_SITEPACKAGES_DIR}/\
>> +${SHORT_PN}.pth
>> +        fi
>>  }
>>  distutils3_do_install[vardepsexclude] = "MACHINE"
>>  
>>
> 
> 
> 

-- 
Jose Lamego | OTC Embedded Platforms & Tools | GDC


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 484 bytes --]

[PATCH 2/3] python3-pip: Remove pth file creation at recipe level

[Jose Lamego]

Handling of installed eggs is now performed at the distutils3 bbclass.

This change removes the pth file creation at recipe level.

[YOCTO #8673]

Signed-off-by: Jose Lamego <jose.a.lamego@linux.intel.com>
---
 meta/recipes-devtools/python/python3-pip_9.0.1.bb | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/meta/recipes-devtools/python/python3-pip_9.0.1.bb b/meta/recipes-devtools/python/python3-pip_9.0.1.bb
index 6ac94bb..9dce107 100644
--- a/meta/recipes-devtools/python/python3-pip_9.0.1.bb
+++ b/meta/recipes-devtools/python/python3-pip_9.0.1.bb
@@ -31,9 +31,6 @@ do_install_append() {
 
     # Install as pip3 and leave pip2 as default
     rm ${D}/${bindir}/pip
-
-    # Installed eggs need to be passed directly to the interpreter via a pth file
-    echo "./${SRCNAME}-${PV}-py${PYTHON_BASEVERSION}.egg" > ${D}${PYTHON_SITEPACKAGES_DIR}/${SRCNAME}-${PV}.pth
 }
 
 RDEPENDS_${PN} = "\
-- 
1.8.3.1

Re: [PATCH 2/3] python3-pip: Remove pth file creation at recipe level

[Jose Lamego]

[-- Attachment #1.1: Type: text/plain, Size: 1094 bytes --]

Ping.

On 01/17/2017 09:42 AM, Jose Lamego wrote:
> Handling of installed eggs is now performed at the distutils3 bbclass.
> 
> This change removes the pth file creation at recipe level.
> 
> [YOCTO #8673]
> 
> Signed-off-by: Jose Lamego <jose.a.lamego@linux.intel.com>
> ---
>  meta/recipes-devtools/python/python3-pip_9.0.1.bb | 3 ---
>  1 file changed, 3 deletions(-)
> 
> diff --git a/meta/recipes-devtools/python/python3-pip_9.0.1.bb b/meta/recipes-devtools/python/python3-pip_9.0.1.bb
> index 6ac94bb..9dce107 100644
> --- a/meta/recipes-devtools/python/python3-pip_9.0.1.bb
> +++ b/meta/recipes-devtools/python/python3-pip_9.0.1.bb
> @@ -31,9 +31,6 @@ do_install_append() {
>  
>      # Install as pip3 and leave pip2 as default
>      rm ${D}/${bindir}/pip
> -
> -    # Installed eggs need to be passed directly to the interpreter via a pth file
> -    echo "./${SRCNAME}-${PV}-py${PYTHON_BASEVERSION}.egg" > ${D}${PYTHON_SITEPACKAGES_DIR}/${SRCNAME}-${PV}.pth
>  }
>  
>  RDEPENDS_${PN} = "\
> 

-- 
Jose Lamego | OTC Embedded Platforms & Tools | GDC


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 501 bytes --]

Re: [PATCH 2/3] python3-pip: Remove pth file creation at recipe level

[Jose Lamego]

[-- Attachment #1.1: Type: text/plain, Size: 1186 bytes --]

Ping.

On 02/08/2017 04:31 PM, Jose Lamego wrote:
> Ping.
> 
> On 01/17/2017 09:42 AM, Jose Lamego wrote:
>> Handling of installed eggs is now performed at the distutils3 bbclass.
>>
>> This change removes the pth file creation at recipe level.
>>
>> [YOCTO #8673]
>>
>> Signed-off-by: Jose Lamego <jose.a.lamego@linux.intel.com>
>> ---
>>  meta/recipes-devtools/python/python3-pip_9.0.1.bb | 3 ---
>>  1 file changed, 3 deletions(-)
>>
>> diff --git a/meta/recipes-devtools/python/python3-pip_9.0.1.bb b/meta/recipes-devtools/python/python3-pip_9.0.1.bb
>> index 6ac94bb..9dce107 100644
>> --- a/meta/recipes-devtools/python/python3-pip_9.0.1.bb
>> +++ b/meta/recipes-devtools/python/python3-pip_9.0.1.bb
>> @@ -31,9 +31,6 @@ do_install_append() {
>>  
>>      # Install as pip3 and leave pip2 as default
>>      rm ${D}/${bindir}/pip
>> -
>> -    # Installed eggs need to be passed directly to the interpreter via a pth file
>> -    echo "./${SRCNAME}-${PV}-py${PYTHON_BASEVERSION}.egg" > ${D}${PYTHON_SITEPACKAGES_DIR}/${SRCNAME}-${PV}.pth
>>  }
>>  
>>  RDEPENDS_${PN} = "\
>>
> 
> 
> 

-- 
Jose Lamego | OTC Embedded Platforms & Tools | GDC


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 484 bytes --]

[PATCH 3/3] python3-setuptools: Remove pth file creation at recipe level

[Jose Lamego]

Handling of installed eggs is now performed at the distutils3 bbclass.

This change removes the pth file creation at recipe level.

[YOCTO #8673]

Signed-off-by: Jose Lamego <jose.a.lamego@linux.intel.com>
---
 meta/recipes-devtools/python/python3-setuptools_31.0.0.bb | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/meta/recipes-devtools/python/python3-setuptools_31.0.0.bb b/meta/recipes-devtools/python/python3-setuptools_31.0.0.bb
index 65af6f0..b37b381 100644
--- a/meta/recipes-devtools/python/python3-setuptools_31.0.0.bb
+++ b/meta/recipes-devtools/python/python3-setuptools_31.0.0.bb
@@ -7,11 +7,8 @@ inherit distutils3
 
 DISTUTILS_INSTALL_ARGS += "--install-lib=${D}${PYTHON_SITEPACKAGES_DIR}"
 
-# The installer puts the wrong path in the setuptools.pth file.  Correct it.
 do_install_append() {
-    rm ${D}${PYTHON_SITEPACKAGES_DIR}/setuptools.pth
     mv ${D}${bindir}/easy_install ${D}${bindir}/easy3_install
-    echo "./${SRCNAME}-${PV}-py${PYTHON_BASEVERSION}.egg" > ${D}${PYTHON_SITEPACKAGES_DIR}/setuptools.pth
 }
 
 RDEPENDS_${PN} = "\
-- 
1.8.3.1

Re: [PATCH 3/3] python3-setuptools: Remove pth file creation at recipe level

[Jose Lamego]

[-- Attachment #1.1: Type: text/plain, Size: 1263 bytes --]

Ping.

On 01/17/2017 09:42 AM, Jose Lamego wrote:
> Handling of installed eggs is now performed at the distutils3 bbclass.
> 
> This change removes the pth file creation at recipe level.
> 
> [YOCTO #8673]
> 
> Signed-off-by: Jose Lamego <jose.a.lamego@linux.intel.com>
> ---
>  meta/recipes-devtools/python/python3-setuptools_31.0.0.bb | 3 ---
>  1 file changed, 3 deletions(-)
> 
> diff --git a/meta/recipes-devtools/python/python3-setuptools_31.0.0.bb b/meta/recipes-devtools/python/python3-setuptools_31.0.0.bb
> index 65af6f0..b37b381 100644
> --- a/meta/recipes-devtools/python/python3-setuptools_31.0.0.bb
> +++ b/meta/recipes-devtools/python/python3-setuptools_31.0.0.bb
> @@ -7,11 +7,8 @@ inherit distutils3
>  
>  DISTUTILS_INSTALL_ARGS += "--install-lib=${D}${PYTHON_SITEPACKAGES_DIR}"
>  
> -# The installer puts the wrong path in the setuptools.pth file.  Correct it.
>  do_install_append() {
> -    rm ${D}${PYTHON_SITEPACKAGES_DIR}/setuptools.pth
>      mv ${D}${bindir}/easy_install ${D}${bindir}/easy3_install
> -    echo "./${SRCNAME}-${PV}-py${PYTHON_BASEVERSION}.egg" > ${D}${PYTHON_SITEPACKAGES_DIR}/setuptools.pth
>  }
>  
>  RDEPENDS_${PN} = "\
> 

-- 
Jose Lamego | OTC Embedded Platforms & Tools | GDC


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 501 bytes --]

Re: [PATCH 3/3] python3-setuptools: Remove pth file creation at recipe level

[Jose Lamego]

[-- Attachment #1.1: Type: text/plain, Size: 1357 bytes --]

Ping.

On 02/08/2017 04:31 PM, Jose Lamego wrote:
> Ping.
> 
> On 01/17/2017 09:42 AM, Jose Lamego wrote:
>> Handling of installed eggs is now performed at the distutils3 bbclass.
>>
>> This change removes the pth file creation at recipe level.
>>
>> [YOCTO #8673]
>>
>> Signed-off-by: Jose Lamego <jose.a.lamego@linux.intel.com>
>> ---
>>  meta/recipes-devtools/python/python3-setuptools_31.0.0.bb | 3 ---
>>  1 file changed, 3 deletions(-)
>>
>> diff --git a/meta/recipes-devtools/python/python3-setuptools_31.0.0.bb b/meta/recipes-devtools/python/python3-setuptools_31.0.0.bb
>> index 65af6f0..b37b381 100644
>> --- a/meta/recipes-devtools/python/python3-setuptools_31.0.0.bb
>> +++ b/meta/recipes-devtools/python/python3-setuptools_31.0.0.bb
>> @@ -7,11 +7,8 @@ inherit distutils3
>>  
>>  DISTUTILS_INSTALL_ARGS += "--install-lib=${D}${PYTHON_SITEPACKAGES_DIR}"
>>  
>> -# The installer puts the wrong path in the setuptools.pth file.  Correct it.
>>  do_install_append() {
>> -    rm ${D}${PYTHON_SITEPACKAGES_DIR}/setuptools.pth
>>      mv ${D}${bindir}/easy_install ${D}${bindir}/easy3_install
>> -    echo "./${SRCNAME}-${PV}-py${PYTHON_BASEVERSION}.egg" > ${D}${PYTHON_SITEPACKAGES_DIR}/setuptools.pth
>>  }
>>  
>>  RDEPENDS_${PN} = "\
>>
> 
> 
> 

-- 
Jose Lamego | OTC Embedded Platforms & Tools | GDC


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 484 bytes --]

Re: [PATCH 0/3] distutils3: add egg files to path

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 724 bytes --]

On 17 January 2017 at 15:42, Jose Lamego <jose.a.lamego@linux.intel.com>
wrote:

> Packages that use egg files or directories for installation may
> not be found when imported at the python3 interpreter.
> egg files/directories path must be included in a pth file to
> be appropriately included in python path.
>

Do we understand why pip and distutils are installing as eggs, instead of
the traditional format?  I thought we
passed --single-version-externally-managed to setuptools to handle this but
I can't see this anywhere (Khem removed it from distutils back in 2013).
For reference, python3-mako just installs the metadata to the egg folder
and installs the Python code in the traditional way.

Ross

[-- Attachment #2: Type: text/html, Size: 1233 bytes --]