[PATCH 1/1] python3-native: Fix pip install issue due to unclean build directory

[PATCH 1/1] python3-native: Fix pip install issue due to unclean build directory

[Alejandro Hernandez]

When installing python3-native sometimes pips default build
directory (which is on the host and is user dependant) is left unclean,
due to this, when python3-core is being installed it tries to use
the same directory producing an error, this explicitly removes
what the previous installation might have left behind, fixing the issue.

Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>
---
 .../python/python3-native_3.4.2.bb                 |  1 +
 .../python3/pip_build_directory_unclean.patch      | 28 ++++++++++++++++++++++
 2 files changed, 29 insertions(+)
 create mode 100644 meta/recipes-devtools/python/python3/pip_build_directory_unclean.patch

diff --git a/meta/recipes-devtools/python/python3-native_3.4.2.bb b/meta/recipes-devtools/python/python3-native_3.4.2.bb
index d98a278..37399be 100644
--- a/meta/recipes-devtools/python/python3-native_3.4.2.bb
+++ b/meta/recipes-devtools/python/python3-native_3.4.2.bb
@@ -26,6 +26,7 @@ file://makerace.patch \
 ${DISTRO_SRC_URI} \
 file://sysconfig.py-add-_PYTHON_PROJECT_SRC.patch \
 file://setup.py-check-cross_compiling-when-get-FLAGS.patch \
+file://pip_build_directory_unclean.patch \
 "
 SRC_URI[md5sum] = "36fc7327c02c6f12fa24fc9ba78039e3"
 SRC_URI[sha256sum] = "1c6d9682d145c056537e477bbfa060ce727f9edd38df1827e0f970dcf04b2def"
diff --git a/meta/recipes-devtools/python/python3/pip_build_directory_unclean.patch b/meta/recipes-devtools/python/python3/pip_build_directory_unclean.patch
new file mode 100644
index 0000000..c0c4617
--- /dev/null
+++ b/meta/recipes-devtools/python/python3/pip_build_directory_unclean.patch
@@ -0,0 +1,28 @@
+Index: Python-3.4.2/Lib/ensurepip/__init__.py
+===================================================================
+--- Python-3.4.2.orig/Lib/ensurepip/__init__.py
++++ Python-3.4.2/Lib/ensurepip/__init__.py
+@@ -3,6 +3,7 @@ import os.path
+ import pkgutil
+ import sys
+ import tempfile
++import pwd
+ 
+ 
+ __all__ = ["version", "bootstrap"]
+@@ -36,6 +37,15 @@ def _run_pip(args, additional_paths=None
+     if additional_paths is not None:
+         sys.path = additional_paths + sys.path
+ 
++    # Explicitly make sure pip build directory was left clean
++    user = pwd.getpwuid(os.getuid())[0]
++    if os.path.exists("/tmp/pip_build_" + user + "/pip-delete-this-directory.txt"):
++        os.remove("/tmp/pip_build_" + user + "/pip-delete-this-directory.txt")
++        try:
++            os.rmdir("/tmp/pip_build_" + user + "/")
++        except OSError:
++            print("Error: Pip build directory (%s) was left unclean, make sure it is clean before continuing\n" % ("/tmp/pip_build_" + user + "/"))
++
+     # Install the bundled software
+     import pip
+     pip.main(args)
-- 
1.8.4.5

Re: [PATCH 1/1] python3-native: Fix pip install issue due to unclean build directory

[Richard Purdie]

On Thu, 2015-04-16 at 09:45 +0000, Alejandro Hernandez wrote:
> When installing python3-native sometimes pips default build
> directory (which is on the host and is user dependant) is left unclean,
> due to this, when python3-core is being installed it tries to use
> the same directory producing an error, this explicitly removes
> what the previous installation might have left behind, fixing the issue.
> 
> Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>
> ---
>  .../python/python3-native_3.4.2.bb                 |  1 +
>  .../python3/pip_build_directory_unclean.patch      | 28 ++++++++++++++++++++++
>  2 files changed, 29 insertions(+)
>  create mode 100644 meta/recipes-devtools/python/python3/pip_build_directory_unclean.patch

The problem here is that two builds of python3-native could be happening
on the same system at the same time. In fact I'd bet that is why its
breaking on the autobuilder. Can we not tell it to use something in
WORKDIR as its temp directory instead?

Cheers,

Richard

Re: [PATCH 1/1] python3-native: Fix pip install issue due to unclean build directory

[Alejandro Hernandez]

On 17/04/15 16:50, Richard Purdie wrote:
> On Thu, 2015-04-16 at 09:45 +0000, Alejandro Hernandez wrote:
>> When installing python3-native sometimes pips default build
>> directory (which is on the host and is user dependant) is left unclean,
>> due to this, when python3-core is being installed it tries to use
>> the same directory producing an error, this explicitly removes
>> what the previous installation might have left behind, fixing the issue.
>>
>> Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>
>> ---
>>   .../python/python3-native_3.4.2.bb                 |  1 +
>>   .../python3/pip_build_directory_unclean.patch      | 28 ++++++++++++++++++++++
>>   2 files changed, 29 insertions(+)
>>   create mode 100644 meta/recipes-devtools/python/python3/pip_build_directory_unclean.patch
> The problem here is that two builds of python3-native could be happening
> on the same system at the same time. In fact I'd bet that is why its
> breaking on the autobuilder. Can we not tell it to use something in
> WORKDIR as its temp directory instead?
>
> Cheers,
>
> Richard
Hmm, that's potentially what is happening, in fact, my first thought was 
to do that and use something within the WORKDIR, but I wasn't able to do 
that because they do something weird to install pip, they bootstrap the 
installation and they use python wheels, so the actual part of the code 
where one could specify a different build directory for pip is inside 
one of the wheels, making it complicated to modify.
The only other thing that occurs to me would be to just make it sleep or 
make it wait somehow (we could modify the queue?) until the directory is 
empty.

I still think this should work, I only saw cases where the build 
directory is left with one empty file, another reason why I decided to 
explicitly delete it instead of forcing it to delete everything inside 
the directory, disabling pip is not an option because it is required 
since python 3.4 as I stated on the upgrade patch, we could try this, 
and if it doesn't work I'll find another workaround, but let me know 
what you think.


Alejandro

Re: [PATCH 1/1] python3-native: Fix pip install issue due to unclean build directory

[Richard Purdie]

On Fri, 2015-04-17 at 19:19 -0500, Alejandro Hernandez wrote:
> On 17/04/15 16:50, Richard Purdie wrote:
> > On Thu, 2015-04-16 at 09:45 +0000, Alejandro Hernandez wrote:
> >> When installing python3-native sometimes pips default build
> >> directory (which is on the host and is user dependant) is left unclean,
> >> due to this, when python3-core is being installed it tries to use
> >> the same directory producing an error, this explicitly removes
> >> what the previous installation might have left behind, fixing the issue.
> >>
> >> Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>
> >> ---
> >>   .../python/python3-native_3.4.2.bb                 |  1 +
> >>   .../python3/pip_build_directory_unclean.patch      | 28 ++++++++++++++++++++++
> >>   2 files changed, 29 insertions(+)
> >>   create mode 100644 meta/recipes-devtools/python/python3/pip_build_directory_unclean.patch
> > The problem here is that two builds of python3-native could be happening
> > on the same system at the same time. In fact I'd bet that is why its
> > breaking on the autobuilder. Can we not tell it to use something in
> > WORKDIR as its temp directory instead?
> >
> > Cheers,
> >
> > Richard
> Hmm, that's potentially what is happening, in fact, my first thought was 
> to do that and use something within the WORKDIR, but I wasn't able to do 
> that because they do something weird to install pip, they bootstrap the 
> installation and they use python wheels, so the actual part of the code 
> where one could specify a different build directory for pip is inside 
> one of the wheels, making it complicated to modify.
> The only other thing that occurs to me would be to just make it sleep or 
> make it wait somehow (we could modify the queue?) until the directory is 
> empty.
> 
> I still think this should work, I only saw cases where the build 
> directory is left with one empty file, another reason why I decided to 
> explicitly delete it instead of forcing it to delete everything inside 
> the directory, disabling pip is not an option because it is required 
> since python 3.4 as I stated on the upgrade patch, we could try this, 
> and if it doesn't work I'll find another workaround, but let me know 
> what you think.

I tried it on the autobuilder and it failed again. I think part of the
problem is that this part of the process runs under pseudo so it thinks
its root, yet can't clear root's pip directory in /tmp.

I suspect we are going to have to tell pip to use WORKDIR somehow, even
if we have to patch it to take some kind of prompt from the environment
for example.

Cheers,

Richard

Re: [PATCH 1/1] python3-native: Fix pip install issue due to unclean build directory

[Alejandro Hernandez]

On 19/04/15 02:49, Richard Purdie wrote:
> On Fri, 2015-04-17 at 19:19 -0500, Alejandro Hernandez wrote:
>> On 17/04/15 16:50, Richard Purdie wrote:
>>> On Thu, 2015-04-16 at 09:45 +0000, Alejandro Hernandez wrote:
>>>> When installing python3-native sometimes pips default build
>>>> directory (which is on the host and is user dependant) is left unclean,
>>>> due to this, when python3-core is being installed it tries to use
>>>> the same directory producing an error, this explicitly removes
>>>> what the previous installation might have left behind, fixing the issue.
>>>>
>>>> Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>
>>>> ---
>>>>    .../python/python3-native_3.4.2.bb                 |  1 +
>>>>    .../python3/pip_build_directory_unclean.patch      | 28 ++++++++++++++++++++++
>>>>    2 files changed, 29 insertions(+)
>>>>    create mode 100644 meta/recipes-devtools/python/python3/pip_build_directory_unclean.patch
> I tried it on the autobuilder and it failed again. I think part of the
> problem is that this part of the process runs under pseudo so it thinks
> its root, yet can't clear root's pip directory in /tmp.
>
> I suspect we are going to have to tell pip to use WORKDIR somehow, even
> if we have to patch it to take some kind of prompt from the environment
> for example.
>
> Cheers,
>
> Richard
>

Yes, the problem does happen when it uses pseudo and it thinks its root, 
I thought removing the files inside the same python code would be able 
to do it, ok I will find another workaround.