virtualenv for toaster and python 3

virtualenv for toaster and python 3

[Richard Purdie]

The move to python3 is unsurprisingly causing some issues in places.
One of them is unfortunately the use of vritualenv, which as I
understand it, toaster currently promotes as its preferred way of being
run.

The issue is that we need "python" to be python v2 but "python3" to be
python v3. It seems with virtualenv you can either have python v2 or
python v3 but not both.

The python developers official party line is that "python" is v2 and
"python3" is v3. There are some distros choosing not to do this (e.g.
Arch Linux) but most are following the official recommendation and I
think OE needs to follow the official recommendation too. Its likely OE
will need to error with sanity test failures if the environment doesn't
conform to this view of the world.

Why do we need this? We have a mixture of code, some of it is ported to
v3, some isn't. Whilst we're aiming to convert all of "our" code to
python v3, even once we've done that, we can't easily change scripts
that ship with the source code we build for example, or expect everyone
to do that for all software they build.

It may be virtualenv can be configured to not take over "python" but
only "python3" but Ed/I haven't found the option, I'm no expert.

The other alternative is to promote the use of something like "pip3
install 'Django>1.8,<1.9' --user" instead. Toaster doesn't need many
modules and those it does need should be installable into the user's
homedir using pip3.

Whilst this means changing the documentation and taking a new approach,
I think its the "least bad" solution we have right now.

Obviously I'm open to other ideas but we really do need a way which
allows mixed versions of python and virtualenv doesn't seem to work for
this.

Cheers,

Richard

Re: virtualenv for toaster and python 3

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 2666 bytes --]

On Tue, May 31, 2016 at 3:45 AM, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> The move to python3 is unsurprisingly causing some issues in places.
> One of them is unfortunately the use of vritualenv, which as I
> understand it, toaster currently promotes as its preferred way of being
> run.
>
> The issue is that we need "python" to be python v2 but "python3" to be
> python v3. It seems with virtualenv you can either have python v2 or
> python v3 but not both.
>
> The python developers official party line is that "python" is v2 and
> "python3" is v3. There are some distros choosing not to do this (e.g.
> Arch Linux) but most are following the official recommendation and I
> think OE needs to follow the official recommendation too. Its likely OE
> will need to error with sanity test failures if the environment doesn't
> conform to this view of the world.
>
> Why do we need this? We have a mixture of code, some of it is ported to
> v3, some isn't. Whilst we're aiming to convert all of "our" code to
> python v3, even once we've done that, we can't easily change scripts
> that ship with the source code we build for example, or expect everyone
> to do that for all software they build.
>
> It may be virtualenv can be configured to not take over "python" but
> only "python3" but Ed/I haven't found the option, I'm no expert.
>
> The other alternative is to promote the use of something like "pip3
> install 'Django>1.8,<1.9' --user" instead. Toaster doesn't need many
> modules and those it does need should be installable into the user's
> homedir using pip3.
>
> Whilst this means changing the documentation and taking a new approach,
> I think its the "least bad" solution we have right now.
>
> Obviously I'm open to other ideas but we really do need a way which
> allows mixed versions of python and virtualenv doesn't seem to work for
> this.


virtualenv is the wrong tool to use for python 3. python 3 includes its own
tool for managing virtual environments, pyvenv. See
https://docs.python.org/dev/library/venv.html.

As an FYI, you can run either virtualenv or pyvenv for an arbitrary python
version by running it with python -m virtualenv or python -m venv, and
virtualenv takes a -p argument to specify the path to the python
executable. But of course that doesn't matter, since we don't need to use
virtualenv in python 3 environments. But still, useful to know that both
python packages provide an __main__.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 3436 bytes --]

Re: virtualenv for toaster and python 3

[Brian Avery]

So,
From my admittedly brief web searches, I found multiple online
references to the fact that, by design, both virtualenv and venv are
designed to 'jail' a particular python version as the default.
Unfortunately, this means there aren't any arguments that allow either
of these approaches to simultaneously support both python2&3.

A hack:
if you really want to maintain both versions with the convention that
python will invoke python2 and python3 will invoke python3 then you
can do:
-------------
python3 -m venv  python3_venv
rm  python3_venv/bin/python
. ./python3_venv/bin/activate
-------------

This results in python resolving to the host version, typically,
/usr/bin/python aka python2. python3 will resolve to
python3_venv/bin/python3. pip will resolve to the python3 pip.  Since
we will need to have modules installed for python3 for toaster, this
is what we want.  I tried some pip installs of both compiled modules
and pure python modules and these were correctly installed as python3
modules.
In this environment if you want to install a python2 modules, pip2
install <foo> will do that.

Since I don't know what side effects this might have, suggesting this
as a "solution" seems silly.  However, I wanted to put this out there
since it did work for me in case someone is stuck...

Unfortunately, this doesn't really solve what we do for toaster...

-bavery




On Tue, May 31, 2016 at 7:09 AM, Christopher Larson <clarson@kergoth.com> wrote:
>
>
> On Tue, May 31, 2016 at 3:45 AM, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
>>
>> The move to python3 is unsurprisingly causing some issues in places.
>> One of them is unfortunately the use of vritualenv, which as I
>> understand it, toaster currently promotes as its preferred way of being
>> run.
>>
>> The issue is that we need "python" to be python v2 but "python3" to be
>> python v3. It seems with virtualenv you can either have python v2 or
>> python v3 but not both.
>>
>> The python developers official party line is that "python" is v2 and
>> "python3" is v3. There are some distros choosing not to do this (e.g.
>> Arch Linux) but most are following the official recommendation and I
>> think OE needs to follow the official recommendation too. Its likely OE
>> will need to error with sanity test failures if the environment doesn't
>> conform to this view of the world.
>>
>> Why do we need this? We have a mixture of code, some of it is ported to
>> v3, some isn't. Whilst we're aiming to convert all of "our" code to
>> python v3, even once we've done that, we can't easily change scripts
>> that ship with the source code we build for example, or expect everyone
>> to do that for all software they build.
>>
>> It may be virtualenv can be configured to not take over "python" but
>> only "python3" but Ed/I haven't found the option, I'm no expert.
>>
>> The other alternative is to promote the use of something like "pip3
>> install 'Django>1.8,<1.9' --user" instead. Toaster doesn't need many
>> modules and those it does need should be installable into the user's
>> homedir using pip3.
>>
>> Whilst this means changing the documentation and taking a new approach,
>> I think its the "least bad" solution we have right now.
>>
>> Obviously I'm open to other ideas but we really do need a way which
>> allows mixed versions of python and virtualenv doesn't seem to work for
>> this.
>
>
> virtualenv is the wrong tool to use for python 3. python 3 includes its own
> tool for managing virtual environments, pyvenv. See
> https://docs.python.org/dev/library/venv.html.
>
> As an FYI, you can run either virtualenv or pyvenv for an arbitrary python
> version by running it with python -m virtualenv or python -m venv, and
> virtualenv takes a -p argument to specify the path to the python executable.
> But of course that doesn't matter, since we don't need to use virtualenv in
> python 3 environments. But still, useful to know that both python packages
> provide an __main__.
> --
> Christopher Larson
> clarson at kergoth dot com
> Founder - BitBake, OpenEmbedded, OpenZaurus
> Maintainer - Tslib
> Senior Software Engineer, Mentor Graphics
>
> --
> _______________________________________________
> toaster mailing list
> toaster@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/toaster
>

Re: virtualenv for toaster and python 3

[Brian Avery]

One possible solution is to use RP's suggestion to leverage the --user
argument (which stores packges in
$HOME/.local/lib/python3.4/site-packages.

>pip3 install --user -r poky/bitbake/toaster-requirements.txt
* works except wsgiref is not python3 compatible yet
>pip3 list installed --local
argparse (1.2.1)
beautifulsoup4 (4.4.1)
Django (1.8.13)

* works to show what is a --user install vs a global site install.
> pip3 uninstall/install --user --upgrade/.... works as expected...

-b




On Tue, May 31, 2016 at 11:28 AM, Brian Avery <avery.brian@gmail.com> wrote:
> So,
> From my admittedly brief web searches, I found multiple online
> references to the fact that, by design, both virtualenv and venv are
> designed to 'jail' a particular python version as the default.
> Unfortunately, this means there aren't any arguments that allow either
> of these approaches to simultaneously support both python2&3.
>
> A hack:
> if you really want to maintain both versions with the convention that
> python will invoke python2 and python3 will invoke python3 then you
> can do:
> -------------
> python3 -m venv  python3_venv
> rm  python3_venv/bin/python
> . ./python3_venv/bin/activate
> -------------
>
> This results in python resolving to the host version, typically,
> /usr/bin/python aka python2. python3 will resolve to
> python3_venv/bin/python3. pip will resolve to the python3 pip.  Since
> we will need to have modules installed for python3 for toaster, this
> is what we want.  I tried some pip installs of both compiled modules
> and pure python modules and these were correctly installed as python3
> modules.
> In this environment if you want to install a python2 modules, pip2
> install <foo> will do that.
>
> Since I don't know what side effects this might have, suggesting this
> as a "solution" seems silly.  However, I wanted to put this out there
> since it did work for me in case someone is stuck...
>
> Unfortunately, this doesn't really solve what we do for toaster...
>
> -bavery
>
>
>
>
> On Tue, May 31, 2016 at 7:09 AM, Christopher Larson <clarson@kergoth.com> wrote:
>>
>>
>> On Tue, May 31, 2016 at 3:45 AM, Richard Purdie
>> <richard.purdie@linuxfoundation.org> wrote:
>>>
>>> The move to python3 is unsurprisingly causing some issues in places.
>>> One of them is unfortunately the use of vritualenv, which as I
>>> understand it, toaster currently promotes as its preferred way of being
>>> run.
>>>
>>> The issue is that we need "python" to be python v2 but "python3" to be
>>> python v3. It seems with virtualenv you can either have python v2 or
>>> python v3 but not both.
>>>
>>> The python developers official party line is that "python" is v2 and
>>> "python3" is v3. There are some distros choosing not to do this (e.g.
>>> Arch Linux) but most are following the official recommendation and I
>>> think OE needs to follow the official recommendation too. Its likely OE
>>> will need to error with sanity test failures if the environment doesn't
>>> conform to this view of the world.
>>>
>>> Why do we need this? We have a mixture of code, some of it is ported to
>>> v3, some isn't. Whilst we're aiming to convert all of "our" code to
>>> python v3, even once we've done that, we can't easily change scripts
>>> that ship with the source code we build for example, or expect everyone
>>> to do that for all software they build.
>>>
>>> It may be virtualenv can be configured to not take over "python" but
>>> only "python3" but Ed/I haven't found the option, I'm no expert.
>>>
>>> The other alternative is to promote the use of something like "pip3
>>> install 'Django>1.8,<1.9' --user" instead. Toaster doesn't need many
>>> modules and those it does need should be installable into the user's
>>> homedir using pip3.
>>>
>>> Whilst this means changing the documentation and taking a new approach,
>>> I think its the "least bad" solution we have right now.
>>>
>>> Obviously I'm open to other ideas but we really do need a way which
>>> allows mixed versions of python and virtualenv doesn't seem to work for
>>> this.
>>
>>
>> virtualenv is the wrong tool to use for python 3. python 3 includes its own
>> tool for managing virtual environments, pyvenv. See
>> https://docs.python.org/dev/library/venv.html.
>>
>> As an FYI, you can run either virtualenv or pyvenv for an arbitrary python
>> version by running it with python -m virtualenv or python -m venv, and
>> virtualenv takes a -p argument to specify the path to the python executable.
>> But of course that doesn't matter, since we don't need to use virtualenv in
>> python 3 environments. But still, useful to know that both python packages
>> provide an __main__.
>> --
>> Christopher Larson
>> clarson at kergoth dot com
>> Founder - BitBake, OpenEmbedded, OpenZaurus
>> Maintainer - Tslib
>> Senior Software Engineer, Mentor Graphics
>>
>> --
>> _______________________________________________
>> toaster mailing list
>> toaster@yoctoproject.org
>> https://lists.yoctoproject.org/listinfo/toaster
>>

Re: virtualenv for toaster and python 3

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 925 bytes --]

On Tue, May 31, 2016 at 11:47 AM, Brian Avery <avery.brian@gmail.com> wrote:

> One possible solution is to use RP's suggestion to leverage the --user
> argument (which stores packges in
> $HOME/.local/lib/python3.4/site-packages.
>
> >pip3 install --user -r poky/bitbake/toaster-requirements.txt
> * works except wsgiref is not python3 compatible yet
> >pip3 list installed --local
> argparse (1.2.1)
> beautifulsoup4 (4.4.1)
> Django (1.8.13)
>
> * works to show what is a --user install vs a global site install.
> > pip3 uninstall/install --user --upgrade/.... works as expected...
>

Cluttering up my ~/.local with app/project specific requirements is not
appropriate. As someone who uses --user quite a bit, I'd be quite annoyed
by this.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 1390 bytes --]

Re: virtualenv for toaster and python 3

[Ed Bartosh]

On Tue, May 31, 2016 at 11:48:01AM -0700, Christopher Larson wrote:
> On Tue, May 31, 2016 at 11:47 AM, Brian Avery <avery.brian@gmail.com> wrote:
> 
> > One possible solution is to use RP's suggestion to leverage the --user
> > argument (which stores packges in
> > $HOME/.local/lib/python3.4/site-packages.
> >
> > >pip3 install --user -r poky/bitbake/toaster-requirements.txt
> > * works except wsgiref is not python3 compatible yet
> > >pip3 list installed --local
> > argparse (1.2.1)
> > beautifulsoup4 (4.4.1)
> > Django (1.8.13)
> >
> > * works to show what is a --user install vs a global site install.
> > > pip3 uninstall/install --user --upgrade/.... works as expected...
> >
> 
> Cluttering up my ~/.local with app/project specific requirements is not
> appropriate. As someone who uses --user quite a bit, I'd be quite annoyed
> by this.

So far pip3 --user looks like the only working approach to me. It's not
ideal, but it's not that bad either. It should be used for its direct purpose
- to install dependencies for 'one user' mode.

For production Toaster instances I'd suggest to install runtime
dependencies on the host system.

BTW, Toaster has only one runtime requirement: Django. argparse and
wsgiref is a legacy and beautifulsoup4 is not a runtime dependency. It's used only
in tests.

--
Regards,
Ed

Re: virtualenv for toaster and python 3

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 2091 bytes --]

On Wed, Jun 1, 2016 at 2:27 AM, Ed Bartosh <ed.bartosh@linux.intel.com>
wrote:

> On Tue, May 31, 2016 at 11:48:01AM -0700, Christopher Larson wrote:
> > On Tue, May 31, 2016 at 11:47 AM, Brian Avery <avery.brian@gmail.com>
> wrote:
> >
> > > One possible solution is to use RP's suggestion to leverage the --user
> > > argument (which stores packges in
> > > $HOME/.local/lib/python3.4/site-packages.
> > >
> > > >pip3 install --user -r poky/bitbake/toaster-requirements.txt
> > > * works except wsgiref is not python3 compatible yet
> > > >pip3 list installed --local
> > > argparse (1.2.1)
> > > beautifulsoup4 (4.4.1)
> > > Django (1.8.13)
> > >
> > > * works to show what is a --user install vs a global site install.
> > > > pip3 uninstall/install --user --upgrade/.... works as expected...
> > >
> >
> > Cluttering up my ~/.local with app/project specific requirements is not
> > appropriate. As someone who uses --user quite a bit, I'd be quite annoyed
> > by this.
>
> So far pip3 --user looks like the only working approach to me. It's not
> ideal, but it's not that bad either. It should be used for its direct
> purpose
> - to install dependencies for 'one user' mode.
>
> For production Toaster instances I'd suggest to install runtime
> dependencies on the host system.
>
> BTW, Toaster has only one runtime requirement: Django. argparse and
> wsgiref is a legacy and beautifulsoup4 is not a runtime dependency. It's
> used only
> in tests.


--user is intended for use by the user,  not your scripts. I choose what I
put there for good reason. Your use case is what virtual environments are
for. If toaster can't handle that, it should be fixed to do so. From the
python documentation:

This scheme is designed to be the most convenient solution for users that
don’t have write permission to the global site-packages directory or don’t
want to install into it.

-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 2853 bytes --]

Re: virtualenv for toaster and python 3

[Ed Bartosh]

On Wed, Jun 01, 2016 at 10:24:52AM -0700, Christopher Larson wrote:
> On Wed, Jun 1, 2016 at 2:27 AM, Ed Bartosh <ed.bartosh@linux.intel.com>
> wrote:
> 
> > On Tue, May 31, 2016 at 11:48:01AM -0700, Christopher Larson wrote:
> > > On Tue, May 31, 2016 at 11:47 AM, Brian Avery <avery.brian@gmail.com>
> > wrote:
> > >
> > > > One possible solution is to use RP's suggestion to leverage the --user
> > > > argument (which stores packges in
> > > > $HOME/.local/lib/python3.4/site-packages.
> > > >
> > > > >pip3 install --user -r poky/bitbake/toaster-requirements.txt
> > > > * works except wsgiref is not python3 compatible yet
> > > > >pip3 list installed --local
> > > > argparse (1.2.1)
> > > > beautifulsoup4 (4.4.1)
> > > > Django (1.8.13)
> > > >
> > > > * works to show what is a --user install vs a global site install.
> > > > > pip3 uninstall/install --user --upgrade/.... works as expected...
> > > >
> > >
> > > Cluttering up my ~/.local with app/project specific requirements is not
> > > appropriate. As someone who uses --user quite a bit, I'd be quite annoyed
> > > by this.
> >
> > So far pip3 --user looks like the only working approach to me. It's not
> > ideal, but it's not that bad either. It should be used for its direct
> > purpose
> > - to install dependencies for 'one user' mode.
> >
> > For production Toaster instances I'd suggest to install runtime
> > dependencies on the host system.
> >
> > BTW, Toaster has only one runtime requirement: Django. argparse and
> > wsgiref is a legacy and beautifulsoup4 is not a runtime dependency. It's
> > used only
> > in tests.
> 
> 
> --user is intended for use by the user,  not your scripts.
True. I'm not saying that toaster should do that. User will run pip3
install --user 'Django>1.8<1.9' or install Django 1.8 any other suitable
way.

> I choose what I put there for good reason. Your use case is what virtual environments are
> for. If toaster can't handle that, it should be fixed to do so.
Toaster can't be fixed to setup an environment where python is python2
and python3 is python3. Neither virtualenv or venv support this.

> From the python documentation:
> 
> This scheme is designed to be the most convenient solution for users that
> don’t have write permission to the global site-packages directory or don’t
> want to install into it.
Yes, and this is exactly what I'm proposing. Sorry for not being clear.

--
Regards,
Ed

Re: virtualenv for toaster and python 3

[Michael Wood]

On 01/06/16 21:12, Ed Bartosh wrote:
> On Wed, Jun 01, 2016 at 10:24:52AM -0700, Christopher Larson wrote:
>> On Wed, Jun 1, 2016 at 2:27 AM, Ed Bartosh <ed.bartosh@linux.intel.com>
>> wrote:
>>
>>> On Tue, May 31, 2016 at 11:48:01AM -0700, Christopher Larson wrote:
>>>> On Tue, May 31, 2016 at 11:47 AM, Brian Avery <avery.brian@gmail.com>
>>> wrote:
>>>>> One possible solution is to use RP's suggestion to leverage the --user
>>>>> argument (which stores packges in
>>>>> $HOME/.local/lib/python3.4/site-packages.
>>>>>
>>>>>> pip3 install --user -r poky/bitbake/toaster-requirements.txt
>>>>> * works except wsgiref is not python3 compatible yet
>>>>>> pip3 list installed --local
>>>>> argparse (1.2.1)
>>>>> beautifulsoup4 (4.4.1)
>>>>> Django (1.8.13)
>>>>>
>>>>> * works to show what is a --user install vs a global site install.
>>>>>> pip3 uninstall/install --user --upgrade/.... works as expected...
>>>> Cluttering up my ~/.local with app/project specific requirements is not
>>>> appropriate. As someone who uses --user quite a bit, I'd be quite annoyed
>>>> by this.
>>> So far pip3 --user looks like the only working approach to me. It's not
>>> ideal, but it's not that bad either. It should be used for its direct
>>> purpose
>>> - to install dependencies for 'one user' mode.
>>>
>>> For production Toaster instances I'd suggest to install runtime
>>> dependencies on the host system.
>>>
>>> BTW, Toaster has only one runtime requirement: Django. argparse and
>>> wsgiref is a legacy and beautifulsoup4 is not a runtime dependency. It's
>>> used only
>>> in tests.
>>
>> --user is intended for use by the user,  not your scripts.
> True. I'm not saying that toaster should do that. User will run pip3
> install --user 'Django>1.8<1.9' or install Django 1.8 any other suitable
> way.
>
>> I choose what I put there for good reason. Your use case is what virtual environments are
>> for. If toaster can't handle that, it should be fixed to do so.
> Toaster can't be fixed to setup an environment where python is python2
> and python3 is python3. Neither virtualenv or venv support this.
>
>>  From the python documentation:
>>
>> This scheme is designed to be the most convenient solution for users that
>> don’t have write permission to the global site-packages directory or don’t
>> want to install into it.
> Yes, and this is exactly what I'm proposing. Sorry for not being clear.
>
> --
> Regards,
> Ed

Did anyone test this, it works for me:

$ virtualenv -p python3 bothpython
$ virtualenv -p python2 bothpython
$ source ./bothpython/bin/activate
$ pip3 install -r ./bitbake/toaster-requirements.txt
$ source toaster start

The only change needed is to fix the toaster script to use python3 to 
detect if the dependencies are installed or not.

diff --git a/bitbake/bin/toaster b/bitbake/bin/toaster
index be1bb9c..e3a0dae 100755
--- a/bitbake/bin/toaster
+++ b/bitbake/bin/toaster
@@ -115,7 +115,7 @@ verify_prereq() {
exp='s/Django\([><=]\+\)\([^,]\+\),\([><=]\+\)\(.\+\)/'
      exp=$exp'import sys,django;version=django.get_version().split(".");'
      exp=$exp'sys.exit(not (version \1 "\2".split(".") and version \3 
"\4".split
-    if ! sed -n "$exp" $reqfile | python - ; then
+    if ! sed -n "$exp" $reqfile | python3 - ; then
          req=`grep ^Django $reqfile`
          echo "This program needs $req"
          echo "Please install with pip install -r $reqfile"


Michael