Re: [OE-core] [PATCH] waf: explicitly run waf under Python 3

Re: [OE-core] [PATCH] waf: explicitly run waf under Python 3

[Richard Purdie]

On Thu, 2020-10-08 at 17:23 +0100, Ross Burton wrote:
> We don't assume Python 2 is on the host and don't expose an
> unversioned
> python in HOSTTOOLS, but waf uses the unversion python binary.
> 
> Typically this is solved by patching waf (see the glmark2 recipe) but
> this can be solved centrally by invoking python3 directly.  All
> wscripts
> that needed Python 2 would have been fixed by now.
> 
> Signed-off-by: Ross Burton <ross.burton@arm.com>
> ---
>  meta/classes/waf.bbclass | 8 ++++----
>  1 file changed, 4 insertions(+), 4 deletions(-)

Please rebase against master-next, Joshua beat you in patch submission
against the class!

Cheers,

Richard

[PATCH] waf: explicitly run waf under Python 3

[Ross Burton]

We don't assume Python 2 is on the host and don't expose an unversioned
python in HOSTTOOLS, but waf uses the unversion python binary.

Typically this is solved by patching waf (see the glmark2 recipe) but
this can be solved centrally by invoking python3 directly.  All wscripts
that needed Python 2 would have been fixed by now.

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 meta/classes/waf.bbclass | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/meta/classes/waf.bbclass b/meta/classes/waf.bbclass
index 309f625a40..df6d4d53e8 100644
--- a/meta/classes/waf.bbclass
+++ b/meta/classes/waf.bbclass
@@ -42,7 +42,7 @@ python waf_preconfigure() {
     subsrcdir = d.getVar('S')
     wafbin = os.path.join(subsrcdir, 'waf')
     try:
-        result = subprocess.check_output([wafbin, '--version'], cwd=subsrcdir, stderr=subprocess.STDOUT)
+        result = subprocess.check_output(['python3', wafbin, '--version'], cwd=subsrcdir, stderr=subprocess.STDOUT)
         version = result.decode('utf-8').split()[1]
         if StrictVersion(version) >= StrictVersion("1.8.7"):
             d.setVar("WAF_EXTRA_CONF", "--bindir=${bindir} --libdir=${libdir}")
@@ -55,16 +55,16 @@ python waf_preconfigure() {
 do_configure[prefuncs] += "waf_preconfigure"
 
 waf_do_configure() {
-	(cd ${S} && ./waf configure -o ${B} --prefix=${prefix} ${WAF_EXTRA_CONF} ${EXTRA_OECONF})
+	(cd ${S} && python3 ./waf configure -o ${B} --prefix=${prefix} ${WAF_EXTRA_CONF} ${EXTRA_OECONF})
 }
 
 do_compile[progress] = "outof:^\[\s*(\d+)/\s*(\d+)\]\s+"
 waf_do_compile()  {
-	(cd ${S} && ./waf build ${@oe.utils.parallel_make_argument(d, '-j%d', limit=64)} ${EXTRA_OEWAF_BUILD})
+	(cd ${S} && python3 ./waf build ${@oe.utils.parallel_make_argument(d, '-j%d', limit=64)} ${EXTRA_OEWAF_BUILD})
 }
 
 waf_do_install() {
-	(cd ${S} && ./waf install --destdir=${D} ${EXTRA_OEWAF_INSTALL})
+	(cd ${S} && python3 ./waf install --destdir=${D} ${EXTRA_OEWAF_INSTALL})
 }
 
 EXPORT_FUNCTIONS do_configure do_compile do_install
-- 
2.25.1

Re: [OE-core] [PATCH] waf: explicitly run waf under Python 3

[Joshua Watt]

[-- Attachment #1: Type: text/plain, Size: 2456 bytes --]

On Wed, Oct 14, 2020, 6:26 AM Ross Burton <ross@burtonini.com> wrote:

> We don't assume Python 2 is on the host and don't expose an unversioned
> python in HOSTTOOLS, but waf uses the unversion python binary.
>
> Typically this is solved by patching waf (see the glmark2 recipe) but
> this can be solved centrally by invoking python3 directly.  All wscripts
> that needed Python 2 would have been fixed by now.
>
> Signed-off-by: Ross Burton <ross.burton@arm.com>
> ---
>  meta/classes/waf.bbclass | 8 ++++----
>  1 file changed, 4 insertions(+), 4 deletions(-)
>
> diff --git a/meta/classes/waf.bbclass b/meta/classes/waf.bbclass
> index 309f625a40..df6d4d53e8 100644
> --- a/meta/classes/waf.bbclass
> +++ b/meta/classes/waf.bbclass
> @@ -42,7 +42,7 @@ python waf_preconfigure() {
>      subsrcdir = d.getVar('S')
>      wafbin = os.path.join(subsrcdir, 'waf')
>      try:
> -        result = subprocess.check_output([wafbin, '--version'],
> cwd=subsrcdir, stderr=subprocess.STDOUT)
> +        result = subprocess.check_output(['python3', wafbin,
> '--version'], cwd=subsrcdir, stderr=subprocess.STDOUT)
>

This will break our internal usage... We still use waf 1 which runs under
python 2 only :(, although we are working on transitioning

         version = result.decode('utf-8').split()[1]
>          if StrictVersion(version) >= StrictVersion("1.8.7"):
>              d.setVar("WAF_EXTRA_CONF", "--bindir=${bindir}
> --libdir=${libdir}")
> @@ -55,16 +55,16 @@ python waf_preconfigure() {
>  do_configure[prefuncs] += "waf_preconfigure"
>
>  waf_do_configure() {
> -       (cd ${S} && ./waf configure -o ${B} --prefix=${prefix}
> ${WAF_EXTRA_CONF} ${EXTRA_OECONF})
> +       (cd ${S} && python3 ./waf configure -o ${B} --prefix=${prefix}
> ${WAF_EXTRA_CONF} ${EXTRA_OECONF})
>  }
>
>  do_compile[progress] = "outof:^\[\s*(\d+)/\s*(\d+)\]\s+"
>  waf_do_compile()  {
> -       (cd ${S} && ./waf build ${@oe.utils.parallel_make_argument(d,
> '-j%d', limit=64)} ${EXTRA_OEWAF_BUILD})
> +       (cd ${S} && python3 ./waf build
> ${@oe.utils.parallel_make_argument(d, '-j%d', limit=64)}
> ${EXTRA_OEWAF_BUILD})
>  }
>
>  waf_do_install() {
> -       (cd ${S} && ./waf install --destdir=${D} ${EXTRA_OEWAF_INSTALL})
> +       (cd ${S} && python3 ./waf install --destdir=${D}
> ${EXTRA_OEWAF_INSTALL})
>  }
>
>  EXPORT_FUNCTIONS do_configure do_compile do_install
> --
> 2.25.1
>
>
> 
>
>

[-- Attachment #2: Type: text/html, Size: 3372 bytes --]

Re: [OE-core] [PATCH] waf: explicitly run waf under Python 3

[Khem Raj]

On Wed, Oct 14, 2020 at 5:41 AM Joshua Watt <JPEWhacker@gmail.com> wrote:
>
>
>
> On Wed, Oct 14, 2020, 6:26 AM Ross Burton <ross@burtonini.com> wrote:
>>
>> We don't assume Python 2 is on the host and don't expose an unversioned
>> python in HOSTTOOLS, but waf uses the unversion python binary.
>>
>> Typically this is solved by patching waf (see the glmark2 recipe) but
>> this can be solved centrally by invoking python3 directly.  All wscripts
>> that needed Python 2 would have been fixed by now.
>>
>> Signed-off-by: Ross Burton <ross.burton@arm.com>
>> ---
>>  meta/classes/waf.bbclass | 8 ++++----
>>  1 file changed, 4 insertions(+), 4 deletions(-)
>>
>> diff --git a/meta/classes/waf.bbclass b/meta/classes/waf.bbclass
>> index 309f625a40..df6d4d53e8 100644
>> --- a/meta/classes/waf.bbclass
>> +++ b/meta/classes/waf.bbclass
>> @@ -42,7 +42,7 @@ python waf_preconfigure() {
>>      subsrcdir = d.getVar('S')
>>      wafbin = os.path.join(subsrcdir, 'waf')
>>      try:
>> -        result = subprocess.check_output([wafbin, '--version'], cwd=subsrcdir, stderr=subprocess.STDOUT)
>> +        result = subprocess.check_output(['python3', wafbin, '--version'], cwd=subsrcdir, stderr=subprocess.STDOUT)
>
>
> This will break our internal usage... We still use waf 1 which runs under python 2 only :(, although we are working on transitioning

this is for master and we have already said goodbye to python2 in
oe-core. perhaps reverse patching waf in your app might be an option ?

>
>>          version = result.decode('utf-8').split()[1]
>>          if StrictVersion(version) >= StrictVersion("1.8.7"):
>>              d.setVar("WAF_EXTRA_CONF", "--bindir=${bindir} --libdir=${libdir}")
>> @@ -55,16 +55,16 @@ python waf_preconfigure() {
>>  do_configure[prefuncs] += "waf_preconfigure"
>>
>>  waf_do_configure() {
>> -       (cd ${S} && ./waf configure -o ${B} --prefix=${prefix} ${WAF_EXTRA_CONF} ${EXTRA_OECONF})
>> +       (cd ${S} && python3 ./waf configure -o ${B} --prefix=${prefix} ${WAF_EXTRA_CONF} ${EXTRA_OECONF})
>>  }
>>
>>  do_compile[progress] = "outof:^\[\s*(\d+)/\s*(\d+)\]\s+"
>>  waf_do_compile()  {
>> -       (cd ${S} && ./waf build ${@oe.utils.parallel_make_argument(d, '-j%d', limit=64)} ${EXTRA_OEWAF_BUILD})
>> +       (cd ${S} && python3 ./waf build ${@oe.utils.parallel_make_argument(d, '-j%d', limit=64)} ${EXTRA_OEWAF_BUILD})
>>  }
>>
>>  waf_do_install() {
>> -       (cd ${S} && ./waf install --destdir=${D} ${EXTRA_OEWAF_INSTALL})
>> +       (cd ${S} && python3 ./waf install --destdir=${D} ${EXTRA_OEWAF_INSTALL})
>>  }
>>
>>  EXPORT_FUNCTIONS do_configure do_compile do_install
>> --
>> 2.25.1
>>
>>
>>
>>
>
> 
>

Re: [OE-core] [PATCH] waf: explicitly run waf under Python 3

[Joshua Watt]

On 10/14/20 4:01 PM, Khem Raj wrote:
> On Wed, Oct 14, 2020 at 5:41 AM Joshua Watt <JPEWhacker@gmail.com> wrote:
>>
>>
>> On Wed, Oct 14, 2020, 6:26 AM Ross Burton <ross@burtonini.com> wrote:
>>> We don't assume Python 2 is on the host and don't expose an unversioned
>>> python in HOSTTOOLS, but waf uses the unversion python binary.
>>>
>>> Typically this is solved by patching waf (see the glmark2 recipe) but
>>> this can be solved centrally by invoking python3 directly.  All wscripts
>>> that needed Python 2 would have been fixed by now.
>>>
>>> Signed-off-by: Ross Burton <ross.burton@arm.com>
>>> ---
>>>   meta/classes/waf.bbclass | 8 ++++----
>>>   1 file changed, 4 insertions(+), 4 deletions(-)
>>>
>>> diff --git a/meta/classes/waf.bbclass b/meta/classes/waf.bbclass
>>> index 309f625a40..df6d4d53e8 100644
>>> --- a/meta/classes/waf.bbclass
>>> +++ b/meta/classes/waf.bbclass
>>> @@ -42,7 +42,7 @@ python waf_preconfigure() {
>>>       subsrcdir = d.getVar('S')
>>>       wafbin = os.path.join(subsrcdir, 'waf')
>>>       try:
>>> -        result = subprocess.check_output([wafbin, '--version'], cwd=subsrcdir, stderr=subprocess.STDOUT)
>>> +        result = subprocess.check_output(['python3', wafbin, '--version'], cwd=subsrcdir, stderr=subprocess.STDOUT)
>>
>> This will break our internal usage... We still use waf 1 which runs under python 2 only :(, although we are working on transitioning
> this is for master and we have already said goodbye to python2 in
> oe-core. perhaps reverse patching waf in your app might be an option ?

a.k.a. upgrading to waf 2 :)

Can we deal with it by importing either `pythonnative` or 
`python3native` in the recipes along with waf.bbclass, e.g.

     inherit waf pythonnative # waf 1

or

     inherit waf python3native # waf 2

Then you get the interpreter than matches the waf in your repo.

I do appreciate that oe-core has dumped python 2 support (and we should 
have *ages* ago also)... but I also don't really want a fire drill when 
most of our applications no longer build on 3.2, especially we can just 
inherit python3native as shown.

>
>>>           version = result.decode('utf-8').split()[1]
>>>           if StrictVersion(version) >= StrictVersion("1.8.7"):
>>>               d.setVar("WAF_EXTRA_CONF", "--bindir=${bindir} --libdir=${libdir}")
>>> @@ -55,16 +55,16 @@ python waf_preconfigure() {
>>>   do_configure[prefuncs] += "waf_preconfigure"
>>>
>>>   waf_do_configure() {
>>> -       (cd ${S} && ./waf configure -o ${B} --prefix=${prefix} ${WAF_EXTRA_CONF} ${EXTRA_OECONF})
>>> +       (cd ${S} && python3 ./waf configure -o ${B} --prefix=${prefix} ${WAF_EXTRA_CONF} ${EXTRA_OECONF})
>>>   }
>>>
>>>   do_compile[progress] = "outof:^\[\s*(\d+)/\s*(\d+)\]\s+"
>>>   waf_do_compile()  {
>>> -       (cd ${S} && ./waf build ${@oe.utils.parallel_make_argument(d, '-j%d', limit=64)} ${EXTRA_OEWAF_BUILD})
>>> +       (cd ${S} && python3 ./waf build ${@oe.utils.parallel_make_argument(d, '-j%d', limit=64)} ${EXTRA_OEWAF_BUILD})
>>>   }
>>>
>>>   waf_do_install() {
>>> -       (cd ${S} && ./waf install --destdir=${D} ${EXTRA_OEWAF_INSTALL})
>>> +       (cd ${S} && python3 ./waf install --destdir=${D} ${EXTRA_OEWAF_INSTALL})
>>>   }
>>>
>>>   EXPORT_FUNCTIONS do_configure do_compile do_install
>>> --
>>> 2.25.1
>>>
>>>
>>>
>>>
>> 
>>

Re: [OE-core] [PATCH] waf: explicitly run waf under Python 3

[Khem Raj]

[-- Attachment #1: Type: text/plain, Size: 4027 bytes --]

On Wed, Oct 14, 2020 at 2:23 PM Joshua Watt <jpewhacker@gmail.com> wrote:

>
> On 10/14/20 4:01 PM, Khem Raj wrote:
> > On Wed, Oct 14, 2020 at 5:41 AM Joshua Watt <JPEWhacker@gmail.com>
> wrote:
> >>
> >>
> >> On Wed, Oct 14, 2020, 6:26 AM Ross Burton <ross@burtonini.com> wrote:
> >>> We don't assume Python 2 is on the host and don't expose an unversioned
> >>> python in HOSTTOOLS, but waf uses the unversion python binary.
> >>>
> >>> Typically this is solved by patching waf (see the glmark2 recipe) but
> >>> this can be solved centrally by invoking python3 directly.  All
> wscripts
> >>> that needed Python 2 would have been fixed by now.
> >>>
> >>> Signed-off-by: Ross Burton <ross.burton@arm.com>
> >>> ---
> >>>   meta/classes/waf.bbclass | 8 ++++----
> >>>   1 file changed, 4 insertions(+), 4 deletions(-)
> >>>
> >>> diff --git a/meta/classes/waf.bbclass b/meta/classes/waf.bbclass
> >>> index 309f625a40..df6d4d53e8 100644
> >>> --- a/meta/classes/waf.bbclass
> >>> +++ b/meta/classes/waf.bbclass
> >>> @@ -42,7 +42,7 @@ python waf_preconfigure() {
> >>>       subsrcdir = d.getVar('S')
> >>>       wafbin = os.path.join(subsrcdir, 'waf')
> >>>       try:
> >>> -        result = subprocess.check_output([wafbin, '--version'],
> cwd=subsrcdir, stderr=subprocess.STDOUT)
> >>> +        result = subprocess.check_output(['python3', wafbin,
> '--version'], cwd=subsrcdir, stderr=subprocess.STDOUT)
> >>
> >> This will break our internal usage... We still use waf 1 which runs
> under python 2 only :(, although we are working on transitioning
> > this is for master and we have already said goodbye to python2 in
> > oe-core. perhaps reverse patching waf in your app might be an option ?
>
> a.k.a. upgrading to waf 2 :)
>
> Can we deal with it by importing either `pythonnative` or
> `python3native` in the recipes along with waf.bbclass, e.g.
>
>      inherit waf pythonnative # waf 1
>
> or
>
>      inherit waf python3native # waf 2
>
> Then you get the interpreter than matches the waf in your repo.



This is going to be a large set of recipes I would imagine and perhaps not
worth it for supporting old version of waf which is not used in any
packages on open layers it will aggravate serialization of waf recipes due
to python native dependency on the other hand there are recipes in meta-Oe
which need this change where patches are pending



>
> I do appreciate that oe-core has dumped python 2 support (and we should
> have *ages* ago also)... but I also don't really want a fire drill when
> most of our applications no longer build on 3.2, especially we can just
> inherit python3native as shown.
>
> >
> >>>           version = result.decode('utf-8').split()[1]
> >>>           if StrictVersion(version) >= StrictVersion("1.8.7"):
> >>>               d.setVar("WAF_EXTRA_CONF", "--bindir=${bindir}
> --libdir=${libdir}")
> >>> @@ -55,16 +55,16 @@ python waf_preconfigure() {
> >>>   do_configure[prefuncs] += "waf_preconfigure"
> >>>
> >>>   waf_do_configure() {
> >>> -       (cd ${S} && ./waf configure -o ${B} --prefix=${prefix}
> ${WAF_EXTRA_CONF} ${EXTRA_OECONF})
> >>> +       (cd ${S} && python3 ./waf configure -o ${B} --prefix=${prefix}
> ${WAF_EXTRA_CONF} ${EXTRA_OECONF})
> >>>   }
> >>>
> >>>   do_compile[progress] = "outof:^\[\s*(\d+)/\s*(\d+)\]\s+"
> >>>   waf_do_compile()  {
> >>> -       (cd ${S} && ./waf build ${@oe.utils.parallel_make_argument(d,
> '-j%d', limit=64)} ${EXTRA_OEWAF_BUILD})
> >>> +       (cd ${S} && python3 ./waf build
> ${@oe.utils.parallel_make_argument(d, '-j%d', limit=64)}
> ${EXTRA_OEWAF_BUILD})
> >>>   }
> >>>
> >>>   waf_do_install() {
> >>> -       (cd ${S} && ./waf install --destdir=${D}
> ${EXTRA_OEWAF_INSTALL})
> >>> +       (cd ${S} && python3 ./waf install --destdir=${D}
> ${EXTRA_OEWAF_INSTALL})
> >>>   }
> >>>
> >>>   EXPORT_FUNCTIONS do_configure do_compile do_install
> >>> --
> >>> 2.25.1
> >>>
> >>>
> >>>
> >>>
> >> 
> >>
>

[-- Attachment #2: Type: text/html, Size: 5639 bytes --]

Re: [OE-core] [PATCH] waf: explicitly run waf under Python 3

[Richard Purdie]

On Wed, 2020-10-14 at 07:41 -0500, Joshua Watt wrote:
> 
> 
> On Wed, Oct 14, 2020, 6:26 AM Ross Burton <ross@burtonini.com> wrote:
> > We don't assume Python 2 is on the host and don't expose an unversioned
> > python in HOSTTOOLS, but waf uses the unversion python binary.
> > 
> > Typically this is solved by patching waf (see the glmark2 recipe) but
> > this can be solved centrally by invoking python3 directly.  All wscripts
> > that needed Python 2 would have been fixed by now.
> > 
> > Signed-off-by: Ross Burton <ross.burton@arm.com>
> > ---
> >  meta/classes/waf.bbclass | 8 ++++----
> >  1 file changed, 4 insertions(+), 4 deletions(-)
> > 
> > diff --git a/meta/classes/waf.bbclass b/meta/classes/waf.bbclass
> > index 309f625a40..df6d4d53e8 100644
> > --- a/meta/classes/waf.bbclass
> > +++ b/meta/classes/waf.bbclass
> > @@ -42,7 +42,7 @@ python waf_preconfigure() {
> >      subsrcdir = d.getVar('S')
> >      wafbin = os.path.join(subsrcdir, 'waf')
> >      try:
> > -        result = subprocess.check_output([wafbin, '--version'], cwd=subsrcdir, stderr=subprocess.STDOUT)
> > +        result = subprocess.check_output(['python3', wafbin, '--version'], cwd=subsrcdir, stderr=subprocess.STDOUT)
> 
> This will break our internal usage... We still use waf 1 which runs under python 2 only :(, although we are working on transitioning

Might I suggest this is parameterised as something like: 

WAFPYTHON = "python3"

then OE-Core can continue on and improve but people with legacy can
handle it better?

Cheers,

Richard

Re: [OE-core] [PATCH] waf: explicitly run waf under Python 3

[Joshua Watt]

On Thu, Oct 15, 2020 at 7:44 AM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> On Wed, 2020-10-14 at 07:41 -0500, Joshua Watt wrote:
> >
> >
> > On Wed, Oct 14, 2020, 6:26 AM Ross Burton <ross@burtonini.com> wrote:
> > > We don't assume Python 2 is on the host and don't expose an unversioned
> > > python in HOSTTOOLS, but waf uses the unversion python binary.
> > >
> > > Typically this is solved by patching waf (see the glmark2 recipe) but
> > > this can be solved centrally by invoking python3 directly.  All wscripts
> > > that needed Python 2 would have been fixed by now.
> > >
> > > Signed-off-by: Ross Burton <ross.burton@arm.com>
> > > ---
> > >  meta/classes/waf.bbclass | 8 ++++----
> > >  1 file changed, 4 insertions(+), 4 deletions(-)
> > >
> > > diff --git a/meta/classes/waf.bbclass b/meta/classes/waf.bbclass
> > > index 309f625a40..df6d4d53e8 100644
> > > --- a/meta/classes/waf.bbclass
> > > +++ b/meta/classes/waf.bbclass
> > > @@ -42,7 +42,7 @@ python waf_preconfigure() {
> > >      subsrcdir = d.getVar('S')
> > >      wafbin = os.path.join(subsrcdir, 'waf')
> > >      try:
> > > -        result = subprocess.check_output([wafbin, '--version'], cwd=subsrcdir, stderr=subprocess.STDOUT)
> > > +        result = subprocess.check_output(['python3', wafbin, '--version'], cwd=subsrcdir, stderr=subprocess.STDOUT)
> >
> > This will break our internal usage... We still use waf 1 which runs under python 2 only :(, although we are working on transitioning
>
> Might I suggest this is parameterised as something like:
>
> WAFPYTHON = "python3"
>
> then OE-Core can continue on and improve but people with legacy can
> handle it better?

That would be fine with me

>
> Cheers,
>
> Richard
>

Re: [OE-core] [PATCH] waf: explicitly run waf under Python 3

[Ross Burton]

On Thu, 15 Oct 2020 at 13:44, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> Might I suggest this is parameterised as something like:
>
> WAFPYTHON = "python3"
>
> then OE-Core can continue on and improve but people with legacy can
> handle it better?

This was the patch I originally had months ago ;)

v2 incoming.

Ross