[PATCH 00/16] porting scripts to python3

[PATCH 00/16] porting scripts to python3

[Ed Bartosh]

Hi,

Please review set of python3 fixes for oe scripts.

As most of the scripts are not covered by tests it was quite hard
to test them. I tested only those which I know how to work with.
It would be greate to have any help with testing.

The following changes since commit dcbfc72303d6782d9108538c830cf2ee7849bf59:

  poky: Add Ubuntu 16.04 to tested/supported list since I run it myself (2016-06-02 10:37:05 +0100)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib ed/oe/python3
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/oe/python3

Ed Bartosh (16):
  scripts: python3: convert iterables to lists
  scripts: python3: use new style except statement
  scripts: python3: rename raw_input to input
  scripts: python3: fix urllib imports
  ksize.py: python3: get rid of strings.join
  scripts: python3: get rid of __future__ imports
  scripts: python3: use new metaclass syntax
  combo-layer: python3: import reduce
  engine.py: python3: rename sys.maxint to sys.maxsize
  scripts: python3: use explicit relative imports
  scripts: python3: replace exec statement with builtin
  engine: python3: replace iteritems() -> items()
  dirsize: python3: fix TypeError: unorderable types
  scripts: python3: decode subprocess output
  combo-layer: python3: use tempfile.TemporaryFile
  scripts: python3: change python to python3 in shebang

 scripts/bitbake-whatchanged                      |  9 +++----
 scripts/buildhistory-collect-srcrevs             |  2 +-
 scripts/cleanup-workdir                          |  4 +--
 scripts/combo-layer                              | 15 ++++++-----
 scripts/contrib/bbvars.py                        | 16 +++++------
 scripts/contrib/devtool-stress.py                |  2 +-
 scripts/contrib/list-packageconfig-flags.py      |  2 +-
 scripts/contrib/verify-homepage.py               |  6 ++---
 scripts/cp-noerror                               | 10 +++----
 scripts/gen-lockedsig-cache                      |  2 +-
 scripts/lib/bsp/engine.py                        | 34 +++++++++++-------------
 scripts/lib/bsp/kernel.py                        | 25 +++++++++--------
 scripts/lib/devtool/__init__.py                  |  2 +-
 scripts/lib/devtool/build.py                     |  2 +-
 scripts/lib/devtool/standard.py                  |  4 +--
 scripts/lib/recipetool/create_buildsys.py        |  2 +-
 scripts/lib/recipetool/create_buildsys_python.py |  6 ++---
 scripts/oe-pkgdata-util                          |  2 +-
 scripts/oe-publish-sdk                           |  2 +-
 scripts/oe-selftest                              |  2 +-
 scripts/oe-trim-schemas                          |  2 +-
 scripts/oepydevshell-internal.py                 |  4 +--
 scripts/opkg-query-helper.py                     |  2 +-
 scripts/pybootchartgui/pybootchartgui/main.py.in |  2 --
 scripts/pybootchartgui/pybootchartgui/parsing.py |  3 ---
 scripts/pythondeps                               |  2 +-
 scripts/relocate_sdk.py                          |  2 +-
 scripts/send-error-report                        | 22 +++++++--------
 scripts/swabber-strace-attach                    |  2 +-
 scripts/sysroot-relativelinks.py                 |  2 +-
 scripts/test-remote-image                        | 10 +++----
 scripts/tiny/dirsize.py                          | 16 +++++------
 scripts/tiny/ksize.py                            |  8 +++---
 scripts/wic                                      |  2 --
 scripts/yocto-bsp                                |  2 +-
 scripts/yocto-kernel                             |  2 +-
 scripts/yocto-layer                              |  2 +-
 37 files changed, 107 insertions(+), 127 deletions(-)

--
Regards,
Ed

[PATCH 01/16] scripts: python3: convert iterables to lists

[Ed Bartosh]

Converted return value of items() keys() and values() to
lists when dictionary is modified in the loop and when
the result is added to the list.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/bitbake-whatchanged                      | 6 +++---
 scripts/combo-layer                              | 2 +-
 scripts/lib/devtool/build.py                     | 2 +-
 scripts/lib/devtool/standard.py                  | 4 ++--
 scripts/lib/recipetool/create_buildsys.py        | 2 +-
 scripts/lib/recipetool/create_buildsys_python.py | 6 +++---
 scripts/oe-pkgdata-util                          | 2 +-
 7 files changed, 12 insertions(+), 12 deletions(-)

diff --git a/scripts/bitbake-whatchanged b/scripts/bitbake-whatchanged
index a20adb2..b05aead 100755
--- a/scripts/bitbake-whatchanged
+++ b/scripts/bitbake-whatchanged
@@ -120,7 +120,7 @@ def print_added(d_new = None, d_old = None):
     Print the newly added tasks
     """
     added = {}
-    for k in d_new.keys():
+    for k in list(d_new.keys()):
         if k not in d_old:
             # Add the new one to added dict, and remove it from
             # d_new, so the remaining ones are the changed ones
@@ -155,7 +155,7 @@ def print_vrchanged(d_new = None, d_old = None, vr = None):
     """
     pvchanged = {}
     counter = 0
-    for k in d_new.keys():
+    for k in list(d_new.keys()):
         if d_new.get(k).get(vr) != d_old.get(k).get(vr):
             counter += 1
             pn, task = split_pntask(k)
@@ -279,7 +279,7 @@ Note:
 
         # Remove the same one from both stamps.
         cnt_unchanged = 0
-        for k in new_dict.keys():
+        for k in list(new_dict.keys()):
             if k in old_dict:
                 cnt_unchanged += 1
                 del(new_dict[k])
diff --git a/scripts/combo-layer b/scripts/combo-layer
index 234d9e4..7c41f92 100755
--- a/scripts/combo-layer
+++ b/scripts/combo-layer
@@ -1186,7 +1186,7 @@ def update_with_history(conf, components, revisions, repos):
         msg = conf_commit_msg(conf, components)
         new_tree = runcmd("git write-tree", **wargs).strip()
         new_rev = runcmd("git commit-tree".split() +
-                         add_p([head] + additional_heads.keys()) +
+                         add_p([head] + list(additional_heads.keys())) +
                          ["-m", msg, new_tree],
                          **wargs).strip()
         # And done! This is the first time we change the HEAD in the actual work tree.
diff --git a/scripts/lib/devtool/build.py b/scripts/lib/devtool/build.py
index 48f6fe1..6be549d 100644
--- a/scripts/lib/devtool/build.py
+++ b/scripts/lib/devtool/build.py
@@ -27,7 +27,7 @@ logger = logging.getLogger('devtool')
 
 
 def _set_file_values(fn, values):
-    remaining = values.keys()
+    remaining = list(values.keys())
 
     def varfunc(varname, origvalue, op, newlines):
         newvalue = values.get(varname, origvalue)
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 08153c6..a2516d6 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -998,7 +998,7 @@ def _export_local_files(srctree, rd, destdir):
         bb.process.run(['git', 'checkout', tree, '--', '.'], cwd=srctree,
                         env=dict(os.environ, GIT_WORK_TREE=destdir,
                                  GIT_INDEX_FILE=tmp_index))
-        new_set = _git_ls_tree(srctree, tree, True).keys()
+        new_set = list(_git_ls_tree(srctree, tree, True).keys())
     elif os.path.isdir(local_files_dir):
         # If not tracked by Git, just copy from working copy
         new_set = _ls_tree(os.path.join(srctree, 'oe-local-files'))
@@ -1309,7 +1309,7 @@ def reset(args, config, basepath, workspace):
         raise DevtoolError("Recipe must be specified, or specify -a/--all to "
                            "reset all recipes")
     if args.all:
-        recipes = workspace.keys()
+        recipes = list(workspace.keys())
     else:
         recipes = [args.recipename]
 
diff --git a/scripts/lib/recipetool/create_buildsys.py b/scripts/lib/recipetool/create_buildsys.py
index de3d9ae..78ae4bc 100644
--- a/scripts/lib/recipetool/create_buildsys.py
+++ b/scripts/lib/recipetool/create_buildsys.py
@@ -682,7 +682,7 @@ class AutotoolsRecipeHandler(RecipeHandler):
                 process_macro(in_keyword, partial)
 
         if extravalues:
-            for k,v in extravalues.items():
+            for k,v in list(extravalues.items()):
                 if v:
                     if v.startswith('$') or v.startswith('@') or v.startswith('%'):
                         del extravalues[k]
diff --git a/scripts/lib/recipetool/create_buildsys_python.py b/scripts/lib/recipetool/create_buildsys_python.py
index 55cce0e..aff13cf 100644
--- a/scripts/lib/recipetool/create_buildsys_python.py
+++ b/scripts/lib/recipetool/create_buildsys_python.py
@@ -361,7 +361,7 @@ class PythonRecipeHandler(RecipeHandler):
 
         # Naive mapping of setup() arguments to PKG-INFO field names
         for d in [info, non_literals]:
-            for key, value in d.items():
+            for key, value in list(d.items()):
                 new_key = _map(key)
                 if new_key != key:
                     del d[key]
@@ -443,7 +443,7 @@ class PythonRecipeHandler(RecipeHandler):
                 elif new_value != value:
                     info[variable] = new_value
             elif hasattr(value, 'items'):
-                for dkey, dvalue in value.items():
+                for dkey, dvalue in list(value.items()):
                     new_list = []
                     for pos, a_value in enumerate(dvalue):
                         new_value = replace_value(search, replace, a_value)
@@ -608,7 +608,7 @@ def gather_setup_info(fileobj):
     visitor.visit(parsed)
 
     non_literals, extensions = {}, []
-    for key, value in visitor.keywords.items():
+    for key, value in list(visitor.keywords.items()):
         if key == 'ext_modules':
             if isinstance(value, list):
                 for ext in value:
diff --git a/scripts/oe-pkgdata-util b/scripts/oe-pkgdata-util
index b39d9b5..b16ecc9 100755
--- a/scripts/oe-pkgdata-util
+++ b/scripts/oe-pkgdata-util
@@ -240,7 +240,7 @@ def lookup_pkg(args):
         sys.exit(1)
 
     if args.reverse:
-        items = mappings.values()
+        items = list(mappings.values())
     else:
         items = []
         for pkg in pkgs:
-- 
2.1.4

[PATCH 02/16] scripts: python3: use new style except statement

[Ed Bartosh]

Changed old syle except statements 'except <exception>, var'
to new style 'except <exception> as var' as old style is not
supported in python3.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/combo-layer       |  4 ++--
 scripts/contrib/bbvars.py | 14 +++++++-------
 scripts/cp-noerror        |  8 ++++----
 scripts/tiny/ksize.py     |  2 +-
 4 files changed, 14 insertions(+), 14 deletions(-)

diff --git a/scripts/combo-layer b/scripts/combo-layer
index 7c41f92..0644cdc 100755
--- a/scripts/combo-layer
+++ b/scripts/combo-layer
@@ -190,7 +190,7 @@ def runcmd(cmd,destdir=None,printerr=True,out=None,env=None):
         err = os.tmpfile()
     try:
         subprocess.check_call(cmd, stdout=out, stderr=err, cwd=destdir, shell=isinstance(cmd, str), env=env or os.environ)
-    except subprocess.CalledProcessError,e:
+    except subprocess.CalledProcessError as e:
         err.seek(0)
         if printerr:
             logger.error("%s" % err.read())
@@ -429,7 +429,7 @@ file_exclude = %s''' % (name, file_filter or '<empty>', repo.get('file_exclude',
                     runcmd('git replace --graft %s %s' % (start, startrev))
             try:
                 runcmd(merge)
-            except Exception, error:
+            except Exception as error:
                 logger.info('''Merging component repository history failed, perhaps because of merge conflicts.
 It may be possible to commit anyway after resolving these conflicts.
 
diff --git a/scripts/contrib/bbvars.py b/scripts/contrib/bbvars.py
index 6f47935..b865dd1 100755
--- a/scripts/contrib/bbvars.py
+++ b/scripts/contrib/bbvars.py
@@ -37,9 +37,9 @@ def recipe_bbvars(recipe):
     vset = set()
     try:
         r = open(recipe)
-    except IOError as (errno, strerror):
+    except IOError as err:
         print('WARNING: Failed to open recipe ', recipe)
-        print(strerror)
+        print(err.args[1])
 
     for line in r:
         # Strip any comments from the line
@@ -71,9 +71,9 @@ def bbvar_is_documented(var, docfiles):
     for doc in docfiles:
         try:
             f = open(doc)
-        except IOError as (errno, strerror):
+        except IOError as err:
             print('WARNING: Failed to open doc ', doc)
-            print(strerror)
+            print(err.args[1])
         for line in f:
             if prog.match(line):
                 return True
@@ -87,8 +87,8 @@ def bbvar_doctag(var, docconf):
 
     try:
         f = open(docconf)
-    except IOError as (errno, strerror):
-        return strerror
+    except IOError as err:
+        return err.args[1]
 
     for line in f:
         m = prog.search(line)
@@ -109,7 +109,7 @@ def main():
     # Collect and validate input
     try:
         opts, args = getopt.getopt(sys.argv[1:], "d:hm:t:T", ["help"])
-    except getopt.GetoptError, err:
+    except getopt.GetoptError as err:
         print('%s' % str(err))
         usage()
         sys.exit(2)
diff --git a/scripts/cp-noerror b/scripts/cp-noerror
index 28eb90d..d8be677 100755
--- a/scripts/cp-noerror
+++ b/scripts/cp-noerror
@@ -33,16 +33,16 @@ def copytree(src, dst, symlinks=False, ignore=None):
                 shutil.copy2(srcname, dstname)
         # catch the Error from the recursive copytree so that we can
         # continue with other files
-        except shutil.Error, err:
+        except shutil.Error as err:
             errors.extend(err.args[0])
-        except EnvironmentError, why:
+        except EnvironmentError as why:
             errors.append((srcname, dstname, str(why)))
     try:
         shutil.copystat(src, dst)
-    except OSError, why:
+    except OSError as why:
         errors.extend((src, dst, str(why)))
     if errors:
-        raise shutil.Error, errors
+        raise shutil.Error(errors)
 
 try:
     copytree(sys.argv[1], sys.argv[2])
diff --git a/scripts/tiny/ksize.py b/scripts/tiny/ksize.py
index 275c983..54b71f8 100755
--- a/scripts/tiny/ksize.py
+++ b/scripts/tiny/ksize.py
@@ -133,7 +133,7 @@ class Report:
 def main():
     try:
         opts, args = getopt.getopt(sys.argv[1:], "dh", ["help"])
-    except getopt.GetoptError, err:
+    except getopt.GetoptError as err:
         print('%s' % str(err))
         usage()
         sys.exit(2)
-- 
2.1.4

[PATCH 03/16] scripts: python3: rename raw_input to input

[Ed Bartosh]

Renamed raw_input to input as raw_input does not
exist in python 3.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/bsp/engine.py        | 12 ++++++------
 scripts/lib/bsp/kernel.py        | 12 ++++++------
 scripts/oepydevshell-internal.py |  2 +-
 scripts/send-error-report        |  6 +++---
 4 files changed, 16 insertions(+), 16 deletions(-)

diff --git a/scripts/lib/bsp/engine.py b/scripts/lib/bsp/engine.py
index 3ab4295..f75454d 100644
--- a/scripts/lib/bsp/engine.py
+++ b/scripts/lib/bsp/engine.py
@@ -464,9 +464,9 @@ class ListInputLine(InputLine):
         choices_str = self.gen_choices_str(choicepairs)
         choices_val_list = self.gen_choices_val_list(choicepairs)
         if checklist:
-            choiceval = default(find_choicevals(raw_input(msg + "\n" + choices_str), choices_val_list), default_choice)
+            choiceval = default(find_choicevals(input(msg + "\n" + choices_str), choices_val_list), default_choice)
         else:
-            choiceval = default(find_choiceval(raw_input(msg + "\n" + choices_str), choices_val_list), default_choice)
+            choiceval = default(find_choiceval(input(msg + "\n" + choices_str), choices_val_list), default_choice)
 
         return choiceval
 
@@ -540,12 +540,12 @@ def get_verified_git_repo(input_str, name):
     """
     msg = input_str.strip() + " "
 
-    giturl = default(raw_input(msg), name)
+    giturl = default(input(msg), name)
 
     while True:
         if verify_git_repo(giturl):
             return giturl
-        giturl = default(raw_input(msg), name)
+        giturl = default(input(msg), name)
 
 
 def get_verified_file(input_str, name, filename_can_be_null):
@@ -555,14 +555,14 @@ def get_verified_file(input_str, name, filename_can_be_null):
     """
     msg = input_str.strip() + " "
 
-    filename = default(raw_input(msg), name)
+    filename = default(input(msg), name)
 
     while True:
         if not filename and filename_can_be_null:
             return filename
         if os.path.isfile(filename):
             return filename
-        filename = default(raw_input(msg), name)
+        filename = default(input(msg), name)
 
 
 def replace_file(replace_this, with_this):
diff --git a/scripts/lib/bsp/kernel.py b/scripts/lib/bsp/kernel.py
index 5bfa663..3c4b798 100644
--- a/scripts/lib/bsp/kernel.py
+++ b/scripts/lib/bsp/kernel.py
@@ -202,8 +202,8 @@ def yocto_kernel_config_rm(scripts_path, machine):
     config_items = read_config_items(scripts_path, machine)
 
     print("Specify the kernel config items to remove:")
-    input = raw_input(gen_choices_str(config_items))
-    rm_choices = input.split()
+    inp = input(gen_choices_str(config_items))
+    rm_choices = inp.split()
     rm_choices.sort()
 
     removed = []
@@ -359,8 +359,8 @@ def yocto_kernel_patch_rm(scripts_path, machine):
     patches = read_patch_items(scripts_path, machine)
 
     print("Specify the patches to remove:")
-    input = raw_input(gen_choices_str(patches))
-    rm_choices = input.split()
+    inp = input(gen_choices_str(patches))
+    rm_choices = inp.split()
     rm_choices.sort()
 
     removed = []
@@ -610,8 +610,8 @@ def yocto_kernel_feature_rm(scripts_path, machine):
     features = read_features(scripts_path, machine)
 
     print("Specify the features to remove:")
-    input = raw_input(gen_choices_str(features))
-    rm_choices = input.split()
+    inp = input(gen_choices_str(features))
+    rm_choices = inp.split()
     rm_choices.sort()
 
     removed = []
diff --git a/scripts/oepydevshell-internal.py b/scripts/oepydevshell-internal.py
index f7b2e4e..2566606 100755
--- a/scripts/oepydevshell-internal.py
+++ b/scripts/oepydevshell-internal.py
@@ -67,7 +67,7 @@ try:
                     i = i[4096:]
                 if sys.stdin in ready:
                     echonocbreak(sys.stdin.fileno())
-                    o = raw_input()
+                    o = input()
                     cbreaknoecho(sys.stdin.fileno())
                     pty.write(o + "\n")
             except (IOError, OSError) as e:
diff --git a/scripts/send-error-report b/scripts/send-error-report
index ed78bd6..7251e27 100755
--- a/scripts/send-error-report
+++ b/scripts/send-error-report
@@ -44,12 +44,12 @@ def getPayloadLimit(url):
 
 def ask_for_contactdetails():
     print("Please enter your name and your email (optionally), they'll be saved in the file you send.")
-    username = raw_input("Name (required): ")
-    email = raw_input("E-mail (not required): ")
+    username = input("Name (required): ")
+    email = input("E-mail (not required): ")
     return username, email
 
 def edit_content(json_file_path):
-    edit = raw_input("Review information before sending? (y/n): ")
+    edit = input("Review information before sending? (y/n): ")
     if 'y' in edit or 'Y' in edit:
         editor = os.environ.get('EDITOR', None)
         if editor:
-- 
2.1.4

[PATCH 04/16] scripts: python3: fix urllib imports

[Ed Bartosh]

Some functions and classes have been moved from urllib[2]
to urllib.request and urllib.error in python 3.

Used new imports to make the code working in python 3.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/contrib/verify-homepage.py |  4 ++--
 scripts/send-error-report          | 14 +++++++-------
 2 files changed, 9 insertions(+), 9 deletions(-)

diff --git a/scripts/contrib/verify-homepage.py b/scripts/contrib/verify-homepage.py
index 265ff65..18bb15b 100755
--- a/scripts/contrib/verify-homepage.py
+++ b/scripts/contrib/verify-homepage.py
@@ -7,7 +7,7 @@
 import sys
 import os
 import subprocess
-import urllib2
+import urllib.request
 
 
 # Allow importing scripts/lib modules
@@ -47,7 +47,7 @@ def verifyHomepage(bbhandler):
             homepage = data.getVar("HOMEPAGE", True)
             if homepage:
                 try:
-                    urllib2.urlopen(homepage, timeout=5)
+                    urllib.request.urlopen(homepage, timeout=5)
                 except Exception:
                     count = count + wgetHomepage(os.path.basename(realfn), homepage)
             checked.append(realfn)
diff --git a/scripts/send-error-report b/scripts/send-error-report
index 7251e27..122ce32 100755
--- a/scripts/send-error-report
+++ b/scripts/send-error-report
@@ -7,7 +7,7 @@
 # Author: Andreea Proca <andreea.b.proca@intel.com>
 # Author: Michael Wood <michael.g.wood@intel.com>
 
-import urllib2
+import urllib.request, urllib.error
 import sys
 import json
 import os
@@ -25,10 +25,10 @@ log = logging.getLogger("send-error-report")
 logging.basicConfig(format='%(levelname)s: %(message)s')
 
 def getPayloadLimit(url):
-    req = urllib2.Request(url, None)
+    req = urllib.request.Request(url, None)
     try:
-        response = urllib2.urlopen(req)
-    except urllib2.URLError as e:
+        response = urllib.request.urlopen(req)
+    except urllib.error.URLError as e:
         # Use this opportunity to bail out if we can't even contact the server
         log.error("Could not contact server: " + url)
         log.error(e.reason)
@@ -136,10 +136,10 @@ def send_data(data, args):
     else:
         url = "http://"+args.server+"/ClientPost/"
 
-    req = urllib2.Request(url, data=data, headers=headers)
+    req = urllib.request.Request(url, data=data, headers=headers)
     try:
-        response = urllib2.urlopen(req)
-    except urllib2.HTTPError, e:
+        response = urllib.request.urlopen(req)
+    except urllib.error.HTTPError as e:
         logging.error(e.reason)
         sys.exit(1)
 
-- 
2.1.4

[PATCH 05/16] ksize.py: python3: get rid of strings.join

[Ed Bartosh]

Used join method instead of strings.join as stings.join
doesn't exist in python 3.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/tiny/ksize.py | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/scripts/tiny/ksize.py b/scripts/tiny/ksize.py
index 54b71f8..587c930 100755
--- a/scripts/tiny/ksize.py
+++ b/scripts/tiny/ksize.py
@@ -28,8 +28,6 @@ import sys
 import getopt
 import os
 from subprocess import *
-from string import join
-
 
 def usage():
     prog = os.path.basename(sys.argv[0])
@@ -66,7 +64,7 @@ class Report:
 
         p = Popen("ls " + path + "/*.o | grep -v built-in.o",
                   shell=True, stdout=PIPE, stderr=PIPE)
-        glob = join(p.communicate()[0].splitlines())
+        glob = ' '.join(p.communicate()[0].splitlines())
         oreport = Report(glob, path + "/*.o")
         oreport.sizes.title = path + "/*.o"
         r.parts.append(oreport)
-- 
2.1.4

[PATCH 06/16] scripts: python3: get rid of __future__ imports

[Ed Bartosh]

Removed print_function and with_statement imports from __future__
as they're supported by python 3 by default.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/bitbake-whatchanged                      | 1 -
 scripts/pybootchartgui/pybootchartgui/main.py.in | 2 --
 scripts/pybootchartgui/pybootchartgui/parsing.py | 3 ---
 scripts/wic                                      | 2 --
 4 files changed, 8 deletions(-)

diff --git a/scripts/bitbake-whatchanged b/scripts/bitbake-whatchanged
index b05aead..55b61d0 100755
--- a/scripts/bitbake-whatchanged
+++ b/scripts/bitbake-whatchanged
@@ -17,7 +17,6 @@
 # along with this program; if not, write to the Free Software
 # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
 
-from __future__ import print_function
 import os
 import sys
 import getopt
diff --git a/scripts/pybootchartgui/pybootchartgui/main.py.in b/scripts/pybootchartgui/pybootchartgui/main.py.in
index 21bb0be..af26bd2 100644
--- a/scripts/pybootchartgui/pybootchartgui/main.py.in
+++ b/scripts/pybootchartgui/pybootchartgui/main.py.in
@@ -16,8 +16,6 @@
 #  You should have received a copy of the GNU General Public License
 #  along with pybootchartgui. If not, see <http://www.gnu.org/licenses/>.
 
-from __future__ import print_function
-
 import sys
 import os
 import optparse
diff --git a/scripts/pybootchartgui/pybootchartgui/parsing.py b/scripts/pybootchartgui/pybootchartgui/parsing.py
index d423b9f..a3a0b0b 100644
--- a/scripts/pybootchartgui/pybootchartgui/parsing.py
+++ b/scripts/pybootchartgui/pybootchartgui/parsing.py
@@ -13,9 +13,6 @@
 #  You should have received a copy of the GNU General Public License
 #  along with pybootchartgui. If not, see <http://www.gnu.org/licenses/>.
 
-
-from __future__ import with_statement
-
 import os
 import string
 import re
diff --git a/scripts/wic b/scripts/wic
index 5cc06f4..9023755 100755
--- a/scripts/wic
+++ b/scripts/wic
@@ -28,8 +28,6 @@
 # AUTHORS
 # Tom Zanussi <tom.zanussi (at] linux.intel.com>
 #
-from __future__ import print_function
-
 __version__ = "0.2.0"
 
 # Python Standard Library modules
-- 
2.1.4

[PATCH 07/16] scripts: python3: use new metaclass syntax

[Ed Bartosh]

Used metaclass=<metaclass> syntax instead old
__metaclass__ = <metaclass> as only new one is supported
in python 3.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/bsp/engine.py | 6 ++----
 scripts/test-remote-image | 8 ++------
 2 files changed, 4 insertions(+), 10 deletions(-)

diff --git a/scripts/lib/bsp/engine.py b/scripts/lib/bsp/engine.py
index f75454d..5b49486 100644
--- a/scripts/lib/bsp/engine.py
+++ b/scripts/lib/bsp/engine.py
@@ -40,12 +40,11 @@ import json
 import subprocess
 import shutil
 
-class Line():
+class Line(metaclass=ABCMeta):
     """
     Generic (abstract) container representing a line that will appear
     in the BSP-generating program.
     """
-    __metaclass__ = ABCMeta
 
     def __init__(self, line):
         self.line = line
@@ -319,11 +318,10 @@ class BooleanInputLine(InputLine):
         return line
 
 
-class ListInputLine(InputLine):
+class ListInputLine(InputLine, metaclass=ABCMeta):
     """
     Base class for List-based Input lines. e.g. Choicelist, Checklist.
     """
-    __metaclass__ = ABCMeta
 
     def __init__(self, props, tag, lineno):
         InputLine.__init__(self, props, tag, lineno)
diff --git a/scripts/test-remote-image b/scripts/test-remote-image
index 7a00db9..698948b 100755
--- a/scripts/test-remote-image
+++ b/scripts/test-remote-image
@@ -92,13 +92,11 @@ def get_args_parser():
     parser.add_argument('--skip-download', required=False, action="store_true", dest="skip_download", default=False, help='Skip downloading the images completely. This needs the correct files to be present in the directory specified by the target profile.')
     return parser
 
-class BaseTargetProfile(object):
+class BaseTargetProfile(object, metaclass=ABCMeta):
     """
     This class defines the meta profile for a specific target (MACHINE type + image type).
     """
 
-    __metaclass__ = ABCMeta
-
     def __init__(self, image_type):
         self.image_type = image_type
 
@@ -191,13 +189,11 @@ class AutoTargetProfile(BaseTargetProfile):
         return controller.get_extra_files()
 
 
-class BaseRepoProfile(object):
+class BaseRepoProfile(object, metaclass=ABCMeta):
     """
     This class defines the meta profile for an images repository.
     """
 
-    __metaclass__ = ABCMeta
-
     def __init__(self, repolink, localdir):
         self.localdir = localdir
         self.repolink = repolink
-- 
2.1.4

[PATCH 08/16] combo-layer: python3: import reduce

[Ed Bartosh]

Reduce is not a builtin function in python3.
It has to be imported from functools.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/combo-layer | 1 +
 1 file changed, 1 insertion(+)

diff --git a/scripts/combo-layer b/scripts/combo-layer
index 0644cdc..0954bb6 100755
--- a/scripts/combo-layer
+++ b/scripts/combo-layer
@@ -33,6 +33,7 @@ import pipes
 import shutil
 from collections import OrderedDict
 from string import Template
+from functools import reduce
 
 __version__ = "0.2.1"
 
-- 
2.1.4

[PATCH 09/16] engine.py: python3: rename sys.maxint to sys.maxsize

[Ed Bartosh]

Renamed sys.maxint -> sys.maxsize as sys.maxint doesn't
exist in python 3.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/bsp/engine.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/scripts/lib/bsp/engine.py b/scripts/lib/bsp/engine.py
index 5b49486..03bd66e 100644
--- a/scripts/lib/bsp/engine.py
+++ b/scripts/lib/bsp/engine.py
@@ -49,7 +49,7 @@ class Line(metaclass=ABCMeta):
     def __init__(self, line):
         self.line = line
         self.generated_line = ""
-        self.prio = sys.maxint
+        self.prio = sys.maxsize
         self.discard = False
 
     @abstractmethod
@@ -154,7 +154,7 @@ class InputLine(Line):
         try:
             self.prio = int(props["prio"])
         except KeyError:
-            self.prio = sys.maxint
+            self.prio = sys.maxsize
 
     def gen(self, context = None):
         try:
@@ -1284,7 +1284,7 @@ class InputLineGroup(InputLine):
     def __init__(self, codeline):
         InputLine.__init__(self, {}, "", 0)
         self.group = []
-        self.prio = sys.maxint
+        self.prio = sys.maxsize
         self.group.append(codeline)
 
     def append(self, line):
-- 
2.1.4

[PATCH 10/16] scripts: python3: use explicit relative imports

[Ed Bartosh]

Implicit relative imports within packages are not supported in
python 3. They have to be converted to explicit imports.

Used 'from .module import' syntax for relative imports.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/bsp/engine.py | 2 +-
 scripts/lib/bsp/kernel.py | 3 +--
 2 files changed, 2 insertions(+), 3 deletions(-)

diff --git a/scripts/lib/bsp/engine.py b/scripts/lib/bsp/engine.py
index 03bd66e..5ff6c98 100644
--- a/scripts/lib/bsp/engine.py
+++ b/scripts/lib/bsp/engine.py
@@ -34,7 +34,7 @@
 import os
 import sys
 from abc import ABCMeta, abstractmethod
-from tags import *
+from .tags import *
 import shlex
 import json
 import subprocess
diff --git a/scripts/lib/bsp/kernel.py b/scripts/lib/bsp/kernel.py
index 3c4b798..fe7133d 100644
--- a/scripts/lib/bsp/kernel.py
+++ b/scripts/lib/bsp/kernel.py
@@ -32,8 +32,7 @@ import shutil
 from tags import *
 import glob
 import subprocess
-from engine import create_context
-
+from .engine import create_context
 
 def find_bblayers():
     """
-- 
2.1.4

[PATCH 11/16] scripts: python3: replace exec statement with builtin

[Ed Bartosh]

Used exec() builtin instead of 'exec' statement as
this statement doesn't exist in python 3.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/bsp/engine.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/lib/bsp/engine.py b/scripts/lib/bsp/engine.py
index 5ff6c98..2b87d33 100644
--- a/scripts/lib/bsp/engine.py
+++ b/scripts/lib/bsp/engine.py
@@ -1362,7 +1362,7 @@ def run_program_lines(linelist, codedump):
         of = open("bspgen.out", "w")
         of.write(buf)
         of.close()
-    exec buf
+    exec(buf)
 
 
 def gen_target(files, context = None):
-- 
2.1.4

[PATCH 12/16] engine: python3: replace iteritems() -> items()

[Ed Bartosh]

Used items() as iteritems() doesn't exist in python 3.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/bsp/engine.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/scripts/lib/bsp/engine.py b/scripts/lib/bsp/engine.py
index 2b87d33..3ed90fa 100644
--- a/scripts/lib/bsp/engine.py
+++ b/scripts/lib/bsp/engine.py
@@ -1385,7 +1385,7 @@ def gen_supplied_property_vals(properties, program_lines):
     Generate user-specified entries for input values instead of
     generating input prompts.
     """
-    for name, val in properties.iteritems():
+    for name, val in properties.items():
         program_line = name + " = \"" + val + "\""
         program_lines.append(program_line)
 
@@ -1621,7 +1621,7 @@ def print_dict(items, indent = 0):
     """
     Print the values in a possibly nested dictionary.
     """
-    for key, val in items.iteritems():
+    for key, val in items.items():
         print("    "*indent + "\"%s\" :" % key)
         if type(val) == dict:
             print("{")
@@ -1654,7 +1654,7 @@ def get_properties(input_lines):
         props = line.props
         item = {}
         name = props["name"]
-        for key, val in props.items():
+        for key, val props.items():
             if not key == "name":
                 item[key] = val
         properties[name] = item
-- 
2.1.4

[PATCH 13/16] dirsize: python3: fix TypeError: unorderable types

[Ed Bartosh]

Python 3 ignores the __cmp__() method and doesn't have cmp() builtin
function. This caused sorted() call to raise
    TypeError: unorderable types: Record() < Record()

Removing __cmp__ method and implementing __lt__ should solve the
problem as __lt__ is the only method needed for sort[ed] to work.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/tiny/dirsize.py | 14 +++++---------
 1 file changed, 5 insertions(+), 9 deletions(-)

diff --git a/scripts/tiny/dirsize.py b/scripts/tiny/dirsize.py
index 5329b86..0b4fbd1 100755
--- a/scripts/tiny/dirsize.py
+++ b/scripts/tiny/dirsize.py
@@ -52,20 +52,16 @@ class Record:
         self.size = 0
         self.records = []
 
-    def __cmp__(this, that):
+    def __lt__(this, that):
         if that is None:
-            return 1
+            return False
         if not isinstance(that, Record):
             raise TypeError
         if len(this.records) > 0 and len(that.records) == 0:
-            return -1
-        if len(this.records) == 0 and len(that.records) > 0:
-            return 1
-        if this.size < that.size:
-            return -1
+            return False
         if this.size > that.size:
-            return 1
-        return 0
+            return False
+        return True
 
     def show(self, minsize):
         total = 0
-- 
2.1.4

[PATCH 14/16] scripts: python3: decode subprocess output

[Ed Bartosh]

stdeout and stderr content returned by subprocess API has different
types in Python 3(bytes) and Python 2(string). Decoding it to
'utf-8' makes it unicode on both pythons.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/cleanup-workdir   |  2 +-
 scripts/combo-layer       |  2 +-
 scripts/lib/bsp/kernel.py | 10 +++++-----
 scripts/oe-selftest       |  2 +-
 4 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/scripts/cleanup-workdir b/scripts/cleanup-workdir
index fee464c..0b2cf99 100755
--- a/scripts/cleanup-workdir
+++ b/scripts/cleanup-workdir
@@ -45,7 +45,7 @@ def run_command(cmd):
     if pipe.returncode != 0:
         print("Execute command '%s' failed." % cmd)
         sys.exit(1)
-    return output
+    return output.decode('utf-8')
 
 def get_cur_arch_dirs(workdir, arch_dirs):
     pattern = workdir + '/(.*?)/'
diff --git a/scripts/combo-layer b/scripts/combo-layer
index 0954bb6..7f0fa48 100755
--- a/scripts/combo-layer
+++ b/scripts/combo-layer
@@ -198,7 +198,7 @@ def runcmd(cmd,destdir=None,printerr=True,out=None,env=None):
         raise e
 
     err.seek(0)
-    output = err.read()
+    output = err.read().decode('utf-8')
     logger.debug("output: %s" % output.replace(chr(0), '\\0'))
     return output
 
diff --git a/scripts/lib/bsp/kernel.py b/scripts/lib/bsp/kernel.py
index fe7133d..91cc79c 100644
--- a/scripts/lib/bsp/kernel.py
+++ b/scripts/lib/bsp/kernel.py
@@ -49,7 +49,7 @@ def find_bblayers():
 
     bitbake_env_cmd = "bitbake -e"
     bitbake_env_lines = subprocess.Popen(bitbake_env_cmd, shell=True,
-                                         stdout=subprocess.PIPE).stdout.read()
+                                         stdout=subprocess.PIPE).stdout.read().decode('utf-8')
 
     if not bitbake_env_lines:
         print("Couldn't get '%s' output, exiting." % bitbake_env_cmd)
@@ -734,7 +734,7 @@ def yocto_kernel_available_features_list(scripts_path, machine):
     feature_url = find_feature_url(giturl)
 
     feature_cmd = "wget -q -O - " + feature_url
-    tmp = subprocess.Popen(feature_cmd, shell=True, stdout=subprocess.PIPE).stdout.read()
+    tmp = subprocess.Popen(feature_cmd, shell=True, stdout=subprocess.PIPE).stdout.read().decode('utf-8')
 
     print("The current set of kernel features available to %s is:\n" % machine)
 
@@ -785,7 +785,7 @@ def get_feature_desc(git_url, feature):
     """
     feature_desc_url = find_feature_desc_url(git_url, feature)
     feature_desc_cmd = "wget -q -O - " + feature_desc_url
-    tmp = subprocess.Popen(feature_desc_cmd, shell=True, stdout=subprocess.PIPE).stdout.read()
+    tmp = subprocess.Popen(feature_desc_cmd, shell=True, stdout=subprocess.PIPE).stdout.read().decode('utf-8')
 
     return find_feature_desc(tmp.split("\n"))
 
@@ -1001,7 +1001,7 @@ def base_branches(context):
     print("Getting branches from remote repo %s..." % giturl)
 
     gitcmd = "git ls-remote %s *heads* 2>&1" % (giturl)
-    tmp = subprocess.Popen(gitcmd, shell=True, stdout=subprocess.PIPE).stdout.read()
+    tmp = subprocess.Popen(gitcmd, shell=True, stdout=subprocess.PIPE).stdout.read().decode('utf-8')
 
     branches = []
 
@@ -1031,7 +1031,7 @@ def all_branches(context):
     print("Getting branches from remote repo %s..." % giturl)
 
     gitcmd = "git ls-remote %s *heads* 2>&1" % (giturl)
-    tmp = subprocess.Popen(gitcmd, shell=True, stdout=subprocess.PIPE).stdout.read()
+    tmp = subprocess.Popen(gitcmd, shell=True, stdout=subprocess.PIPE).stdout.read().decode('utf-8')
 
     branches = []
 
diff --git a/scripts/oe-selftest b/scripts/oe-selftest
index b1ecf7f..df76f94 100755
--- a/scripts/oe-selftest
+++ b/scripts/oe-selftest
@@ -371,7 +371,7 @@ def coverage_setup(coverage_source, coverage_include, coverage_omit):
     import subprocess
     builddir = os.environ.get("BUILDDIR")
     pokydir = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))
-    curcommit= subprocess.check_output(["git", "--git-dir", os.path.join(pokydir, ".git"), "rev-parse", "HEAD"])
+    curcommit= subprocess.check_output(["git", "--git-dir", os.path.join(pokydir, ".git"), "rev-parse", "HEAD"]).decode('utf-8')
     coveragerc = "%s/.coveragerc" % builddir
     data_file = "%s/.coverage." % builddir
     data_file += datetime.datetime.now().strftime('%Y%m%dT%H%M%S')
-- 
2.1.4

[PATCH 15/16] combo-layer: python3: use tempfile.TemporaryFile

[Ed Bartosh]

Used tempfile.TemporaryFile() API instead of deprecated
os.tmpfile().

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/combo-layer | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/combo-layer b/scripts/combo-layer
index 7f0fa48..eaa7f5a 100755
--- a/scripts/combo-layer
+++ b/scripts/combo-layer
@@ -185,10 +185,10 @@ def runcmd(cmd,destdir=None,printerr=True,out=None,env=None):
     """
     logger.debug("run cmd '%s' in %s" % (cmd, os.getcwd() if destdir is None else destdir))
     if not out:
-        out = os.tmpfile()
+        out = tempfile.TemporaryFile()
         err = out
     else:
-        err = os.tmpfile()
+        err = tempfile.TemporaryFile()
     try:
         subprocess.check_call(cmd, stdout=out, stderr=err, cwd=destdir, shell=isinstance(cmd, str), env=env or os.environ)
     except subprocess.CalledProcessError as e:
-- 
2.1.4

[PATCH 16/16] scripts: python3: change python to python3 in shebang

[Ed Bartosh]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/bitbake-whatchanged                 | 2 +-
 scripts/buildhistory-collect-srcrevs        | 2 +-
 scripts/cleanup-workdir                     | 2 +-
 scripts/combo-layer                         | 2 +-
 scripts/contrib/bbvars.py                   | 2 +-
 scripts/contrib/devtool-stress.py           | 2 +-
 scripts/contrib/list-packageconfig-flags.py | 2 +-
 scripts/contrib/verify-homepage.py          | 2 +-
 scripts/cp-noerror                          | 2 +-
 scripts/gen-lockedsig-cache                 | 2 +-
 scripts/lib/devtool/__init__.py             | 2 +-
 scripts/oe-publish-sdk                      | 2 +-
 scripts/oe-trim-schemas                     | 2 +-
 scripts/oepydevshell-internal.py            | 2 +-
 scripts/opkg-query-helper.py                | 2 +-
 scripts/pythondeps                          | 2 +-
 scripts/relocate_sdk.py                     | 2 +-
 scripts/send-error-report                   | 2 +-
 scripts/swabber-strace-attach               | 2 +-
 scripts/sysroot-relativelinks.py            | 2 +-
 scripts/test-remote-image                   | 2 +-
 scripts/tiny/dirsize.py                     | 2 +-
 scripts/tiny/ksize.py                       | 2 +-
 scripts/yocto-bsp                           | 2 +-
 scripts/yocto-kernel                        | 2 +-
 scripts/yocto-layer                         | 2 +-
 26 files changed, 26 insertions(+), 26 deletions(-)

diff --git a/scripts/bitbake-whatchanged b/scripts/bitbake-whatchanged
index 55b61d0..0207777 100755
--- a/scripts/bitbake-whatchanged
+++ b/scripts/bitbake-whatchanged
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # ex:ts=4:sw=4:sts=4:et
 # -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
 
diff --git a/scripts/buildhistory-collect-srcrevs b/scripts/buildhistory-collect-srcrevs
index 79d2657..8a03580 100755
--- a/scripts/buildhistory-collect-srcrevs
+++ b/scripts/buildhistory-collect-srcrevs
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 #
 # Collects the recorded SRCREV values from buildhistory and reports on them
 #
diff --git a/scripts/cleanup-workdir b/scripts/cleanup-workdir
index 0b2cf99..86eae37 100755
--- a/scripts/cleanup-workdir
+++ b/scripts/cleanup-workdir
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 # Copyright (c) 2012 Wind River Systems, Inc.
 #
diff --git a/scripts/combo-layer b/scripts/combo-layer
index eaa7f5a..8f57ba5 100755
--- a/scripts/combo-layer
+++ b/scripts/combo-layer
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # ex:ts=4:sw=4:sts=4:et
 # -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
 #
diff --git a/scripts/contrib/bbvars.py b/scripts/contrib/bbvars.py
index b865dd1..d8d0594 100755
--- a/scripts/contrib/bbvars.py
+++ b/scripts/contrib/bbvars.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
diff --git a/scripts/contrib/devtool-stress.py b/scripts/contrib/devtool-stress.py
index 8cf92ca..2723491 100755
--- a/scripts/contrib/devtool-stress.py
+++ b/scripts/contrib/devtool-stress.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 # devtool stress tester
 #
diff --git a/scripts/contrib/list-packageconfig-flags.py b/scripts/contrib/list-packageconfig-flags.py
index 22d0c49..b8327e4 100755
--- a/scripts/contrib/list-packageconfig-flags.py
+++ b/scripts/contrib/list-packageconfig-flags.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
diff --git a/scripts/contrib/verify-homepage.py b/scripts/contrib/verify-homepage.py
index 18bb15b..61a047c4 100755
--- a/scripts/contrib/verify-homepage.py
+++ b/scripts/contrib/verify-homepage.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 # This script can be used to verify HOMEPAGE values for all recipes in
 # the current configuration.
diff --git a/scripts/cp-noerror b/scripts/cp-noerror
index d8be677..35eb211 100755
--- a/scripts/cp-noerror
+++ b/scripts/cp-noerror
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 #
 # Allow copying of $1 to $2 but if files in $1 disappear during the copy operation,
 # don't error.
diff --git a/scripts/gen-lockedsig-cache b/scripts/gen-lockedsig-cache
index 0986a21..26e9b63 100755
--- a/scripts/gen-lockedsig-cache
+++ b/scripts/gen-lockedsig-cache
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 import os
 import sys
diff --git a/scripts/lib/devtool/__init__.py b/scripts/lib/devtool/__init__.py
index 7005363..77b1fd9 100644
--- a/scripts/lib/devtool/__init__.py
+++ b/scripts/lib/devtool/__init__.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 # Development tool - utility functions for plugins
 #
diff --git a/scripts/oe-publish-sdk b/scripts/oe-publish-sdk
index 55872f2..4fe8974 100755
--- a/scripts/oe-publish-sdk
+++ b/scripts/oe-publish-sdk
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 # OpenEmbedded SDK publishing tool
 
diff --git a/scripts/oe-trim-schemas b/scripts/oe-trim-schemas
index 29fb3a1..66a1b8d 100755
--- a/scripts/oe-trim-schemas
+++ b/scripts/oe-trim-schemas
@@ -1,4 +1,4 @@
-#! /usr/bin/env python
+#! /usr/bin/env python3
 
 import sys
 try:
diff --git a/scripts/oepydevshell-internal.py b/scripts/oepydevshell-internal.py
index 2566606..7761f66 100755
--- a/scripts/oepydevshell-internal.py
+++ b/scripts/oepydevshell-internal.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 import os
 import sys
diff --git a/scripts/opkg-query-helper.py b/scripts/opkg-query-helper.py
index 2fb1a78..ce89491 100755
--- a/scripts/opkg-query-helper.py
+++ b/scripts/opkg-query-helper.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 # OpenEmbedded opkg query helper utility
 #
diff --git a/scripts/pythondeps b/scripts/pythondeps
index f1e6452..590b976 100755
--- a/scripts/pythondeps
+++ b/scripts/pythondeps
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 #
 # Determine dependencies of python scripts or available python modules in a search path.
 #
diff --git a/scripts/relocate_sdk.py b/scripts/relocate_sdk.py
index 99fca86..e47b4d9 100755
--- a/scripts/relocate_sdk.py
+++ b/scripts/relocate_sdk.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 #
 # Copyright (c) 2012 Intel Corporation
 #
diff --git a/scripts/send-error-report b/scripts/send-error-report
index 122ce32..ff23552 100755
--- a/scripts/send-error-report
+++ b/scripts/send-error-report
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 # Sends an error report (if the report-error class was enabled) to a
 # remote server.
diff --git a/scripts/swabber-strace-attach b/scripts/swabber-strace-attach
index f258987..e8f3258 100755
--- a/scripts/swabber-strace-attach
+++ b/scripts/swabber-strace-attach
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 import os
 import sys
 import subprocess
diff --git a/scripts/sysroot-relativelinks.py b/scripts/sysroot-relativelinks.py
index ac26367..e44eba2 100755
--- a/scripts/sysroot-relativelinks.py
+++ b/scripts/sysroot-relativelinks.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 import sys
 import os
 
diff --git a/scripts/test-remote-image b/scripts/test-remote-image
index 698948b..27b1cae 100755
--- a/scripts/test-remote-image
+++ b/scripts/test-remote-image
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 # Copyright (c) 2014 Intel Corporation
 #
diff --git a/scripts/tiny/dirsize.py b/scripts/tiny/dirsize.py
index 0b4fbd1..ddccc5a 100755
--- a/scripts/tiny/dirsize.py
+++ b/scripts/tiny/dirsize.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 #
 # Copyright (c) 2011, Intel Corporation.
 # All rights reserved.
diff --git a/scripts/tiny/ksize.py b/scripts/tiny/ksize.py
index 587c930..b9d2b19 100755
--- a/scripts/tiny/ksize.py
+++ b/scripts/tiny/ksize.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 #
 # Copyright (c) 2011, Intel Corporation.
 # All rights reserved.
diff --git a/scripts/yocto-bsp b/scripts/yocto-bsp
index ce30e55..6a422c3 100755
--- a/scripts/yocto-bsp
+++ b/scripts/yocto-bsp
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # ex:ts=4:sw=4:sts=4:et
 # -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
 #
diff --git a/scripts/yocto-kernel b/scripts/yocto-kernel
index daaad07..fd97cb7 100755
--- a/scripts/yocto-kernel
+++ b/scripts/yocto-kernel
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # ex:ts=4:sw=4:sts=4:et
 # -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
 #
diff --git a/scripts/yocto-layer b/scripts/yocto-layer
index 356972e..a5267f2 100755
--- a/scripts/yocto-layer
+++ b/scripts/yocto-layer
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # ex:ts=4:sw=4:sts=4:et
 # -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
 #
-- 
2.1.4

Re: [PATCH 01/16] scripts: python3: convert iterables to lists

[Leonardo Sandoval]

Ed,

what is the reason for this change?


On 06/02/2016 05:12 AM, Ed Bartosh wrote:
> Converted return value of items() keys() and values() to
> lists when dictionary is modified in the loop and when
> the result is added to the list.
>
> Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
> ---
>   scripts/bitbake-whatchanged                      | 6 +++---
>   scripts/combo-layer                              | 2 +-
>   scripts/lib/devtool/build.py                     | 2 +-
>   scripts/lib/devtool/standard.py                  | 4 ++--
>   scripts/lib/recipetool/create_buildsys.py        | 2 +-
>   scripts/lib/recipetool/create_buildsys_python.py | 6 +++---
>   scripts/oe-pkgdata-util                          | 2 +-
>   7 files changed, 12 insertions(+), 12 deletions(-)
>
> diff --git a/scripts/bitbake-whatchanged b/scripts/bitbake-whatchanged
> index a20adb2..b05aead 100755
> --- a/scripts/bitbake-whatchanged
> +++ b/scripts/bitbake-whatchanged
> @@ -120,7 +120,7 @@ def print_added(d_new = None, d_old = None):
>       Print the newly added tasks
>       """
>       added = {}
> -    for k in d_new.keys():
> +    for k in list(d_new.keys()):
>           if k not in d_old:
>               # Add the new one to added dict, and remove it from
>               # d_new, so the remaining ones are the changed ones
> @@ -155,7 +155,7 @@ def print_vrchanged(d_new = None, d_old = None, vr = None):
>       """
>       pvchanged = {}
>       counter = 0
> -    for k in d_new.keys():
> +    for k in list(d_new.keys()):
>           if d_new.get(k).get(vr) != d_old.get(k).get(vr):
>               counter += 1
>               pn, task = split_pntask(k)
> @@ -279,7 +279,7 @@ Note:
>   
>           # Remove the same one from both stamps.
>           cnt_unchanged = 0
> -        for k in new_dict.keys():
> +        for k in list(new_dict.keys()):
>               if k in old_dict:
>                   cnt_unchanged += 1
>                   del(new_dict[k])
> diff --git a/scripts/combo-layer b/scripts/combo-layer
> index 234d9e4..7c41f92 100755
> --- a/scripts/combo-layer
> +++ b/scripts/combo-layer
> @@ -1186,7 +1186,7 @@ def update_with_history(conf, components, revisions, repos):
>           msg = conf_commit_msg(conf, components)
>           new_tree = runcmd("git write-tree", **wargs).strip()
>           new_rev = runcmd("git commit-tree".split() +
> -                         add_p([head] + additional_heads.keys()) +
> +                         add_p([head] + list(additional_heads.keys())) +
>                            ["-m", msg, new_tree],
>                            **wargs).strip()
>           # And done! This is the first time we change the HEAD in the actual work tree.
> diff --git a/scripts/lib/devtool/build.py b/scripts/lib/devtool/build.py
> index 48f6fe1..6be549d 100644
> --- a/scripts/lib/devtool/build.py
> +++ b/scripts/lib/devtool/build.py
> @@ -27,7 +27,7 @@ logger = logging.getLogger('devtool')
>   
>   
>   def _set_file_values(fn, values):
> -    remaining = values.keys()
> +    remaining = list(values.keys())
>   
>       def varfunc(varname, origvalue, op, newlines):
>           newvalue = values.get(varname, origvalue)
> diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
> index 08153c6..a2516d6 100644
> --- a/scripts/lib/devtool/standard.py
> +++ b/scripts/lib/devtool/standard.py
> @@ -998,7 +998,7 @@ def _export_local_files(srctree, rd, destdir):
>           bb.process.run(['git', 'checkout', tree, '--', '.'], cwd=srctree,
>                           env=dict(os.environ, GIT_WORK_TREE=destdir,
>                                    GIT_INDEX_FILE=tmp_index))
> -        new_set = _git_ls_tree(srctree, tree, True).keys()
> +        new_set = list(_git_ls_tree(srctree, tree, True).keys())
>       elif os.path.isdir(local_files_dir):
>           # If not tracked by Git, just copy from working copy
>           new_set = _ls_tree(os.path.join(srctree, 'oe-local-files'))
> @@ -1309,7 +1309,7 @@ def reset(args, config, basepath, workspace):
>           raise DevtoolError("Recipe must be specified, or specify -a/--all to "
>                              "reset all recipes")
>       if args.all:
> -        recipes = workspace.keys()
> +        recipes = list(workspace.keys())
>       else:
>           recipes = [args.recipename]
>   
> diff --git a/scripts/lib/recipetool/create_buildsys.py b/scripts/lib/recipetool/create_buildsys.py
> index de3d9ae..78ae4bc 100644
> --- a/scripts/lib/recipetool/create_buildsys.py
> +++ b/scripts/lib/recipetool/create_buildsys.py
> @@ -682,7 +682,7 @@ class AutotoolsRecipeHandler(RecipeHandler):
>                   process_macro(in_keyword, partial)
>   
>           if extravalues:
> -            for k,v in extravalues.items():
> +            for k,v in list(extravalues.items()):
>                   if v:
>                       if v.startswith('$') or v.startswith('@') or v.startswith('%'):
>                           del extravalues[k]
> diff --git a/scripts/lib/recipetool/create_buildsys_python.py b/scripts/lib/recipetool/create_buildsys_python.py
> index 55cce0e..aff13cf 100644
> --- a/scripts/lib/recipetool/create_buildsys_python.py
> +++ b/scripts/lib/recipetool/create_buildsys_python.py
> @@ -361,7 +361,7 @@ class PythonRecipeHandler(RecipeHandler):
>   
>           # Naive mapping of setup() arguments to PKG-INFO field names
>           for d in [info, non_literals]:
> -            for key, value in d.items():
> +            for key, value in list(d.items()):
>                   new_key = _map(key)
>                   if new_key != key:
>                       del d[key]
> @@ -443,7 +443,7 @@ class PythonRecipeHandler(RecipeHandler):
>                   elif new_value != value:
>                       info[variable] = new_value
>               elif hasattr(value, 'items'):
> -                for dkey, dvalue in value.items():
> +                for dkey, dvalue in list(value.items()):
>                       new_list = []
>                       for pos, a_value in enumerate(dvalue):
>                           new_value = replace_value(search, replace, a_value)
> @@ -608,7 +608,7 @@ def gather_setup_info(fileobj):
>       visitor.visit(parsed)
>   
>       non_literals, extensions = {}, []
> -    for key, value in visitor.keywords.items():
> +    for key, value in list(visitor.keywords.items()):
>           if key == 'ext_modules':
>               if isinstance(value, list):
>                   for ext in value:
> diff --git a/scripts/oe-pkgdata-util b/scripts/oe-pkgdata-util
> index b39d9b5..b16ecc9 100755
> --- a/scripts/oe-pkgdata-util
> +++ b/scripts/oe-pkgdata-util
> @@ -240,7 +240,7 @@ def lookup_pkg(args):
>           sys.exit(1)
>   
>       if args.reverse:
> -        items = mappings.values()
> +        items = list(mappings.values())
>       else:
>           items = []
>           for pkg in pkgs:

Re: [PATCH 01/16] scripts: python3: convert iterables to lists

[Ed Bartosh]

On Thu, Jun 02, 2016 at 08:55:21AM -0500, Leonardo Sandoval wrote:
> Ed,
> 
> what is the reason for this change?
> 
The reason is that in python 3 dict.keys() returns iterator unlike in
pyton 2 where it returns list. If dictionary is changed inside the loop
(which is the case here) "RuntimeError: dictionary changed size during iteration" will be raised.

Another case is when result of dict.keys() is added to the list. This
kind of operations would raise TypeError: unsupported operand type(s)
for +: 'dict_keys' and 'list'


> 
> On 06/02/2016 05:12 AM, Ed Bartosh wrote:
> >Converted return value of items() keys() and values() to
> >lists when dictionary is modified in the loop and when
> >the result is added to the list.
> >
> >Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
> >---
> >  scripts/bitbake-whatchanged                      | 6 +++---
> >  scripts/combo-layer                              | 2 +-
> >  scripts/lib/devtool/build.py                     | 2 +-
> >  scripts/lib/devtool/standard.py                  | 4 ++--
> >  scripts/lib/recipetool/create_buildsys.py        | 2 +-
> >  scripts/lib/recipetool/create_buildsys_python.py | 6 +++---
> >  scripts/oe-pkgdata-util                          | 2 +-
> >  7 files changed, 12 insertions(+), 12 deletions(-)
> >
> >diff --git a/scripts/bitbake-whatchanged b/scripts/bitbake-whatchanged
> >index a20adb2..b05aead 100755
> >--- a/scripts/bitbake-whatchanged
> >+++ b/scripts/bitbake-whatchanged
> >@@ -120,7 +120,7 @@ def print_added(d_new = None, d_old = None):
> >      Print the newly added tasks
> >      """
> >      added = {}
> >-    for k in d_new.keys():
> >+    for k in list(d_new.keys()):
> >          if k not in d_old:
> >              # Add the new one to added dict, and remove it from
> >              # d_new, so the remaining ones are the changed ones
> >@@ -155,7 +155,7 @@ def print_vrchanged(d_new = None, d_old = None, vr = None):
> >      """
> >      pvchanged = {}
> >      counter = 0
> >-    for k in d_new.keys():
> >+    for k in list(d_new.keys()):
> >          if d_new.get(k).get(vr) != d_old.get(k).get(vr):
> >              counter += 1
> >              pn, task = split_pntask(k)
> >@@ -279,7 +279,7 @@ Note:
> >          # Remove the same one from both stamps.
> >          cnt_unchanged = 0
> >-        for k in new_dict.keys():
> >+        for k in list(new_dict.keys()):
> >              if k in old_dict:
> >                  cnt_unchanged += 1
> >                  del(new_dict[k])
> >diff --git a/scripts/combo-layer b/scripts/combo-layer
> >index 234d9e4..7c41f92 100755
> >--- a/scripts/combo-layer
> >+++ b/scripts/combo-layer
> >@@ -1186,7 +1186,7 @@ def update_with_history(conf, components, revisions, repos):
> >          msg = conf_commit_msg(conf, components)
> >          new_tree = runcmd("git write-tree", **wargs).strip()
> >          new_rev = runcmd("git commit-tree".split() +
> >-                         add_p([head] + additional_heads.keys()) +
> >+                         add_p([head] + list(additional_heads.keys())) +
> >                           ["-m", msg, new_tree],
> >                           **wargs).strip()
> >          # And done! This is the first time we change the HEAD in the actual work tree.
> >diff --git a/scripts/lib/devtool/build.py b/scripts/lib/devtool/build.py
> >index 48f6fe1..6be549d 100644
> >--- a/scripts/lib/devtool/build.py
> >+++ b/scripts/lib/devtool/build.py
> >@@ -27,7 +27,7 @@ logger = logging.getLogger('devtool')
> >  def _set_file_values(fn, values):
> >-    remaining = values.keys()
> >+    remaining = list(values.keys())
> >      def varfunc(varname, origvalue, op, newlines):
> >          newvalue = values.get(varname, origvalue)
> >diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
> >index 08153c6..a2516d6 100644
> >--- a/scripts/lib/devtool/standard.py
> >+++ b/scripts/lib/devtool/standard.py
> >@@ -998,7 +998,7 @@ def _export_local_files(srctree, rd, destdir):
> >          bb.process.run(['git', 'checkout', tree, '--', '.'], cwd=srctree,
> >                          env=dict(os.environ, GIT_WORK_TREE=destdir,
> >                                   GIT_INDEX_FILE=tmp_index))
> >-        new_set = _git_ls_tree(srctree, tree, True).keys()
> >+        new_set = list(_git_ls_tree(srctree, tree, True).keys())
> >      elif os.path.isdir(local_files_dir):
> >          # If not tracked by Git, just copy from working copy
> >          new_set = _ls_tree(os.path.join(srctree, 'oe-local-files'))
> >@@ -1309,7 +1309,7 @@ def reset(args, config, basepath, workspace):
> >          raise DevtoolError("Recipe must be specified, or specify -a/--all to "
> >                             "reset all recipes")
> >      if args.all:
> >-        recipes = workspace.keys()
> >+        recipes = list(workspace.keys())
> >      else:
> >          recipes = [args.recipename]
> >diff --git a/scripts/lib/recipetool/create_buildsys.py b/scripts/lib/recipetool/create_buildsys.py
> >index de3d9ae..78ae4bc 100644
> >--- a/scripts/lib/recipetool/create_buildsys.py
> >+++ b/scripts/lib/recipetool/create_buildsys.py
> >@@ -682,7 +682,7 @@ class AutotoolsRecipeHandler(RecipeHandler):
> >                  process_macro(in_keyword, partial)
> >          if extravalues:
> >-            for k,v in extravalues.items():
> >+            for k,v in list(extravalues.items()):
> >                  if v:
> >                      if v.startswith('$') or v.startswith('@') or v.startswith('%'):
> >                          del extravalues[k]
> >diff --git a/scripts/lib/recipetool/create_buildsys_python.py b/scripts/lib/recipetool/create_buildsys_python.py
> >index 55cce0e..aff13cf 100644
> >--- a/scripts/lib/recipetool/create_buildsys_python.py
> >+++ b/scripts/lib/recipetool/create_buildsys_python.py
> >@@ -361,7 +361,7 @@ class PythonRecipeHandler(RecipeHandler):
> >          # Naive mapping of setup() arguments to PKG-INFO field names
> >          for d in [info, non_literals]:
> >-            for key, value in d.items():
> >+            for key, value in list(d.items()):
> >                  new_key = _map(key)
> >                  if new_key != key:
> >                      del d[key]
> >@@ -443,7 +443,7 @@ class PythonRecipeHandler(RecipeHandler):
> >                  elif new_value != value:
> >                      info[variable] = new_value
> >              elif hasattr(value, 'items'):
> >-                for dkey, dvalue in value.items():
> >+                for dkey, dvalue in list(value.items()):
> >                      new_list = []
> >                      for pos, a_value in enumerate(dvalue):
> >                          new_value = replace_value(search, replace, a_value)
> >@@ -608,7 +608,7 @@ def gather_setup_info(fileobj):
> >      visitor.visit(parsed)
> >      non_literals, extensions = {}, []
> >-    for key, value in visitor.keywords.items():
> >+    for key, value in list(visitor.keywords.items()):
> >          if key == 'ext_modules':
> >              if isinstance(value, list):
> >                  for ext in value:
> >diff --git a/scripts/oe-pkgdata-util b/scripts/oe-pkgdata-util
> >index b39d9b5..b16ecc9 100755
> >--- a/scripts/oe-pkgdata-util
> >+++ b/scripts/oe-pkgdata-util
> >@@ -240,7 +240,7 @@ def lookup_pkg(args):
> >          sys.exit(1)
> >      if args.reverse:
> >-        items = mappings.values()
> >+        items = list(mappings.values())
> >      else:
> >          items = []
> >          for pkg in pkgs:
> 

-- 
--
Regards,
Ed

Re: [PATCH 12/16] engine: python3: replace iteritems() -> items()

[Ibarra Lopez, Humberto]

This patch introduced a syntax error. Sent a patch to fix it http://lists.openembedded.org/pipermail/openembedded-core/2016-June/122522.html

Humberto

> -----Original Message-----
> From: openembedded-core-bounces@lists.openembedded.org
> [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf
> Of Ed Bartosh
> Sent: Thursday, June 2, 2016 5:13 AM
> To: openembedded-core@lists.openembedded.org
> Subject: [OE-core] [PATCH 12/16] engine: python3: replace iteritems() ->
> items()
> 
> Used items() as iteritems() doesn't exist in python 3.
> 
> Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
> ---
>  scripts/lib/bsp/engine.py | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/scripts/lib/bsp/engine.py b/scripts/lib/bsp/engine.py index
> 2b87d33..3ed90fa 100644
> --- a/scripts/lib/bsp/engine.py
> +++ b/scripts/lib/bsp/engine.py
> @@ -1385,7 +1385,7 @@ def gen_supplied_property_vals(properties,
> program_lines):
>      Generate user-specified entries for input values instead of
>      generating input prompts.
>      """
> -    for name, val in properties.iteritems():
> +    for name, val in properties.items():
>          program_line = name + " = \"" + val + "\""
>          program_lines.append(program_line)
> 
> @@ -1621,7 +1621,7 @@ def print_dict(items, indent = 0):
>      """
>      Print the values in a possibly nested dictionary.
>      """
> -    for key, val in items.iteritems():
> +    for key, val in items.items():
>          print("    "*indent + "\"%s\" :" % key)
>          if type(val) == dict:
>              print("{")
> @@ -1654,7 +1654,7 @@ def get_properties(input_lines):
>          props = line.props
>          item = {}
>          name = props["name"]
> -        for key, val in props.items():
> +        for key, val props.items():
>              if not key == "name":
>                  item[key] = val
>          properties[name] = item
> --
> 2.1.4
> 
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core