[PATCH 00/11][auh]: Add support of new upstream mechanism and improvments

[PATCH 00/11][auh]: Add support of new upstream mechanism and improvments

[Aníbal Limón]

The changes can be found at,

http://git.yoctoproject.org/cgit/cgit.cgi/auto-upgrade-helper/log/?h=alimon/devel

Aníbal Limón (11):
  upgradehelper.py: Fix status_msg in pkg_upgrade_handler
  upgradehelper.py: Make usage of new upstream detection mechanism
  upgradehelper.py: Use settings.from as email for commits.
  upgradehelper.py: Improve usage when try to upgrade one recipe.
  upgrade_helper.py: Make notice of License change into email.
  upgradehelper.py: Add specific information about machines build
    succeed.
  upgradehelper.py: Move recipe work into upgradehelper work dir.
  upgradehelper.py: Fix interactive mode when recipe upgrade finish.
  upgradehelper.py: Change policy for send emails and fix error passing
  upgradehelper.py: Change print to logger info.
  README: Edit Maintainece section to add current maintainer.

 README           |   3 +-
 recipe.py        |  16 ++--
 upgradehelper.py | 227 ++++++++++++++++++++++++++++++-------------------------
 3 files changed, 139 insertions(+), 107 deletions(-)

-- 
1.8.4.5

[PATCH 01/11][auh] upgradehelper.py: Fix status_msg in pkg_upgrade_handler

[Aníbal Limón]

When is called pkg_upgrade_handler for generate email and save
build history at end sometimes status_msg is undefined.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 upgradehelper.py | 16 +++++++++-------
 1 file changed, 9 insertions(+), 7 deletions(-)

diff --git a/upgradehelper.py b/upgradehelper.py
index ddd0835..366efb3 100755
--- a/upgradehelper.py
+++ b/upgradehelper.py
@@ -173,8 +173,6 @@ class Updater(object):
         self.email_handler = Email(settings)
         self.statistics = Statistics()
 
-
-
     def _get_env(self):
         stdout = self.bb.env(self.pn)
 
@@ -217,6 +215,11 @@ class Updater(object):
         self.recipe = recipe(self.env, self.new_ver, self.interactive, self.workdir,
                              self.recipe_dir, self.bb, self.git)
 
+    def _get_status_msg(self, err):
+        if err:
+            return str(err)
+        else:
+            return "Succeeded"
 
     def _create_workdir(self):
         self.workdir = self.uh_dir + "/" + self.pn
@@ -344,7 +347,6 @@ class Updater(object):
     def pkg_upgrade_handler(self, err):
         if err and self.patch_file:
             answer = "N"
-            status_msg = str(err)
             if self.interactive:
                 I(" %s: Do you want to keep the changes? (y/N)" % self.pn)
                 answer = sys.stdin.readline().strip().upper()
@@ -354,8 +356,6 @@ class Updater(object):
                 self.git.reset_hard(1)
                 self.git.clean_untracked()
                 return
-        elif not err:
-            status_msg = "Succeeded"
 
         status = type(err).__name__
 
@@ -384,7 +384,8 @@ class Updater(object):
             else:
                 subject += " FAILED"
 
-            msg_body = self.mail_header % (self.pn, self.new_ver, status_msg)
+            msg_body = self.mail_header % (self.pn, self.new_ver,
+                    self._get_status_msg(err))
 
             if err is None:
                 msg_body += self.next_steps_info % os.path.basename(self.patch_file)
@@ -653,7 +654,8 @@ class UniverseUpdater(Updater):
     # overriding the base method
     def pkg_upgrade_handler(self, err):
         super(UniverseUpdater, self).pkg_upgrade_handler(self)
-        self.update_history(self.pn, self.new_ver, self.maintainer, status_msg)
+        self.update_history(self.pn, self.new_ver, self.maintainer,
+                self._get_status_msg(err))
 
     def run(self):
         self.update_master()
-- 
1.8.4.5

[PATCH 02/11][auh] upgradehelper.py: Make usage of new upstream detection mechanism

[Aníbal Limón]

Change logic to use new values in checkpkg file generated with new
 oe.recipeutils.get_recipe_upstream_version and distrodata class.

Remove CSV parsing by hand use csv module and also add debugging when
discard recipes.

[YOCTO #7488]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 upgradehelper.py | 96 ++++++++++++++++++++++++++++----------------------------
 1 file changed, 48 insertions(+), 48 deletions(-)

diff --git a/upgradehelper.py b/upgradehelper.py
index 366efb3..6af79a0 100755
--- a/upgradehelper.py
+++ b/upgradehelper.py
@@ -189,19 +189,6 @@ class Updater(object):
         self.env = bb_env
         self.recipe_dir = os.path.dirname(self.env['FILE'])
 
-    def _parse_checkpkg_line(self, line):
-        m = re.match("^([^ \t]*)[ \t]+([^ \t]*)[ \t]+([^ \t]*)[ \t]+.*", line)
-        if m:
-            res = (m.group(1), m.group(2), m.group(3))
-            m = re.search("<([^ \t]+@[^ \t]+)>", line)
-            if m:
-                maintainer = m.group(1)
-            else:
-                maintainer = None
-            return res + (maintainer,)
-
-        return (None, None, None, None)
-
     def _detect_recipe_type(self):
         if self.env['SRC_URI'].find("ftp://") != -1 or  \
                 self.env['SRC_URI'].find("http://") != -1 or \
@@ -309,7 +296,41 @@ class Updater(object):
                     C(" \"distrodata.bbclass\" not inherited. Consider adding "
                       "the following to your local.conf:\n\n"
                       "INHERIT =+ \"distrodata\"\n")
-                    exit(1)
+                else:
+                    C(line)
+
+            exit(1)
+
+    def _parse_checkpkg_file(self, file_path):
+        import csv
+
+        pkgs_list = []
+
+        with open(file_path, "r") as f:
+            reader = csv.reader(f, delimiter='\t')
+            for row in reader:
+                if reader.line_num == 1: # skip header line
+                    continue
+
+                pn = row[0]
+                cur_ver = row[1]
+                next_ver = row[2]
+                status = row[11]
+                maintainer = row[14]
+                no_upgrade_reason = row[15]
+
+                if status == 'UPDATE' and not no_upgrade_reason:
+                    pkgs_list.append((pn, next_ver, maintainer))
+                else:
+                    if no_upgrade_reason:
+                        D(" Skip package %s (status = %s, current version = %s," \
+                            " next version = %s, no upgrade reason = %s)" %
+                            (pn, status, cur_ver, next_ver, no_upgrade_reason))
+                    else:
+                        D(" Skip package %s (status = %s, current version = %s," \
+                            " next version = %s)" %
+                            (pn, status, cur_ver, next_ver))
+        return pkgs_list
 
     def _get_packages_to_upgrade(self, packages=None):
         if packages is None:
@@ -323,25 +344,7 @@ class Updater(object):
 
         self._check_upstream_versions(packages)
 
-        pkgs_list = []
-
-        with open(get_build_dir() + "/tmp/log/checkpkg.csv") as csv:
-            # Skip header line
-            next(csv)
-            for line in csv:
-                (pn, cur_ver, next_ver, maintainer) = self._parse_checkpkg_line(line)
-
-                if (pn, cur_ver, next_ver, maintainer) == (None, None, None, None):
-                    continue
-
-                if cur_ver != next_ver and next_ver != "N/A" and \
-                        next_ver != "INVALID":
-                    pkgs_list.append((pn, next_ver, maintainer))
-                else:
-                    W(" Skip package %s (current version = %s, next version = %s)" %
-                        (pn, cur_ver, next_ver))
-
-        return pkgs_list
+        return self._parse_checkpkg_file(get_build_dir() + "/tmp/log/checkpkg.csv")
 
     # this function will be called at the end of each recipe upgrade
     def pkg_upgrade_handler(self, err):
@@ -455,7 +458,6 @@ class Updater(object):
             W("No recipes attempted, not sending status mail!")
 
     def run(self, package_list=None):
-#[lp]        pkgs_to_upgrade = self._order_list(self._get_packages_to_upgrade(package_list))
         pkgs_to_upgrade = self._get_packages_to_upgrade(package_list)
 
         total_pkgs = len(pkgs_to_upgrade)
@@ -521,24 +523,27 @@ class UniverseUpdater(Updater):
     # checks if maintainer is in whitelist and that the recipe itself is not
     # blacklisted: python, gcc, etc. Also, check the history if the recipe
     # hasn't already been tried
-    def pkg_upgradable(self, pn, next_ver, maintainer):
+    def _pkg_upgradable(self, pn, next_ver, maintainer):
         if not maintainer:
-            D("Skipping upgrade of %s: no maintainer" % pn)
+            D(" Skipping upgrade of %s: no maintainer" % pn)
             return False
 
         if "blacklist" in settings:
             for p in settings["blacklist"].split():
                 if p == pn:
+                    D(" Skipping upgrade of %s: blacklist" % pn)
                     return False
 
         if "maintainers_whitelist" in settings:
             found = False
             for m in settings["maintainers_whitelist"].split():
-                if m == maintainer:
+                if maintainer.find(m) != -1:
                     found = True
                     break
 
-            if not found:
+            if found == False:
+                D(" Skipping upgrade of %s: maintainer \"%s\" not in whitelist" %
+                        (pn, maintainer))
                 return False
 
         if pn in self.history:
@@ -553,12 +558,14 @@ class UniverseUpdater(Updater):
                         self.history[pn][3] == str(Error())) and retry_delta > 7:
                     return True
 
+                D(" Skipping upgrade of %s: is in history and not 7 days passed" % pn)
                 return False
 
         # drop native/cross/cross-canadian recipes. We deal with native
         # when upgrading the main recipe but we keep away of cross* pkgs...
         # for now
         if pn.find("cross") != -1 or pn.find("native") != -1:
+            D(" Skipping upgrade of %s: is cross or native" % pn)
             return False
 
         return True
@@ -616,22 +623,15 @@ class UniverseUpdater(Updater):
               " check date are the same ...")
 
         pkgs_list = []
-
-        with open(last_checkpkg_file, "r") as csv:
-            for line in csv:
-                (pn, cur_ver, next_ver, maintainer) = self._parse_checkpkg_line(line)
-                if (pn, cur_ver, next_ver, maintainer) != (None, None, None, None) and \
-                        cur_ver != next_ver and next_ver != "N/A" and \
-                        next_ver != "INVALID":
-                    if self.pkg_upgradable(pn, next_ver, maintainer):
-                        pkgs_list.append((pn, next_ver, maintainer))
+        for pkg in self._parse_checkpkg_file(last_checkpkg_file):
+            if self._pkg_upgradable(pkg[0], pkg[1], pkg[2]):
+                pkgs_list.append(pkg)
 
         # Update last_checkpkg_run only after the version check has been completed
         with open(get_build_dir() + "/upgrade-helper/last_checkpkg_run", "w+") as last_check:
             last_check.write(current_date + "," + cur_master_commit + "," +
                              last_checkpkg_file)
 
-
         print("########### The list of recipes to be upgraded ############")
         for p, v, m in pkgs_list:
             print("%s,%s,%s" % (p, v, m))
-- 
1.8.4.5

[PATCH 03/11][auh] upgradehelper.py: Use settings.from as email for commits.

[Aníbal Limón]

[YOCTO #7489]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 upgradehelper.py | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/upgradehelper.py b/upgradehelper.py
index 6af79a0..4a07d7b 100755
--- a/upgradehelper.py
+++ b/upgradehelper.py
@@ -148,10 +148,8 @@ class Updater(object):
         self.bb = Bitbake(get_build_dir())
         self.buildhistory = BuildHistory(get_build_dir())
         self.git = None
-        if send_email:
-            self.author = "Upgrade Helper <uh@not.set>"
-        else:
-            self.author = None
+        self.author_email = settings.get('from', 'uh@not.set')
+        self.author = "Upgrade Helper <%s>" % self.author_email
         self.skip_compilation = skip_compilation
         self.interactive = not auto_mode
         self.send_email = send_email
-- 
1.8.4.5

[PATCH 04/11][auh] upgradehelper.py: Improve usage when try to upgrade one recipe.

[Aníbal Limón]

When try to upgrade only one recipe is needed to specify the version
also when enable send emails it need to have a maintainer to avoid
exception.

[YOCTO #7489]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 upgradehelper.py | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/upgradehelper.py b/upgradehelper.py
index 4a07d7b..74f9c0a 100755
--- a/upgradehelper.py
+++ b/upgradehelper.py
@@ -66,10 +66,15 @@ def parse_cmdline():
                                      formatter_class=argparse.RawTextHelpFormatter,
                                      epilog=help_text)
     parser.add_argument("recipe", nargs="+", help="recipe to be upgraded")
+
     parser.add_argument("-t", "--to_version",
                         help="version to upgrade the recipe to")
+    parser.add_argument("-m", "--maintainer",
+                        help="maintainer of the recipe")
+
     parser.add_argument("-a", "--auto-mode", action="store_true", default=False,
                         help="disable interactive mode")
+
     parser.add_argument("-d", "--debug-level", type=int, default=4, choices=range(1, 6),
                         help="set the debug level: CRITICAL=1, ERROR=2, WARNING=3, INFO=4, DEBUG=5")
     parser.add_argument("-e", "--send-emails", action="store_true", default=False,
@@ -686,7 +691,15 @@ if __name__ == "__main__":
         updater.run()
     elif len(args.recipe) >= 1:
         if len(args.recipe) == 1:
-            pkg_list = [(args.recipe[0], args.to_version, None)]
+            if not args.to_version:
+                E(" For upgrade only one recipe you must specify --to_version\n")
+                exit(1)
+
+            if not args.maintainer and args.send_emails:
+                E(" For upgrade only one recipe and send email you must specify --maintainer\n")
+                exit(1)
+
+            pkg_list = [(args.recipe[0], args.to_version, args.maintainer)]
         else:
             pkg_list = []
             for pkg in args.recipe:
-- 
1.8.4.5

[PATCH 05/11][auh] upgrade_helper.py: Make notice of License change into email.

[Aníbal Limón]

When license change it need to be specified into email to maintainers
for review it.

Add get_license_diff_file_name method to recipe class for get it
into email handler and write information about it.

[YOCTO 7186]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 recipe.py        |  7 +++++++
 upgradehelper.py | 53 ++++++++++++++++++++++++++++++-----------------------
 2 files changed, 37 insertions(+), 23 deletions(-)

diff --git a/recipe.py b/recipe.py
index 8ecd2a2..55eb5bf 100644
--- a/recipe.py
+++ b/recipe.py
@@ -394,6 +394,13 @@ class Recipe(object):
 
         return False
 
+    def get_license_diff_file_name(self):
+        file_name = None
+        if not self.license_diff_file is None:
+            file_name = os.path.basename(self.license_diff_file)
+
+        return file_name
+
     def _get_failed_recipes(self, output):
         failed_tasks = dict()
         machine = None
diff --git a/upgradehelper.py b/upgradehelper.py
index 74f9c0a..953f86e 100755
--- a/upgradehelper.py
+++ b/upgradehelper.py
@@ -124,26 +124,6 @@ def parse_config_file(config_file):
     return (settings, maintainer_override)
 
 class Updater(object):
-    mail_header = \
-        "Hello,\n\nYou are receiving this email because you are the maintainer\n" \
-        "of *%s* recipe and this is to let you know that the automatic attempt\n" \
-        "to upgrade the recipe to *%s* has %s.\n\n"
-
-    next_steps_info = \
-        "The recipe has been successfully compiled for all major architectures.\n\n" \
-        "Next steps:\n" \
-        "    - apply the patch: git am %s\n" \
-        "    - check that required patches have not been removed from the recipe\n" \
-        "    - compile an image that contains the package\n" \
-        "    - perform some basic sanity tests\n" \
-        "    - amend the patch and sign it off: git commit -s --reset-author --amend\n" \
-        "    - send it to the list\n\n" \
-
-    mail_footer = \
-        "Attached are the patch and the logs (+ license file diff) in case of failure.\n\n" \
-        "Regards,\nThe Upgrade Helper"
-
-
     def __init__(self, auto_mode=False, send_email=False, skip_compilation=False):
 
         self.uh_dir = get_build_dir() + "/upgrade-helper"
@@ -375,6 +355,29 @@ class Updater(object):
             self.git.clean_untracked()
 
         if self.send_email:
+            mail_header = \
+                "Hello,\n\nYou are receiving this email because you are the maintainer\n" \
+                "of *%s* recipe and this is to let you know that the automatic attempt\n" \
+                "to upgrade the recipe to *%s* has %s.\n\n"
+
+            license_change_info = \
+                "*LICENSE CHANGED* please review the %s file and update the LICENSE\n" \
+                "variable in the recipe if is needed.\n\n"
+
+            next_steps_info = \
+                "The recipe has been successfully compiled for all major architectures.\n\n" \
+                "Next steps:\n" \
+                "    - apply the patch: git am %s\n" \
+                "    - check that required patches have not been removed from the recipe\n" \
+                "    - compile an image that contains the package\n" \
+                "    - perform some basic sanity tests\n" \
+                "    - amend the patch and sign it off: git commit -s --reset-author --amend\n" \
+                "    - send it to the list\n\n" \
+
+            mail_footer = \
+                "Attached are the patch, license diff (if change) and bitbake log.\n\n" \
+                "Regards,\nThe Upgrade Helper"
+
             # don't bother maintainer with mail if the recipe is already up to date
             if status == "UpgradeNotNeededError":
                 return
@@ -390,13 +393,17 @@ class Updater(object):
             else:
                 subject += " FAILED"
 
-            msg_body = self.mail_header % (self.pn, self.new_ver,
+            msg_body = mail_header % (self.pn, self.new_ver,
                     self._get_status_msg(err))
 
+            license_diff_fn = self.recipe.get_license_diff_file_name()
+            if license_diff_fn:
+                msg_body += license_change_info % license_diff_fn
+
             if err is None:
-                msg_body += self.next_steps_info % os.path.basename(self.patch_file)
+                msg_body += next_steps_info % os.path.basename(self.patch_file)
 
-            msg_body += self.mail_footer
+            msg_body += mail_footer
 
             # Add possible attachments to email
             attachments = []
-- 
1.8.4.5

[PATCH 06/11][auh] upgradehelper.py: Add specific information about machines build succeed.

[Aníbal Limón]

Use build machines in email to maintainers instead of put that the build
succeed in major arch's, this give to the maintainer non ambiguos info.

[YOCTO #7489]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 upgradehelper.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/upgradehelper.py b/upgradehelper.py
index 953f86e..266435d 100755
--- a/upgradehelper.py
+++ b/upgradehelper.py
@@ -365,7 +365,7 @@ class Updater(object):
                 "variable in the recipe if is needed.\n\n"
 
             next_steps_info = \
-                "The recipe has been successfully compiled for all major architectures.\n\n" \
+                "The recipe has been successfully compiled for machines %s.\n\n" \
                 "Next steps:\n" \
                 "    - apply the patch: git am %s\n" \
                 "    - check that required patches have not been removed from the recipe\n" \
@@ -401,7 +401,8 @@ class Updater(object):
                 msg_body += license_change_info % license_diff_fn
 
             if err is None:
-                msg_body += next_steps_info % os.path.basename(self.patch_file)
+                msg_body += next_steps_info % (', '.join(self.machines),
+                        os.path.basename(self.patch_file))
 
             msg_body += mail_footer
 
-- 
1.8.4.5

[PATCH 07/11][auh] upgradehelper.py: Move recipe work into upgradehelper work dir.

[Aníbal Limón]

[YOCTO #7489]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 upgradehelper.py | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/upgradehelper.py b/upgradehelper.py
index 266435d..19753a6 100755
--- a/upgradehelper.py
+++ b/upgradehelper.py
@@ -126,10 +126,14 @@ def parse_config_file(config_file):
 class Updater(object):
     def __init__(self, auto_mode=False, send_email=False, skip_compilation=False):
 
-        self.uh_dir = get_build_dir() + "/upgrade-helper"
+        self.uh_dir = os.path.join(get_build_dir(), "upgrade-helper")
         if not os.path.exists(self.uh_dir):
             os.mkdir(self.uh_dir)
 
+        self.uh_work_dir = os.path.join(self.uh_dir, "work")
+        if not os.path.exists(self.uh_work_dir):
+            os.mkdir(self.uh_work_dir)
+
         self.bb = Bitbake(get_build_dir())
         self.buildhistory = BuildHistory(get_build_dir())
         self.git = None
@@ -192,7 +196,7 @@ class Updater(object):
             return "Succeeded"
 
     def _create_workdir(self):
-        self.workdir = self.uh_dir + "/" + self.pn
+        self.workdir = os.path.join(self.uh_work_dir, self.pn)
 
         if not os.path.exists(self.workdir):
             os.mkdir(self.workdir)
-- 
1.8.4.5

[PATCH 08/11][auh] upgradehelper.py: Fix interactive mode when recipe upgrade finish.

[Aníbal Limón]

The validation for dropping changes needs to be defined only when
interactive mode is true if not this causes to drop the changes
for every failure.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 upgradehelper.py | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/upgradehelper.py b/upgradehelper.py
index 19753a6..b1f075d 100755
--- a/upgradehelper.py
+++ b/upgradehelper.py
@@ -335,11 +335,10 @@ class Updater(object):
 
     # this function will be called at the end of each recipe upgrade
     def pkg_upgrade_handler(self, err):
-        if err and self.patch_file:
+        if err and self.patch_file and self.interactive:
             answer = "N"
-            if self.interactive:
-                I(" %s: Do you want to keep the changes? (y/N)" % self.pn)
-                answer = sys.stdin.readline().strip().upper()
+            I(" %s: Do you want to keep the changes? (y/N)" % self.pn)
+            answer = sys.stdin.readline().strip().upper()
 
             if answer == '' or answer == 'N':
                 I(" %s: Dropping changes from git ..." % self.pn)
-- 
1.8.4.5

[PATCH 09/11][auh] upgradehelper.py: Change policy for send emails and fix error passing

[Aníbal Limón]

Now send emails almost in all cases this give the maintainer
patches and diff to continue work also if the build isn't
succesful.

[YOCTO #7489]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 upgradehelper.py | 19 ++++++++++---------
 1 file changed, 10 insertions(+), 9 deletions(-)

diff --git a/upgradehelper.py b/upgradehelper.py
index b1f075d..a4aa0ab 100755
--- a/upgradehelper.py
+++ b/upgradehelper.py
@@ -381,8 +381,11 @@ class Updater(object):
                 "Attached are the patch, license diff (if change) and bitbake log.\n\n" \
                 "Regards,\nThe Upgrade Helper"
 
-            # don't bother maintainer with mail if the recipe is already up to date
-            if status == "UpgradeNotNeededError":
+            if status == "Error" or status == "FetchError" or\
+               status == "UpgradeNotNeededError":
+                D( "%s: Don't send email to maintainer because the error was " \
+                   "%s and the information isn't useful, please review it." \
+                    % (self.pn, status))
                 return
 
             if self.maintainer in maintainer_override:
@@ -478,6 +481,7 @@ class Updater(object):
 
         attempted_pkgs = 0
         for self.pn, self.new_ver, self.maintainer in pkgs_to_upgrade:
+            error = None
             self.recipe = None
             attempted_pkgs += 1
             I(" ATTEMPT PACKAGE %d/%d" % (attempted_pkgs, total_pkgs))
@@ -489,12 +493,9 @@ class Updater(object):
                     step()
 
                 I(" %s: Upgrade SUCCESSFUL! Please test!" % self.pn)
-                error = None
-            except UpgradeNotNeededError as e:
-                I(" %s: %s" % (self.pn, e.message))
-                error = e
-            except Error as e:
-                E(" %s: %s" % (self.pn, e.message))
+            except Exception as e:
+                if isinstance(e, Error):
+                    E(" %s: %s" % (self.pn, e.message))
                 E(" %s: Upgrade FAILED! Logs and/or file diffs are available in %s" % (self.pn, self.workdir))
                 error = e
 
@@ -667,7 +668,7 @@ class UniverseUpdater(Updater):
 
     # overriding the base method
     def pkg_upgrade_handler(self, err):
-        super(UniverseUpdater, self).pkg_upgrade_handler(self)
+        super(UniverseUpdater, self).pkg_upgrade_handler(err)
         self.update_history(self.pn, self.new_ver, self.maintainer,
                 self._get_status_msg(err))
 
-- 
1.8.4.5

Re: [PATCH 09/11][auh] upgradehelper.py: Change policy for send emails and fix error passing

[Paul Eggleton]

On Wednesday 10 June 2015 16:28:50 Aníbal Limón wrote:
> Now send emails almost in all cases this give the maintainer
> patches and diff to continue work also if the build isn't
> succesful.
>
> [YOCTO #7489]
> 
> Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
> ---
>  upgradehelper.py | 19 ++++++++++---------
>  1 file changed, 10 insertions(+), 9 deletions(-)
> 
> diff --git a/upgradehelper.py b/upgradehelper.py
> index b1f075d..a4aa0ab 100755
> --- a/upgradehelper.py
> +++ b/upgradehelper.py
> @@ -381,8 +381,11 @@ class Updater(object):
>                  "Attached are the patch, license diff (if change) and
> bitbake log.\n\n" \ "Regards,\nThe Upgrade Helper"
> 
> -            # don't bother maintainer with mail if the recipe is already up
> to date -            if status == "UpgradeNotNeededError":
> +            if status == "Error" or status == "FetchError" or\
> +               status == "UpgradeNotNeededError":
> +                D( "%s: Don't send email to maintainer because the error
> was " \ +                   "%s and the information isn't useful, please
> review it." \ +                    % (self.pn, status))

Obviously this is in the existing code, but I'm not hugely keen on checking 
the error name as a string - I'd much rather we checked it with isinstance 
here. That might require us to have all errors that the maintainer can fix as 
subclasses of a particular class though.

Also, do you have an idea of the types of exception that might now trigger an 
email that don't inherit from our "Error" class here but are actually worth 
telling the maintainer about? I can certainly think about situations where 
bugs in the script here could trigger a large number of emails to maintainers 
which we probably wouldn't want.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 09/11][auh] upgradehelper.py: Change policy for send emails and fix error passing

[Aníbal Limón]

Hi Paul,

Comments below,

Kind regards,
     alimon

On 11/06/15 15:51, Paul Eggleton wrote:
> On Wednesday 10 June 2015 16:28:50 Aníbal Limón wrote:
>> Now send emails almost in all cases this give the maintainer
>> patches and diff to continue work also if the build isn't
>> succesful.
>>
>> [YOCTO #7489]
>>
>> Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
>> ---
>>   upgradehelper.py | 19 ++++++++++---------
>>   1 file changed, 10 insertions(+), 9 deletions(-)
>>
>> diff --git a/upgradehelper.py b/upgradehelper.py
>> index b1f075d..a4aa0ab 100755
>> --- a/upgradehelper.py
>> +++ b/upgradehelper.py
>> @@ -381,8 +381,11 @@ class Updater(object):
>>                   "Attached are the patch, license diff (if change) and
>> bitbake log.\n\n" \ "Regards,\nThe Upgrade Helper"
>>
>> -            # don't bother maintainer with mail if the recipe is already up
>> to date -            if status == "UpgradeNotNeededError":
>> +            if status == "Error" or status == "FetchError" or\
>> +               status == "UpgradeNotNeededError":
>> +                D( "%s: Don't send email to maintainer because the error
>> was " \ +                   "%s and the information isn't useful, please
>> review it." \ +                    % (self.pn, status))
> Obviously this is in the existing code, but I'm not hugely keen on checking
> the error name as a string - I'd much rather we checked it with isinstance
> here. That might require us to have all errors that the maintainer can fix as
> subclasses of a particular class though.
Yes i'll change the string testing for isinstance.
>
> Also, do you have an idea of the types of exception that might now trigger an
> email that don't inherit from our "Error" class here but are actually worth
> telling the maintainer about? I can certainly think about situations where
> bugs in the script here could trigger a large number of emails to maintainers
> which we probably wouldn't want.
That's good observation i filter all the errors that don't are in the 
class of Error,
also i avoided Error/FetchError/UpgradeNotNeeded because when happens no 
usueful info is
generated.
>
> Cheers,
> Paul
>

Re: [PATCH 09/11][auh] upgradehelper.py: Change policy for send emails and fix error passing

[Paul Eggleton]

On Thursday 11 June 2015 15:56:57 Aníbal Limón wrote:
> Hi Paul,
> 
> Comments below,
> 
> Kind regards,
>      alimon
> 
> On 11/06/15 15:51, Paul Eggleton wrote:
> > On Wednesday 10 June 2015 16:28:50 Aníbal Limón wrote:
> >> Now send emails almost in all cases this give the maintainer
> >> patches and diff to continue work also if the build isn't
> >> succesful.
> >> 
> >> [YOCTO #7489]
> >> 
> >> Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
> >> ---
> >> 
> >>   upgradehelper.py | 19 ++++++++++---------
> >>   1 file changed, 10 insertions(+), 9 deletions(-)
> >> 
> >> diff --git a/upgradehelper.py b/upgradehelper.py
> >> index b1f075d..a4aa0ab 100755
> >> --- a/upgradehelper.py
> >> +++ b/upgradehelper.py
> >> 
> >> @@ -381,8 +381,11 @@ class Updater(object):
> >>                   "Attached are the patch, license diff (if change) and
> >> 
> >> bitbake log.\n\n" \ "Regards,\nThe Upgrade Helper"
> >> 
> >> -            # don't bother maintainer with mail if the recipe is already
> >> up to date -            if status == "UpgradeNotNeededError":
> >> +            if status == "Error" or status == "FetchError" or\
> >> +               status == "UpgradeNotNeededError":
> >> +                D( "%s: Don't send email to maintainer because the error
> >> was " \ +                   "%s and the information isn't useful, please
> >> review it." \ +                    % (self.pn, status))
> > 
> > Obviously this is in the existing code, but I'm not hugely keen on
> > checking
> > the error name as a string - I'd much rather we checked it with isinstance
> > here. That might require us to have all errors that the maintainer can fix
> > as subclasses of a particular class though.
> 
> Yes i'll change the string testing for isinstance.
> 
> > Also, do you have an idea of the types of exception that might now trigger
> > an email that don't inherit from our "Error" class here but are actually
> > worth telling the maintainer about? I can certainly think about
> > situations where bugs in the script here could trigger a large number of
> > emails to maintainers which we probably wouldn't want.
> 
> That's good observation i filter all the errors that don't are in the
> class of Error,
> also i avoided Error/FetchError/UpgradeNotNeeded because when happens no
> usueful info is
> generated.

Sorry I'm not quite following - you mean you did filter out exceptions that 
don't inherit from Error or you will change it to do that? Unless I'm missing 
something, this patch now catches any exception of class Exception and passes 
it into this function, but we only skip Error/FetchError/UpgradeNotNeeded - so 
for example a SyntaxError will get sent after this change, right? Surely we 
don't want that?

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 09/11][auh] upgradehelper.py: Change policy for send emails and fix error passing

[Aníbal Limón]

On 11/06/15 16:05, Paul Eggleton wrote:
> On Thursday 11 June 2015 15:56:57 Aníbal Limón wrote:
>> Hi Paul,
>>
>> Comments below,
>>
>> Kind regards,
>>       alimon
>>
>> On 11/06/15 15:51, Paul Eggleton wrote:
>>> On Wednesday 10 June 2015 16:28:50 Aníbal Limón wrote:
>>>> Now send emails almost in all cases this give the maintainer
>>>> patches and diff to continue work also if the build isn't
>>>> succesful.
>>>>
>>>> [YOCTO #7489]
>>>>
>>>> Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
>>>> ---
>>>>
>>>>    upgradehelper.py | 19 ++++++++++---------
>>>>    1 file changed, 10 insertions(+), 9 deletions(-)
>>>>
>>>> diff --git a/upgradehelper.py b/upgradehelper.py
>>>> index b1f075d..a4aa0ab 100755
>>>> --- a/upgradehelper.py
>>>> +++ b/upgradehelper.py
>>>>
>>>> @@ -381,8 +381,11 @@ class Updater(object):
>>>>                    "Attached are the patch, license diff (if change) and
>>>>
>>>> bitbake log.\n\n" \ "Regards,\nThe Upgrade Helper"
>>>>
>>>> -            # don't bother maintainer with mail if the recipe is already
>>>> up to date -            if status == "UpgradeNotNeededError":
>>>> +            if status == "Error" or status == "FetchError" or\
>>>> +               status == "UpgradeNotNeededError":
>>>> +                D( "%s: Don't send email to maintainer because the error
>>>> was " \ +                   "%s and the information isn't useful, please
>>>> review it." \ +                    % (self.pn, status))
>>> Obviously this is in the existing code, but I'm not hugely keen on
>>> checking
>>> the error name as a string - I'd much rather we checked it with isinstance
>>> here. That might require us to have all errors that the maintainer can fix
>>> as subclasses of a particular class though.
>> Yes i'll change the string testing for isinstance.
>>
>>> Also, do you have an idea of the types of exception that might now trigger
>>> an email that don't inherit from our "Error" class here but are actually
>>> worth telling the maintainer about? I can certainly think about
>>> situations where bugs in the script here could trigger a large number of
>>> emails to maintainers which we probably wouldn't want.
>> That's good observation i filter all the errors that don't are in the
>> class of Error,
>> also i avoided Error/FetchError/UpgradeNotNeeded because when happens no
>> usueful info is
>> generated.
> Sorry I'm not quite following - you mean you did filter out exceptions that
> don't inherit from Error or you will change it to do that? Unless I'm missing
> something, this patch now catches any exception of class Exception and passes
> it into this function, but we only skip Error/FetchError/UpgradeNotNeeded - so
> for example a SyntaxError will get sent after this change, right? Surely we
> don't want that?

Now it's filtering Error/FetchError/UpgradeNotNeeded errors but you are
right that we need to only take into account exceptions inherit Error to
avoid send emails with system exceptions like Syntax and IO.

I'll change the code for guarantee that.

Cheers,
     alimon


>
> Cheers,
> Paul
>

[PATCH 10/11][auh] upgradehelper.py: Change print to logger info.

[Aníbal Limón]

Print sentences need to be displayed as information in the logger
this unifies logger facilities.

[YOCTO #7489]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 recipe.py        | 9 ++++-----
 upgradehelper.py | 8 ++++----
 2 files changed, 8 insertions(+), 9 deletions(-)

diff --git a/recipe.py b/recipe.py
index 55eb5bf..6fd8f24 100644
--- a/recipe.py
+++ b/recipe.py
@@ -376,11 +376,11 @@ class Recipe(object):
                   "been updated! View diff? (Y/n)" % (self.env['PN'], license_file))
                 answer = sys.stdin.readline().strip().upper()
                 if answer == '' or answer == 'Y':
-                    print(" ################ Licence file diff #################")
+                    I(" ################ Licence file diff #################")
                     with open(self.license_diff_file) as diff:
-                        print("%s" % diff.read())
-                    print(" ####################################################")
-                print("Retry compilation? (Y/n)")
+                        I("%s" % diff.read())
+                    I(" ####################################################")
+                I(" Retry compilation? (Y/n)")
                 answer = sys.stdin.readline().strip().upper()
                 if answer == '' or answer == 'Y':
                     return True
@@ -647,7 +647,6 @@ class Recipe(object):
 
                     if not self._license_issue_handled(log_file):
                         raise LicenseError()
-
                     #retry
                     self.compile(machine)
                 elif failed_task == "do_fetch":
diff --git a/upgradehelper.py b/upgradehelper.py
index a4aa0ab..be15e1f 100755
--- a/upgradehelper.py
+++ b/upgradehelper.py
@@ -514,7 +514,7 @@ class Updater(object):
                     exit(0)
 
         if (attempted_pkgs > 1):
-            print("%s" % self.statistics.pkg_stats())
+            I("%s" % self.statistics.pkg_stats())
             if self.send_email:
                 self.send_status_mail()
 
@@ -647,10 +647,10 @@ class UniverseUpdater(Updater):
             last_check.write(current_date + "," + cur_master_commit + "," +
                              last_checkpkg_file)
 
-        print("########### The list of recipes to be upgraded ############")
+        I(" ########### The list of recipes to be upgraded ############")
         for p, v, m in pkgs_list:
-            print("%s,%s,%s" % (p, v, m))
-        print("############################################################")
+            I(" %s, %s, %s" % (p, v, m))
+        I(" ############################################################")
 
         return pkgs_list
 
-- 
1.8.4.5

[PATCH 11/11][auh] README: Edit Maintainece section to add current maintainer.

[Aníbal Limón]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 README | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/README b/README
index ec3f955..dbcc755 100644
--- a/README
+++ b/README
@@ -112,7 +112,8 @@ The latest version of the code can always be found here:
   http://git.yoctoproject.org/cgit/cgit.cgi/auto-upgrade-helper/
 
 Contributions are welcome. Please send patches / pull requests to
-yocto@yoctoproject.org with '[auh]' in the subject.
+yocto@yoctoproject.org with '[auh]' in the subject also CC the
+current maintainer: Aníbal Limón <anibal.limon@linux.intel.com>.
 
 
 License
-- 
1.8.4.5