[Buildroot] [buildroot-test 0/5] improve modularity by adding an

[Buildroot] [buildroot-test 0/5] improve modularity by adding an

[Victor Huesca]

This patch series replace the manual argument parsing by a python
argument parser. The goal is to improve the modularity to help the
integration of new information to the mails.

Victor Huesca (5):
  utils/daily-mail: introduce argparse to ease modularity of the script
  utils/daily-mail: allow to pass a date in arguments
  utils/daily-mail: allow to specify branches from command-line
  utils/daily-mail: allow to select only a subset of emails to send
  utils/daily-mail: allow to select only a subset of data to send in
    emails

 utils/daily-mail | 64 +++++++++++++++++++++++++++++++++++-------------
 1 file changed, 47 insertions(+), 17 deletions(-)

-- 
2.21.0

[Buildroot] [buildroot-test 1/5] utils/daily-mail: introduce argparse to ease modularity of the script

[Victor Huesca]

Having an actual argument parser instead of manually checking for
command-line arguments will allow future patches to add divers options
to the script. This patch only introduce the parser with the 'dry-run'
argument.

Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
---
 utils/daily-mail | 15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)

diff --git a/utils/daily-mail b/utils/daily-mail
index 2d5e684..d4c69f6 100755
--- a/utils/daily-mail
+++ b/utils/daily-mail
@@ -12,6 +12,7 @@ import localconfig
 import csv
 from collections import defaultdict
 import math
+import argparse
 
 sys.path.append(os.path.join(localconfig.brbase, "utils"))
 import getdeveloperlib  # noqa: E402
@@ -379,7 +380,14 @@ def calculate_notifications(results):
     return notifications
 
 
+def parse_args():
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--dry-run', action='store_true', help='Do not send email, only show output')
+    return parser.parse_args()
+
+
 def __main__():
+    args = parse_args()
     yesterday = date.today() - timedelta(1)
     yesterday_str = yesterday.strftime('%Y-%m-%d')
     branches = get_branches()
@@ -392,15 +400,12 @@ def __main__():
     results = get_build_results(db, yesterday_str, branches)
     results_by_reason = get_build_results_grouped_by_reason(db, yesterday_str, branches)
     notifications = calculate_notifications(results)
-    dry_run = False
-    if len(sys.argv) == 2 and sys.argv[1] == "--dry-run":
-        dry_run = True
     smtp = smtplib.SMTP(localconfig.smtphost, localconfig.smtpport)
     smtp.starttls()
     smtp.login(localconfig.smtpuser, localconfig.smtppass)
-    developers_email(smtp, branches, notifications, yesterday_str, dry_run)
+    developers_email(smtp, branches, notifications, yesterday_str, args.dry_run)
     global_email(smtp, results, results_by_reason, yesterday_str,
-                 overall_stats, dry_run)
+                 overall_stats, args.dry_run)
     smtp.quit()
 
 
-- 
2.21.0

[Buildroot] [buildroot-test 2/5] utils/daily-mail: allow to pass a date in arguments

[Victor Huesca]

This allows to use a custorm date instead of the hardcoded yesterday.

Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
---
 utils/daily-mail | 16 +++++++++++++---
 1 file changed, 13 insertions(+), 3 deletions(-)

diff --git a/utils/daily-mail b/utils/daily-mail
index d4c69f6..cb7402f 100755
--- a/utils/daily-mail
+++ b/utils/daily-mail
@@ -13,10 +13,13 @@ import csv
 from collections import defaultdict
 import math
 import argparse
+import re
 
 sys.path.append(os.path.join(localconfig.brbase, "utils"))
 import getdeveloperlib  # noqa: E402
 
+RE_DATE = re.compile(r'^\d\d\d\d-\d\d-\d\d$')
+
 baseurl = "autobuild.buildroot.net"
 http_baseurl = "http://" + baseurl
 
@@ -383,13 +386,20 @@ def calculate_notifications(results):
 def parse_args():
     parser = argparse.ArgumentParser()
     parser.add_argument('--dry-run', action='store_true', help='Do not send email, only show output')
-    return parser.parse_args()
+    parser.add_argument('--date', action='store', help='Use this date instead of yesterday')
+    args = parser.parse_args()
+    if args.date and not RE_DATE.match(args.date):
+        parser.error('date must be in format YYYY-MM-DD')
+    return args
 
 
 def __main__():
     args = parse_args()
-    yesterday = date.today() - timedelta(1)
-    yesterday_str = yesterday.strftime('%Y-%m-%d')
+    if args.date:
+        yesterday_str = args.date
+    else:
+        yesterday = date.today() - timedelta(1)
+        yesterday_str = yesterday.strftime('%Y-%m-%d')
     branches = get_branches()
 
     db = _mysql.connect(host=localconfig.host,
-- 
2.21.0

[Buildroot] [buildroot-test 3/5] utils/daily-mail: allow to specify branches from command-line

[Victor Huesca]

This patch allows to specify branches to include in emails instead of
reading the 'branches' file. If no branch is provided, the default
'branches' file is used.

Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
---
 utils/daily-mail | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/utils/daily-mail b/utils/daily-mail
index cb7402f..040ee15 100755
--- a/utils/daily-mail
+++ b/utils/daily-mail
@@ -387,6 +387,7 @@ def parse_args():
     parser = argparse.ArgumentParser()
     parser.add_argument('--dry-run', action='store_true', help='Do not send email, only show output')
     parser.add_argument('--date', action='store', help='Use this date instead of yesterday')
+    parser.add_argument('--branches', action='store', nargs='+', help='List of branches (blank separated)')
     args = parser.parse_args()
     if args.date and not RE_DATE.match(args.date):
         parser.error('date must be in format YYYY-MM-DD')
@@ -400,7 +401,7 @@ def __main__():
     else:
         yesterday = date.today() - timedelta(1)
         yesterday_str = yesterday.strftime('%Y-%m-%d')
-    branches = get_branches()
+    branches = args.branches if args.branches else get_branches()
 
     db = _mysql.connect(host=localconfig.host,
                         user=localconfig.user,
-- 
2.21.0

[Buildroot] [buildroot-test 4/5] utils/daily-mail: allow to select only a subset of emails to send

[Victor Huesca]

This patch allows to send the global email and the per-developer email
separately by specifying the destination as command-line argument.
The default behavior -- when no destination is provided -- is to select
both the global and developers emails.

Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
---
 utils/daily-mail | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/utils/daily-mail b/utils/daily-mail
index 040ee15..4407438 100755
--- a/utils/daily-mail
+++ b/utils/daily-mail
@@ -388,6 +388,8 @@ def parse_args():
     parser.add_argument('--dry-run', action='store_true', help='Do not send email, only show output')
     parser.add_argument('--date', action='store', help='Use this date instead of yesterday')
     parser.add_argument('--branches', action='store', nargs='+', help='List of branches (blank separated)')
+    parser.add_argument('--dest', action='store', nargs='+', choices=['dev', 'global'],
+                        help='List of emails type to send (ie. global, dev)')
     args = parser.parse_args()
     if args.date and not RE_DATE.match(args.date):
         parser.error('date must be in format YYYY-MM-DD')
@@ -402,6 +404,7 @@ def __main__():
         yesterday = date.today() - timedelta(1)
         yesterday_str = yesterday.strftime('%Y-%m-%d')
     branches = args.branches if args.branches else get_branches()
+    email_dest = set(args.dest) if args.dest else {'dev', 'global'}
 
     db = _mysql.connect(host=localconfig.host,
                         user=localconfig.user,
@@ -414,9 +417,11 @@ def __main__():
     smtp = smtplib.SMTP(localconfig.smtphost, localconfig.smtpport)
     smtp.starttls()
     smtp.login(localconfig.smtpuser, localconfig.smtppass)
-    developers_email(smtp, branches, notifications, yesterday_str, args.dry_run)
-    global_email(smtp, results, results_by_reason, yesterday_str,
-                 overall_stats, args.dry_run)
+    if 'dev' in email_dest:
+        developers_email(smtp, branches, notifications, yesterday_str, args.dry_run)
+    if 'global' in email_dest:
+        global_email(smtp, results, results_by_reason, yesterday_str,
+                     overall_stats, args.dry_run)
     smtp.quit()
 
 
-- 
2.21.0

[Buildroot] [buildroot-test 5/5] utils/daily-mail: allow to select only a subset of data to send in emails

[Victor Huesca]

This patch adds a modular way to select a subset of the available
information collected by this script. Currently the range of possibilities
is limited to enable or disable the autobuild results, but it allow
future patches to easily add new kind of data to this script.

Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
---
 utils/daily-mail | 25 +++++++++++++++++--------
 1 file changed, 17 insertions(+), 8 deletions(-)

diff --git a/utils/daily-mail b/utils/daily-mail
index 4407438..20e5886 100755
--- a/utils/daily-mail
+++ b/utils/daily-mail
@@ -231,7 +231,7 @@ def developers_email(smtp, branches, notifications, datestr, dry_run):
     for dev, notif in notifications.iteritems():
         to = dev.name
         email_from = localconfig.fromaddr
-        subject = "[{}] Your build results for {}".format(baseurl, datestr)
+        subject = "[{}] Your daily results for {}".format(baseurl, datestr)
         contents = "Hello,\n\n"
         contents += textwrap.fill("This is the list of Buildroot build failures that occurred on {}, "
                                   "and for which you are a registered architecture developer or package "
@@ -304,11 +304,12 @@ def global_email(smtp, results, results_by_reason, datestr, overall, dry_run):
     '''
     to = "buildroot at buildroot.org"
     email_from = localconfig.fromaddr
-    subject = "[{}] Build results for {}".format(baseurl, datestr)
+    subject = "[{}] Daily results for {}".format(baseurl, datestr)
     contents = "Hello,\n\n"
-    contents += "Build statistics for {}\n".format(datestr)
-    contents += "===============================\n\n"
-    contents += "      branch |  OK | NOK | TIM | TOT |\n"
+    if len(overall) != 0:
+        contents += "Autobuild statistics for {}\n".format(datestr)
+        contents += "===============================\n\n"
+        contents += "      branch |  OK | NOK | TIM | TOT |\n"
     for branch in sorted(overall.iterkeys()):
         stats = overall[branch]
         if stats[3] == 0:
@@ -390,6 +391,8 @@ def parse_args():
     parser.add_argument('--branches', action='store', nargs='+', help='List of branches (blank separated)')
     parser.add_argument('--dest', action='store', nargs='+', choices=['dev', 'global'],
                         help='List of emails type to send (ie. global, dev)')
+    parser.add_argument('--data', action='store', nargs='+', choices=['autobuild'],
+                        help='List of information to add in emails (blank separated)')
     args = parser.parse_args()
     if args.date and not RE_DATE.match(args.date):
         parser.error('date must be in format YYYY-MM-DD')
@@ -405,14 +408,20 @@ def __main__():
         yesterday_str = yesterday.strftime('%Y-%m-%d')
     branches = args.branches if args.branches else get_branches()
     email_dest = set(args.dest) if args.dest else {'dev', 'global'}
+    email_data = set(args.data) if args.data else {'autobuild'}
 
     db = _mysql.connect(host=localconfig.host,
                         user=localconfig.user,
                         passwd=localconfig.passwd,
                         db=localconfig.db)
-    overall_stats = get_overall_stats(db, yesterday_str, branches)
-    results = get_build_results(db, yesterday_str, branches)
-    results_by_reason = get_build_results_grouped_by_reason(db, yesterday_str, branches)
+    if 'autobuild' in email_data:
+        overall_stats = get_overall_stats(db, yesterday_str, branches)
+        results = get_build_results(db, yesterday_str, branches)
+        results_by_reason = get_build_results_grouped_by_reason(db, yesterday_str, branches)
+    else:
+        overall_stats = {}
+        results = {}
+        results_by_reason = {}
     notifications = calculate_notifications(results)
     smtp = smtplib.SMTP(localconfig.smtphost, localconfig.smtpport)
     smtp.starttls()
-- 
2.21.0

[Buildroot] [buildroot-test 0/5] improve modularity by adding an

[Thomas Petazzoni]

On Sun,  4 Aug 2019 17:25:31 +0200
Victor Huesca <victor.huesca@bootlin.com> wrote:

> Victor Huesca (5):
>   utils/daily-mail: introduce argparse to ease modularity of the script
>   utils/daily-mail: allow to pass a date in arguments
>   utils/daily-mail: allow to specify branches from command-line
>   utils/daily-mail: allow to select only a subset of emails to send
>   utils/daily-mail: allow to select only a subset of data to send in
>     emails

Applied, thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com