[Buildroot] [PATCH buildroot-test 00/10] add gitlab-ci jobs to the daily-mail

[Buildroot] [PATCH buildroot-test 00/10] add gitlab-ci jobs to the daily-mail

[Victor Huesca]

This patch series adds gitlab-ci job failures to the daily-mail, this
includes both defconfig build failures and runtime tests failures.


Patches 1-3: gitlab-ci infrastructure for defconfigs and runtime-tests.
Patches 4-5: add defconfigs and runtime-tests to per-developer
 notifications.
Patches 6-8: add the formatting logic for defconfig and runtime-tests.
Patches 9-10: adjust command-line arguments and enable gitlab-ci
 integration.

Victor Huesca (10):
  utils/daily-mail: new data: gitlab-ci jobs
  utils/daily-mail: new gitlab-ci data: defconfigs
  utils/daily-mail: new gitlab-ci data: runtime tests
  utils/daily-mail: add defconfig support in developers notifications
  utils/daily-mail: add runtime tests support in developers
    notifications
  utils/daily-mail: prepare the mail formatting to support gitlab-ci
    results
  utils/daily-mail: add formatting for defconfig results
  utils/daily-mail: add formatting for runtime-test results
  utils/daily-mail: add argument for gitlab-ci related data
  utils/daily-mail: enable gitlab-ci emails

 utils/daily-mail | 342 +++++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 313 insertions(+), 29 deletions(-)

-- 
2.21.0

[Buildroot] [PATCH buildroot-test 01/10] utils/daily-mail: new data: gitlab-ci jobs

[Victor Huesca]

Buildroot have a mirror repository on gitlab.com that is used to run
continious integration tasks. Among these tasks, a complete checkup of
defconfigs and runtime-tests is regularly performed.

This patch allows to retrieve tests that have failed over a given period
of time. It is based on the (very imperfect) gitlab-ci API.

Due to multiple limitations of the API (pagination of results, crucial
data requiring a dedicated request, etc.), this implementation performs
multiple optimisation to be able to retreive the tasks needed.

Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
---
 utils/daily-mail | 157 ++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 156 insertions(+), 1 deletion(-)

diff --git a/utils/daily-mail b/utils/daily-mail
index baf0e0b..2d4ed1c 100755
--- a/utils/daily-mail
+++ b/utils/daily-mail
@@ -7,7 +7,7 @@ import textwrap
 import mysmtplib as smtplib
 from email.mime.text import MIMEText
 from email.utils import formatdate
-from datetime import date, timedelta
+from datetime import date, timedelta, datetime
 import localconfig
 import csv
 from collections import defaultdict
@@ -16,6 +16,10 @@ import argparse
 import re
 import json
 from packaging import version
+import certifi
+from urllib3 import HTTPSConnectionPool
+from urllib3.exceptions import HTTPError
+from multiprocessing import Pool as ProcessPool
 
 sys.path.append(os.path.join(localconfig.brbase, "utils"))
 import getdeveloperlib  # noqa: E402
@@ -27,6 +31,7 @@ baseurl = "autobuild.buildroot.net"
 http_baseurl = "http://" + baseurl
 
 developers = getdeveloperlib.parse_developers(localconfig.brbase)
+gitlab_ci_http_pool = None
 
 
 def get_branches():
@@ -469,6 +474,156 @@ def get_outdated_pkgs(path):
     return sorted(s, key=lambda pkg: pkg['name'])
 
 
+def request_paginated_url(pool, url, page_size=100, page_name='page', **request_kwargs):
+    '''
+    Simple helper to retreive data from call paginated url.
+    Request's result is assumed to be json list.
+    '''
+    request_kwargs.setdefault('fields', {})
+    request_kwargs['fields']['per_page'] = page_size
+    results = []
+    page = 1
+    while "len(res) == page_size":
+        request_kwargs['fields'][page_name] = page
+        try:
+            req = pool.request('GET', url, **request_kwargs)
+        except HTTPError:
+            print('error: there were an error with gitlab api. '
+                  'Please make sure your identification token is right and try again.')
+            break
+        if req.status != 200:
+            print('error: there were an error with gitlab api. '
+                  'Please make sure your identification token is right and try again.')
+            break
+        res = json.loads(req.data)
+        results += res
+        page += 1
+        if len(res) != page_size:
+            break
+    return results
+
+
+def get_pipeline_details(project_id, pipeline, **request_kwargs):
+    pool = gitlab_ci_http_pool
+    pipeline_url = '/api/v4/projects/{}/pipelines/{}'.format(project_id, pipeline)
+    try:
+        pipeline_req = pool.request('GET', pipeline_url, **request_kwargs)
+    except HTTPError:
+        print('error: there were an error while fetching the gitlab-ci pipeline\'s details. '
+              'Please make sure your identification token is right and try again.')
+        return {}
+    if pipeline_req.status != 200:
+        print('error: there were an error while fetching the gitlab-ci pipeline\'s details. '
+              'Please make sure your identification token is right and try again.')
+        return {}
+    infos = json.loads(pipeline_req.data)
+    return infos
+
+
+def get_pipeline_jobs(project_id, pipeline, kinds=None, **request_kwargs):
+    pool = gitlab_ci_http_pool
+    jobs_url = '/api/v4/projects/{}/pipelines/{}/jobs'.format(project_id, pipeline)
+    jobs = request_paginated_url(pool, jobs_url, **request_kwargs)
+    for job in jobs:
+        # Remove the unnecessary 'pipeline' field
+        del job['pipeline']
+        # Backup the actual name and set the job's kind and name depending on the naming scheme
+        job['full_name'] = job['name']
+        job['kind'] = 'other'
+    # Filter failed jobs then sort them
+    jobs = (job for job in jobs if job['status'] == 'failed')
+    if kinds:
+        jobs = (job for job in jobs if job['kind'] in kinds)
+    return sorted(jobs, key=lambda j: (j['name'], j['kind']))
+
+
+def get_gitlab_ci_pipelines(http_pool, process_pool, project_id, date, date_range=None):
+    '''
+    Returns all gitlab-ci pipelines ran after the giver `min_date`
+    '''
+    date_to = date
+    date_from = date - date_range if date_range else date
+    per_page = process_pool._processes
+    if per_page > 100:
+        raise ValueError('gitlab-ci supports a max pagination of 100 results per page. '
+                         'The pool passed as argument must have 100 or less processes.')
+    fields = {'per_page': per_page}
+    url = '/api/v4/projects/{}/pipelines'.format(project_id)
+    page = 1
+    pipelines = []
+
+    # Fetch pipelines `nb_processes` by `nb_processes` until `from_date` in reached
+    while True:
+        # List pipelines
+        fields['page'] = page
+        page += 1
+        try:
+            cur_pipelines = http_pool.request('GET', url, fields=fields)
+        except HTTPError:
+            print('error: there were an error while fetching the gitlab-ci pipelines. '
+                  'Please make sure your identification token is right and try again.')
+            return pipelines
+        if cur_pipelines.status != 200:
+            print('error: there were an error while fetching the gitlab-ci pipeline\'s details. '
+                  'Please make sure your identification token is right and try again.')
+            return pipelines
+        cur_pipelines = json.loads(cur_pipelines.data)
+
+        # Handle cases where the requested date is older than all pipelines
+        if len(cur_pipelines) == 0:
+            return pipelines
+
+        # Request more infos about each pipeline (for instance the date)
+        pipelines_infos = [process_pool.apply_async(get_pipeline_details, (project_id, p['id']))
+                           for p in cur_pipelines]
+        pipelines_infos = [p.get() for p in pipelines_infos]
+
+        # Add pipeline in the right period, break once the from_date is found
+        for p in pipelines_infos:
+            cur_date = datetime.strptime(p['finished_at'], '%Y-%m-%dT%H:%M:%S.%fZ').date()
+            if cur_date < date_from:
+                return pipelines
+            if cur_date > date_to:
+                continue
+            pipelines.append(p)
+
+    return pipelines
+
+
+def get_gitlab_ci(branches, date, date_range=None, kinds=None):
+    '''
+    Retreive failed jobs from gitlab-ci grouped by pipelines
+    for the specified date.
+    If `kinds` list is provided, only jobs matching one of the
+    given kind will be returned.
+    '''
+    header = {'PRIVATE-TOKEN': localconfig.gitlab_user_token}
+    project_id = localconfig.gitlab_project_id
+    failed_filter = {'fields': {'status': 'failed'}}
+
+    # Build pools
+    global gitlab_ci_http_pool
+    gitlab_ci_http_pool = HTTPSConnectionPool('gitlab.com', port=443, cert_reqs='CERT_REQUIRED',
+                                              ca_certs=certifi.where(), timeout=30, headers=header)
+    process_pool = ProcessPool(processes=64)
+
+    # Fetch pipelines 64 by 64 until `date_from` in reached
+    pipelines = get_gitlab_ci_pipelines(gitlab_ci_http_pool, process_pool, project_id, date, date_range)
+
+    # Drop unnecessary branches
+    pipelines = [p for p in pipelines if p['ref'] in branches]
+
+    # Get jobs ran by those pipelines
+    pipelines_jobs = [process_pool.apply_async(get_pipeline_jobs, (project_id, p['id'], kinds),
+                                               failed_filter) for p in pipelines]
+    for pipeline, jobs in zip(pipelines, pipelines_jobs):
+        pipeline['jobs'] = jobs.get()
+
+    # Group pipelines by branch
+    pipelines = {branch: [p for p in pipelines if p['ref'] == branch] for branch in branches}
+    return pipelines
+
+
 def calculate_notifications(results, outdated_pkgs):
     '''
     Prepare the notifications{} dict for the notifications to individual
-- 
2.21.0

[Buildroot] [PATCH buildroot-test 02/10] utils/daily-mail: new gitlab-ci data: defconfigs

[Victor Huesca]

This patch adds defconfigs to the gitlab-ci data introduced by the
previous patch.

Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
---
 utils/daily-mail | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/utils/daily-mail b/utils/daily-mail
index 2d4ed1c..9dc9729 100755
--- a/utils/daily-mail
+++ b/utils/daily-mail
@@ -528,8 +528,12 @@ def get_pipeline_jobs(project_id, pipeline, kinds=None, **request_kwargs):
         # Remove the unnecessary 'pipeline' field
         del job['pipeline']
         # Backup the actual name and set the job's kind and name depending on the naming scheme
-        job['full_name'] = job['name']
-        job['kind'] = 'other'
+        name = job['full_name'] = job['name']
+        if name.endswith('_defconfig'):
+            job['kind'] = 'defconfig'
+            job['name'] = name[:-10]
+        else:
+            job['kind'] = 'other'
     # Filter failed jobs then sort them
     jobs = (job for job in jobs if job['status'] == 'failed')
     if kinds:
-- 
2.21.0

[Buildroot] [PATCH buildroot-test 03/10] utils/daily-mail: new gitlab-ci data: runtime tests

[Victor Huesca]

This patch adds runtime tests to the recognized gitlab-ci results.

Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
---
 utils/daily-mail | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/utils/daily-mail b/utils/daily-mail
index 9dc9729..53500a6 100755
--- a/utils/daily-mail
+++ b/utils/daily-mail
@@ -532,6 +532,9 @@ def get_pipeline_jobs(project_id, pipeline, kinds=None, **request_kwargs):
         if name.endswith('_defconfig'):
             job['kind'] = 'defconfig'
             job['name'] = name[:-10]
+        elif name.startswith('tests.'):
+            job['kind'] = 'runtime-test'
+            job['name'] = name.split('.')[-1]
         else:
             job['kind'] = 'other'
     # Filter failed jobs then sort them
-- 
2.21.0

[Buildroot] [PATCH buildroot-test 04/10] utils/daily-mail: add defconfig support in developers notifications

[Victor Huesca]

This patch adds defconfig job failures in the 'Notification' structure.
This will allow to inform developers about failures related to the
defconfigs they maintains.

This patch relys on the gitlab-ci infrasture introduced earlier on in
this series as well as the patch adding support for defconfig in the
getdeveloperlib already applied to master.

Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
---
 utils/daily-mail | 28 ++++++++++++++++++++++++++--
 1 file changed, 26 insertions(+), 2 deletions(-)

diff --git a/utils/daily-mail b/utils/daily-mail
index 53500a6..5a44fad 100755
--- a/utils/daily-mail
+++ b/utils/daily-mail
@@ -77,6 +77,7 @@ class Notification:
     def __init__(self):
         self.arch_notifications = defaultdict(list)
         self.package_notifications = defaultdict(list)
+        self.defconfig_notifications = defaultdict(list)
         self.package_version_notification = []
 
 
@@ -166,6 +167,23 @@ def add_package_notification(branch, notifications, build_result):
     build_result['orphan'] = orphan
 
 
+def add_defconfig_notification(branch, notifications, job):
+    '''
+    Add to the notifications{} dict notifications that are related to
+    defconfig "maintainers".
+    '''
+    orphan = True
+    for dev in developers:
+        if job['name'] in dev.defconfigs:
+            orphan = False
+            notif = notifications.setdefault(dev, Notification())
+            notif.defconfig_notifications[branch].append(job)
+    if orphan:
+        job['orphan'] = True
+        notif = notifications.setdefault(get_orphan_developer(), Notification())
+        notif.defconfig_notifications[branch].append(job)
+
+
 def add_outdated_pkg_notification(notifications, package):
     '''
     Add to the notifications{} dict notifications that are related to
@@ -631,7 +649,7 @@ def get_gitlab_ci(branches, date, date_range=None, kinds=None):
     return pipelines
 
 
-def calculate_notifications(results, outdated_pkgs):
+def calculate_notifications(results, outdated_pkgs, gitlab_results):
     '''
     Prepare the notifications{} dict for the notifications to individual
     developers, based on architecture developers, package developers,
@@ -649,6 +667,12 @@ def calculate_notifications(results, outdated_pkgs):
     for pkg in outdated_pkgs:
         add_outdated_pkg_notification(notifications, pkg)
 
+    for branch, pipelines in gitlab_results.iteritems():
+        for pipeline in pipelines:
+            for job in pipeline['jobs']:
+                if job['kind'] == 'defconfig':
+                    add_defconfig_notification(branch, notifications, job)
+
     return notifications
 
 
@@ -691,7 +715,7 @@ def __main__():
         results = {}
         results_by_reason = {}
     outdated_pkgs = get_outdated_pkgs(localconfig.pkg_stats) if 'outdated' in email_data else []
-    notifications = calculate_notifications(results, outdated_pkgs)
+    notifications = calculate_notifications(results, outdated_pkgs, {})
     smtp = smtplib.SMTP(localconfig.smtphost, localconfig.smtpport)
     smtp.starttls()
     smtp.login(localconfig.smtpuser, localconfig.smtppass)
-- 
2.21.0

[Buildroot] [PATCH buildroot-test 05/10] utils/daily-mail: add runtime tests support in developers notifications

[Victor Huesca]

This patch adds runtime-test job failures in the 'Notification'
structure.
This will allow to inform developers about failures related to the
runtime-tests they maintains.

This patch relys on the gitlab-ci infrasture introduced earlier on in
this series as well as the patch adding support for runtime-tests in the
getdeveloperlib already applied to master.

Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
---
 utils/daily-mail | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/utils/daily-mail b/utils/daily-mail
index 5a44fad..5e8799a 100755
--- a/utils/daily-mail
+++ b/utils/daily-mail
@@ -78,6 +78,7 @@ class Notification:
         self.arch_notifications = defaultdict(list)
         self.package_notifications = defaultdict(list)
         self.defconfig_notifications = defaultdict(list)
+        self.runtime_test_notifications = defaultdict(list)
         self.package_version_notification = []
 
 
@@ -184,6 +185,23 @@ def add_defconfig_notification(branch, notifications, job):
         notif.defconfig_notifications[branch].append(job)
 
 
+def add_runtime_test_notification(branch, notifications, job):
+    '''
+    Add to the notifications{} dict notifications that are related to
+    runtime-tests "maintainers".
+    '''
+    orphan = True
+    for dev in developers:
+        if job['full_name'] in dev.runtime_tests:
+            orphan = False
+            notif = notifications.setdefault(dev, Notification())
+            notif.runtime_test_notifications[branch].append(job)
+    if orphan:
+        job['orphan'] = True
+        notif = notifications.setdefault(get_orphan_developer(), Notification())
+        notif.runtime_test_notifications[branch].append(job)
+
+
 def add_outdated_pkg_notification(notifications, package):
     '''
     Add to the notifications{} dict notifications that are related to
@@ -672,6 +690,8 @@ def calculate_notifications(results, outdated_pkgs, gitlab_results):
             for job in pipeline['jobs']:
                 if job['kind'] == 'defconfig':
                     add_defconfig_notification(branch, notifications, job)
+                elif job['kind'] == 'runtime-test':
+                    add_runtime_test_notification(branch, notifications, job)
 
     return notifications
 
-- 
2.21.0

[Buildroot] [PATCH buildroot-test 06/10] utils/daily-mail: prepare the mail formatting to support gitlab-ci results

[Victor Huesca]

This patch allows to add gitlab-ci related results to mails.

Since every gitlab-ci result share the same information, a single
function was preferred to factorize this part for now (it could be split
once different gitlab-ci results have specific formatting).

Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
---
 utils/daily-mail | 68 ++++++++++++++++++++++++++++++++++++++----------
 1 file changed, 54 insertions(+), 14 deletions(-)

diff --git a/utils/daily-mail b/utils/daily-mail
index 5e8799a..9f0673b 100755
--- a/utils/daily-mail
+++ b/utils/daily-mail
@@ -246,6 +246,12 @@ def shrink_str(string, length, align='right', fill='...'):
         raise ValueError('align must be either `left`, `right` or `center`')
 
 
+def fmt_duration(**timedelta_args):
+    '''Small helper to format a duration'''
+    t = datetime(1990, 1, 1) + timedelta(**timedelta_args)
+    return t.strftime('%Hh%Mm%Ss').lstrip('0:hms')
+
+
 def show_results(results, show_status, show_orphan=False):
     contents = ""
     # Header
@@ -314,6 +320,32 @@ def show_outdated(packages, show_orphan=False):
     return contents
 
 
+def show_gitlab_ci_jobs(jobs, job_column_description='test-name', job_column_length=40, show_orphan=False):
+    # Header
+    contents = ''
+    if show_orphan:
+        contents += '{:>{}} | {:^61} | {:>8} | {:^5}\n'.format(job_column_description, job_column_length,
+                                                               'link to the job', 'duration', 'orph?')
+        contents += '{0:->{1}}-+-{0:-^61}-+-{0:->8} | {0:-^5}\n'.format('', job_column_length)
+    else:
+        contents += '{:>{}} | {:^61} | {:>8}\n'.format(job_column_description, job_column_length,
+                                                       'link to the job', 'duration')
+        contents += '{0:->{1}}-+-{0:-^61}-+-{0:->8}\n'.format('', job_column_length)
+
+    # Data
+    for job in sorted(jobs, key=lambda j: (j['kind'], j['name'])):
+        name = shrink_str(job['name'], job_column_length, align='left')
+        duration = fmt_duration(seconds=job['duration'])
+        orphan = 'ORPH' if job.get('orphan') else ''
+
+        contents += '{:>{}} | {:^61} | {:>8}'.format(name, job_column_length, job['web_url'], duration)
+        if show_orphan:
+            contents += ' | {:^5}'.format(orphan)
+        contents += '\n'
+
+    return contents
+
+
 def developers_email(smtp, branches, notifications, datestr, dry_run):
     '''
     Send the e-mails to the individual developers
@@ -387,10 +419,8 @@ def developers_email(smtp, branches, notifications, datestr, dry_run):
             print "To: {}".format(dev.name)
 
 
-def global_email_branch_result(results, results_by_reason, branch):
-    contents = "Results for branch '{}'\n".format(branch)
-    contents += "=====================" + "=" * len(branch) + "\n\n"
-    contents += "Classification of failures by reason\n"
+def global_email_branch_result(results, results_by_reason):
+    contents = "Classification of failures by reason\n"
     contents += "------------------------------------\n\n"
     for r in results_by_reason:
         reason = shrink_str(r['reason'], 30)
@@ -400,11 +430,14 @@ def global_email_branch_result(results, results_by_reason, branch):
     contents += "Detail of failures\n"
     contents += "------------------\n\n"
     contents += show_results(results, show_status=True, show_orphan=True)
-    contents += "\n"
     return contents
 
 
-def global_email(smtp, results, results_by_reason, datestr, overall, outdated_pkgs, dry_run):
+def global_email_branch_gitlab_ci(pipelines):
+    pass  # Implement this to support gitlab-ci related results
+
+
+def global_email(smtp, branches, results, results_by_reason, datestr, overall, outdated_pkgs, gitlab_ci, dry_run):
     '''
     Send the global e-mail to the mailing list
     '''
@@ -421,14 +454,21 @@ def global_email(smtp, results, results_by_reason, datestr, overall, outdated_pk
         if stats[3] == 0:
             continue
         contents += "  {:^10} | {:^3} | {:^3} | {:^3} | {:^3} |\n".format(branch, stats[0], stats[1], stats[2], stats[3])
-    if len(overall) != 0:
+    if any(map(len, (overall, gitlab_ci))):
         contents += "\n"
-    for branch in results.keys():
-        if len(results[branch]) == 0:
+    for branch in branches:
+        res = results.get(branch, [])
+        git = gitlab_ci.get(branch, [])
+        if not any(map(len, (res, git))):
             continue
-        contents += global_email_branch_result(results[branch], results_by_reason[branch], branch)
-    if len(overall) != 0:
-        contents += "\n"
+        contents += "Results for branch '{}'\n".format(branch)
+        contents += "=====================" + "=" * len(branch) + "\n\n"
+        if len(res) != 0:
+            contents += global_email_branch_result(res, results_by_reason[branch])
+            contents += "\n\n"
+        if len(git) != 0:
+            contents += global_email_branch_gitlab_ci(git)
+            contents += "\n\n"
     if outdated_pkgs:
         contents += "Packages having a newer version\n"
         contents += "===============================\n\n"
@@ -742,8 +782,8 @@ def __main__():
     if 'dev' in email_dest:
         developers_email(smtp, branches, notifications, date_str, args.dry_run)
     if 'global' in email_dest:
-        global_email(smtp, results, results_by_reason, date_str,
-                     overall_stats, outdated_pkgs, args.dry_run)
+        global_email(smtp, branches, results, results_by_reason, date_str,
+                     overall_stats, outdated_pkgs, {}, args.dry_run)
     smtp.quit()
 
 
-- 
2.21.0

[Buildroot] [PATCH buildroot-test 07/10] utils/daily-mail: add formatting for defconfig results

[Victor Huesca]

This patch uses the gitlab-ci results generic function to format
defconfigs in mails.

Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
---
 utils/daily-mail | 33 ++++++++++++++++++++++++---------
 1 file changed, 24 insertions(+), 9 deletions(-)

diff --git a/utils/daily-mail b/utils/daily-mail
index 9f0673b..f8a4bfb 100755
--- a/utils/daily-mail
+++ b/utils/daily-mail
@@ -355,14 +355,14 @@ def developers_email(smtp, branches, notifications, datestr, dry_run):
         email_from = localconfig.fromaddr
         subject = "[{}] Your daily results for {}".format(baseurl, datestr)
         contents = "Hello,\n\n"
-
-        if notif.arch_notifications or notif.package_notifications:
-            contents += "Autobuilder failures\n"
-            contents += "====================\n\n"
+        if notif.arch_notifications or notif.package_notifications or notif.defconfig_notifications:
+            contents += "Recent build failures\n"
+            contents += "=====================\n\n"
             contents += textwrap.fill("This is the list of Buildroot build failures that occurred on {}, "
-                                      "and for which you are a registered architecture developer or package "
-                                      "developer. Please help us improving the quality of Buildroot by "
-                                      "investigating those build failures and sending patches to fix them. "
+                                      "and for which you are a registered architecture developer, package "
+                                      "developer or defconfig developer. Please help us improving the "
+                                      "quality of Buildroot by investigating those build failures and "
+                                      "sending patches to fix them. "
                                       "Thanks!".format(datestr))
             contents += "\n\n"
         show_orphan = dev.name == ORPHAN_DEVELOPER
@@ -370,8 +370,9 @@ def developers_email(smtp, branches, notifications, datestr, dry_run):
         for branch in branches:
             archs = notif.arch_notifications.get(branch, [])
             packages = notif.package_notifications.get(branch, [])
+            defconfigs = notif.defconfig_notifications.get(branch, [])
 
-            if len(archs) == 0 and len(packages) == 0:
+            if not any(map(len, (archs, packages, defconfigs))):
                 continue
 
             contents += "Results for the '{}' branch\n".format(branch)
@@ -384,6 +385,11 @@ def developers_email(smtp, branches, notifications, datestr, dry_run):
             if len(packages) != 0:
                 contents += "Build failures related to your packages:\n\n"
                 contents += show_results(packages, show_status=False, show_orphan=show_orphan)
+                contents += "\n\n"
+            if len(defconfigs) != 0:
+                contents += "Build failures related to your defconfigs:\n\n"
+                contents += show_gitlab_ci_jobs(defconfigs, job_column_description='defconfig',
+                                                job_column_length=33, show_orphan=show_orphan)
 
             contents += "\n"
 
@@ -434,7 +440,16 @@ def global_email_branch_result(results, results_by_reason):
 
 
 def global_email_branch_gitlab_ci(pipelines):
-    pass  # Implement this to support gitlab-ci related results
+    defconfigs = [j for p in pipelines for j in p['jobs'] if j['kind'] == 'defconfig']
+
+    contents = ""
+    if len(defconfigs) != 0:
+        contents += 'Detail of defconfig failures\n'
+        contents += '----------------------------\n\n'
+        contents += show_gitlab_ci_jobs(defconfigs, job_column_description='defconfig',
+                                        job_column_length=33, show_orphan=True)
+        contents += "\n\n"
+    return contents
 
 
 def global_email(smtp, branches, results, results_by_reason, datestr, overall, outdated_pkgs, gitlab_ci, dry_run):
-- 
2.21.0

[Buildroot] [PATCH buildroot-test 08/10] utils/daily-mail: add formatting for runtime-test results

[Victor Huesca]

This patch uses the gitlab-ci results generic function to format
runtime-test in mails.

Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
---
 utils/daily-mail | 32 +++++++++++++++++++++++++-------
 1 file changed, 25 insertions(+), 7 deletions(-)

diff --git a/utils/daily-mail b/utils/daily-mail
index f8a4bfb..0510c07 100755
--- a/utils/daily-mail
+++ b/utils/daily-mail
@@ -355,14 +355,19 @@ def developers_email(smtp, branches, notifications, datestr, dry_run):
         email_from = localconfig.fromaddr
         subject = "[{}] Your daily results for {}".format(baseurl, datestr)
         contents = "Hello,\n\n"
-        if notif.arch_notifications or notif.package_notifications or notif.defconfig_notifications:
-            contents += "Recent build failures\n"
-            contents += "=====================\n\n"
+
+        archs = notif.arch_notifications
+        packages = notif.package_notifications
+        defconfigs = notif.defconfig_notifications
+        runtime_tests = notif.runtime_test_notifications
+        if archs or packages or defconfigs or runtime_tests:
+            contents += "Recent build failures and runtime-tests failures\n"
+            contents += "================================================\n\n"
             contents += textwrap.fill("This is the list of Buildroot build failures that occurred on {}, "
                                       "and for which you are a registered architecture developer, package "
-                                      "developer or defconfig developer. Please help us improving the "
-                                      "quality of Buildroot by investigating those build failures and "
-                                      "sending patches to fix them. "
+                                      "developer or defconfig developer. This list also include runtime "
+                                      "tests failures. Please help us improving the quality of Buildroot by "
+                                      "investigating those build failures and sending patches to fix them. "
                                       "Thanks!".format(datestr))
             contents += "\n\n"
         show_orphan = dev.name == ORPHAN_DEVELOPER
@@ -371,8 +376,9 @@ def developers_email(smtp, branches, notifications, datestr, dry_run):
             archs = notif.arch_notifications.get(branch, [])
             packages = notif.package_notifications.get(branch, [])
             defconfigs = notif.defconfig_notifications.get(branch, [])
+            runtimes = notif.runtime_test_notifications.get(branch, [])
 
-            if not any(map(len, (archs, packages, defconfigs))):
+            if not any(map(len, (archs, packages, defconfigs, runtimes))):
                 continue
 
             contents += "Results for the '{}' branch\n".format(branch)
@@ -390,6 +396,11 @@ def developers_email(smtp, branches, notifications, datestr, dry_run):
                 contents += "Build failures related to your defconfigs:\n\n"
                 contents += show_gitlab_ci_jobs(defconfigs, job_column_description='defconfig',
                                                 job_column_length=33, show_orphan=show_orphan)
+                contents += "\n\n"
+            if len(runtimes) != 0:
+                contents += "Build failures related to your runtime tests:\n\n"
+                contents += show_gitlab_ci_jobs(runtimes, job_column_description='runtime-test',
+                                                job_column_length=25, show_orphan=show_orphan)
 
             contents += "\n"
 
@@ -441,6 +452,7 @@ def global_email_branch_result(results, results_by_reason):
 
 def global_email_branch_gitlab_ci(pipelines):
     defconfigs = [j for p in pipelines for j in p['jobs'] if j['kind'] == 'defconfig']
+    runtime_tests = [j for p in pipelines for j in p['jobs'] if j['kind'] == 'runtime-test']
 
     contents = ""
     if len(defconfigs) != 0:
@@ -449,6 +461,12 @@ def global_email_branch_gitlab_ci(pipelines):
         contents += show_gitlab_ci_jobs(defconfigs, job_column_description='defconfig',
                                         job_column_length=33, show_orphan=True)
         contents += "\n\n"
+    if len(runtime_tests) != 0:
+        contents += "Detail of runtime-test failures\n"
+        contents += "-------------------------------\n\n"
+        contents += show_gitlab_ci_jobs(runtime_tests, job_column_description='runtime-test',
+                                        job_column_length=25, show_orphan=True)
+        contents += "\n\n"
     return contents
 
 
-- 
2.21.0

[Buildroot] [PATCH buildroot-test 09/10] utils/daily-mail: add argument for gitlab-ci related data

[Victor Huesca]

This patch provides two new data to the args parser to allow enabling
defconfigs, runtime-tests and sanity checks in global/developer emails.
These tree data are grouped together in a gitlab-ci data group as they
are fetched at the same time.

This new list will be use by the next patch to enable gitlab-ci jobs in
email.

Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
---
 utils/daily-mail | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)

diff --git a/utils/daily-mail b/utils/daily-mail
index 0510c07..6b0a1b4 100755
--- a/utils/daily-mail
+++ b/utils/daily-mail
@@ -776,7 +776,8 @@ def parse_args():
     parser.add_argument('--branches', action='store', nargs='+', help='List of branches (blank separated)')
     parser.add_argument('--dest', action='store', nargs='+', choices=['dev', 'global'],
                         help='List of emails type to send (ie. global, dev)')
-    parser.add_argument('--data', action='store', nargs='+', choices=['autobuild', 'outdated'],
+    parser.add_argument('--data', action='store', nargs='+',
+                        choices=['autobuild', 'outdated', 'defconfig', 'runtime-test'],
                         help='List of information to add in emails (blank separated)')
     args = parser.parse_args()
     if args.date and not RE_DATE.match(args.date):
@@ -787,13 +788,13 @@ def parse_args():
 def __main__():
     args = parse_args()
     if args.date:
-        date_str = args.date
+        date_struct = datetime.strptime(args.date, '%Y-%m-%d').date()
     else:
-        yesterday = date.today() - timedelta(1)
-        date_str = yesterday.strftime('%Y-%m-%d')
-    branches = args.branches if args.branches else get_branches()
+        date_struct = date.today() - timedelta(1)
+    date_str = date_struct.isoformat()
+    branches = args.branches or get_branches()
     email_dest = set(args.dest) if args.dest else {'dev', 'global'}
-    email_data = set(args.data) if args.data else {'autobuild', 'outdated'}
+    email_data = set(args.data) if args.data else {'autobuild', 'outdated', 'defconfig', 'runtime-test'}
 
     db = _mysql.connect(host=localconfig.host,
                         user=localconfig.user,
-- 
2.21.0

[Buildroot] [PATCH buildroot-test 10/10] utils/daily-mail: enable gitlab-ci emails

[Victor Huesca]

This patch enable all gitlab-ci related results added by previous
patches from the series in both the global and per-developer emails.

Defconfigs and runtime-tests entries are added in the per-branch results
for both global and dev-mails.

Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
---
 utils/daily-mail | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/utils/daily-mail b/utils/daily-mail
index 6b0a1b4..c249376 100755
--- a/utils/daily-mail
+++ b/utils/daily-mail
@@ -809,7 +809,11 @@ def __main__():
         results = {}
         results_by_reason = {}
     outdated_pkgs = get_outdated_pkgs(localconfig.pkg_stats) if 'outdated' in email_data else []
-    notifications = calculate_notifications(results, outdated_pkgs, {})
+    if email_data & {'defconfig', 'runtime-test'}:
+        gitlab_ci = get_gitlab_ci(branches, date_struct, kinds=email_data)
+    else:
+        gitlab_ci = {}
+    notifications = calculate_notifications(results, outdated_pkgs, gitlab_ci)
     smtp = smtplib.SMTP(localconfig.smtphost, localconfig.smtpport)
     smtp.starttls()
     smtp.login(localconfig.smtpuser, localconfig.smtppass)
@@ -817,7 +821,7 @@ def __main__():
         developers_email(smtp, branches, notifications, date_str, args.dry_run)
     if 'global' in email_dest:
         global_email(smtp, branches, results, results_by_reason, date_str,
-                     overall_stats, outdated_pkgs, {}, args.dry_run)
+                     overall_stats, outdated_pkgs, gitlab_ci, args.dry_run)
     smtp.quit()
 
 
-- 
2.21.0

[Buildroot] [PATCH buildroot-test 04/10] utils/daily-mail: add defconfig support in developers notifications

[Titouan Christophe]

Hello Victor and all,

On 8/29/19 3:23 PM, Victor Huesca wrote:
> This patch adds defconfig job failures in the 'Notification' structure.
> This will allow to inform developers about failures related to the
> defconfigs they maintains.
> 
> This patch relys on the gitlab-ci infrasture introduced earlier on in
> this series as well as the patch adding support for defconfig in the
> getdeveloperlib already applied to master.
> 
> Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
> ---
>   utils/daily-mail | 28 ++++++++++++++++++++++++++--
>   1 file changed, 26 insertions(+), 2 deletions(-)
> 
> diff --git a/utils/daily-mail b/utils/daily-mail
> index 53500a6..5a44fad 100755
> --- a/utils/daily-mail
> +++ b/utils/daily-mail
> @@ -77,6 +77,7 @@ class Notification:
>       def __init__(self):
>           self.arch_notifications = defaultdict(list)
>           self.package_notifications = defaultdict(list)
> +        self.defconfig_notifications = defaultdict(list)
>           self.package_version_notification = []
>   
>   
> @@ -166,6 +167,23 @@ def add_package_notification(branch, notifications, build_result):
>       build_result['orphan'] = orphan
>   
>   
> +def add_defconfig_notification(branch, notifications, job):
> +    '''
> +    Add to the notifications{} dict notifications that are related to
> +    defconfig "maintainers".
> +    '''
> +    orphan = True
> +    for dev in developers:
> +        if job['name'] in dev.defconfigs:
> +            orphan = False
> +            notif = notifications.setdefault(dev, Notification())
> +            notif.defconfig_notifications[branch].append(job)
> +    if orphan:
> +        job['orphan'] = True
> +        notif = notifications.setdefault(get_orphan_developer(), Notification())
> +        notif.defconfig_notifications[branch].append(job)
> +
> +
>   def add_outdated_pkg_notification(notifications, package):
>       '''
>       Add to the notifications{} dict notifications that are related to
> @@ -631,7 +649,7 @@ def get_gitlab_ci(branches, date, date_range=None, kinds=None):
>       return pipelines
>   
>   
> -def calculate_notifications(results, outdated_pkgs):
> +def calculate_notifications(results, outdated_pkgs, gitlab_results):
>       '''
>       Prepare the notifications{} dict for the notifications to individual
>       developers, based on architecture developers, package developers,
> @@ -649,6 +667,12 @@ def calculate_notifications(results, outdated_pkgs):
>       for pkg in outdated_pkgs:
>           add_outdated_pkg_notification(notifications, pkg)
>   
> +    for branch, pipelines in gitlab_results.iteritems():

Not sure if relevant, but dict.iteritems() does not exists anymore in 
Py3, and dict.items() now returns a "view" 
(https://docs.python.org/3.0/whatsnew/3.0.html#views-and-iterators-instead-of-lists).

> +        for pipeline in pipelines:
> +            for job in pipeline['jobs']:
> +                if job['kind'] == 'defconfig':
> +                    add_defconfig_notification(branch, notifications, job)
> +
>       return notifications
>   
>   
> @@ -691,7 +715,7 @@ def __main__():
>           results = {}
>           results_by_reason = {}
>       outdated_pkgs = get_outdated_pkgs(localconfig.pkg_stats) if 'outdated' in email_data else []
> -    notifications = calculate_notifications(results, outdated_pkgs)
> +    notifications = calculate_notifications(results, outdated_pkgs, {})
>       smtp = smtplib.SMTP(localconfig.smtphost, localconfig.smtpport)
>       smtp.starttls()
>       smtp.login(localconfig.smtpuser, localconfig.smtppass)
> 

Regards,

Titouan

[Buildroot] [PATCH buildroot-test 00/10] add gitlab-ci jobs to the daily-mail

[Thomas Petazzoni]

Hello Victor,

On Thu, 29 Aug 2019 15:23:38 +0200
Victor Huesca <victor.huesca@bootlin.com> wrote:

> Victor Huesca (10):
>   utils/daily-mail: new data: gitlab-ci jobs
>   utils/daily-mail: new gitlab-ci data: defconfigs
>   utils/daily-mail: new gitlab-ci data: runtime tests
>   utils/daily-mail: add defconfig support in developers notifications
>   utils/daily-mail: add runtime tests support in developers
>     notifications
>   utils/daily-mail: prepare the mail formatting to support gitlab-ci
>     results
>   utils/daily-mail: add formatting for defconfig results
>   utils/daily-mail: add formatting for runtime-test results
>   utils/daily-mail: add argument for gitlab-ci related data
>   utils/daily-mail: enable gitlab-ci emails

Thanks for this patch series! I have made a number of changes, and
applied it.

The main changes are:

 - More filtering of pipelines: only failed pipelines are retrieved

 - Earlier filtering of pipelines: we can filter pipelines by branch
   and status before getting the details for each pipeline

 - Removal of the job duration in the e-mails, it is not a very useful
   information IMO

 - Further splitting of some of the patches

So, hopefully this Saturday we should see e-mails about defconfig
failures, since we have at least one still failing on master.

Thanks a lot for this work!

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com