[Buildroot] [PATCH 0/2] gitlab-ci: fix and handle the weekly scheduled pipeline (branch yem/gitlab-ci-schedule)

[Buildroot] [PATCH 0/2] gitlab-ci: fix and handle the weekly scheduled pipeline (branch yem/gitlab-ci-schedule)

[Yann E. MORIN]

Hello All!

Our weekly scheduled pipeline no longer builds the defconfigs (it only
checks them for validity), and also no longer builds the runtime tests
either.

This seems to have been caused by the fact that a scheduled pipeline is
(no longer) recognised as a triggerred pipeline by gitlab-CI, and so
does not match our current heurostocs.

This series fixes the priority of pipeline creation conditions, and adds
the scheduled condition, both to the defconfigs and the runtime tests.

note however that the jobs limit if 500 on gitlab.com, and the current
weekly schedule would created ~588 jobs. As a consequence, the pipeline
*will* fail.

But until we find a better solution, let's at least fix the conditions
under which it should fail! :-]

Here is a result of the new scheduled pipeline:
    https://gitlab.com/ymorin/buildroot/-/pipelines/192690611 (parent)
    https://gitlab.com/ymorin/buildroot/-/pipelines/192690931 (child)


Regards,
Yann E. MORIN.


The following changes since commit bb216e3d8ade12442db794cbe5780e53bc082be8

  package/sentry-native: fix double variable assignment (2020-09-21 21:39:36 +0200)


are available in the git repository at:

  https://git.busybox.net/~ymorin/git/buildroot

for you to fetch changes up to d01c55362b599b6b41c5e914f46644ac4088e243

  support/scripts: handle scheduled pipelines (2020-09-21 22:50:45 +0200)


----------------------------------------------------------------
Yann E. MORIN (2):
      support/scripts: prioritize conditions for pipeline creation
      support/scripts: handle scheduled pipelines

 support/scripts/generate-gitlab-ci-yml | 64 +++++++++++++++++++---------------
 1 file changed, 35 insertions(+), 29 deletions(-)

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 1/2] support/scripts: prioritize conditions for pipeline creation

[Yann E. MORIN]

When multiple conditions match simultaneously, even though that should
not happen in practice, we want the more "important" one to win over
the less "important" ones. For example, a tag is more important than a
branch name or a trigger.

Currently, the latest condition to match takes precendence over any
previous one, while we want the exact opposite.

Fix that with proper fallbacks in else-blocks.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
---
 support/scripts/generate-gitlab-ci-yml | 58 +++++++++++++-------------
 1 file changed, 29 insertions(+), 29 deletions(-)

diff --git a/support/scripts/generate-gitlab-ci-yml b/support/scripts/generate-gitlab-ci-yml
index e42943953c..faa52e85b2 100755
--- a/support/scripts/generate-gitlab-ci-yml
+++ b/support/scripts/generate-gitlab-ci-yml
@@ -49,26 +49,26 @@ gen_defconfigs() {
     if [ -n "${CI_COMMIT_TAG}" ]; then
         # For tags, create a pipeline.
         template=base
-    fi
-    if [ -n "${CI_PIPELINE_TRIGGERED}" ]; then
+    elif [ -n "${CI_PIPELINE_TRIGGERED}" ]; then
         # For pipeline created by using a trigger token.
         template=base
+    else
+        case "${CI_COMMIT_REF_NAME}" in
+            # For master, next, and maintenance branches, only check the defconfigs
+            (master|next|????.??.x)
+                template=check
+                ext=_check
+            ;;
+            # For the branch or tag name named *-defconfigs, create a pipeline.
+            (*-defconfigs)
+                template=base
+            ;;
+            (*-*_defconfig)
+                defconfigs=( "${CI_COMMIT_REF_NAME##*-}" )
+                template=base
+            ;;
+        esac
     fi
-    case "${CI_COMMIT_REF_NAME}" in
-        # For master, next, and maintenance branches, only check the defconfigs
-        (master|next|????.??.x)
-            template=check
-            ext=_check
-        ;;
-        # For the branch or tag name named *-defconfigs, create a pipeline.
-        (*-defconfigs)
-            template=base
-        ;;
-        (*-*_defconfig)
-            defconfigs=( "${CI_COMMIT_REF_NAME##*-}" )
-            template=base
-        ;;
-    esac
 
     if [ -n "${template}" ]; then
         for cfg in "${defconfigs[@]}"; do
@@ -91,21 +91,21 @@ gen_tests() {
     if [ -n "${CI_COMMIT_TAG}" ]; then
         # For tags, create a pipeline.
         run_tests=true
-    fi
-    if [ -n "${CI_PIPELINE_TRIGGERED}" ]; then
+    elif [ -n "${CI_PIPELINE_TRIGGERED}" ]; then
         # For pipeline created by using a trigger token.
         run_tests=true
+    else
+        case "${CI_COMMIT_REF_NAME}" in
+            # For the branch or tag name named *-runtime-tests, create a pipeline.
+            (*-runtime-tests)
+                run_tests=true
+            ;;
+            (*-tests.*)
+                tests=( "${CI_COMMIT_REF_NAME##*-}" )
+                run_tests=true
+            ;;
+        esac
     fi
-    case "${CI_COMMIT_REF_NAME}" in
-        # For the branch or tag name named *-runtime-tests, create a pipeline.
-        (*-runtime-tests)
-            run_tests=true
-        ;;
-        (*-tests.*)
-            tests=( "${CI_COMMIT_REF_NAME##*-}" )
-            run_tests=true
-        ;;
-    esac
 
     if ${run_tests}; then
         printf '%s: { extends: .runtime_test_base }\n' "${tests[@]}"
-- 
2.20.1

[Buildroot] [PATCH 2/2] support/scripts: handle scheduled pipelines

[Yann E. MORIN]

We currently only handle triggerred pipelines, i.e. pipelines that
have been created via the PAI with a token.

However, the weekly trigger is not a trigger, it is a schedule.

Unbeknownst to us, it seems the behaviour of scheduled pipeline has
changed, and they are no longer treated as a triggerred pipeline, and
the corresponding variable is not set; instead, the scheduled property
is reflected in the CI_PIPELINE_SOURCE variable, that is aptly set to
'schedule'.

Use that to recognise our weekly scheduled pipeline.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
---
 support/scripts/generate-gitlab-ci-yml | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/support/scripts/generate-gitlab-ci-yml b/support/scripts/generate-gitlab-ci-yml
index faa52e85b2..01d850d09c 100755
--- a/support/scripts/generate-gitlab-ci-yml
+++ b/support/scripts/generate-gitlab-ci-yml
@@ -52,6 +52,9 @@ gen_defconfigs() {
     elif [ -n "${CI_PIPELINE_TRIGGERED}" ]; then
         # For pipeline created by using a trigger token.
         template=base
+    elif [ "${CI_PIPELINE_SOURCE}" = "schedule" ]; then
+        # For scheduled pipelines
+        template=base
     else
         case "${CI_COMMIT_REF_NAME}" in
             # For master, next, and maintenance branches, only check the defconfigs
@@ -94,6 +97,9 @@ gen_tests() {
     elif [ -n "${CI_PIPELINE_TRIGGERED}" ]; then
         # For pipeline created by using a trigger token.
         run_tests=true
+    elif [ "${CI_PIPELINE_SOURCE}" = "schedule" ]; then
+        # For scheduled pipelines
+        run_tests=true
     else
         case "${CI_COMMIT_REF_NAME}" in
             # For the branch or tag name named *-runtime-tests, create a pipeline.
-- 
2.20.1

[Buildroot] [PATCH 1/2] support/scripts: prioritize conditions for pipeline creation

[Thomas Petazzoni]

On Mon, 21 Sep 2020 23:08:41 +0200
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> When multiple conditions match simultaneously, even though that should
> not happen in practice, we want the more "important" one to win over
> the less "important" ones. For example, a tag is more important than a
> branch name or a trigger.
> 
> Currently, the latest condition to match takes precendence over any
> previous one, while we want the exact opposite.
> 
> Fix that with proper fallbacks in else-blocks.
> 
> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> ---
>  support/scripts/generate-gitlab-ci-yml | 58 +++++++++++++-------------
>  1 file changed, 29 insertions(+), 29 deletions(-)

Applied to master, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com