[Buildroot] [PATCH 0/2 v4] gitlab-ci: allow running test-pkg (branch yem/test-pkg-in-gitlab-ci)

[Buildroot] [PATCH 0/2 v4] gitlab-ci: allow running test-pkg (branch yem/test-pkg-in-gitlab-ci)

[Yann E. MORIN]

Hello All!

This series is a respin of the previous work by Romain:
    http://lists.busybox.net/pipermail/buildroot/2021-June/thread.html#313369

Yann has done quite some extensive rework, so much so that it is not
sane to apply without further review.

So, the big changes from Romain's work, are:

  - utils/test-kpkg only lists, does not generate yaml output

  - support/scripts/generate-gitlab-ci-yml generates the yaml blurb from
    that list

  - an empty config fragment is now an error

  - a config fragment that yields no test is also an error

  - some eye-candy and code-styles in the shell scripts...

Regards,
Yann E. MORIN.


----------------------------------------------------------------
Romain Naour (2):
      support/misc/gitlab-ci.yml.in
      utils/test-pkg: add gitlab-ci support

 .gitlab-ci.yml                         |  3 +++
 support/misc/gitlab-ci.yml.in          | 38 ++++++++++++++++++++++++-----
 support/scripts/generate-gitlab-ci-yml | 30 ++++++++++++++++++++++-
 utils/test-pkg                         | 44 ++++++++++++++++++++++++++--------
 4 files changed, 98 insertions(+), 17 deletions(-)

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 1/2 v4] support/misc/gitlab-ci.yml.in

[Yann E. MORIN]

From: Romain Naour <romain.naour@gmail.com>

In a followup commit, the make command used to log and display the last
lines on error will be used in another job.
Factorize it by introducing .run_make template.

Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
---
 support/misc/gitlab-ci.yml.in | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)

diff --git a/support/misc/gitlab-ci.yml.in b/support/misc/gitlab-ci.yml.in
index fcfff5c6aa..1ee3772154 100644
--- a/support/misc/gitlab-ci.yml.in
+++ b/support/misc/gitlab-ci.yml.in
@@ -26,20 +26,24 @@
         paths:
             - .config
 
+.run_make: &run_make
+    |
+      make O=${OUTPUT_DIR} > >(tee build.log |grep '>>>') 2>&1 || {
+          echo 'Failed build last output'
+          tail -200 build.log
+          exit 1
+      }
+
 .defconfig_base:
     before_script:
         - DEFCONFIG_NAME=${CI_JOB_NAME}
+        - OUTPUT_DIR=output
     script:
         - echo "Configure Buildroot for ${DEFCONFIG_NAME}"
         - make ${DEFCONFIG_NAME}
         - ./support/scripts/check-dotconfig.py .config ./configs/${DEFCONFIG_NAME}
         - echo 'Build buildroot'
-        - |
-            make > >(tee build.log |grep '>>>') 2>&1 || {
-                echo 'Failed build last output'
-                tail -200 build.log
-                exit 1
-            }
+        - *run_make
         - |
             ./support/scripts/boot-qemu-image.py "${DEFCONFIG_NAME}" > >(tee runtime-test.log) 2>&1 || {
                 echo 'Failed runtime test last output'
-- 
2.25.1

[Buildroot] [PATCH 1/2 v4] support/misc/gitlab-ci.yml.in

[Romain Naour]

Le 27/06/2021 ? 16:09, Yann E. MORIN a ?crit?:
> From: Romain Naour <romain.naour@gmail.com>
> 
> In a followup commit, the make command used to log and display the last
> lines on error will be used in another job.
> Factorize it by introducing .run_make template.
> 
> Signed-off-by: Romain Naour <romain.naour@gmail.com>
> Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>

Reviewed-by: Romain Naour <romain.naour@gmail.com>

Best regards,
Romain


> ---
>  support/misc/gitlab-ci.yml.in | 16 ++++++++++------
>  1 file changed, 10 insertions(+), 6 deletions(-)
> 
> diff --git a/support/misc/gitlab-ci.yml.in b/support/misc/gitlab-ci.yml.in
> index fcfff5c6aa..1ee3772154 100644
> --- a/support/misc/gitlab-ci.yml.in
> +++ b/support/misc/gitlab-ci.yml.in
> @@ -26,20 +26,24 @@
>          paths:
>              - .config
>  
> +.run_make: &run_make
> +    |
> +      make O=${OUTPUT_DIR} > >(tee build.log |grep '>>>') 2>&1 || {
> +          echo 'Failed build last output'
> +          tail -200 build.log
> +          exit 1
> +      }
> +
>  .defconfig_base:
>      before_script:
>          - DEFCONFIG_NAME=${CI_JOB_NAME}
> +        - OUTPUT_DIR=output
>      script:
>          - echo "Configure Buildroot for ${DEFCONFIG_NAME}"
>          - make ${DEFCONFIG_NAME}
>          - ./support/scripts/check-dotconfig.py .config ./configs/${DEFCONFIG_NAME}
>          - echo 'Build buildroot'
> -        - |
> -            make > >(tee build.log |grep '>>>') 2>&1 || {
> -                echo 'Failed build last output'
> -                tail -200 build.log
> -                exit 1
> -            }
> +        - *run_make
>          - |
>              ./support/scripts/boot-qemu-image.py "${DEFCONFIG_NAME}" > >(tee runtime-test.log) 2>&1 || {
>                  echo 'Failed runtime test last output'
> 

[Buildroot] [PATCH 1/2 v4] support/misc/gitlab-ci.yml.in

[Thomas Petazzoni]

Hello,

On Sun, 27 Jun 2021 16:09:58 +0200
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> From: Romain Naour <romain.naour@gmail.com>
> 
> In a followup commit, the make command used to log and display the last
> lines on error will be used in another job.
> Factorize it by introducing .run_make template.

The commit title.. is missing a commit title :-)

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 2/2 v4] utils/test-pkg: add gitlab-ci support

[Yann E. MORIN]

From: Romain Naour <romain.naour@gmail.com>

The gitlab-ci support in test-pkg allows to parallelize the test-pkg
work into several gitlab jobs. It's much faster than local serialized
testing.

We allow that with two main changes:

  - the test-pkg script is enhanced to be able to only list the builds
    that are not skipped (i.e. for which the configuration fragment
    yield an actual, valid .config)

    We add a new option, --list-only, which returns that list to stdout,
    and still outputs the previous traces, but to stderr (for post-
    mortem analysis, and for local testing to have quick feedback)

  - the script that generates the pipline,  uses that list to create one
    job for each test.

    This is triggered when the last commit log of the series (HEAD),
    contains the 'test-pkg config:' directive all alone on its own line,
    followed by a configuration fragment snippet to be used as input to
    test-pkg.

    If the user provides an empty fragment, this is considered an error:
    indeed, without a fragment (and the package name), there is noway to
    know what to test.

    Furtheremore, if that fragment yields an empty list of tests, then
    there is nothing to test either, so that is also considered an
    error.

As far as Gitlab-CI itslef is concerned, we need to propagate a bit of
information from the parent pipeline to the chile pipeline. Indeed, all
the per-test config files are generated in the parent pipeline, but used
in the child.

So we export the parent pipeline ID, and use that to retrieve all the
per-test .config files, as per the gitlab-CI documentation [0] (we can't
save the whole of the "br-test/" directory, because that will hit the
5-MiB limit in the public Gitlab-CI instance).

[0] https://docs.gitlab.com/ee/ci/yaml/README.html#artifact-downloads-to-child-pipelines

Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
[yann.morin.1998 at free.fr:
  - do not inject yml-knowledge in test-pkg, make it totally agnostic to
    the testing infra: just list matching builds
  - generate the actual yml snippet in support/scripts/generate-gitlab-ci-yml,
    using the list emitted by test-pkg
  - some code-style-candies...
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>

---
v4: reworked by Yann:
    tst-pkg only lists, does not generate the yml code
    The yml code is generated by support/scripts/generate-gitlab-ci-yml
    Empty fragment is an error
    Fragment that yierlds no test is also an error

v3: Implement Arnout's review: http://lists.busybox.net/pipermail/buildroot/2021-May/310656.html
    Enable artifacts download from child-pipeline

v2: Rework this patch following Arnout review
    use CI_COMMIT_DESCRIPTION
    remove .config from artifacts but keep images directory since
    it can be useful for further issue investigation
    use the "br-test-pkg" prefix for test-pkg jobs
---
 .gitlab-ci.yml                         |  3 ++
 support/misc/gitlab-ci.yml.in          | 22 +++++++++++++
 support/scripts/generate-gitlab-ci-yml | 30 +++++++++++++++++-
 utils/test-pkg                         | 44 ++++++++++++++++++++------
 4 files changed, 88 insertions(+), 11 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index e85ac32033..31eb495859 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -13,6 +13,7 @@ generate-gitlab-ci-yml:
   artifacts:
     paths:
       - generated-gitlab-ci.yml
+      - br-test-pkg/*/.config
 
 buildroot-pipeline:
   stage: build
@@ -21,3 +22,5 @@ buildroot-pipeline:
       - artifact: generated-gitlab-ci.yml
         job: generate-gitlab-ci-yml
     strategy: depend
+  variables:
+    PARENT_PIPELINE_ID: $CI_PIPELINE_ID
diff --git a/support/misc/gitlab-ci.yml.in b/support/misc/gitlab-ci.yml.in
index 1ee3772154..be7951b3d2 100644
--- a/support/misc/gitlab-ci.yml.in
+++ b/support/misc/gitlab-ci.yml.in
@@ -80,3 +80,25 @@
             - test-output/*/.config
             - test-output/*/images/*
 
+.test_pkg:
+    stage: build
+    before_script:
+        - OUTPUT_DIR=${CI_JOB_NAME}
+    script:
+        - echo "Configure Buildroot for ${OUTPUT_DIR}"
+        - make O=${OUTPUT_DIR} syncconfig
+        - make O=${OUTPUT_DIR} savedefconfig
+        - echo 'Build buildroot'
+        - *run_make
+    needs:
+        - pipeline: $PARENT_PIPELINE_ID
+          job: generate-gitlab-ci-yml
+    artifacts:
+        when: always
+        expire_in: 2 weeks
+        paths:
+            - build.log
+            - br-test-pkg/*/.config
+            - br-test-pkg/*/defconfig
+            - br-test-pkg/*/build/build-time.log
+            - br-test-pkg/*/build/packages-file-list*.txt
diff --git a/support/scripts/generate-gitlab-ci-yml b/support/scripts/generate-gitlab-ci-yml
index 3f498e08fd..fc0f8cedf7 100755
--- a/support/scripts/generate-gitlab-ci-yml
+++ b/support/scripts/generate-gitlab-ci-yml
@@ -23,7 +23,7 @@ _EOF_
 
 gen_tests() {
     local -a basics defconfigs runtimes
-    local do_basics do_defconfigs do_runtime
+    local do_basics do_defconfigs do_runtime do_testpkg
     local defconfigs_ext cfg tst
 
     basics=( DEVELOPERS flake8 package )
@@ -77,9 +77,33 @@ gen_tests() {
         esac
     fi
 
+    # Retrieve defconfig for test-pkg from the git commit message (if any)
+    echo "$CI_COMMIT_DESCRIPTION" \
+        | sed -n '/^test-pkg config:$/,/^$/p' \
+        > defconfig.frag
+
+    if [ -s defconfig.frag ]; then
+        sed -i 1d defconfig.frag
+        if [ ! -s defconfig.frag ]; then
+            printf "Empty configuration fragment.\n" >&2; exit 1
+        fi
+        # Use --all since we expect the user having already pre-tested the new package
+        # with the default subset of toolchains.
+        do_testpkg=( $( ./utils/test-pkg \
+                            --all --list-only \
+                            --config-snippet defconfig.frag \
+                            --build-dir br-test-pkg \
+                     )
+                   )
+        if [ "${#do_testpkg[@]}" -eq 0 ]; then
+            printf "Configuration fragment enables no test.\n" >&2; exit 1
+        fi
+    fi
+
     # If nothing else, at least do the basics to generate a valid pipeline
     if [    -z "${do_defconfigs}" \
          -a -z "${do_runtime}" \
+         -a -z "${do_testpkg}" \
        ]
     then
         do_basics=true
@@ -101,6 +125,10 @@ gen_tests() {
     if ${do_runtime:-false}; then
         printf '%s: { extends: .runtime_test_base }\n' "${runtimes[@]}"
     fi
+
+    if [ -n "${do_testpkg}" ]; then
+        printf '%s: { extends: .test_pkg }\n' "${do_test_pkg[@]}"
+    fi
 }
 
 main "${@}"
diff --git a/utils/test-pkg b/utils/test-pkg
index a317d8c17a..2349674069 100755
--- a/utils/test-pkg
+++ b/utils/test-pkg
@@ -10,15 +10,19 @@ do_clean() {
     fi
 }
 
+trace() {
+    printf "${@}"
+}
+
 main() {
     local o O opts
-    local cfg dir pkg random toolchains_csv toolchain all number mode
+    local cfg dir pkg random toolchains_csv toolchain all number mode list_only
     local ret nb nb_skip nb_fail nb_legal nb_tc build_dir keep
     local -a toolchains
     local pkg_br_name
 
-    o='hakc:d:n:p:r:t:'
-    O='help,all,keep,config-snippet:,build-dir:,number:,package:,random:,toolchains-csv:'
+    o='hakgc:d:n:p:r:t:'
+    O='help,all,keep,config-snippet:,build-dir:,list-only,number:,package:,random:,toolchains-csv:'
     opts="$(getopt -n "${my_name}" -o "${o}" -l "${O}" -- "${@}")"
     eval set -- "${opts}"
 
@@ -27,6 +31,7 @@ main() {
     keep=0
     number=0
     mode=0
+    list_only=0
     toolchains_csv="${TOOLCHAINS_CSV}"
     while [ ${#} -gt 0 ]; do
         case "${1}" in
@@ -39,6 +44,10 @@ main() {
         (-k|--keep)
             keep=1; shift 1
             ;;
+        (-l|--list-only)
+            list_only=1; shift 1
+            trace() { printf "${@}" >&2; }
+            ;;
         (-c|--config-snippet)
             cfg="${2}"; shift 2
             ;;
@@ -118,6 +127,11 @@ main() {
         printf "error: no toolchain found (networking issue?)\n" >&2; exit 1
     fi
 
+    if [ -n "${list_file}" ]; then
+        # Running in list-only implies keeping the build directories.
+        keep=1
+    fi
+
     nb=0
     nb_skip=0
     nb_fail=0
@@ -126,17 +140,21 @@ main() {
         : $((nb++))
         toolchain="$(basename "${toolchainconfig}" .config)"
         build_dir="${dir}/${toolchain}"
-        printf "%40s [%*d/%d]: " "${toolchain}" ${#nb_tc} ${nb} ${nb_tc}
-        build_one "${build_dir}" "${toolchainconfig}" "${cfg}" "${pkg}" && ret=0 || ret=${?}
+        trace "%40s [%*d/%d]: " "${toolchain}" ${#nb_tc} ${nb} ${nb_tc}
+        build_one "${build_dir}" "${toolchainconfig}" "${cfg}" "${pkg}" "${list_only}" && ret=0 || ret=${?}
         case ${ret} in
-        (0) printf "OK\n";;
-        (1) : $((nb_skip++)); printf "SKIPPED\n";;
-        (2) : $((nb_fail++)); printf "FAILED\n";;
-        (3) : $((nb_legal++)); printf "FAILED\n";;
+        (0) trace "OK\n"
+            if [ ${list_only} -eq 1 ]; then
+                printf '%s\n' "$build_dir"
+            fi
+            ;;
+        (1) : $((nb_skip++)); trace "SKIPPED\n";;
+        (2) : $((nb_fail++)); trace "FAILED\n";;
+        (3) : $((nb_legal++)); trace "FAILED\n";;
         esac
     done
 
-    printf "%d builds, %d skipped, %d build failed, %d legal-info failed\n" \
+    trace "%d builds, %d skipped, %d build failed, %d legal-info failed\n" \
         ${nb} ${nb_skip} ${nb_fail} ${nb_legal}
 
     return $((nb_fail + nb_legal))
@@ -147,6 +165,7 @@ build_one() {
     local toolchainconfig="${2}"
     local cfg="${3}"
     local pkg="${4}"
+    local defer="${5}"
 
     mkdir -p "${dir}"
 
@@ -166,6 +185,11 @@ build_one() {
     # Remove file, it's empty anyway.
     rm -f "${dir}/missing.config"
 
+    # Defer building the job to the caller (e.g. a gitlab pipeline)
+    if [ ${defer} -eq 1 ]; then
+        return 0
+    fi
+
     if [ -n "${pkg}" ]; then
         if ! make O="${dir}" "${pkg}-dirclean" >> "${dir}/logfile" 2>&1; then
             return 2
-- 
2.25.1

[Buildroot] [PATCH 2/2 v4] utils/test-pkg: add gitlab-ci support

[Romain Naour]

Le 27/06/2021 ? 16:09, Yann E. MORIN a ?crit?:
> From: Romain Naour <romain.naour@gmail.com>
> 
> The gitlab-ci support in test-pkg allows to parallelize the test-pkg
> work into several gitlab jobs. It's much faster than local serialized
> testing.
> 
> We allow that with two main changes:
> 
>   - the test-pkg script is enhanced to be able to only list the builds
>     that are not skipped (i.e. for which the configuration fragment
>     yield an actual, valid .config)
> 
>     We add a new option, --list-only, which returns that list to stdout,
>     and still outputs the previous traces, but to stderr (for post-
>     mortem analysis, and for local testing to have quick feedback)
> 
>   - the script that generates the pipline,  uses that list to create one
                                    pipeline
>     job for each test.
> 
>     This is triggered when the last commit log of the series (HEAD),
>     contains the 'test-pkg config:' directive all alone on its own line,
>     followed by a configuration fragment snippet to be used as input to
>     test-pkg.
> 
>     If the user provides an empty fragment, this is considered an error:
>     indeed, without a fragment (and the package name), there is noway to
>     know what to test.
> 
>     Furtheremore, if that fragment yields an empty list of tests, then
      Furthermore
>     there is nothing to test either, so that is also considered an
>     error.
> 
> As far as Gitlab-CI itslef is concerned, we need to propagate a bit of
                      itself
> information from the parent pipeline to the chile pipeline. Indeed, all
                                              child
> the per-test config files are generated in the parent pipeline, but used
> in the child.
> 
> So we export the parent pipeline ID, and use that to retrieve all the
> per-test .config files, as per the gitlab-CI documentation [0] (we can't
> save the whole of the "br-test/" directory, because that will hit the
> 5-MiB limit in the public Gitlab-CI instance).
> 
> [0] https://docs.gitlab.com/ee/ci/yaml/README.html#artifact-downloads-to-child-pipelines
> 
> Signed-off-by: Romain Naour <romain.naour@gmail.com>
> Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
> [yann.morin.1998 at free.fr:
>   - do not inject yml-knowledge in test-pkg, make it totally agnostic to
>     the testing infra: just list matching builds
>   - generate the actual yml snippet in support/scripts/generate-gitlab-ci-yml,
>     using the list emitted by test-pkg
>   - some code-style-candies...
> ]

Thanks for the final rework!

> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> 
> ---
> v4: reworked by Yann:
>     tst-pkg only lists, does not generate the yml code
>     The yml code is generated by support/scripts/generate-gitlab-ci-yml
>     Empty fragment is an error
>     Fragment that yierlds no test is also an error
> 
> v3: Implement Arnout's review: http://lists.busybox.net/pipermail/buildroot/2021-May/310656.html
>     Enable artifacts download from child-pipeline
> 
> v2: Rework this patch following Arnout review
>     use CI_COMMIT_DESCRIPTION
>     remove .config from artifacts but keep images directory since
>     it can be useful for further issue investigation
>     use the "br-test-pkg" prefix for test-pkg jobs
> ---
>  .gitlab-ci.yml                         |  3 ++
>  support/misc/gitlab-ci.yml.in          | 22 +++++++++++++
>  support/scripts/generate-gitlab-ci-yml | 30 +++++++++++++++++-
>  utils/test-pkg                         | 44 ++++++++++++++++++++------
>  4 files changed, 88 insertions(+), 11 deletions(-)
> 
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index e85ac32033..31eb495859 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -13,6 +13,7 @@ generate-gitlab-ci-yml:
>    artifacts:
>      paths:
>        - generated-gitlab-ci.yml
> +      - br-test-pkg/*/.config
>  
>  buildroot-pipeline:
>    stage: build
> @@ -21,3 +22,5 @@ buildroot-pipeline:
>        - artifact: generated-gitlab-ci.yml
>          job: generate-gitlab-ci-yml
>      strategy: depend
> +  variables:
> +    PARENT_PIPELINE_ID: $CI_PIPELINE_ID
> diff --git a/support/misc/gitlab-ci.yml.in b/support/misc/gitlab-ci.yml.in
> index 1ee3772154..be7951b3d2 100644
> --- a/support/misc/gitlab-ci.yml.in
> +++ b/support/misc/gitlab-ci.yml.in
> @@ -80,3 +80,25 @@
>              - test-output/*/.config
>              - test-output/*/images/*
>  
> +.test_pkg:
> +    stage: build
> +    before_script:
> +        - OUTPUT_DIR=${CI_JOB_NAME}
> +    script:
> +        - echo "Configure Buildroot for ${OUTPUT_DIR}"
> +        - make O=${OUTPUT_DIR} syncconfig
> +        - make O=${OUTPUT_DIR} savedefconfig
> +        - echo 'Build buildroot'
> +        - *run_make
> +    needs:
> +        - pipeline: $PARENT_PIPELINE_ID
> +          job: generate-gitlab-ci-yml
> +    artifacts:
> +        when: always
> +        expire_in: 2 weeks
> +        paths:
> +            - build.log
> +            - br-test-pkg/*/.config
> +            - br-test-pkg/*/defconfig
> +            - br-test-pkg/*/build/build-time.log
> +            - br-test-pkg/*/build/packages-file-list*.txt
> diff --git a/support/scripts/generate-gitlab-ci-yml b/support/scripts/generate-gitlab-ci-yml
> index 3f498e08fd..fc0f8cedf7 100755
> --- a/support/scripts/generate-gitlab-ci-yml
> +++ b/support/scripts/generate-gitlab-ci-yml
> @@ -23,7 +23,7 @@ _EOF_
>  
>  gen_tests() {
>      local -a basics defconfigs runtimes
> -    local do_basics do_defconfigs do_runtime
> +    local do_basics do_defconfigs do_runtime do_testpkg
>      local defconfigs_ext cfg tst
>  
>      basics=( DEVELOPERS flake8 package )
> @@ -77,9 +77,33 @@ gen_tests() {
>          esac
>      fi
>  
> +    # Retrieve defconfig for test-pkg from the git commit message (if any)
> +    echo "$CI_COMMIT_DESCRIPTION" \
> +        | sed -n '/^test-pkg config:$/,/^$/p' \
> +        > defconfig.frag
> +
> +    if [ -s defconfig.frag ]; then
> +        sed -i 1d defconfig.frag
> +        if [ ! -s defconfig.frag ]; then
> +            printf "Empty configuration fragment.\n" >&2; exit 1
> +        fi
> +        # Use --all since we expect the user having already pre-tested the new package
> +        # with the default subset of toolchains.
> +        do_testpkg=( $( ./utils/test-pkg \
> +                            --all --list-only \
> +                            --config-snippet defconfig.frag \
> +                            --build-dir br-test-pkg \
> +                     )
> +                   )
> +        if [ "${#do_testpkg[@]}" -eq 0 ]; then
> +            printf "Configuration fragment enables no test.\n" >&2; exit 1
> +        fi
> +    fi
> +
>      # If nothing else, at least do the basics to generate a valid pipeline
>      if [    -z "${do_defconfigs}" \
>           -a -z "${do_runtime}" \
> +         -a -z "${do_testpkg}" \
>         ]
>      then
>          do_basics=true
> @@ -101,6 +125,10 @@ gen_tests() {
>      if ${do_runtime:-false}; then
>          printf '%s: { extends: .runtime_test_base }\n' "${runtimes[@]}"
>      fi
> +
> +    if [ -n "${do_testpkg}" ]; then
> +        printf '%s: { extends: .test_pkg }\n' "${do_test_pkg[@]}"

s/do_test_pkg/do_testpkg/

With that fixed:
https://gitlab.com/kubu93/buildroot/-/pipelines/327870496

Best regards,
Romain


> +    fi
>  }
>  
>  main "${@}"
> diff --git a/utils/test-pkg b/utils/test-pkg
> index a317d8c17a..2349674069 100755
> --- a/utils/test-pkg
> +++ b/utils/test-pkg
> @@ -10,15 +10,19 @@ do_clean() {
>      fi
>  }
>  
> +trace() {
> +    printf "${@}"
> +}
> +
>  main() {
>      local o O opts
> -    local cfg dir pkg random toolchains_csv toolchain all number mode
> +    local cfg dir pkg random toolchains_csv toolchain all number mode list_only
>      local ret nb nb_skip nb_fail nb_legal nb_tc build_dir keep
>      local -a toolchains
>      local pkg_br_name
>  
> -    o='hakc:d:n:p:r:t:'
> -    O='help,all,keep,config-snippet:,build-dir:,number:,package:,random:,toolchains-csv:'
> +    o='hakgc:d:n:p:r:t:'
> +    O='help,all,keep,config-snippet:,build-dir:,list-only,number:,package:,random:,toolchains-csv:'
>      opts="$(getopt -n "${my_name}" -o "${o}" -l "${O}" -- "${@}")"
>      eval set -- "${opts}"
>  
> @@ -27,6 +31,7 @@ main() {
>      keep=0
>      number=0
>      mode=0
> +    list_only=0
>      toolchains_csv="${TOOLCHAINS_CSV}"
>      while [ ${#} -gt 0 ]; do
>          case "${1}" in
> @@ -39,6 +44,10 @@ main() {
>          (-k|--keep)
>              keep=1; shift 1
>              ;;
> +        (-l|--list-only)
> +            list_only=1; shift 1
> +            trace() { printf "${@}" >&2; }
> +            ;;
>          (-c|--config-snippet)
>              cfg="${2}"; shift 2
>              ;;
> @@ -118,6 +127,11 @@ main() {
>          printf "error: no toolchain found (networking issue?)\n" >&2; exit 1
>      fi
>  
> +    if [ -n "${list_file}" ]; then
> +        # Running in list-only implies keeping the build directories.
> +        keep=1
> +    fi
> +
>      nb=0
>      nb_skip=0
>      nb_fail=0
> @@ -126,17 +140,21 @@ main() {
>          : $((nb++))
>          toolchain="$(basename "${toolchainconfig}" .config)"
>          build_dir="${dir}/${toolchain}"
> -        printf "%40s [%*d/%d]: " "${toolchain}" ${#nb_tc} ${nb} ${nb_tc}
> -        build_one "${build_dir}" "${toolchainconfig}" "${cfg}" "${pkg}" && ret=0 || ret=${?}
> +        trace "%40s [%*d/%d]: " "${toolchain}" ${#nb_tc} ${nb} ${nb_tc}
> +        build_one "${build_dir}" "${toolchainconfig}" "${cfg}" "${pkg}" "${list_only}" && ret=0 || ret=${?}
>          case ${ret} in
> -        (0) printf "OK\n";;
> -        (1) : $((nb_skip++)); printf "SKIPPED\n";;
> -        (2) : $((nb_fail++)); printf "FAILED\n";;
> -        (3) : $((nb_legal++)); printf "FAILED\n";;
> +        (0) trace "OK\n"
> +            if [ ${list_only} -eq 1 ]; then
> +                printf '%s\n' "$build_dir"
> +            fi
> +            ;;
> +        (1) : $((nb_skip++)); trace "SKIPPED\n";;
> +        (2) : $((nb_fail++)); trace "FAILED\n";;
> +        (3) : $((nb_legal++)); trace "FAILED\n";;
>          esac
>      done
>  
> -    printf "%d builds, %d skipped, %d build failed, %d legal-info failed\n" \
> +    trace "%d builds, %d skipped, %d build failed, %d legal-info failed\n" \
>          ${nb} ${nb_skip} ${nb_fail} ${nb_legal}
>  
>      return $((nb_fail + nb_legal))
> @@ -147,6 +165,7 @@ build_one() {
>      local toolchainconfig="${2}"
>      local cfg="${3}"
>      local pkg="${4}"
> +    local defer="${5}"
>  
>      mkdir -p "${dir}"
>  
> @@ -166,6 +185,11 @@ build_one() {
>      # Remove file, it's empty anyway.
>      rm -f "${dir}/missing.config"
>  
> +    # Defer building the job to the caller (e.g. a gitlab pipeline)
> +    if [ ${defer} -eq 1 ]; then
> +        return 0
> +    fi
> +
>      if [ -n "${pkg}" ]; then
>          if ! make O="${dir}" "${pkg}-dirclean" >> "${dir}/logfile" 2>&1; then
>              return 2
> 

[Buildroot] [PATCH 2/2 v4] utils/test-pkg: add gitlab-ci support

[Thomas Petazzoni]

Hello,

On Sun, 27 Jun 2021 16:09:59 +0200
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

>   - the script that generates the pipline,  uses that list to create one
>     job for each test.
> 
>     This is triggered when the last commit log of the series (HEAD),
>     contains the 'test-pkg config:' directive all alone on its own line,
>     followed by a configuration fragment snippet to be used as input to
>     test-pkg.

Hm, this is not really trivial to use. It definitely needs some
documentation somewhere.

> +    # Retrieve defconfig for test-pkg from the git commit message (if any)
> +    echo "$CI_COMMIT_DESCRIPTION" \
> +        | sed -n '/^test-pkg config:$/,/^$/p' \
> +        > defconfig.frag

So what happens with this defconfig.frag file? It is left there after
the script has executed?

> +
> +    if [ -s defconfig.frag ]; then
> +        sed -i 1d defconfig.frag
> +        if [ ! -s defconfig.frag ]; then
> +            printf "Empty configuration fragment.\n" >&2; exit 1
> +        fi
> +        # Use --all since we expect the user having already pre-tested the new package
> +        # with the default subset of toolchains.
> +        do_testpkg=( $( ./utils/test-pkg \
> +                            --all --list-only \
> +                            --config-snippet defconfig.frag \
> +                            --build-dir br-test-pkg \

Same question about the br-test-pkg directory, it stays there after the
script has executed?

Shouldn't both of these be using a temporary directory?

Note: I am not sure I have understood how the .config files generated
by test-pkg make their way to the Gitlab builds, so perhaps I'm missing
something.


> +        (-l|--list-only)
> +            list_only=1; shift 1
> +            trace() { printf "${@}" >&2; }

I find "list-only" to be misleading, because it is not much more than
listing: it is generating the .config for each test configuration. So
what about --no-build, or --gen-config-only, or something like that?

To me --list-only, would really only list, i.e it wouldn't generate
anything.

> +    if [ -n "${list_file}" ]; then
> +        # Running in list-only implies keeping the build directories.
> +        keep=1
> +    fi

I'm not sure I like this semantic. Perhaps we should simply rely on the
user passing -k ? Especially since this feature is most likely only
going to be used by the generate-gitlab-ci-yml machinery ?

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 2/2 v4] utils/test-pkg: add gitlab-ci support

[Yann E. MORIN]

Thomas, All,

On 2021-06-27 18:09 +0200, Thomas Petazzoni spake thusly:
> On Sun, 27 Jun 2021 16:09:59 +0200
> "Yann E. MORIN" <yann.morin.1998@free.fr> wrote:
> >   - the script that generates the pipline,  uses that list to create one
> >     job for each test.
> > 
> >     This is triggered when the last commit log of the series (HEAD),
> >     contains the 'test-pkg config:' directive all alone on its own line,
> >     followed by a configuration fragment snippet to be used as input to
> >     test-pkg.
> Hm, this is not really trivial to use. It definitely needs some
> documentation somewhere.

Right. Like all the rest of the automatic gitlab suport for branches and
tags! ;-)

Oh, in fact there is a terse, very limited blurb about that in our
manual, hidden away toward the end of 22.7.1:
    https://buildroot.org/downloads/manual/manual.html#_using_the_run_tests_framework

It is still very undocumented overall...

> > +    # Retrieve defconfig for test-pkg from the git commit message (if any)
> > +    echo "$CI_COMMIT_DESCRIPTION" \
> > +        | sed -n '/^test-pkg config:$/,/^$/p' \
> > +        > defconfig.frag
> So what happens with this defconfig.frag file? It is left there after
> the script has executed?

Yes, but this script is only expected to be run by the gitlab-ci infra,
which runs in a container that is trashed afterwards. So anything that
is left after the build and is not listed as an artefact, is lost.

And when debugging localy, it is nice to keep the temporary files to
investigate.

> > +    if [ -s defconfig.frag ]; then
> > +        sed -i 1d defconfig.frag
> > +        if [ ! -s defconfig.frag ]; then
> > +            printf "Empty configuration fragment.\n" >&2; exit 1
> > +        fi
> > +        # Use --all since we expect the user having already pre-tested the new package
> > +        # with the default subset of toolchains.
> > +        do_testpkg=( $( ./utils/test-pkg \
> > +                            --all --list-only \
> > +                            --config-snippet defconfig.frag \
> > +                            --build-dir br-test-pkg \
> Same question about the br-test-pkg directory, it stays there after the
> script has executed?

Yes, but unlike defconfig.frag:

  - br-test-pkg/*/.config are listed as artefact, so they are saved
    before the container exits

  - when the container exits, everything else is lost

> Shouldn't both of these be using a temporary directory?

We don't care about that when running in the gitlab-ci infra, because
everything gets lost after the job is done.

> Note: I am not sure I have understood how the .config files generated
> by test-pkg make their way to the Gitlab builds, so perhaps I'm missing
> something.

They are listed as artefacts, so they are saved before the job in the
parent pipeline ends. Then the jobs in the child pipeline inherit them
via the 'needs' directive, and thus they are extracted for all those
jobs.

Yes, each test job get all the br-test-pkg/*/.config files, even if each
job only uses one. But since the job name is the name of the directory
in which to do the build, each job indeed builds only one configuration.

> > +        (-l|--list-only)
> > +            list_only=1; shift 1
> > +            trace() { printf "${@}" >&2; }
> I find "list-only" to be misleading, because it is not much more than
> listing: it is generating the .config for each test configuration. So
> what about --no-build, or --gen-config-only, or something like that?
> 
> To me --list-only, would really only list, i.e it wouldn't generate
> anything.

Initially, the option was named --generate-only, but this was when
test-pkg itself was generating the yaml fragment (which was not nice at
all). So I renamed it to --list-only, butI was not very happy either...

What about --check-buildable-and-list ? Or --prepare-config-and-list ?
--check-if-buildable-then-list-and-keep-config-file ? I obviously have
no good idea... ;-]

The problem is that it is non-trivial to only list the jobs, because
what we really need as input to test-pkg are:
  - a config snippet
  - a list of toolchain definitions

However, to generate the list of jobs, we need to iterate over all the
bundled toolchains defintions, check that it generates a valid
configuration, and if so, generate a job definiton for that.

So, if we just re-emit the toolchain definition, we also need to save
the config fragment, to re-run test-pkg again in each job, to regenerate
a valid configuration, then build it.

However, after the first step, we already know that the configuration is
valid: we even have a .config file for that configuration, which is all
that we need as an input to Buildroot.

So that's why we only need to save the br-test-pkg/*/.config as artefact
of the parent job, and build those in children jobs.

> > +    if [ -n "${list_file}" ]; then
> > +        # Running in list-only implies keeping the build directories.
> > +        keep=1
> > +    fi
> I'm not sure I like this semantic. Perhaps we should simply rely on the
> user passing -k ? Especially since this feature is most likely only
> going to be used by the generate-gitlab-ci-yml machinery ?

ACK, I was not sure either...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'