Re: [PATCH 2/3] gitlab-ci.yml: Add a job to build EDK2 firmware binaries

[Laszlo Ersek <lersek@redhat.com>]

On 01/06/20 19:46, Philippe Mathieu-Daudé wrote:
> Add a GitLab job to build the EDK2 firmware binaries.
> This job is only built when the roms/edk2/ submodule is updated,
> when a git-ref starts with 'edk2' or when the last commit contains
> 'EDK2'.

keyword "or"; okay.

> 
> GitLab CI generates an artifacts.zip file containing the firmware
> binaries.
> 
> With edk2-stable201905, the job took 40 minutes 26 seconds,
> the artifacts.zip takes 10MiB.
> 
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>  .gitlab-ci-edk2.yml | 37 +++++++++++++++++++++++++++++++++++++
>  .gitlab-ci.yml      |  3 +++
>  MAINTAINERS         |  3 ++-
>  3 files changed, 42 insertions(+), 1 deletion(-)
>  create mode 100644 .gitlab-ci-edk2.yml
> 
> diff --git a/.gitlab-ci-edk2.yml b/.gitlab-ci-edk2.yml
> new file mode 100644
> index 0000000000..abfaf52874
> --- /dev/null
> +++ b/.gitlab-ci-edk2.yml
> @@ -0,0 +1,37 @@
> +build-edk2:
> + rules: # Only run this job when ...
> + - changes: # ... roms/edk2/ is modified (submodule updated)
> +   - roms/edk2/*
> +   when: always
> + - if: '$CI_COMMIT_REF_NAME =~ /^edk2/' # ... the branch/tag starts with 'edk2'

(1) can you add "or" in the comment here?

> +   when: always
> + - if: '$CI_COMMIT_MESSAGE =~ /edk2/i' # last commit description contains 'EDK2'

(2) ditto

> +   when: always
> + artifacts:
> +   paths: # 'artifacts.zip' will contains the following files:
> +   - pc-bios/edk2*bz2
> +   - pc-bios/edk2-licenses.txt
> +   - edk2-stdout.log
> +   - edk2-stderr.log
> + image: ubuntu:16.04 # Use Ubuntu Xenial
> + before_script: # Install packages requiered to build EDK2
> + - apt-get update --quiet --quiet
> + - DEBIAN_FRONTEND=noninteractive
> +   apt-get install --assume-yes --no-install-recommends --quiet --quiet
> +     build-essential
> +     ca-certificates
> +     dos2unix
> +     gcc-aarch64-linux-gnu
> +     gcc-arm-linux-gnueabi
> +     git
> +     iasl
> +     make
> +     nasm
> +     python
> +     uuid-dev
> + script: # Clone the required submodules and build EDK2
> + - git submodule update --init roms/edk2

yes, this is needed; qemu users are used to updating top-level
submodules (which is why we didn't try to automate that away in the edk2
build stuff)

> + - git -C roms/edk2 submodule update --init

(3) but this should not be necessary. See the "submodules" target in
"roms/Makefile.edk2".

> + - export JOBS=$(($(getconf _NPROCESSORS_ONLN) + 1))
> + - echo "=== Using ${JOBS} simultaneous jobs ==="
> + - make -j${JOBS} -C roms efi 1>edk2-stdout.log 2> >(tee -a edk2-stderr.log >&2)

Process substitution is a nifty feature, but perhaps we can do without
it, for simplicity. (I realize this is bash-only; I just like to
minimize the use of non-portable features if there is a portable
replacement that is also simple.)

Redirections are processed in the order they appear on the command line
[1], *after* stdout/stdin is redirected for pipelining [2]:

[1]
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_07

"If more than one redirection operator is specified with a command, the
order of evaluation is from beginning to end."

[2]
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_02

"The standard input, standard output, or both of a command shall be
considered to be assigned by the pipeline before any redirection
specified by redirection operators that are part of the command"


(4) Therefore, the following should work:

  make -j${JOBS} -C roms efi 2>&1 1>edk2-stdout.log \
  | tee -a edk2-stderr.log >&2

Untested, of course :)

Looks OK otherwise.

Thanks!
Laszlo

> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index ebcef0ebe9..f799246047 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -1,3 +1,6 @@
> +include:
> +  - local: '/.gitlab-ci-edk2.yml'
> +
>  before_script:
>   - apt-get update -qq
>   - apt-get install -y -qq flex bison libglib2.0-dev libpixman-1-dev genisoimage
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 8571327881..22a1fd5824 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -2346,6 +2346,7 @@ F: roms/edk2
>  F: roms/edk2-*
>  F: tests/data/uefi-boot-images/
>  F: tests/uefi-test-tools/
> +F: .gitlab-ci-edk2.yml
>  
>  Usermode Emulation
>  ------------------
> @@ -2689,7 +2690,7 @@ W: https://cirrus-ci.com/github/qemu/qemu
>  GitLab Continuous Integration
>  M: Thomas Huth <thuth@redhat.com>
>  S: Maintained
> -F: .gitlab-ci.yml
> +F: .gitlab-ci*.yml
>  
>  Guest Test Compilation Support
>  M: Alex Bennée <alex.bennee@linaro.org>
>