[PATCH-for-8.0] gitlab-ci: Remove job building EDK2 firmware binaries

[PATCH-for-8.0] gitlab-ci: Remove job building EDK2 firmware binaries

[Philippe Mathieu-Daudé]

When we introduced this Gitlab-CI job in commit 71920809ce
("gitlab-ci.yml: Add jobs to build EDK2 firmware binaries"),
the naive plan was to have reproducible binaries by downloading
what this job would build, testing it and eventually committing
it. With retrospective, nothing happened 3 years later and this
job is just bitrotting:

  Step 1/3 : FROM ubuntu:18.04
  18.04: Pulling from library/ubuntu
  mediaType in manifest should be
  'application/vnd.docker.distribution.manifest.v2+json' not
  'application/vnd.oci.image.manifest.v1+json'

Remove this job to avoid wasting maintenance and CI ressources.

Reported-by: Palmer Dabbelt <palmer@rivosinc.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
Supersedes: <20230222155341.10127-1-palmer@rivosinc.com>
---
 .gitlab-ci.d/edk2.yml         | 85 -----------------------------------
 .gitlab-ci.d/edk2/Dockerfile  | 27 -----------
 .gitlab-ci.d/qemu-project.yml |  1 -
 MAINTAINERS                   |  2 -
 4 files changed, 115 deletions(-)
 delete mode 100644 .gitlab-ci.d/edk2.yml
 delete mode 100644 .gitlab-ci.d/edk2/Dockerfile

diff --git a/.gitlab-ci.d/edk2.yml b/.gitlab-ci.d/edk2.yml
deleted file mode 100644
index 314e101745..0000000000
--- a/.gitlab-ci.d/edk2.yml
+++ /dev/null
@@ -1,85 +0,0 @@
-# All jobs needing docker-edk2 must use the same rules it uses.
-.edk2_job_rules:
-  rules:
-    # Forks don't get pipelines unless QEMU_CI=1 or QEMU_CI=2 is set
-    - if: '$QEMU_CI != "1" && $QEMU_CI != "2" && $CI_PROJECT_NAMESPACE != "qemu-project"'
-      when: never
-
-    # In forks, if QEMU_CI=1 is set, then create manual job
-    # if any of the files affecting the build are touched
-    - if: '$QEMU_CI == "1" && $CI_PROJECT_NAMESPACE != "qemu-project"'
-      changes:
-        - .gitlab-ci.d/edk2.yml
-        - .gitlab-ci.d/edk2/Dockerfile
-        - roms/edk2/*
-      when: manual
-
-    # In forks, if QEMU_CI=1 is set, then create manual job
-    # if the branch/tag starts with 'edk2'
-    - if: '$QEMU_CI == "1" && $CI_PROJECT_NAMESPACE != "qemu-project" && $CI_COMMIT_REF_NAME =~ /^edk2/'
-      when: manual
-
-    # In forks, if QEMU_CI=1 is set, then create manual job
-    # if last commit msg contains 'EDK2' (case insensitive)
-    - if: '$QEMU_CI == "1" && $CI_PROJECT_NAMESPACE != "qemu-project" && $CI_COMMIT_MESSAGE =~ /edk2/i'
-      when: manual
-
-    # Run if any files affecting the build output are touched
-    - changes:
-        - .gitlab-ci.d/edk2.yml
-        - .gitlab-ci.d/edk2/Dockerfile
-        - roms/edk2/*
-      when: on_success
-
-    # Run if the branch/tag starts with 'edk2'
-    - if: '$CI_COMMIT_REF_NAME =~ /^edk2/'
-      when: on_success
-
-    # Run if last commit msg contains 'EDK2' (case insensitive)
-    - if: '$CI_COMMIT_MESSAGE =~ /edk2/i'
-      when: on_success
-
-docker-edk2:
-  extends: .edk2_job_rules
-  stage: containers
-  image: docker:19.03.1
-  services:
-    - docker:19.03.1-dind
-  variables:
-    GIT_DEPTH: 3
-    IMAGE_TAG: $CI_REGISTRY_IMAGE:edk2-cross-build
-    # We don't use TLS
-    DOCKER_HOST: tcp://docker:2375
-    DOCKER_TLS_CERTDIR: ""
-  before_script:
-    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
-  script:
-    - docker pull $IMAGE_TAG || true
-    - docker build --cache-from $IMAGE_TAG --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
-                                           --tag $IMAGE_TAG .gitlab-ci.d/edk2
-    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
-    - docker push $IMAGE_TAG
-
-build-edk2:
-  extends: .edk2_job_rules
-  stage: build
-  needs: ['docker-edk2']
-  artifacts:
-    paths: # 'artifacts.zip' will contains the following files:
-      - pc-bios/edk2*bz2
-      - pc-bios/edk2-licenses.txt
-      - edk2-stdout.log
-      - edk2-stderr.log
-  image: $CI_REGISTRY_IMAGE:edk2-cross-build
-  variables:
-    GIT_DEPTH: 3
-  script: # Clone the required submodules and build EDK2
-    - git submodule update --init roms/edk2
-    - git -C roms/edk2 submodule update --init --
-       ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3
-       BaseTools/Source/C/BrotliCompress/brotli
-       CryptoPkg/Library/OpensslLib/openssl
-       MdeModulePkg/Library/BrotliCustomDecompressLib/brotli
-    - export JOBS=$(($(getconf _NPROCESSORS_ONLN) + 1))
-    - echo "=== Using ${JOBS} simultaneous jobs ==="
-    - make -j${JOBS} -C roms efi 2>&1 1>edk2-stdout.log | tee -a edk2-stderr.log >&2
diff --git a/.gitlab-ci.d/edk2/Dockerfile b/.gitlab-ci.d/edk2/Dockerfile
deleted file mode 100644
index bbe50ff832..0000000000
--- a/.gitlab-ci.d/edk2/Dockerfile
+++ /dev/null
@@ -1,27 +0,0 @@
-#
-# Docker image to cross-compile EDK2 firmware binaries
-#
-FROM ubuntu:18.04
-
-MAINTAINER Philippe Mathieu-Daudé <f4bug@amsat.org>
-
-# Install packages required to build EDK2
-RUN apt update \
-    && \
-    \
-    DEBIAN_FRONTEND=noninteractive \
-    apt install --assume-yes --no-install-recommends \
-        build-essential \
-        ca-certificates \
-        dos2unix \
-        gcc-aarch64-linux-gnu \
-        gcc-arm-linux-gnueabi \
-        git \
-        iasl \
-        make \
-        nasm \
-        python3 \
-        uuid-dev \
-    && \
-    \
-    rm -rf /var/lib/apt/lists/*
diff --git a/.gitlab-ci.d/qemu-project.yml b/.gitlab-ci.d/qemu-project.yml
index 691d9bf5dc..a7ed447fe4 100644
--- a/.gitlab-ci.d/qemu-project.yml
+++ b/.gitlab-ci.d/qemu-project.yml
@@ -4,7 +4,6 @@
 include:
   - local: '/.gitlab-ci.d/base.yml'
   - local: '/.gitlab-ci.d/stages.yml'
-  - local: '/.gitlab-ci.d/edk2.yml'
   - local: '/.gitlab-ci.d/opensbi.yml'
   - local: '/.gitlab-ci.d/containers.yml'
   - local: '/.gitlab-ci.d/crossbuilds.yml'
diff --git a/MAINTAINERS b/MAINTAINERS
index 3f24bc2787..b45b540fbd 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -3329,8 +3329,6 @@ F: roms/edk2
 F: roms/edk2-*
 F: tests/data/uefi-boot-images/
 F: tests/uefi-test-tools/
-F: .gitlab-ci.d/edk2.yml
-F: .gitlab-ci.d/edk2/
 
 VT-d Emulation
 M: Michael S. Tsirkin <mst@redhat.com>
-- 
2.38.1

Re: [PATCH-for-8.0] gitlab-ci: Remove job building EDK2 firmware binaries

[Peter Maydell]

On Fri, 10 Mar 2023 at 13:33, Philippe Mathieu-Daudé <philmd@linaro.org> wrote:
>
> When we introduced this Gitlab-CI job in commit 71920809ce
> ("gitlab-ci.yml: Add jobs to build EDK2 firmware binaries"),
> the naive plan was to have reproducible binaries by downloading
> what this job would build, testing it and eventually committing
> it. With retrospective, nothing happened 3 years later and this
> job is just bitrotting:
>
>   Step 1/3 : FROM ubuntu:18.04
>   18.04: Pulling from library/ubuntu
>   mediaType in manifest should be
>   'application/vnd.docker.distribution.manifest.v2+json' not
>   'application/vnd.oci.image.manifest.v1+json'
>
> Remove this job to avoid wasting maintenance and CI ressources.

Does the same thing hold for the opensbi job ?

thanks
-- PMM

Re: [PATCH-for-8.0] gitlab-ci: Remove job building EDK2 firmware binaries

[Philippe Mathieu-Daudé]

On 10/3/23 14:38, Peter Maydell wrote:
> On Fri, 10 Mar 2023 at 13:33, Philippe Mathieu-Daudé <philmd@linaro.org> wrote:
>>
>> When we introduced this Gitlab-CI job in commit 71920809ce
>> ("gitlab-ci.yml: Add jobs to build EDK2 firmware binaries"),
>> the naive plan was to have reproducible binaries by downloading
>> what this job would build, testing it and eventually committing
>> it. With retrospective, nothing happened 3 years later and this
>> job is just bitrotting:
>>
>>    Step 1/3 : FROM ubuntu:18.04
>>    18.04: Pulling from library/ubuntu
>>    mediaType in manifest should be
>>    'application/vnd.docker.distribution.manifest.v2+json' not
>>    'application/vnd.oci.image.manifest.v1+json'
>>
>> Remove this job to avoid wasting maintenance and CI ressources.
> 
> Does the same thing hold for the opensbi job ?

Cc'ing Bin, I have no idea.

Re: [PATCH-for-8.0] gitlab-ci: Remove job building EDK2 firmware binaries

[Bin Meng]

On Fri, Mar 10, 2023 at 9:50 PM Philippe Mathieu-Daudé
<philmd@linaro.org> wrote:
>
> On 10/3/23 14:38, Peter Maydell wrote:
> > On Fri, 10 Mar 2023 at 13:33, Philippe Mathieu-Daudé <philmd@linaro.org> wrote:
> >>
> >> When we introduced this Gitlab-CI job in commit 71920809ce
> >> ("gitlab-ci.yml: Add jobs to build EDK2 firmware binaries"),
> >> the naive plan was to have reproducible binaries by downloading
> >> what this job would build, testing it and eventually committing
> >> it. With retrospective, nothing happened 3 years later and this
> >> job is just bitrotting:
> >>
> >>    Step 1/3 : FROM ubuntu:18.04
> >>    18.04: Pulling from library/ubuntu
> >>    mediaType in manifest should be
> >>    'application/vnd.docker.distribution.manifest.v2+json' not
> >>    'application/vnd.oci.image.manifest.v1+json'
> >>
> >> Remove this job to avoid wasting maintenance and CI ressources.
> >
> > Does the same thing hold for the opensbi job ?
>
> Cc'ing Bin, I have no idea.
>

The OpenSBI job now builds fine. I have no preference on keeping vs.
removing it.

I remember our previous goal was to create CI jobs for every pc-bios
image but apparently that never happened.

Regards,
Bin

Re: [PATCH-for-8.0] gitlab-ci: Remove job building EDK2 firmware binaries

[Philippe Mathieu-Daudé]

On 13/3/23 03:09, Bin Meng wrote:
> On Fri, Mar 10, 2023 at 9:50 PM Philippe Mathieu-Daudé
> <philmd@linaro.org> wrote:
>>
>> On 10/3/23 14:38, Peter Maydell wrote:
>>> On Fri, 10 Mar 2023 at 13:33, Philippe Mathieu-Daudé <philmd@linaro.org> wrote:
>>>>
>>>> When we introduced this Gitlab-CI job in commit 71920809ce
>>>> ("gitlab-ci.yml: Add jobs to build EDK2 firmware binaries"),
>>>> the naive plan was to have reproducible binaries by downloading
>>>> what this job would build, testing it and eventually committing
>>>> it. With retrospective, nothing happened 3 years later and this
>>>> job is just bitrotting:
>>>>
>>>>     Step 1/3 : FROM ubuntu:18.04
>>>>     18.04: Pulling from library/ubuntu
>>>>     mediaType in manifest should be
>>>>     'application/vnd.docker.distribution.manifest.v2+json' not
>>>>     'application/vnd.oci.image.manifest.v1+json'
>>>>
>>>> Remove this job to avoid wasting maintenance and CI ressources.
>>>
>>> Does the same thing hold for the opensbi job ?
>>
>> Cc'ing Bin, I have no idea.
>>
> 
> The OpenSBI job now builds fine. I have no preference on keeping vs.
> removing it.
> 
> I remember our previous goal was to create CI jobs for every pc-bios
> image but apparently that never happened.

Yes, and I don't see interest in the community (neither worry that
pc-bios/ images committed are built on each maintainer workstations).

If it isn't consumed by QEMU, then better remove that job and save
precious CI minutes. I presume OpenSBI itself is already tested
by its mainstream project.

Re: [PATCH-for-8.0] gitlab-ci: Remove job building EDK2 firmware binaries

[Bin Meng]

Hi Philippe,

On Mon, Mar 13, 2023 at 4:51 PM Philippe Mathieu-Daudé
<philmd@linaro.org> wrote:
>
> On 13/3/23 03:09, Bin Meng wrote:
> > On Fri, Mar 10, 2023 at 9:50 PM Philippe Mathieu-Daudé
> > <philmd@linaro.org> wrote:
> >>
> >> On 10/3/23 14:38, Peter Maydell wrote:
> >>> On Fri, 10 Mar 2023 at 13:33, Philippe Mathieu-Daudé <philmd@linaro.org> wrote:
> >>>>
> >>>> When we introduced this Gitlab-CI job in commit 71920809ce
> >>>> ("gitlab-ci.yml: Add jobs to build EDK2 firmware binaries"),
> >>>> the naive plan was to have reproducible binaries by downloading
> >>>> what this job would build, testing it and eventually committing
> >>>> it. With retrospective, nothing happened 3 years later and this
> >>>> job is just bitrotting:
> >>>>
> >>>>     Step 1/3 : FROM ubuntu:18.04
> >>>>     18.04: Pulling from library/ubuntu
> >>>>     mediaType in manifest should be
> >>>>     'application/vnd.docker.distribution.manifest.v2+json' not
> >>>>     'application/vnd.oci.image.manifest.v1+json'
> >>>>
> >>>> Remove this job to avoid wasting maintenance and CI ressources.
> >>>
> >>> Does the same thing hold for the opensbi job ?
> >>
> >> Cc'ing Bin, I have no idea.
> >>
> >
> > The OpenSBI job now builds fine. I have no preference on keeping vs.
> > removing it.
> >
> > I remember our previous goal was to create CI jobs for every pc-bios
> > image but apparently that never happened.
>
> Yes, and I don't see interest in the community (neither worry that
> pc-bios/ images committed are built on each maintainer workstations).
>
> If it isn't consumed by QEMU, then better remove that job and save
> precious CI minutes. I presume OpenSBI itself is already tested
> by its mainstream project.

Not sure what does "consumed" here mean?

QEMU uses OpenSBI images on RISC-V machines by default.

Regards,
Bin

Re: [PATCH-for-8.0] gitlab-ci: Remove job building EDK2 firmware binaries

[Philippe Mathieu-Daudé]

On 13/3/23 10:35, Bin Meng wrote:
> Hi Philippe,
> 
> On Mon, Mar 13, 2023 at 4:51 PM Philippe Mathieu-Daudé
> <philmd@linaro.org> wrote:
>>
>> On 13/3/23 03:09, Bin Meng wrote:
>>> On Fri, Mar 10, 2023 at 9:50 PM Philippe Mathieu-Daudé
>>> <philmd@linaro.org> wrote:
>>>>
>>>> On 10/3/23 14:38, Peter Maydell wrote:
>>>>> On Fri, 10 Mar 2023 at 13:33, Philippe Mathieu-Daudé <philmd@linaro.org> wrote:
>>>>>>
>>>>>> When we introduced this Gitlab-CI job in commit 71920809ce
>>>>>> ("gitlab-ci.yml: Add jobs to build EDK2 firmware binaries"),
>>>>>> the naive plan was to have reproducible binaries by downloading
>>>>>> what this job would build, testing it and eventually committing
>>>>>> it. With retrospective, nothing happened 3 years later and this
>>>>>> job is just bitrotting:
>>>>>>
>>>>>>      Step 1/3 : FROM ubuntu:18.04
>>>>>>      18.04: Pulling from library/ubuntu
>>>>>>      mediaType in manifest should be
>>>>>>      'application/vnd.docker.distribution.manifest.v2+json' not
>>>>>>      'application/vnd.oci.image.manifest.v1+json'
>>>>>>
>>>>>> Remove this job to avoid wasting maintenance and CI ressources.
>>>>>
>>>>> Does the same thing hold for the opensbi job ?
>>>>
>>>> Cc'ing Bin, I have no idea.
>>>>
>>>
>>> The OpenSBI job now builds fine. I have no preference on keeping vs.
>>> removing it.
>>>
>>> I remember our previous goal was to create CI jobs for every pc-bios
>>> image but apparently that never happened.
>>
>> Yes, and I don't see interest in the community (neither worry that
>> pc-bios/ images committed are built on each maintainer workstations).
>>
>> If it isn't consumed by QEMU, then better remove that job and save
>> precious CI minutes. I presume OpenSBI itself is already tested
>> by its mainstream project.
> 
> Not sure what does "consumed" here mean?
> 
> QEMU uses OpenSBI images on RISC-V machines by default.

QEMU repository allows building QEMU system binaries which 'consume'
the following (committed) files:
- pc-bios/opensbi-riscv32-generic-fw_dynamic.bin
- pc-bios/opensbi-riscv64-generic-fw_dynamic.bin

We don't need to run the build-opensbi job to run QEMU: we use the
prebuilt images.

Re: [PATCH-for-8.0] gitlab-ci: Remove job building EDK2 firmware binaries

[Bin Meng]

On Mon, Mar 13, 2023 at 6:01 PM Philippe Mathieu-Daudé
<philmd@linaro.org> wrote:
>
> On 13/3/23 10:35, Bin Meng wrote:
> > Hi Philippe,
> >
> > On Mon, Mar 13, 2023 at 4:51 PM Philippe Mathieu-Daudé
> > <philmd@linaro.org> wrote:
> >>
> >> On 13/3/23 03:09, Bin Meng wrote:
> >>> On Fri, Mar 10, 2023 at 9:50 PM Philippe Mathieu-Daudé
> >>> <philmd@linaro.org> wrote:
> >>>>
> >>>> On 10/3/23 14:38, Peter Maydell wrote:
> >>>>> On Fri, 10 Mar 2023 at 13:33, Philippe Mathieu-Daudé <philmd@linaro.org> wrote:
> >>>>>>
> >>>>>> When we introduced this Gitlab-CI job in commit 71920809ce
> >>>>>> ("gitlab-ci.yml: Add jobs to build EDK2 firmware binaries"),
> >>>>>> the naive plan was to have reproducible binaries by downloading
> >>>>>> what this job would build, testing it and eventually committing
> >>>>>> it. With retrospective, nothing happened 3 years later and this
> >>>>>> job is just bitrotting:
> >>>>>>
> >>>>>>      Step 1/3 : FROM ubuntu:18.04
> >>>>>>      18.04: Pulling from library/ubuntu
> >>>>>>      mediaType in manifest should be
> >>>>>>      'application/vnd.docker.distribution.manifest.v2+json' not
> >>>>>>      'application/vnd.oci.image.manifest.v1+json'
> >>>>>>
> >>>>>> Remove this job to avoid wasting maintenance and CI ressources.
> >>>>>
> >>>>> Does the same thing hold for the opensbi job ?
> >>>>
> >>>> Cc'ing Bin, I have no idea.
> >>>>
> >>>
> >>> The OpenSBI job now builds fine. I have no preference on keeping vs.
> >>> removing it.
> >>>
> >>> I remember our previous goal was to create CI jobs for every pc-bios
> >>> image but apparently that never happened.
> >>
> >> Yes, and I don't see interest in the community (neither worry that
> >> pc-bios/ images committed are built on each maintainer workstations).
> >>
> >> If it isn't consumed by QEMU, then better remove that job and save
> >> precious CI minutes. I presume OpenSBI itself is already tested
> >> by its mainstream project.
> >
> > Not sure what does "consumed" here mean?
> >
> > QEMU uses OpenSBI images on RISC-V machines by default.
>
> QEMU repository allows building QEMU system binaries which 'consume'
> the following (committed) files:
> - pc-bios/opensbi-riscv32-generic-fw_dynamic.bin
> - pc-bios/opensbi-riscv64-generic-fw_dynamic.bin
>
> We don't need to run the build-opensbi job to run QEMU: we use the
> prebuilt images.

Okay, thanks for the clarification. I believe that's the case for
every pc-bios image? If yes, we don't need to build these pc-bios
images in QEMU CI.

Regards,
Bin