From: "Alex Bennée" <alex.bennee@linaro.org>
To: peter.maydell@linaro.org
Cc: "Thomas Huth" <thuth@redhat.com>,
"Daniel P. Berrangé" <berrange@redhat.com>,
"Philippe Mathieu-Daudé" <philmd@redhat.com>,
qemu-devel@nongnu.org,
"Wainer dos Santos Moschetta" <wainersm@redhat.com>,
"Alex Bennée" <alex.bennee@linaro.org>
Subject: [PULL 19/41] gitlab: build all container images during CI
Date: Tue, 7 Jul 2020 08:08:36 +0100 [thread overview]
Message-ID: <20200707070858.6622-20-alex.bennee@linaro.org> (raw)
In-Reply-To: <20200707070858.6622-1-alex.bennee@linaro.org>
From: Daniel P. Berrangé <berrange@redhat.com>
We have a number of container images in tests/docker/dockerfiles
that are intended to provide well defined environments for doing
test builds. We want our CI system to use these containers too.
This introduces builds of all of them as the first stage in the
CI, so that the built containers are available for later build
jobs. The containers are setup to use the GitLab container
registry as the cache, so we only pay the penalty of the full
build when the dockerfiles change. The main qemu-project/qemu
repo is used as a second cache, so that users forking QEMU will
see a fast turnaround time on their CI jobs.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20200622153318.751107-3-berrange@redhat.com>
[AJB: tweak the tag format]
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Acked-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20200701135652.1366-22-alex.bennee@linaro.org>
diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
new file mode 100644
index 000000000000..2afd01f1d179
--- /dev/null
+++ b/.gitlab-ci.d/containers.yml
@@ -0,0 +1,245 @@
+.container_job_template: &container_job_definition
+ image: docker:stable
+ stage: containers
+ services:
+ - docker:dind
+ before_script:
+ - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest"
+ - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/$NAME:latest"
+ - docker info
+ - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
+ script:
+ - docker pull "$TAG" || docker pull "$COMMON_TAG" || true
+ - sed -i -e "s,FROM qemu/,FROM $CI_REGISTRY_IMAGE/qemu/," tests/docker/dockerfiles/$NAME.docker
+ - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "tests/docker/dockerfiles/$NAME.docker" tests/docker/dockerfiles
+ - docker push "$TAG"
+ after_script:
+ - docker logout
+
+amd64-centos7-container:
+ <<: *container_job_definition
+ variables:
+ NAME: centos7
+
+amd64-centos8-container:
+ <<: *container_job_definition
+ variables:
+ NAME: centos8
+
+amd64-debian10-container:
+ <<: *container_job_definition
+ variables:
+ NAME: debian10
+
+amd64-debian11-container:
+ <<: *container_job_definition
+ variables:
+ NAME: debian11
+
+amd64-debian9-container:
+ <<: *container_job_definition
+ variables:
+ NAME: debian9
+
+amd64-debian9-mxe-container:
+ <<: *container_job_definition
+ stage: containers-layer2
+ needs: ['amd64-debian9-container']
+ variables:
+ NAME: debian9-mxe
+
+alpha-debian-cross-container:
+ <<: *container_job_definition
+ stage: containers-layer2
+ needs: ['amd64-debian10-container']
+ variables:
+ NAME: debian-alpha-cross
+
+amd64-debian-cross-container:
+ <<: *container_job_definition
+ stage: containers-layer2
+ needs: ['amd64-debian10-container']
+ variables:
+ NAME: debian-amd64-cross
+
+amd64-debian-container:
+ <<: *container_job_definition
+ stage: containers-layer2
+ needs: ['amd64-debian10-container']
+ variables:
+ NAME: debian-amd64
+
+arm64-debian-cross-container:
+ <<: *container_job_definition
+ stage: containers-layer2
+ needs: ['amd64-debian10-container']
+ variables:
+ NAME: debian-arm64-cross
+
+arm64-test-debian-cross-container:
+ <<: *container_job_definition
+ stage: containers-layer2
+ needs: ['amd64-debian11-container']
+ variables:
+ NAME: debian-arm64-test-cross
+
+armel-debian-cross-container:
+ <<: *container_job_definition
+ stage: containers-layer2
+ needs: ['amd64-debian10-container']
+ variables:
+ NAME: debian-armel-cross
+
+armhf-debian-cross-container:
+ <<: *container_job_definition
+ stage: containers-layer2
+ needs: ['amd64-debian10-container']
+ variables:
+ NAME: debian-armhf-cross
+
+hppa-debian-cross-container:
+ <<: *container_job_definition
+ stage: containers-layer2
+ needs: ['amd64-debian10-container']
+ variables:
+ NAME: debian-hppa-cross
+
+m68k-debian-cross-container:
+ <<: *container_job_definition
+ stage: containers-layer2
+ needs: ['amd64-debian10-container']
+ variables:
+ NAME: debian-m68k-cross
+
+mips64-debian-cross-container:
+ <<: *container_job_definition
+ stage: containers-layer2
+ needs: ['amd64-debian10-container']
+ variables:
+ NAME: debian-mips64-cross
+
+mips64el-debian-cross-container:
+ <<: *container_job_definition
+ stage: containers-layer2
+ needs: ['amd64-debian10-container']
+ variables:
+ NAME: debian-mips64el-cross
+
+mips-debian-cross-container:
+ <<: *container_job_definition
+ stage: containers-layer2
+ needs: ['amd64-debian10-container']
+ variables:
+ NAME: debian-mips-cross
+
+mipsel-debian-cross-container:
+ <<: *container_job_definition
+ stage: containers-layer2
+ needs: ['amd64-debian10-container']
+ variables:
+ NAME: debian-mipsel-cross
+
+powerpc-debian-cross-container:
+ <<: *container_job_definition
+ stage: containers-layer2
+ needs: ['amd64-debian10-container']
+ variables:
+ NAME: debian-powerpc-cross
+
+ppc64-debian-cross-container:
+ <<: *container_job_definition
+ stage: containers-layer2
+ needs: ['amd64-debian10-container']
+ variables:
+ NAME: debian-ppc64-cross
+
+ppc64el-debian-cross-container:
+ <<: *container_job_definition
+ stage: containers-layer2
+ needs: ['amd64-debian10-container']
+ variables:
+ NAME: debian-ppc64el-cross
+
+riscv64-debian-cross-container:
+ <<: *container_job_definition
+ stage: containers-layer2
+ needs: ['amd64-debian10-container']
+ variables:
+ NAME: debian-riscv64-cross
+
+s390x-debian-cross-container:
+ <<: *container_job_definition
+ stage: containers-layer2
+ needs: ['amd64-debian10-container']
+ variables:
+ NAME: debian-s390x-cross
+
+sh4-debian-cross-container:
+ <<: *container_job_definition
+ stage: containers-layer2
+ needs: ['amd64-debian10-container']
+ variables:
+ NAME: debian-sh4-cross
+
+sparc64-debian-cross-container:
+ <<: *container_job_definition
+ stage: containers-layer2
+ needs: ['amd64-debian10-container']
+ variables:
+ NAME: debian-sparc64-cross
+
+tricore-debian-cross-container:
+ <<: *container_job_definition
+ stage: containers-layer2
+ needs: ['amd64-debian9-container']
+ variables:
+ NAME: debian-tricore-cross
+
+win32-debian-cross-container:
+ <<: *container_job_definition
+ stage: containers-layer3
+ needs: ['amd64-debian9-mxe-container']
+ variables:
+ NAME: debian-win32-cross
+
+win64-debian-cross-container:
+ <<: *container_job_definition
+ stage: containers-layer3
+ needs: ['amd64-debian9-mxe-container']
+ variables:
+ NAME: debian-win64-cross
+
+xtensa-debian-cross-container:
+ <<: *container_job_definition
+ variables:
+ NAME: debian-xtensa-cross
+
+cris-fedora-cross-container:
+ <<: *container_job_definition
+ variables:
+ NAME: fedora-cris-cross
+
+amd64-fedora-container:
+ <<: *container_job_definition
+ variables:
+ NAME: fedora
+
+i386-fedora-cross-container:
+ <<: *container_job_definition
+ variables:
+ NAME: fedora-i386-cross
+
+amd64-ubuntu1804-container:
+ <<: *container_job_definition
+ variables:
+ NAME: ubuntu1804
+
+amd64-ubuntu2004-container:
+ <<: *container_job_definition
+ variables:
+ NAME: ubuntu2004
+
+amd64-ubuntu-container:
+ <<: *container_job_definition
+ variables:
+ NAME: ubuntu
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 9fdc752ea636..72d688875fd7 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -1,10 +1,13 @@
stages:
- containers
+ - containers-layer2
+ - containers-layer3
- build
include:
- local: '/.gitlab-ci.d/edk2.yml'
- local: '/.gitlab-ci.d/opensbi.yml'
+ - local: '/.gitlab-ci.d/containers.yml'
.update_apt_template: &before_script_apt
before_script:
--
2.20.1
next prev parent reply other threads:[~2020-07-07 7:25 UTC|newest]
Thread overview: 56+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-07-07 7:08 [PULL 00/41] testing updates (vm, gitlab, misc build fixes) Alex Bennée
2020-07-07 7:08 ` [PULL 01/41] crypto/linux_keyring: fix 'secret_keyring' configure test Alex Bennée
2020-07-07 7:08 ` [PULL 02/41] util/coroutine: Cleanup start_switch_fiber_ for TSAN Alex Bennée
2020-07-07 7:08 ` [PULL 03/41] tests/vm: pass args through to BaseVM's __init__ Alex Bennée
2020-07-07 7:08 ` [PULL 04/41] tests/vm: Add configuration to basevm.py Alex Bennée
2020-07-10 13:50 ` Alex Bennée
2020-07-07 7:08 ` [PULL 05/41] tests/vm: Added configuration file support Alex Bennée
2020-07-07 7:08 ` [PULL 06/41] tests/vm: Add common Ubuntu python module Alex Bennée
2020-07-07 7:08 ` [PULL 07/41] tests/vm: Added a new script for ubuntu.aarch64 Alex Bennée
2020-07-07 7:08 ` [PULL 08/41] tests/vm: Added a new script for centos.aarch64 Alex Bennée
2020-07-07 7:08 ` [PULL 09/41] tests/vm: change scripts to use self._config Alex Bennée
2020-07-07 7:08 ` [PULL 10/41] python/qemu: Add ConsoleSocket for optional use in QEMUMachine Alex Bennée
2020-07-10 19:20 ` John Snow
2020-07-11 16:15 ` Robert Foley
2020-07-11 17:45 ` Alex Bennée
2020-07-13 13:57 ` John Snow
2020-07-13 14:16 ` Philippe Mathieu-Daudé
2020-07-13 14:37 ` Eduardo Habkost
2020-07-07 7:08 ` [PULL 11/41] tests/vm: Add workaround to consume console Alex Bennée
2020-07-07 7:08 ` [PULL 12/41] tests/vm: switch from optsparse to argparse Alex Bennée
2020-07-07 7:08 ` [PULL 13/41] tests/vm: allow us to take advantage of MTTCG Alex Bennée
2020-07-07 7:08 ` [PULL 14/41] tests/docker: check for an parameters not empty string Alex Bennée
2020-07-07 7:08 ` [PULL 15/41] tests/docker: change tag naming scheme of our images Alex Bennée
2020-07-07 7:08 ` [PULL 16/41] .gitignore: un-ignore .gitlab-ci.d Alex Bennée
2020-07-07 7:08 ` [PULL 17/41] gitlab-ci: Fix the change rules after moving the YML files Alex Bennée
2020-07-07 7:08 ` [PULL 18/41] gitlab: introduce explicit "container" and "build" stages Alex Bennée
2020-07-07 7:08 ` Alex Bennée [this message]
2020-07-07 7:08 ` [PULL 20/41] gitlab: convert jobs to use custom built containers Alex Bennée
2020-07-07 7:08 ` [PULL 21/41] gitlab: build containers with buildkit and metadata Alex Bennée
2020-07-07 7:08 ` [PULL 22/41] tests/docker: add --registry support to tooling Alex Bennée
2020-07-07 7:08 ` [PULL 23/41] tests/docker: add packages needed for check-acceptance Alex Bennée
2020-07-07 7:08 ` [PULL 24/41] tests/acceptance: skip s390x_ccw_vrtio_tcg on GitLab Alex Bennée
2020-07-07 7:08 ` [PULL 25/41] tests/acceptance: fix dtb path for machine_rx_gdbsim Alex Bennée
2020-07-07 7:08 ` [PULL 26/41] tests/acceptance: skip multicore mips_malta tests on GitLab Alex Bennée
2020-07-07 7:08 ` [PULL 27/41] tests/acceptance: skip LinuxInitrd 2gib with v4.16 " Alex Bennée
2020-07-07 7:08 ` [PULL 28/41] gitlab: add acceptance testing to system builds Alex Bennée
2020-09-08 19:54 ` Philippe Mathieu-Daudé
2020-09-09 6:03 ` Thomas Huth
2020-07-07 7:08 ` [PULL 29/41] tests/tcg: add more default compilers to configure.sh Alex Bennée
2020-07-07 7:08 ` [PULL 30/41] tests/docker: add a linux-user testing focused image Alex Bennée
2020-07-07 7:08 ` [PULL 31/41] linux-user/elfload: use MAP_FIXED_NOREPLACE in pgb_reserved_va Alex Bennée
2020-07-07 7:08 ` [PULL 32/41] gitlab: enable check-tcg for linux-user tests Alex Bennée
2020-07-07 7:08 ` [PULL 33/41] gitlab: add avocado asset caching Alex Bennée
2020-07-07 7:08 ` [PULL 34/41] gitlab: split build-disabled into two phases Alex Bennée
2020-07-07 7:08 ` [PULL 35/41] gitlab: limit re-builds of the containers Alex Bennée
2020-07-07 7:08 ` [PULL 36/41] containers.yml: build with docker.py tooling Alex Bennée
2020-07-07 7:08 ` [PULL 37/41] testing: add check-build target Alex Bennée
2020-07-07 7:08 ` [PULL 38/41] shippable: pull images from registry instead of building Alex Bennée
2020-07-07 7:08 ` [PULL 39/41] travis.yml: Test also the other targets on s390x Alex Bennée
2020-07-07 7:08 ` [PULL 40/41] tests/qht-bench: Adjust testing rate by -1 Alex Bennée
2020-07-07 7:08 ` [PULL 41/41] tests/qht-bench: Adjust threshold computation Alex Bennée
2020-07-07 9:39 ` [PULL 00/41] testing updates (vm, gitlab, misc build fixes) Alex Bennée
2020-07-09 11:31 ` Peter Maydell
2020-07-09 12:24 ` Philippe Mathieu-Daudé
2020-07-09 13:04 ` Peter Maydell
2020-07-09 15:46 ` Alex Bennée
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200707070858.6622-20-alex.bennee@linaro.org \
--to=alex.bennee@linaro.org \
--cc=berrange@redhat.com \
--cc=peter.maydell@linaro.org \
--cc=philmd@redhat.com \
--cc=qemu-devel@nongnu.org \
--cc=thuth@redhat.com \
--cc=wainersm@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).