* [XEN PATCH 0/3] CI: Allow automatic rebuild of rolling release containers
@ 2024-11-15 17:07 Anthony PERARD
2024-11-15 17:07 ` [XEN PATCH 3/3] CI: New stage "containers" to rebuild some containers Anthony PERARD
` (3 more replies)
0 siblings, 4 replies; 11+ messages in thread
From: Anthony PERARD @ 2024-11-15 17:07 UTC (permalink / raw)
To: xen-devel
Cc: Andrew Cooper, Anthony PERARD, Doug Goldstein, Stefano Stabellini
Patch series available in this git branch:
https://xenbits.xenproject.org/git-http/people/aperard/xen-unstable.git br.gitlab-ci-rebuild-arch-container-v1
Hi,
This patch series would allow to setup a scheduled pipeline on GitLab which
would rebuild containers and run a test on them immediately (the container is
updated even if the test fail, but at least we can find out about it sooner
rather than later).
To enable this, just running a pipeline with XEN_CI_REBUILD_CONTAINERS=1 will
do.
I intend to add a scheduled pipeline to run once a month.
Thanks,
Anthony PERARD (3):
CI: Remove deprecated "only:variables" in favor of "rules:if"
CI: Define XEN_REGISTRY variable
CI: New stage "containers" to rebuild some containers
.gitlab-ci.yml | 5 +++
automation/gitlab-ci/build.yaml | 61 ++++++++++++++++++----------
automation/gitlab-ci/containers.yaml | 29 +++++++++++++
automation/gitlab-ci/test.yaml | 30 +++++++++-----
4 files changed, 93 insertions(+), 32 deletions(-)
create mode 100644 automation/gitlab-ci/containers.yaml
--
Anthony Perard | Vates XCP-ng Developer
XCP-ng & Xen Orchestra - Vates solutions
web: https://vates.tech
^ permalink raw reply [flat|nested] 11+ messages in thread* [XEN PATCH 3/3] CI: New stage "containers" to rebuild some containers 2024-11-15 17:07 [XEN PATCH 0/3] CI: Allow automatic rebuild of rolling release containers Anthony PERARD @ 2024-11-15 17:07 ` Anthony PERARD 2024-11-16 0:34 ` Stefano Stabellini 2024-11-15 17:07 ` [XEN PATCH 2/3] CI: Define XEN_REGISTRY variable Anthony PERARD ` (2 subsequent siblings) 3 siblings, 1 reply; 11+ messages in thread From: Anthony PERARD @ 2024-11-15 17:07 UTC (permalink / raw) To: xen-devel Cc: Andrew Cooper, Anthony PERARD, Doug Goldstein, Stefano Stabellini Rebuild rolling release containers when XEN_CI_REBUILD_CONTAINERS is set. This is to be use with a scheduled pipeline. When $XEN_CI_REBUILD_CONTAINERS is set, only build jobs related to the containers been rebuild will be executed. Build jobs that are using one of the containers been rebuild should wait for the container to be rebuild. If it's a normal pipeline, those dependency are simply ignored. Signed-off-by: Anthony PERARD <anthony.perard@vates.tech> --- .gitlab-ci.yml | 2 + automation/gitlab-ci/build.yaml | 57 ++++++++++++++++++---------- automation/gitlab-ci/containers.yaml | 29 ++++++++++++++ automation/gitlab-ci/test.yaml | 13 +++++++ 4 files changed, 81 insertions(+), 20 deletions(-) create mode 100644 automation/gitlab-ci/containers.yaml diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 941e5822e8..dab9171086 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -8,11 +8,13 @@ workflow: - when: always stages: + - containers - analyze - build - test include: + - 'automation/gitlab-ci/containers.yaml' - 'automation/gitlab-ci/analyze.yaml' - 'automation/gitlab-ci/build.yaml' - 'automation/gitlab-ci/test.yaml' diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml index 1ca6764225..e01eec1423 100644 --- a/automation/gitlab-ci/build.yaml +++ b/automation/gitlab-ci/build.yaml @@ -3,6 +3,9 @@ image: ${XEN_REGISTRY}/${CONTAINER} script: - ./automation/scripts/build 2>&1 | tee build.log + rules: + - if: $XEN_CI_REBUILD_CONTAINERS && $XEN_CI_RUN_AFTER_CONTAINER_BUILD + - if: $XEN_CI_REBUILD_CONTAINERS == null artifacts: paths: - binaries/ @@ -255,6 +258,8 @@ .test-jobs-artifact-common: stage: build needs: [] + rules: + - if: $XEN_CI_REBUILD_CONTAINERS == null # Arm test artifacts @@ -516,17 +521,24 @@ alpine-3.18-clang-debug: variables: CONTAINER: alpine:3.18 -archlinux-gcc: - extends: .gcc-x86-64-build +.container-archlinux-current: variables: CONTAINER: archlinux:current + XEN_CI_RUN_AFTER_CONTAINER_BUILD: true + needs: + - job: container-archlinux-current + optional: true allow_failure: true +archlinux-gcc: + extends: + - .gcc-x86-64-build + - .container-archlinux-current + archlinux-gcc-debug: - extends: .gcc-x86-64-build-debug - variables: - CONTAINER: archlinux:current - allow_failure: true + extends: + - .gcc-x86-64-build-debug + - .container-archlinux-current centos-7-gcc: extends: .gcc-x86-64-build @@ -657,29 +669,34 @@ opensuse-leap-15.6-gcc-debug: variables: CONTAINER: opensuse:leap-15.6-x86_64 -opensuse-tumbleweed-clang: - extends: .clang-x86-64-build +.container-opensuse-tumbleweed-x86_64: variables: CONTAINER: opensuse:tumbleweed-x86_64 + XEN_CI_RUN_AFTER_CONTAINER_BUILD: true + needs: + - job: container-opensuse-tumbleweed-x86_64 + optional: true allow_failure: true +opensuse-tumbleweed-clang: + extends: + - .clang-x86-64-build + - .container-opensuse-tumbleweed-x86_64 + opensuse-tumbleweed-clang-debug: - extends: .clang-x86-64-build-debug - variables: - CONTAINER: opensuse:tumbleweed-x86_64 - allow_failure: true + extends: + - .clang-x86-64-build-debug + - .container-opensuse-tumbleweed-x86_64 opensuse-tumbleweed-gcc: - extends: .gcc-x86-64-build - variables: - CONTAINER: opensuse:tumbleweed-x86_64 - allow_failure: true + extends: + - .gcc-x86-64-build + - .container-opensuse-tumbleweed-x86_64 opensuse-tumbleweed-gcc-debug: - extends: .gcc-x86-64-build-debug - variables: - CONTAINER: opensuse:tumbleweed-x86_64 - allow_failure: true + extends: + - .gcc-x86-64-build-debug + - .container-opensuse-tumbleweed-x86_64 # PowerPC builds (x86 cross) debian-11-ppc64le-gcc: diff --git a/automation/gitlab-ci/containers.yaml b/automation/gitlab-ci/containers.yaml new file mode 100644 index 0000000000..25e8bdc34b --- /dev/null +++ b/automation/gitlab-ci/containers.yaml @@ -0,0 +1,29 @@ +.container-build-tmpl: + stage: containers + image: docker:stable + tags: + - container-builder + rules: + - if: $XEN_CI_REBUILD_CONTAINERS + services: + - docker:dind + before_script: + - apk add make + - docker info + - docker login -u $CI_DEPLOY_USER -p $CI_DEPLOY_PASSWORD $CI_REGISTRY + script: + - make -C automation/build PUSH=1 REGISTRY=${XEN_REGISTRY} ${CONTAINER/:/\/} + after_script: + - docker logout + +container-archlinux-current: + extends: + - .container-build-tmpl + variables: + CONTAINER: "archlinux:current" + +container-opensuse-tumbleweed-x86_64: + extends: + - .container-build-tmpl + variables: + CONTAINER: "opensuse:tumbleweed-x86_64" diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml index 1822e3ea5f..e4d4f1f3c9 100644 --- a/automation/gitlab-ci/test.yaml +++ b/automation/gitlab-ci/test.yaml @@ -1,6 +1,16 @@ +.test-rules: + # Only add rules which prevent jobs from been added to the pipeline. They + # should all have "when: never". + pre_rules: + - if: $XEN_CI_REBUILD_CONTAINERS + when: never + .test-jobs-common: stage: test image: ${XEN_REGISTRY}/${CONTAINER} + rules: + - !reference [.test-rules, pre_rules] + - when: always .arm64-test-needs: &arm64-test-needs - alpine-3.18-arm64-rootfs-export @@ -99,6 +109,7 @@ - '*.dtb' when: always rules: + - !reference [.test-rules, pre_rules] - if: $XILINX_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true" tags: - xilinx @@ -117,6 +128,7 @@ - '*.log' when: always rules: + - !reference [.test-rules, pre_rules] - if: $XILINX_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true" tags: - xilinx @@ -136,6 +148,7 @@ - '*.log' when: always rules: + - !reference [.test-rules, pre_rules] - if: $QUBES_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true" tags: - qubes-hw2 -- Anthony Perard | Vates XCP-ng Developer XCP-ng & Xen Orchestra - Vates solutions web: https://vates.tech ^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [XEN PATCH 3/3] CI: New stage "containers" to rebuild some containers 2024-11-15 17:07 ` [XEN PATCH 3/3] CI: New stage "containers" to rebuild some containers Anthony PERARD @ 2024-11-16 0:34 ` Stefano Stabellini 2024-11-18 14:03 ` Anthony PERARD 0 siblings, 1 reply; 11+ messages in thread From: Stefano Stabellini @ 2024-11-16 0:34 UTC (permalink / raw) To: Anthony PERARD Cc: xen-devel, Andrew Cooper, Doug Goldstein, Stefano Stabellini On Fri, 15 Nov 2024, Anthony PERARD wrote: > Rebuild rolling release containers when XEN_CI_REBUILD_CONTAINERS is > set. This is to be use with a scheduled pipeline. > > When $XEN_CI_REBUILD_CONTAINERS is set, only build jobs related to the > containers been rebuild will be executed. > > Build jobs that are using one of the containers been rebuild should > wait for the container to be rebuild. If it's a normal pipeline, those > dependency are simply ignored. This is a fantastic contribution, thanks Anthony! I think we can simplify this patch by removing all stages except for "containers" on the scheduled pipeline with XEN_CI_REBUILD_CONTAINERS set to true. I think it is a good idea to have a special schedule pipeline for this, and we should exploit the fact that it is special and only use it to rebuild the containers. If we want to, we can have a second scheduled pipeline to do a full normal run afterwards. This way, there is no need to carry the changes to build.yaml or test.yaml that are a bit hard to read/understand for someone unfamiliar with gitlab. When XEN_CI_REBUILD_CONTAINERS == true we only do the containers stage. > Signed-off-by: Anthony PERARD <anthony.perard@vates.tech> > --- > .gitlab-ci.yml | 2 + > automation/gitlab-ci/build.yaml | 57 ++++++++++++++++++---------- > automation/gitlab-ci/containers.yaml | 29 ++++++++++++++ > automation/gitlab-ci/test.yaml | 13 +++++++ > 4 files changed, 81 insertions(+), 20 deletions(-) > create mode 100644 automation/gitlab-ci/containers.yaml > > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml > index 941e5822e8..dab9171086 100644 > --- a/.gitlab-ci.yml > +++ b/.gitlab-ci.yml > @@ -8,11 +8,13 @@ workflow: > - when: always > > stages: > + - containers > - analyze > - build > - test > > include: > + - 'automation/gitlab-ci/containers.yaml' > - 'automation/gitlab-ci/analyze.yaml' > - 'automation/gitlab-ci/build.yaml' > - 'automation/gitlab-ci/test.yaml' > diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml > index 1ca6764225..e01eec1423 100644 > --- a/automation/gitlab-ci/build.yaml > +++ b/automation/gitlab-ci/build.yaml > @@ -3,6 +3,9 @@ > image: ${XEN_REGISTRY}/${CONTAINER} > script: > - ./automation/scripts/build 2>&1 | tee build.log > + rules: > + - if: $XEN_CI_REBUILD_CONTAINERS && $XEN_CI_RUN_AFTER_CONTAINER_BUILD > + - if: $XEN_CI_REBUILD_CONTAINERS == null > artifacts: > paths: > - binaries/ > @@ -255,6 +258,8 @@ > .test-jobs-artifact-common: > stage: build > needs: [] > + rules: > + - if: $XEN_CI_REBUILD_CONTAINERS == null > > # Arm test artifacts > > @@ -516,17 +521,24 @@ alpine-3.18-clang-debug: > variables: > CONTAINER: alpine:3.18 > > -archlinux-gcc: > - extends: .gcc-x86-64-build > +.container-archlinux-current: > variables: > CONTAINER: archlinux:current > + XEN_CI_RUN_AFTER_CONTAINER_BUILD: true > + needs: > + - job: container-archlinux-current > + optional: true > allow_failure: true > > +archlinux-gcc: > + extends: > + - .gcc-x86-64-build > + - .container-archlinux-current > + > archlinux-gcc-debug: > - extends: .gcc-x86-64-build-debug > - variables: > - CONTAINER: archlinux:current > - allow_failure: true > + extends: > + - .gcc-x86-64-build-debug > + - .container-archlinux-current > > centos-7-gcc: > extends: .gcc-x86-64-build > @@ -657,29 +669,34 @@ opensuse-leap-15.6-gcc-debug: > variables: > CONTAINER: opensuse:leap-15.6-x86_64 > > -opensuse-tumbleweed-clang: > - extends: .clang-x86-64-build > +.container-opensuse-tumbleweed-x86_64: > variables: > CONTAINER: opensuse:tumbleweed-x86_64 > + XEN_CI_RUN_AFTER_CONTAINER_BUILD: true > + needs: > + - job: container-opensuse-tumbleweed-x86_64 > + optional: true > allow_failure: true > > +opensuse-tumbleweed-clang: > + extends: > + - .clang-x86-64-build > + - .container-opensuse-tumbleweed-x86_64 > + > opensuse-tumbleweed-clang-debug: > - extends: .clang-x86-64-build-debug > - variables: > - CONTAINER: opensuse:tumbleweed-x86_64 > - allow_failure: true > + extends: > + - .clang-x86-64-build-debug > + - .container-opensuse-tumbleweed-x86_64 > > opensuse-tumbleweed-gcc: > - extends: .gcc-x86-64-build > - variables: > - CONTAINER: opensuse:tumbleweed-x86_64 > - allow_failure: true > + extends: > + - .gcc-x86-64-build > + - .container-opensuse-tumbleweed-x86_64 > > opensuse-tumbleweed-gcc-debug: > - extends: .gcc-x86-64-build-debug > - variables: > - CONTAINER: opensuse:tumbleweed-x86_64 > - allow_failure: true > + extends: > + - .gcc-x86-64-build-debug > + - .container-opensuse-tumbleweed-x86_64 > > # PowerPC builds (x86 cross) > debian-11-ppc64le-gcc: > diff --git a/automation/gitlab-ci/containers.yaml b/automation/gitlab-ci/containers.yaml > new file mode 100644 > index 0000000000..25e8bdc34b > --- /dev/null > +++ b/automation/gitlab-ci/containers.yaml > @@ -0,0 +1,29 @@ > +.container-build-tmpl: > + stage: containers > + image: docker:stable > + tags: > + - container-builder > + rules: > + - if: $XEN_CI_REBUILD_CONTAINERS > + services: > + - docker:dind > + before_script: > + - apk add make > + - docker info > + - docker login -u $CI_DEPLOY_USER -p $CI_DEPLOY_PASSWORD $CI_REGISTRY > + script: > + - make -C automation/build PUSH=1 REGISTRY=${XEN_REGISTRY} ${CONTAINER/:/\/} > + after_script: > + - docker logout > + > +container-archlinux-current: > + extends: > + - .container-build-tmpl > + variables: > + CONTAINER: "archlinux:current" > + > +container-opensuse-tumbleweed-x86_64: > + extends: > + - .container-build-tmpl > + variables: > + CONTAINER: "opensuse:tumbleweed-x86_64" > diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml > index 1822e3ea5f..e4d4f1f3c9 100644 > --- a/automation/gitlab-ci/test.yaml > +++ b/automation/gitlab-ci/test.yaml > @@ -1,6 +1,16 @@ > +.test-rules: > + # Only add rules which prevent jobs from been added to the pipeline. They > + # should all have "when: never". > + pre_rules: > + - if: $XEN_CI_REBUILD_CONTAINERS > + when: never > + > .test-jobs-common: > stage: test > image: ${XEN_REGISTRY}/${CONTAINER} > + rules: > + - !reference [.test-rules, pre_rules] > + - when: always > > .arm64-test-needs: &arm64-test-needs > - alpine-3.18-arm64-rootfs-export > @@ -99,6 +109,7 @@ > - '*.dtb' > when: always > rules: > + - !reference [.test-rules, pre_rules] > - if: $XILINX_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true" > tags: > - xilinx > @@ -117,6 +128,7 @@ > - '*.log' > when: always > rules: > + - !reference [.test-rules, pre_rules] > - if: $XILINX_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true" > tags: > - xilinx > @@ -136,6 +148,7 @@ > - '*.log' > when: always > rules: > + - !reference [.test-rules, pre_rules] > - if: $QUBES_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true" > tags: > - qubes-hw2 > -- > > > Anthony Perard | Vates XCP-ng Developer > > XCP-ng & Xen Orchestra - Vates solutions > > web: https://vates.tech > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [XEN PATCH 3/3] CI: New stage "containers" to rebuild some containers 2024-11-16 0:34 ` Stefano Stabellini @ 2024-11-18 14:03 ` Anthony PERARD 2024-11-19 0:04 ` Stefano Stabellini 0 siblings, 1 reply; 11+ messages in thread From: Anthony PERARD @ 2024-11-18 14:03 UTC (permalink / raw) To: Stefano Stabellini; +Cc: xen-devel, Andrew Cooper, Doug Goldstein On Fri, Nov 15, 2024 at 04:34:21PM -0800, Stefano Stabellini wrote: > On Fri, 15 Nov 2024, Anthony PERARD wrote: > > Rebuild rolling release containers when XEN_CI_REBUILD_CONTAINERS is > > set. This is to be use with a scheduled pipeline. > > > > When $XEN_CI_REBUILD_CONTAINERS is set, only build jobs related to the > > containers been rebuild will be executed. > > > > Build jobs that are using one of the containers been rebuild should > > wait for the container to be rebuild. If it's a normal pipeline, those > > dependency are simply ignored. > > This is a fantastic contribution, thanks Anthony! > > I think we can simplify this patch by removing all stages except for > "containers" on the scheduled pipeline with XEN_CI_REBUILD_CONTAINERS > set to true. > > I think it is a good idea to have a special schedule pipeline for this, > and we should exploit the fact that it is special and only use it to > rebuild the containers. If we want to, we can have a second scheduled > pipeline to do a full normal run afterwards. > > This way, there is no need to carry the changes to build.yaml or > test.yaml that are a bit hard to read/understand for someone unfamiliar > with gitlab. When XEN_CI_REBUILD_CONTAINERS == true we only do the > containers stage. So, you mean having this following change instead of make changes to build.yaml and test.yaml, right? diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -14,7 +14,15 @@ stages: - test include: - - 'automation/gitlab-ci/containers.yaml' - - 'automation/gitlab-ci/analyze.yaml' - - 'automation/gitlab-ci/build.yaml' - - 'automation/gitlab-ci/test.yaml' + - local: 'automation/gitlab-ci/containers.yaml' + rules: + - if: $XEN_CI_REBUILD_CONTAINERS + - local: 'automation/gitlab-ci/analyze.yaml' + rules: + - if: $XEN_CI_REBUILD_CONTAINERS == null + - local: 'automation/gitlab-ci/build.yaml' + rules: + - if: $XEN_CI_REBUILD_CONTAINERS == null + - local: 'automation/gitlab-ci/test.yaml' + rules: + - if: $XEN_CI_REBUILD_CONTAINERS == null Or I guess we can also compare to the string "true" and write the expression as if: $XEN_CI_REBUILD_CONTAINERS == "true" and if: $XEN_CI_REBUILD_CONTAINERS != "true" Cheers, -- Anthony Perard | Vates XCP-ng Developer XCP-ng & Xen Orchestra - Vates solutions web: https://vates.tech ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [XEN PATCH 3/3] CI: New stage "containers" to rebuild some containers 2024-11-18 14:03 ` Anthony PERARD @ 2024-11-19 0:04 ` Stefano Stabellini 0 siblings, 0 replies; 11+ messages in thread From: Stefano Stabellini @ 2024-11-19 0:04 UTC (permalink / raw) To: Anthony PERARD Cc: Stefano Stabellini, xen-devel, Andrew Cooper, Doug Goldstein On Mon, 18 Nov 2024, Anthony PERARD wrote: > On Fri, Nov 15, 2024 at 04:34:21PM -0800, Stefano Stabellini wrote: > > On Fri, 15 Nov 2024, Anthony PERARD wrote: > > > Rebuild rolling release containers when XEN_CI_REBUILD_CONTAINERS is > > > set. This is to be use with a scheduled pipeline. > > > > > > When $XEN_CI_REBUILD_CONTAINERS is set, only build jobs related to the > > > containers been rebuild will be executed. > > > > > > Build jobs that are using one of the containers been rebuild should > > > wait for the container to be rebuild. If it's a normal pipeline, those > > > dependency are simply ignored. > > > > This is a fantastic contribution, thanks Anthony! > > > > I think we can simplify this patch by removing all stages except for > > "containers" on the scheduled pipeline with XEN_CI_REBUILD_CONTAINERS > > set to true. > > > > I think it is a good idea to have a special schedule pipeline for this, > > and we should exploit the fact that it is special and only use it to > > rebuild the containers. If we want to, we can have a second scheduled > > pipeline to do a full normal run afterwards. > > > > This way, there is no need to carry the changes to build.yaml or > > test.yaml that are a bit hard to read/understand for someone unfamiliar > > with gitlab. When XEN_CI_REBUILD_CONTAINERS == true we only do the > > containers stage. > > So, you mean having this following change instead of make changes to > build.yaml and test.yaml, right? > > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml > --- a/.gitlab-ci.yml > +++ b/.gitlab-ci.yml > @@ -14,7 +14,15 @@ stages: > - test > > include: > - - 'automation/gitlab-ci/containers.yaml' > - - 'automation/gitlab-ci/analyze.yaml' > - - 'automation/gitlab-ci/build.yaml' > - - 'automation/gitlab-ci/test.yaml' > + - local: 'automation/gitlab-ci/containers.yaml' > + rules: > + - if: $XEN_CI_REBUILD_CONTAINERS > + - local: 'automation/gitlab-ci/analyze.yaml' > + rules: > + - if: $XEN_CI_REBUILD_CONTAINERS == null > + - local: 'automation/gitlab-ci/build.yaml' > + rules: > + - if: $XEN_CI_REBUILD_CONTAINERS == null > + - local: 'automation/gitlab-ci/test.yaml' > + rules: > + - if: $XEN_CI_REBUILD_CONTAINERS == null > > > Or I guess we can also compare to the string "true" and write the > expression as > if: $XEN_CI_REBUILD_CONTAINERS == "true" > and > if: $XEN_CI_REBUILD_CONTAINERS != "true" Yes exactly, either one of the above is fine! It is much simpler and clearer than having to add (admittedly non-obvious) checks to build.yaml and test.yaml. Thanks! ^ permalink raw reply [flat|nested] 11+ messages in thread
* [XEN PATCH 2/3] CI: Define XEN_REGISTRY variable 2024-11-15 17:07 [XEN PATCH 0/3] CI: Allow automatic rebuild of rolling release containers Anthony PERARD 2024-11-15 17:07 ` [XEN PATCH 3/3] CI: New stage "containers" to rebuild some containers Anthony PERARD @ 2024-11-15 17:07 ` Anthony PERARD 2024-11-16 0:34 ` Stefano Stabellini 2024-11-15 17:07 ` [XEN PATCH 1/3] CI: Remove deprecated "only:variables" in favor of "rules:if" Anthony PERARD 2024-11-15 17:13 ` [XEN PATCH 0/3] CI: Allow automatic rebuild of rolling release containers Andrew Cooper 3 siblings, 1 reply; 11+ messages in thread From: Anthony PERARD @ 2024-11-15 17:07 UTC (permalink / raw) To: xen-devel Cc: Andrew Cooper, Anthony PERARD, Doug Goldstein, Stefano Stabellini This allow to change the registry used for container in a single place, and could be controlled via other mean. Signed-off-by: Anthony PERARD <anthony.perard@vates.tech> --- .gitlab-ci.yml | 3 +++ automation/gitlab-ci/build.yaml | 4 ++-- automation/gitlab-ci/test.yaml | 2 +- 3 files changed, 6 insertions(+), 3 deletions(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index ef4484e09a..941e5822e8 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -1,3 +1,6 @@ +variables: + XEN_REGISTRY: registry.gitlab.com/xen-project/xen + workflow: rules: - if: $CI_COMMIT_BRANCH =~ /^(master|smoke|^coverity-tested\/.*|stable-.*)$/ diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml index ce39b0ee21..1ca6764225 100644 --- a/automation/gitlab-ci/build.yaml +++ b/automation/gitlab-ci/build.yaml @@ -1,6 +1,6 @@ .build-tmpl: &build stage: build - image: registry.gitlab.com/xen-project/xen/${CONTAINER} + image: ${XEN_REGISTRY}/${CONTAINER} script: - ./automation/scripts/build 2>&1 | tee build.log artifacts: @@ -208,7 +208,7 @@ .yocto-test: stage: build - image: registry.gitlab.com/xen-project/xen/${CONTAINER} + image: ${XEN_REGISTRY}/${CONTAINER} script: - ./automation/build/yocto/build-yocto.sh -v --log-dir=./logs --xen-dir=`pwd` ${YOCTO_BOARD} ${YOCTO_OUTPUT} variables: diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml index ab5c8be0cd..1822e3ea5f 100644 --- a/automation/gitlab-ci/test.yaml +++ b/automation/gitlab-ci/test.yaml @@ -1,6 +1,6 @@ .test-jobs-common: stage: test - image: registry.gitlab.com/xen-project/xen/${CONTAINER} + image: ${XEN_REGISTRY}/${CONTAINER} .arm64-test-needs: &arm64-test-needs - alpine-3.18-arm64-rootfs-export -- Anthony Perard | Vates XCP-ng Developer XCP-ng & Xen Orchestra - Vates solutions web: https://vates.tech ^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [XEN PATCH 2/3] CI: Define XEN_REGISTRY variable 2024-11-15 17:07 ` [XEN PATCH 2/3] CI: Define XEN_REGISTRY variable Anthony PERARD @ 2024-11-16 0:34 ` Stefano Stabellini 0 siblings, 0 replies; 11+ messages in thread From: Stefano Stabellini @ 2024-11-16 0:34 UTC (permalink / raw) To: Anthony PERARD Cc: xen-devel, Andrew Cooper, Doug Goldstein, Stefano Stabellini On Fri, 15 Nov 2024, Anthony PERARD wrote: > This allow to change the registry used for container in a single > place, and could be controlled via other mean. > > Signed-off-by: Anthony PERARD <anthony.perard@vates.tech> Reviewed-by: Stefano Stabellini <sstabellini@kernel.org> > --- > .gitlab-ci.yml | 3 +++ > automation/gitlab-ci/build.yaml | 4 ++-- > automation/gitlab-ci/test.yaml | 2 +- > 3 files changed, 6 insertions(+), 3 deletions(-) > > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml > index ef4484e09a..941e5822e8 100644 > --- a/.gitlab-ci.yml > +++ b/.gitlab-ci.yml > @@ -1,3 +1,6 @@ > +variables: > + XEN_REGISTRY: registry.gitlab.com/xen-project/xen > + > workflow: > rules: > - if: $CI_COMMIT_BRANCH =~ /^(master|smoke|^coverity-tested\/.*|stable-.*)$/ > diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml > index ce39b0ee21..1ca6764225 100644 > --- a/automation/gitlab-ci/build.yaml > +++ b/automation/gitlab-ci/build.yaml > @@ -1,6 +1,6 @@ > .build-tmpl: &build > stage: build > - image: registry.gitlab.com/xen-project/xen/${CONTAINER} > + image: ${XEN_REGISTRY}/${CONTAINER} > script: > - ./automation/scripts/build 2>&1 | tee build.log > artifacts: > @@ -208,7 +208,7 @@ > > .yocto-test: > stage: build > - image: registry.gitlab.com/xen-project/xen/${CONTAINER} > + image: ${XEN_REGISTRY}/${CONTAINER} > script: > - ./automation/build/yocto/build-yocto.sh -v --log-dir=./logs --xen-dir=`pwd` ${YOCTO_BOARD} ${YOCTO_OUTPUT} > variables: > diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml > index ab5c8be0cd..1822e3ea5f 100644 > --- a/automation/gitlab-ci/test.yaml > +++ b/automation/gitlab-ci/test.yaml > @@ -1,6 +1,6 @@ > .test-jobs-common: > stage: test > - image: registry.gitlab.com/xen-project/xen/${CONTAINER} > + image: ${XEN_REGISTRY}/${CONTAINER} > > .arm64-test-needs: &arm64-test-needs > - alpine-3.18-arm64-rootfs-export > -- > > > Anthony Perard | Vates XCP-ng Developer > > XCP-ng & Xen Orchestra - Vates solutions > > web: https://vates.tech > ^ permalink raw reply [flat|nested] 11+ messages in thread
* [XEN PATCH 1/3] CI: Remove deprecated "only:variables" in favor of "rules:if" 2024-11-15 17:07 [XEN PATCH 0/3] CI: Allow automatic rebuild of rolling release containers Anthony PERARD 2024-11-15 17:07 ` [XEN PATCH 3/3] CI: New stage "containers" to rebuild some containers Anthony PERARD 2024-11-15 17:07 ` [XEN PATCH 2/3] CI: Define XEN_REGISTRY variable Anthony PERARD @ 2024-11-15 17:07 ` Anthony PERARD 2024-11-16 0:34 ` Stefano Stabellini 2024-11-15 17:13 ` [XEN PATCH 0/3] CI: Allow automatic rebuild of rolling release containers Andrew Cooper 3 siblings, 1 reply; 11+ messages in thread From: Anthony PERARD @ 2024-11-15 17:07 UTC (permalink / raw) To: xen-devel Cc: Andrew Cooper, Anthony PERARD, Doug Goldstein, Stefano Stabellini Also, this prevent using "rules", like in the ".test-jobs-common" template. https://docs.gitlab.com/ee/ci/yaml/#only--except Signed-off-by: Anthony PERARD <anthony.perard@vates.tech> --- automation/gitlab-ci/test.yaml | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml index f5dd4de757..ab5c8be0cd 100644 --- a/automation/gitlab-ci/test.yaml +++ b/automation/gitlab-ci/test.yaml @@ -98,9 +98,8 @@ - '*.log' - '*.dtb' when: always - only: - variables: - - $XILINX_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true" + rules: + - if: $XILINX_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true" tags: - xilinx @@ -117,9 +116,8 @@ - smoke.serial - '*.log' when: always - only: - variables: - - $XILINX_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true" + rules: + - if: $XILINX_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true" tags: - xilinx @@ -137,9 +135,8 @@ - smoke.serial - '*.log' when: always - only: - variables: - - $QUBES_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true" + rules: + - if: $QUBES_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true" tags: - qubes-hw2 -- Anthony Perard | Vates XCP-ng Developer XCP-ng & Xen Orchestra - Vates solutions web: https://vates.tech ^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [XEN PATCH 1/3] CI: Remove deprecated "only:variables" in favor of "rules:if" 2024-11-15 17:07 ` [XEN PATCH 1/3] CI: Remove deprecated "only:variables" in favor of "rules:if" Anthony PERARD @ 2024-11-16 0:34 ` Stefano Stabellini 0 siblings, 0 replies; 11+ messages in thread From: Stefano Stabellini @ 2024-11-16 0:34 UTC (permalink / raw) To: Anthony PERARD Cc: xen-devel, Andrew Cooper, Doug Goldstein, Stefano Stabellini On Fri, 15 Nov 2024, Anthony PERARD wrote: > Also, this prevent using "rules", like in the ".test-jobs-common" > template. > > https://docs.gitlab.com/ee/ci/yaml/#only--except > > Signed-off-by: Anthony PERARD <anthony.perard@vates.tech> Reviewed-by: Stefano Stabellini <sstabellini@kernel.org> > --- > automation/gitlab-ci/test.yaml | 15 ++++++--------- > 1 file changed, 6 insertions(+), 9 deletions(-) > > diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml > index f5dd4de757..ab5c8be0cd 100644 > --- a/automation/gitlab-ci/test.yaml > +++ b/automation/gitlab-ci/test.yaml > @@ -98,9 +98,8 @@ > - '*.log' > - '*.dtb' > when: always > - only: > - variables: > - - $XILINX_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true" > + rules: > + - if: $XILINX_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true" > tags: > - xilinx > > @@ -117,9 +116,8 @@ > - smoke.serial > - '*.log' > when: always > - only: > - variables: > - - $XILINX_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true" > + rules: > + - if: $XILINX_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true" > tags: > - xilinx > > @@ -137,9 +135,8 @@ > - smoke.serial > - '*.log' > when: always > - only: > - variables: > - - $QUBES_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true" > + rules: > + - if: $QUBES_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true" > tags: > - qubes-hw2 > > -- > > > Anthony Perard | Vates XCP-ng Developer > > XCP-ng & Xen Orchestra - Vates solutions > > web: https://vates.tech > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [XEN PATCH 0/3] CI: Allow automatic rebuild of rolling release containers 2024-11-15 17:07 [XEN PATCH 0/3] CI: Allow automatic rebuild of rolling release containers Anthony PERARD ` (2 preceding siblings ...) 2024-11-15 17:07 ` [XEN PATCH 1/3] CI: Remove deprecated "only:variables" in favor of "rules:if" Anthony PERARD @ 2024-11-15 17:13 ` Andrew Cooper 2024-11-18 12:48 ` Anthony PERARD 3 siblings, 1 reply; 11+ messages in thread From: Andrew Cooper @ 2024-11-15 17:13 UTC (permalink / raw) To: Anthony PERARD, xen-devel; +Cc: Doug Goldstein, Stefano Stabellini On 15/11/2024 5:07 pm, Anthony PERARD wrote: > Patch series available in this git branch: > https://xenbits.xenproject.org/git-http/people/aperard/xen-unstable.git br.gitlab-ci-rebuild-arch-container-v1 > > Hi, > > This patch series would allow to setup a scheduled pipeline on GitLab which > would rebuild containers and run a test on them immediately (the container is > updated even if the test fail, but at least we can find out about it sooner > rather than later). > > To enable this, just running a pipeline with XEN_CI_REBUILD_CONTAINERS=1 will > do. > > I intend to add a scheduled pipeline to run once a month. Oh excellent. Thanks for looking into this. One thing I was going to look into doing was to push the new containers to a temporary repository (or a temp name in the main repository), kick off a regular pipeline overriding image:, and on success doing a rename. That would avoid pushing a container with problems into main CI. ~Andrew ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [XEN PATCH 0/3] CI: Allow automatic rebuild of rolling release containers 2024-11-15 17:13 ` [XEN PATCH 0/3] CI: Allow automatic rebuild of rolling release containers Andrew Cooper @ 2024-11-18 12:48 ` Anthony PERARD 0 siblings, 0 replies; 11+ messages in thread From: Anthony PERARD @ 2024-11-18 12:48 UTC (permalink / raw) To: Andrew Cooper; +Cc: xen-devel, Doug Goldstein, Stefano Stabellini On Fri, Nov 15, 2024 at 05:13:13PM +0000, Andrew Cooper wrote: > On 15/11/2024 5:07 pm, Anthony PERARD wrote: > > Patch series available in this git branch: > > https://xenbits.xenproject.org/git-http/people/aperard/xen-unstable.git br.gitlab-ci-rebuild-arch-container-v1 > > > > Hi, > > > > This patch series would allow to setup a scheduled pipeline on GitLab which > > would rebuild containers and run a test on them immediately (the container is > > updated even if the test fail, but at least we can find out about it sooner > > rather than later). > > > > To enable this, just running a pipeline with XEN_CI_REBUILD_CONTAINERS=1 will > > do. > > > > I intend to add a scheduled pipeline to run once a month. > > Oh excellent. Thanks for looking into this. > > One thing I was going to look into doing was to push the new containers > to a temporary repository (or a temp name in the main repository), kick > off a regular pipeline overriding image:, and on success doing a rename. > > That would avoid pushing a container with problems into main CI. I tried something like that before, but I probably didn't approch the problem from the right angle: [RFC XEN PATCH 0/7] automation, RFC prototype, Have GitLab CI built its own containers https://lore.kernel.org/xen-devel/20230302175332.56052-1-anthony.perard@citrix.com/#r I was trying to automatically rebuild container whose Dockerfile had changed, while also having logic to rebuild containers based on rolling-release distributions. But I did end-up duplicating "build.yaml". There's logic in that old series to test a container before "renaming it" (which involve pulling then pushing the container with a new tag) With this new series, I've introduced XEN_REGISTRY, so we could schedule a pipeline by setting XEN_REGISTRY variable to a staging area, and finish the pipeline by pushing to the main location. But, the two containers been updated in this series are used in jobs that are allowed to fail, so updating them don't have any consequence. Also, there's no value in keeping to test with outdated container as that mean we test with an environment that doesn't exist anymore. Cheers, -- Anthony Perard | Vates XCP-ng Developer XCP-ng & Xen Orchestra - Vates solutions web: https://vates.tech ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2024-11-19 0:05 UTC | newest] Thread overview: 11+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2024-11-15 17:07 [XEN PATCH 0/3] CI: Allow automatic rebuild of rolling release containers Anthony PERARD 2024-11-15 17:07 ` [XEN PATCH 3/3] CI: New stage "containers" to rebuild some containers Anthony PERARD 2024-11-16 0:34 ` Stefano Stabellini 2024-11-18 14:03 ` Anthony PERARD 2024-11-19 0:04 ` Stefano Stabellini 2024-11-15 17:07 ` [XEN PATCH 2/3] CI: Define XEN_REGISTRY variable Anthony PERARD 2024-11-16 0:34 ` Stefano Stabellini 2024-11-15 17:07 ` [XEN PATCH 1/3] CI: Remove deprecated "only:variables" in favor of "rules:if" Anthony PERARD 2024-11-16 0:34 ` Stefano Stabellini 2024-11-15 17:13 ` [XEN PATCH 0/3] CI: Allow automatic rebuild of rolling release containers Andrew Cooper 2024-11-18 12:48 ` Anthony PERARD
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.