[XEN PATCH v2 0/3] CI: Allow automatic rebuild of rolling release containers

[XEN PATCH v2 0/3] CI: Allow automatic rebuild of rolling release containers

[Anthony PERARD]

Patch series available in this git branch:
https://xenbits.xenproject.org/git-http/people/aperard/xen-unstable.git br.gitlab-ci-rebuild-arch-container-v2

v2:
- remove the logic to test containers once rebuilt, and only do container build
  in the pipeline.

Hi,

This patch series would allow to setup a scheduled pipeline on GitLab which
would rebuild containers.

To enable this, just running a pipeline with XEN_CI_REBUILD_CONTAINERS=1 will
do.

I intend to add a scheduled pipeline to run once a month.

Thanks,

Anthony PERARD (3):
  CI: Remove deprecated "only:variables" in favor of "rules:if"
  CI: Define XEN_REGISTRY variable
  CI: New stage "containers" to rebuild some containers

 .gitlab-ci.yml                       | 19 +++++++++++++++---
 automation/gitlab-ci/build.yaml      |  4 ++--
 automation/gitlab-ci/containers.yaml | 29 ++++++++++++++++++++++++++++
 automation/gitlab-ci/test.yaml       | 17 +++++++---------
 4 files changed, 54 insertions(+), 15 deletions(-)
 create mode 100644 automation/gitlab-ci/containers.yaml

-- 


Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

[XEN PATCH v2 2/3] CI: Define XEN_REGISTRY variable

[Anthony PERARD]

This allow to change the registry used for container in a single
place, and could be controlled via other mean.

Signed-off-by: Anthony PERARD <anthony.perard@vates.tech>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
---
 .gitlab-ci.yml                  | 3 +++
 automation/gitlab-ci/build.yaml | 4 ++--
 automation/gitlab-ci/test.yaml  | 2 +-
 3 files changed, 6 insertions(+), 3 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index ef4484e09a..941e5822e8 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -1,3 +1,6 @@
+variables:
+  XEN_REGISTRY: registry.gitlab.com/xen-project/xen
+
 workflow:
   rules:
     - if: $CI_COMMIT_BRANCH =~ /^(master|smoke|^coverity-tested\/.*|stable-.*)$/
diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index ce39b0ee21..1ca6764225 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -1,6 +1,6 @@
 .build-tmpl: &build
   stage: build
-  image: registry.gitlab.com/xen-project/xen/${CONTAINER}
+  image: ${XEN_REGISTRY}/${CONTAINER}
   script:
     - ./automation/scripts/build 2>&1 | tee build.log
   artifacts:
@@ -208,7 +208,7 @@
 
 .yocto-test:
   stage: build
-  image: registry.gitlab.com/xen-project/xen/${CONTAINER}
+  image: ${XEN_REGISTRY}/${CONTAINER}
   script:
     - ./automation/build/yocto/build-yocto.sh -v --log-dir=./logs --xen-dir=`pwd` ${YOCTO_BOARD} ${YOCTO_OUTPUT}
   variables:
diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
index ab5c8be0cd..1822e3ea5f 100644
--- a/automation/gitlab-ci/test.yaml
+++ b/automation/gitlab-ci/test.yaml
@@ -1,6 +1,6 @@
 .test-jobs-common:
   stage: test
-  image: registry.gitlab.com/xen-project/xen/${CONTAINER}
+  image: ${XEN_REGISTRY}/${CONTAINER}
 
 .arm64-test-needs: &arm64-test-needs
   - alpine-3.18-arm64-rootfs-export
-- 


Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

[XEN PATCH v2 1/3] CI: Remove deprecated "only:variables" in favor of "rules:if"

[Anthony PERARD]

Also, this prevent using "rules", like in the ".test-jobs-common"
template.

https://docs.gitlab.com/ee/ci/yaml/#only--except

Signed-off-by: Anthony PERARD <anthony.perard@vates.tech>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
---
 automation/gitlab-ci/test.yaml | 15 ++++++---------
 1 file changed, 6 insertions(+), 9 deletions(-)

diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
index f5dd4de757..ab5c8be0cd 100644
--- a/automation/gitlab-ci/test.yaml
+++ b/automation/gitlab-ci/test.yaml
@@ -98,9 +98,8 @@
       - '*.log'
       - '*.dtb'
     when: always
-  only:
-    variables:
-      - $XILINX_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true"
+  rules:
+    - if: $XILINX_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true"
   tags:
     - xilinx
 
@@ -117,9 +116,8 @@
       - smoke.serial
       - '*.log'
     when: always
-  only:
-    variables:
-      - $XILINX_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true"
+  rules:
+    - if: $XILINX_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true"
   tags:
     - xilinx
 
@@ -137,9 +135,8 @@
       - smoke.serial
       - '*.log'
     when: always
-  only:
-    variables:
-      - $QUBES_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true"
+  rules:
+    - if: $QUBES_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true"
   tags:
     - qubes-hw2
 
-- 


Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

[XEN PATCH v2 3/3] CI: New stage "containers" to rebuild some containers

[Anthony PERARD]

Rebuild rolling release containers when XEN_CI_REBUILD_CONTAINERS is
set. This is to be use with a scheduled pipeline.

Signed-off-by: Anthony PERARD <anthony.perard@vates.tech>
---

Notes:
    v2:
    - remove the logic to test containers once rebuilt.
    - only include the config with jobs for the "containers" stage wen
      XEN_CI_REBUILD_CONTAINERS is set.

 .gitlab-ci.yml                       | 16 ++++++++++++---
 automation/gitlab-ci/containers.yaml | 29 ++++++++++++++++++++++++++++
 2 files changed, 42 insertions(+), 3 deletions(-)
 create mode 100644 automation/gitlab-ci/containers.yaml

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 941e5822e8..5a9b8b7228 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -8,11 +8,21 @@ workflow:
     - when: always
 
 stages:
+  - containers
   - analyze
   - build
   - test
 
 include:
-  - 'automation/gitlab-ci/analyze.yaml'
-  - 'automation/gitlab-ci/build.yaml'
-  - 'automation/gitlab-ci/test.yaml'
+  - local: 'automation/gitlab-ci/containers.yaml'
+    rules:
+      - if: $XEN_CI_REBUILD_CONTAINERS
+  - local: 'automation/gitlab-ci/analyze.yaml'
+    rules:
+      - if: $XEN_CI_REBUILD_CONTAINERS == null
+  - local: 'automation/gitlab-ci/build.yaml'
+    rules:
+      - if: $XEN_CI_REBUILD_CONTAINERS == null
+  - local: 'automation/gitlab-ci/test.yaml'
+    rules:
+      - if: $XEN_CI_REBUILD_CONTAINERS == null
diff --git a/automation/gitlab-ci/containers.yaml b/automation/gitlab-ci/containers.yaml
new file mode 100644
index 0000000000..25e8bdc34b
--- /dev/null
+++ b/automation/gitlab-ci/containers.yaml
@@ -0,0 +1,29 @@
+.container-build-tmpl:
+  stage: containers
+  image: docker:stable
+  tags:
+    - container-builder
+  rules:
+    - if: $XEN_CI_REBUILD_CONTAINERS
+  services:
+    - docker:dind
+  before_script:
+    - apk add make
+    - docker info
+    - docker login -u $CI_DEPLOY_USER -p $CI_DEPLOY_PASSWORD $CI_REGISTRY
+  script:
+    - make -C automation/build PUSH=1 REGISTRY=${XEN_REGISTRY} ${CONTAINER/:/\/}
+  after_script:
+    - docker logout
+
+container-archlinux-current:
+  extends:
+    - .container-build-tmpl
+  variables:
+    CONTAINER: "archlinux:current"
+
+container-opensuse-tumbleweed-x86_64:
+  extends:
+    - .container-build-tmpl
+  variables:
+    CONTAINER: "opensuse:tumbleweed-x86_64"
-- 


Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

Re: [XEN PATCH v2 3/3] CI: New stage "containers" to rebuild some containers

[Stefano Stabellini]

On Tue, 19 Nov 2024, Anthony PERARD wrote:
> Rebuild rolling release containers when XEN_CI_REBUILD_CONTAINERS is
> set. This is to be use with a scheduled pipeline.
> 
> Signed-off-by: Anthony PERARD <anthony.perard@vates.tech>

Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>


> ---
> 
> Notes:
>     v2:
>     - remove the logic to test containers once rebuilt.
>     - only include the config with jobs for the "containers" stage wen
>       XEN_CI_REBUILD_CONTAINERS is set.
> 
>  .gitlab-ci.yml                       | 16 ++++++++++++---
>  automation/gitlab-ci/containers.yaml | 29 ++++++++++++++++++++++++++++
>  2 files changed, 42 insertions(+), 3 deletions(-)
>  create mode 100644 automation/gitlab-ci/containers.yaml
> 
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 941e5822e8..5a9b8b7228 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -8,11 +8,21 @@ workflow:
>      - when: always
>  
>  stages:
> +  - containers
>    - analyze
>    - build
>    - test
>  
>  include:
> -  - 'automation/gitlab-ci/analyze.yaml'
> -  - 'automation/gitlab-ci/build.yaml'
> -  - 'automation/gitlab-ci/test.yaml'
> +  - local: 'automation/gitlab-ci/containers.yaml'
> +    rules:
> +      - if: $XEN_CI_REBUILD_CONTAINERS
> +  - local: 'automation/gitlab-ci/analyze.yaml'
> +    rules:
> +      - if: $XEN_CI_REBUILD_CONTAINERS == null
> +  - local: 'automation/gitlab-ci/build.yaml'
> +    rules:
> +      - if: $XEN_CI_REBUILD_CONTAINERS == null
> +  - local: 'automation/gitlab-ci/test.yaml'
> +    rules:
> +      - if: $XEN_CI_REBUILD_CONTAINERS == null
> diff --git a/automation/gitlab-ci/containers.yaml b/automation/gitlab-ci/containers.yaml
> new file mode 100644
> index 0000000000..25e8bdc34b
> --- /dev/null
> +++ b/automation/gitlab-ci/containers.yaml
> @@ -0,0 +1,29 @@
> +.container-build-tmpl:
> +  stage: containers
> +  image: docker:stable
> +  tags:
> +    - container-builder
> +  rules:
> +    - if: $XEN_CI_REBUILD_CONTAINERS
> +  services:
> +    - docker:dind
> +  before_script:
> +    - apk add make
> +    - docker info
> +    - docker login -u $CI_DEPLOY_USER -p $CI_DEPLOY_PASSWORD $CI_REGISTRY
> +  script:
> +    - make -C automation/build PUSH=1 REGISTRY=${XEN_REGISTRY} ${CONTAINER/:/\/}
> +  after_script:
> +    - docker logout
> +
> +container-archlinux-current:
> +  extends:
> +    - .container-build-tmpl
> +  variables:
> +    CONTAINER: "archlinux:current"
> +
> +container-opensuse-tumbleweed-x86_64:
> +  extends:
> +    - .container-build-tmpl
> +  variables:
> +    CONTAINER: "opensuse:tumbleweed-x86_64"
> -- 
> 
> 
> Anthony Perard | Vates XCP-ng Developer
> 
> XCP-ng & Xen Orchestra - Vates solutions
> 
> web: https://vates.tech
> 

Re: [XEN PATCH v2 3/3] CI: New stage "containers" to rebuild some containers

[Anthony PERARD]

On Tue, Nov 19, 2024 at 01:47:16PM -0800, Stefano Stabellini wrote:
> On Tue, 19 Nov 2024, Anthony PERARD wrote:
> > Rebuild rolling release containers when XEN_CI_REBUILD_CONTAINERS is
> > set. This is to be use with a scheduled pipeline.
> > 
> > Signed-off-by: Anthony PERARD <anthony.perard@vates.tech>
> 
> Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>

Thanks!

I've scheduled a pipeline to run once a month so the Arch Linux and
Tumbleweed containers we'll get rebuilt regularly. Both container are
already up-to-date because I failed to use a different registry during
one of my test, but the build-jobs still works.

Cheers,

-- 

Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech