Re: [RFC XEN PATCH 0/7] automation, RFC prototype, Have GitLab CI built its own containers

[Anthony PERARD <anthony.perard@citrix.com>]

On Thu, Mar 02, 2023 at 06:48:35PM -0800, Stefano Stabellini wrote:
> On Thu, 2 Mar 2023, Anthony PERARD wrote:
> > Patch series available in this git branch:
> > https://xenbits.xen.org/git-http/people/aperard/xen-unstable.git br.gitlab-containers-auto-rebuild-v1
> > 
> > Hi,
> > 
> > I have done some research to be able to build containers in the CI. This works
> > only for x86 containers as I've setup only a single x86 gitlab-runner to be
> > able to run docker-in-docker.
> > 
> > The runner is setup to only accept jobs from a branch that is "protected" in
> > gitlab. Also, one need credential to push to the container register, those are
> > called "Group deploy tokens", and I've set the variables CI_DEPLOY_USER and
> > CI_DEPLOY_PASSWORD in the project "xen-project/xen" (variables only visible on
> > a pipeline running on a protected branch).
> > 
> > These patch introduce quite a lot of redundancies in jobs, 2 new jobs per
> > containers which build/push containers, and duplicate most of build.yaml.
> > This mean that if we go with this, we have to duplicate and keep in sync many
> > jobs.
> > 
> > To avoid having to do the duplicated jobs by hand, I could look at
> > creating a script that use "build.yaml" as input and produce the 3
> > stages needed to update a container, but that script would need to be
> > run by hand, as gitlab can't really use it, unless ..
> > 
> > I've look at generated pipeline, and they look like this in gitlab:
> >     https://gitlab.com/xen-project/people/anthonyper/xen/-/pipelines/777665738
> > But the result of the generated/child pipeline doesn't seems to be taken into
> > account in the original pipeline, which make me think that we can't use them to
> > generate "build.yaml". But maybe the could be use for generating the pipeline
> > that will update a container.
> > Doc:
> >     https://docs.gitlab.com/ee/ci/pipelines/downstream_pipelines.html#dynamic-child-pipelines
> > 
> > So, with all of this, is it reasonable to test containers before
> > pushing them to production? Or is it to much work? We could simply have jobs
> > tacking care of rebuilding a container and pushing them to production without
> > testing.
> 
> I don't think it is a good idea to duplicate build.yaml, also because
> some of the containers are used in the testing stage too, so an updated
> container could be OK during the build phase and break the testing
> phase. We would need to duplicate both build.yaml and test.yaml, which
> is not feasible.
> 
> In practice today people either:
> 1) re-build a container locally & test it locally before pushing
> 2) re-build a container locally, docker push it, then run a private
>    gitlab pipeline, if it passes send out a patch to xen-devel
> 
> 1) is not affected by this series
> 2) is also not affected because by the time the pipeline is created, the
> container is already updated
> 
> However, there are cases where it would definitely be nice to have a
> "button" to press to update a container. For instance, when a pipeline
> failis due to a Debian unstable apt-get failure, which can be easily fixed
> by updating the Debian unstable container.
> 
> So I think it would be nice to have jobs that can automatically update
> the build containers but I would set them to manually trigger instead of
> automatically (when: manual).

What I was looking at with this work was to be able to have container
been rebuild automatically on a schedule. We have/had containers that
were 3yr old, and when it's a container that supposed to test the lasted
version of a distro, or a rolling release distro, they are kind of
useless if they aren't rebuild regularly. So I was looking to take the
human out of the loop and have computers the tedious work of rebuilding a
container every month or two.

Containers that needs to be rebuilt regularly to stay relevant are
archlinux, debian/unstable, fedora/latest, propably opensuse/leap and
opensuse/tumbleweed. I don't know if they are others.


> Alternatively, we could move the "containers.yaml" stage to be the first
> step, rebuild the containers and push them to a "staging" area
> (registry.gitlab.com/xen-project/xen/staging), run the build and test
> steps fetching from the staging area instead of the regular, if all
> tests pass, then push the containers to
> registry.gitlab.com/xen-project/xen as last step.

Sounds good, I can look into how easily it would be to use a different
registry to run a pipeline.

Cheers,

-- 
Anthony PERARD