Re: [PATCH v4 16/22] tests/docker: add script for automating container refresh

["Alex Bennée" <alex.bennee@linaro.org>]

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Mon, Jul 05, 2021 at 02:44:34PM +0100, Alex Bennée wrote:
>> 
>> Daniel P. Berrangé <berrange@redhat.com> writes:
>> 
>> > This introduces
>> >
>> >   https://gitlab.com/libvirt/libvirt-ci
>> >
>> > as a git submodule at tests/docker/libvirt-ci
>> >
>> > This submodule only needs to be checked out when needing to re-generate
>> > the files in tests/docker/dockerfiles.
>> >
>> > When a new build pre-requisite is needed for QEMU, it should be added to
>> > the libvirt-ci project 'qemu.yml' file, and the submodule updated to the
>> > new commit.
>> 
>> It seems a bit weird to have the canonical description of QEMU
>> dependencies live in another project does it not?
>
> Yes, this is something I've been struggling with, since there
> are varying use cases here.
>
> The "lcitool" command was originally written to automate the
> provisioning of virtual machines, suitable to act as runners
> for a CI tool.
>
> The VMs would be suitable for building a variety of projects,
> so would need to be installed with the dependancies of all
> projects. It makes sense to have the list of dependancies
> in one central place. If you have them kept in each respective
> project's git repo, then you have to checkout 20 git repos to
> get their dependancies, before you can provision the VM.
>
> It still supports VM provisioning, but now also supports
> the Dockerfile generation too in parallel. With the
> dockerfiles, you still have a need to access the dependancy
> information from outside the main project. For example,
> when building libvirt-perl.git, it wants to know the
> dependancies needed by libvirt.git, so that it can do
> a chained build of the two.
>
> Potentially libvirt would also want to build qemu.git
> first, so it can then test libvirt with latest QEMU.
>
> So these things all end up driving towards the idea of
> storing the build dependancies separate from the project.
>
> It is definitely sub-optimal though, in that it introduces
> a synchronization problem between the 2 respective git
> repos for changes.
>
> For libvirt we've mostly just accepted that pain of needing
> to merge stuff in lock-step, but I think it is worse when
> dealing with QEMU becasue the subsystem maintainer model
> means stuff merges in 2 phases, so there's not a ideal
> synchronization point.
>
>> > The 'make docker-refresh' target will then re-create all the
>> > dockerfiles with updated package lists. This ensures that all the
>> > containers get exactly the same build pre-requisite packages installed.
>> >
>> > It also facilitates the addition of containers targetting new distros
>> > or updating existing containers to new versions of the same distro,
>> > where packages might have been renamed.
>> >
>> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
>> > ---
>> >  docs/devel/testing.rst              | 34 ++++++++++++++++--
>> >  tests/docker/Makefile.include       | 12 +++++++
>> >  tests/docker/dockerfiles-refresh.py | 56 +++++++++++++++++++++++++++++
>> >  3 files changed, 100 insertions(+), 2 deletions(-)
>> >  create mode 100755 tests/docker/dockerfiles-refresh.py
>> >
>> > diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
>> > index 4e42392810..7882db85d4 100644
>> > --- a/docs/devel/testing.rst
>> > +++ b/docs/devel/testing.rst
>> > @@ -372,8 +372,38 @@ Along with many other images, the ``centos8`` image is defined in a Dockerfile
>> >  in ``tests/docker/dockerfiles/``, called ``centos8.docker``. ``make docker-help``
>> >  command will list all the available images.
>> >  
>> > -To add a new image, simply create a new ``.docker`` file under the
>> > -``tests/docker/dockerfiles/`` directory.
>> > +Most of the existing Dockerfiles were written by hand, simply by creating a
>> > +a new ``.docker`` file under the ``tests/docker/dockerfiles/`` directory.
>> > +This has led to an inconsistent set of packages being present across the
>> > +different containers.
>> > +
>> > +Thus going forward, QEMU is aiming to automatically generate the Dockerfiles
>> > +using the ``lcitool`` program provided by the ``libvirt-ci`` project:
>> > +
>> > +  https://gitlab.com/libvirt/libvirt-ci
>> > +
>> > +In that project, there is a ``qemu.yml`` file defining the list of build
>> > +pre-requisites needed by QEMU. This is processed together with the
>> > +``mappings.yml`` file to compute the distro specific list of package names.
>> > +The package names are then fed into a generator which emits a well structured
>> > +dockerfile. The set of dockerfiles which are auto-generated is defined in
>> > +the ``tests/docker/dockerfiles-refresh.py`` script.
>> > +
>> > +When preparing a patch series that changes dockerfiles managed by ``libvirt-ci``
>> > +tools, the following steps should be takenL
>> > +
>> > + * Fork the ``libvirt-ci`` project on gitlab
>> > +
>> > + * Prepare changes to its ``qemu.yml`` file and optionally ``mappings.yml``
>> > +   to define the packages to be added to QEMU's dockerfiles.
>> > +
>> > + * In QEMU run ``make docker-refresh LCITOOL=/path/to/libvirt-ci/lcitool``
>> > +   to re-create the dockerfiles in ``tests/docker/dockerfiles``
>> 
>> If lcitool could be a pre-requisite (even as a developer only one)
>> should we consider having a submodule and QEMU mirror of it?
>
> I did have a submodule in the previous posting, but that creates its
> own pain, because there's a chicken and egg problem to updates. Stuff
> won't want to be merged into libvirt-ci.git until it is accepted by
> a QEMU maintainer, but we need the submodule update ready before
> it can be accepted by the QEMU maintainer. There's no nice way to
> break that cycle without introducing a bit of manual work and
> synchoronization at time of merge to master, which is not desirable
> for QEMU IMHO

Can't lcitool be improved to accept out-of-its-tree metadata? 

>> > + * Submit your changes to QEMU in the normal manner
>> > +
>> > + * Submit ``libvirt-ci`` changes as a merge request, linking to the
>> > +   QEMU patch series that uses them.
>> 
>> This just seems clunky and likely to therefor not get used. I would
>> prefer keeping the meta-data within the project, maybe with a check that
>> ensures the dockerfiles have not gone out of sync with their "idealised"
>> form.
>
>
> Regards,
> Daniel


-- 
Alex Bennée