Re: [RFC] container images for migration-test

[Fabiano Rosas <farosas@suse.de>]

Peter Xu <peterx@redhat.com> writes:

> On Mon, Sep 14, 2026 at 08:58:09PM -0300, Fabiano Rosas wrote:
>> Hi, I just wanted to share and ask for feedback on some container images
>> I've been building as part of my testing of migration pull-requests.
>> 
>> From a Dockerfile from ./tests/docker/dockerfiles/ I'm producing the
>> following:
>> 
>> $ docker images
>> qemu-build-base:11.0.0                        b6efa198e93a        5.8GB
>> qemu-rootfs-base:11.0.0                       72bcb5418f4d       4.08GB
>> qemu:11.0.0                                   018a3f0f4dcc          2GB
>> 
>> This last image is stripped (for size) of everything but a few libs and
>> binaries (qemu, gdb, migration-test, sh) and has an entrypoint of:
>> 
>>  CMD ["qemu-system-x86_64"]
>> 
>> The point is that I can wrap it in a script called qemu-system-x86_64
>> and run it interchangeably with any other qemu binary during
>> migration-test:
>> 
>>  QTEST_QEMU_BINARY_SRC=./build-11.0.0/qemu-system-x86_64 \
>>  QTEST_QEMU_BINARY=./build-master/qemu-system-x86_64 \
>>  ./tests/qtest/migration-test
>> 
>> (note that migration-test spawns the qemu binaries itself)
>> 
>> I run tests against the previous 3 QEMU releases. At every release I
>> build the container for the new release and at every PR I build the
>> container for that git hash using the latest tag as a base.
>
> I'm not a container person,

Me neither, lol. Last container work I did was back when multiarch
manifests were being added to docker. Maybe 2016? I (clearly) haven't
kept up with the progress in the area.

> e.g. I did test my recent channel fix on
> qemu-7.2 by simply building it with build-7.2/.. but after learning it the
> idea using a link to kickoff the container is very interesting on its own
> already.
>

Yeah, the issue is that `git checkout <old version>` and try to build it
is pretty hit or miss.

>> 
>> The reasons that lead me to this solution are:
>> 
>> 1) Migration compat testing is special, the two migration peers can run
>> in different OS versions, so their QEMU builds can be quite different. I
>> cannot test using *just* my dev machine, neither using *just* a
>> container. I need two hosts. Or two containers running different OSes
>> that can both be reachable by the migration-test binary running in
>> either of the OSes.
>
> True..  One thing I want to make us on the same page is, this is about
> distro, not OS?  Say, the kernel is still the same.
>

Yes, it's about what it takes to get a working QEMU binary. I'm not
trying to cover a realistic migration scenario, I'm simply trying to
test across QEMU versions *somehow*.

> So we have three things (1) QEMU (2) distro (3) kernel/OS.  Here the extra
> coverage is (2) only.
>

It's not really about extra coverage, it's about (what I think should
be) the minimum migration testing done by upstream, i.e. ensuring one
binary we release is migration-compatible with another binary we
release. Today we test n vs. n-1 in the CI and I think that's not
enough.

> My gut feeling is distro specific thing causing migration failure is
> extremely rare; I can't even think of any in my work history.  Meanwhile,
> it's still not the "real migration", because kernel is shared.  Anything
> real will need to happen on two real and different hosts, ultimately, to
> make it cover all things.  It's more likely to me that the kernel
> difference may fail a migration than distro difference.
>

You right, but that feels like another problem for another day. It
starts to get away from what the QEMU community can do. I've seen a lot
of cpu-specific issues in the field actually, so even swapping kernels
would miss a few bugs.

> So I still agree having such variance is better, though the hope is what we
> currently do e.g. with compat migration test is good enough.
>
> Said that, if part of this RFC is to see if we want to introduce that too
> for the compat test, no objections either.  Really, it's a matter of if it
> will help us find more bugs.
>

Right, I'm not sure yet. Just sharing because I feel like keeping this
stuff private is a bad approach. In general, if a maintainer relies on
something to do their work, it should be under the community
umbrella. Of course there are matters of _workflow_, and every person
has a preference, but maybe the tooling could be shared.

>> 
>> 2) Neither the QEMU build nor the distro repos are reliable enough to
>> allow me to just "go back in time" and rebuild at any moment, I need to
>> keep the artifacts cached to avoid rebuilding. Otherwise it comes time
>> to send a PR and I discover that some python piece has changed or the
>> distro messed up the older repository.
>
> This is definitely true.  While studying your test approach, I also noticed
> that there's a "toolbox" tool that should have done many of what your
> script did: "enter" the toolbox will transparently map many things (files
> under /dev/, or $HOME) automatically.  At least when thinking this as
> something I use personally, toolbox seems working.  But again I'm not
> container person, maybe there's some difference, just FYI.
>

I had no idea this existed, I'll take a look, thanks!

>> 
>> Any thoughts on this? Could it be useful to anyone? Maybe the stable
>> branch has some similar issues?
>
> I'd be surprised if Michael would need to test stable branch migrations!
> That's IMHO too much work for him.. if it needs to be more than what CI
> covers.  After all, migration is only a submodule in stable maint pov.
>

Oh, not migration, but the idea of building an older codebase. I know
he's dealt with distro repos being broken as well.

>
> That's a good approach.
>
> If you want to go further, you could directly host it on dockerhub, for
> debian or fedora hosts, so other don't need to rebuild anything.
> And everytime there is a new release, we would just add the new image.
>

It would save storage as well. I'm currently keeping a 7.2 build just
because Peter found a bug in it.

> $ ./scripts/qemu-time-machine.sh qemu-system-aarch64 11.1.2 "$@"
> podman run -it --rm \
> -w $(pwd) -v $(pwd):$(pwd) \
> docker.io/qemu-releases/qemu-system-aarch64:11.1.2 \
> /usr/bin/qemu-system-aarch64 "$@"
>
> You can drastically optimize size by using multi-stage builds.
> All images can be based on a common deps image, which contain all
> libraries qemu depend on (~140MB from what I can see on my debian
> system), and then all qemu-* images would be FROM:deps, and they only
> add a single binary inside.
>

Ah true, so I probably don't need the full distro build. And keeping the
build around is also kinda pointless if it can just be rebuilt. I
haven't yet encountered a scenario where a rebuild was necessary
actually.

> Final result would be 140MB for base deps + binary size per image on
> top, (from 3MB to 40MB) per binary. And we could have the whole
> collection of qemu binaries available for everyone with a single script.
>

Good to know, I haven't put too much effort in reducing the size of the
images. Now I'm thinking that if they get quite small, we could also
host a image just with the (non-unit) tests and maybe use it for some
bisecting once a test fails.

> Regards,
> Pierrick