[RFC] container images for migration-test

[RFC] container images for migration-test

[Fabiano Rosas]

Hi, I just wanted to share and ask for feedback on some container images
I've been building as part of my testing of migration pull-requests.

From a Dockerfile from ./tests/docker/dockerfiles/ I'm producing the
following:

$ docker images
qemu-build-base:11.0.0                        b6efa198e93a        5.8GB
qemu-rootfs-base:11.0.0                       72bcb5418f4d       4.08GB
qemu:11.0.0                                   018a3f0f4dcc          2GB

This last image is stripped (for size) of everything but a few libs and
binaries (qemu, gdb, migration-test, sh) and has an entrypoint of:

 CMD ["qemu-system-x86_64"]

The point is that I can wrap it in a script called qemu-system-x86_64
and run it interchangeably with any other qemu binary during
migration-test:

 QTEST_QEMU_BINARY_SRC=./build-11.0.0/qemu-system-x86_64 \
 QTEST_QEMU_BINARY=./build-master/qemu-system-x86_64 \
 ./tests/qtest/migration-test

(note that migration-test spawns the qemu binaries itself)

I run tests against the previous 3 QEMU releases. At every release I
build the container for the new release and at every PR I build the
container for that git hash using the latest tag as a base.

The reasons that lead me to this solution are:

1) Migration compat testing is special, the two migration peers can run
in different OS versions, so their QEMU builds can be quite different. I
cannot test using *just* my dev machine, neither using *just* a
container. I need two hosts. Or two containers running different OSes
that can both be reachable by the migration-test binary running in
either of the OSes.

2) Neither the QEMU build nor the distro repos are reliable enough to
allow me to just "go back in time" and rebuild at any moment, I need to
keep the artifacts cached to avoid rebuilding. Otherwise it comes time
to send a PR and I discover that some python piece has changed or the
distro messed up the older repository.

Any thoughts on this? Could it be useful to anyone? Maybe the stable
branch has some similar issues?

I have a messy script that builds everything, but here are the important
pieces:

- Dockerfile.rootfs.base ==> qemu-rootfs-base-11.1.0
This is just ./tests/docker/dockerfiles/opensuse-leap.docker with a sed
to remove ncurses-devel (suse repo issue)


- Dockerfile.build.base ==> qemu-build-base:11.1.0
FROM qemu-rootfs-base:11.1.0

# actual source, mounted RO
RUN mkdir /src

# not the actual build dir, only the artifacts that are needed
RUN mkdir /build

WORKDIR /build-tmp-11.1.0

RUN --mount=type=cache,target=/build-tmp-11.1.0,rw \
    --mount=type=bind,from=qemu-src-11.1.0,target=/src,ro \
    /src/configure \
    --target-list=x86_64-softmmu,aarch64-softmmu,ppc64-softmmu,s390x-softmmu \
    --disable-plugins --disable-modules --enable-werror --enable-debug \
    --disable-docs --disable-tools && \
    make -j14 && { echo 'build ok'; } || { echo 'build failed'; exit 1; }; \
    cp -rL qemu-bundle /build && \
    cp -rL scripts /build && \
    find . -maxdepth 1 -name "qemu-system-*" -type f -exec cp {} /build \; && \
    find . -maxdepth 1 -name "tests" -type d -exec cp -r {} /build/tests \; ;

RUN --mount=type=cache,target=/build-tmp-11.1.0,rw \
    find /build -maxdepth 1 -name "qemu-system-*" -type f -exec cp -s {} /bin \; && \
    find /build/tests/qtest -maxdepth 1 -name "migration-test" -type f -exec ln -sfT {} /bin/migration-test \; ;
---


- Dockerfile ==> qemu:11.1.0
FROM qemu-build-base:11.1.0 AS runtime

WORKDIR /build

# isolate the library dependencies of qemu, python and gdb
RUN mkdir /libs64 && \
    cp -L $(ldd qemu-system-x86_64 | awk '{print $3}' | xargs) /libs64 && \
    cp -L $(ldd /usr/bin/gdb | awk '{print $3}' | xargs) /libs64 && \
    cp -L $(ldd /usr/bin/python3 | awk '{print $3}' | xargs) /libs64 && \
    cp -L $(ldd /bin/sh | awk '{print $3}' | xargs) /libs64 && \
    cp -L $(ldd /bin/sh | grep ld-linux | awk '{print $1}' | xargs) /libs64 && \
    cp -L $(ldd /usr/lib64/$(readlink $(which python3))/site-packages/rpm/_rpm*.so | awk '{print $3}' | xargs) /libs64
# rpm stuff above is because gdb has its own python runtime

RUN mkdir -p /slash/usr/share && \
    cp -r /usr/lib64/python* /libs64 && \
    ln -sfT /lib64 /slash/usr/lib64

RUN mkdir /libs && \
    cp -r /usr/lib/rpm* /libs

# only these binaries in the image to keep it small
RUN mkdir /bins && \
    cp -L /bin/sh /bins/ && \
    cp -L /bin/ls /bins/ && \
    cp -L /bin/echo /bins/ && \
    cp -L /usr/bin/cat /bins/ && \
    cp -L /usr/bin/gdb /bins/ && \
    cp -L /usr/bin/python3 /bins/ && \
    cp -L /usr/bin/docker /bins/ && \
    cp -L /usr/bin/env /bins/ && \
    cp -rL /usr/share/gdb /slash/usr/share

# symlinks
RUN cp -d /bin/migration-test /bins/ && \
    cp -d /bin/qemu-system-* /bins/

FROM scratch
COPY --from=runtime /slash /
COPY --from=runtime /libs64 /lib64
COPY --from=runtime /libs /usr/lib
COPY --from=runtime /bins /bin
COPY --from=runtime /build /build
ENV PYTHONPATH="/lib64/python3.13"
ENV TMPDIR="/tmp/migtmp"
WORKDIR /build

# -version helps debugging, this is overriden when calling qemu with
# proper args
CMD ["qemu-system-x86_64", "-version"]
---


- The wrappers. One for each of qemu-system-x86_64, sh, migration-test,
  these live in a build-<version> directory.
---
#!/bin/sh

# pass the environment into the container 
echo > "/tmp/migtmp/envfile"
env | while IFS= read -r line; do
    case "$line" in
    QTEST*|QEMU*|PYTHON*)
        echo "$line" >> "/tmp/migtmp/envfile"
        ;;
    esac
done

bin=qemu-system-x86_64
exec docker run --rm -w /build \
                -v /tmp/migtmp:/tmp/migtmp \
                -v /home/farosas/archive:/archive \
                -v /var/run/docker.sock:/var/run/docker.sock \
                --network host \
                --env-file /tmp/migtmp/envfile \
                --cap-add=SYS_PTRACE \
                --security-opt seccomp=unconfined \
                --device /dev/kvm \
                --device /dev/userfaultfd:/dev/userfaultfd \
                -it qemu:11.1.0 $bin $(printf "%s\n" "$@")
---

- The snippet of the test script that uses all of this:

cd ./archive/build-11.0.0

# container-relative
CURRENT_QEMU=/archive/build-latest

if [ $1 = "src" ]; then
     info "forward..."
     run_qtest PYTHON=/bin/python3 TMPDIR=/tmp/migtmp \
     QTEST_QEMU_BINARY=${CURRENT_QEMU}/qemu-system-${ARCH} \
     QTEST_QEMU_BINARY_SRC=./qemu-system-${ARCH} ./tests/qtest/${TEST}
else
     info "backward..."
     run_qtest PYTHON=/bin/python3 TMPDIR=/tmp/migtmp \
     QTEST_QEMU_BINARY=${CURRENT_QEMU}/qemu-system-${ARCH} \
     QTEST_QEMU_BINARY_DST=./qemu-system-${ARCH} ./tests/qtest/${TEST}
fi
---

Re: [RFC] container images for migration-test

[Peter Xu]

On Mon, Sep 14, 2026 at 08:58:09PM -0300, Fabiano Rosas wrote:
> Hi, I just wanted to share and ask for feedback on some container images
> I've been building as part of my testing of migration pull-requests.
> 
> From a Dockerfile from ./tests/docker/dockerfiles/ I'm producing the
> following:
> 
> $ docker images
> qemu-build-base:11.0.0                        b6efa198e93a        5.8GB
> qemu-rootfs-base:11.0.0                       72bcb5418f4d       4.08GB
> qemu:11.0.0                                   018a3f0f4dcc          2GB
> 
> This last image is stripped (for size) of everything but a few libs and
> binaries (qemu, gdb, migration-test, sh) and has an entrypoint of:
> 
>  CMD ["qemu-system-x86_64"]
> 
> The point is that I can wrap it in a script called qemu-system-x86_64
> and run it interchangeably with any other qemu binary during
> migration-test:
> 
>  QTEST_QEMU_BINARY_SRC=./build-11.0.0/qemu-system-x86_64 \
>  QTEST_QEMU_BINARY=./build-master/qemu-system-x86_64 \
>  ./tests/qtest/migration-test
> 
> (note that migration-test spawns the qemu binaries itself)
> 
> I run tests against the previous 3 QEMU releases. At every release I
> build the container for the new release and at every PR I build the
> container for that git hash using the latest tag as a base.

I'm not a container person, e.g. I did test my recent channel fix on
qemu-7.2 by simply building it with build-7.2/.. but after learning it the
idea using a link to kickoff the container is very interesting on its own
already.

> 
> The reasons that lead me to this solution are:
> 
> 1) Migration compat testing is special, the two migration peers can run
> in different OS versions, so their QEMU builds can be quite different. I
> cannot test using *just* my dev machine, neither using *just* a
> container. I need two hosts. Or two containers running different OSes
> that can both be reachable by the migration-test binary running in
> either of the OSes.

True..  One thing I want to make us on the same page is, this is about
distro, not OS?  Say, the kernel is still the same.

So we have three things (1) QEMU (2) distro (3) kernel/OS.  Here the extra
coverage is (2) only.

My gut feeling is distro specific thing causing migration failure is
extremely rare; I can't even think of any in my work history.  Meanwhile,
it's still not the "real migration", because kernel is shared.  Anything
real will need to happen on two real and different hosts, ultimately, to
make it cover all things.  It's more likely to me that the kernel
difference may fail a migration than distro difference.

So I still agree having such variance is better, though the hope is what we
currently do e.g. with compat migration test is good enough.

Said that, if part of this RFC is to see if we want to introduce that too
for the compat test, no objections either.  Really, it's a matter of if it
will help us find more bugs.

> 
> 2) Neither the QEMU build nor the distro repos are reliable enough to
> allow me to just "go back in time" and rebuild at any moment, I need to
> keep the artifacts cached to avoid rebuilding. Otherwise it comes time
> to send a PR and I discover that some python piece has changed or the
> distro messed up the older repository.

This is definitely true.  While studying your test approach, I also noticed
that there's a "toolbox" tool that should have done many of what your
script did: "enter" the toolbox will transparently map many things (files
under /dev/, or $HOME) automatically.  At least when thinking this as
something I use personally, toolbox seems working.  But again I'm not
container person, maybe there's some difference, just FYI.

> 
> Any thoughts on this? Could it be useful to anyone? Maybe the stable
> branch has some similar issues?

I'd be surprised if Michael would need to test stable branch migrations!
That's IMHO too much work for him.. if it needs to be more than what CI
covers.  After all, migration is only a submodule in stable maint pov.

So, I'm not sure whether some of those should be in upstream repo, let me
know your thoughts.  Besides, I'll definitely play more with toolbox and
your scripts some day.

Thanks for sharing!

> 
> I have a messy script that builds everything, but here are the important
> pieces:
> 
> - Dockerfile.rootfs.base ==> qemu-rootfs-base-11.1.0
> This is just ./tests/docker/dockerfiles/opensuse-leap.docker with a sed
> to remove ncurses-devel (suse repo issue)
> 
> 
> - Dockerfile.build.base ==> qemu-build-base:11.1.0
> FROM qemu-rootfs-base:11.1.0
> 
> # actual source, mounted RO
> RUN mkdir /src
> 
> # not the actual build dir, only the artifacts that are needed
> RUN mkdir /build
> 
> WORKDIR /build-tmp-11.1.0
> 
> RUN --mount=type=cache,target=/build-tmp-11.1.0,rw \
>     --mount=type=bind,from=qemu-src-11.1.0,target=/src,ro \
>     /src/configure \
>     --target-list=x86_64-softmmu,aarch64-softmmu,ppc64-softmmu,s390x-softmmu \
>     --disable-plugins --disable-modules --enable-werror --enable-debug \
>     --disable-docs --disable-tools && \
>     make -j14 && { echo 'build ok'; } || { echo 'build failed'; exit 1; }; \
>     cp -rL qemu-bundle /build && \
>     cp -rL scripts /build && \
>     find . -maxdepth 1 -name "qemu-system-*" -type f -exec cp {} /build \; && \
>     find . -maxdepth 1 -name "tests" -type d -exec cp -r {} /build/tests \; ;
> 
> RUN --mount=type=cache,target=/build-tmp-11.1.0,rw \
>     find /build -maxdepth 1 -name "qemu-system-*" -type f -exec cp -s {} /bin \; && \
>     find /build/tests/qtest -maxdepth 1 -name "migration-test" -type f -exec ln -sfT {} /bin/migration-test \; ;
> ---
> 
> 
> - Dockerfile ==> qemu:11.1.0
> FROM qemu-build-base:11.1.0 AS runtime
> 
> WORKDIR /build
> 
> # isolate the library dependencies of qemu, python and gdb
> RUN mkdir /libs64 && \
>     cp -L $(ldd qemu-system-x86_64 | awk '{print $3}' | xargs) /libs64 && \
>     cp -L $(ldd /usr/bin/gdb | awk '{print $3}' | xargs) /libs64 && \
>     cp -L $(ldd /usr/bin/python3 | awk '{print $3}' | xargs) /libs64 && \
>     cp -L $(ldd /bin/sh | awk '{print $3}' | xargs) /libs64 && \
>     cp -L $(ldd /bin/sh | grep ld-linux | awk '{print $1}' | xargs) /libs64 && \
>     cp -L $(ldd /usr/lib64/$(readlink $(which python3))/site-packages/rpm/_rpm*.so | awk '{print $3}' | xargs) /libs64
> # rpm stuff above is because gdb has its own python runtime
> 
> RUN mkdir -p /slash/usr/share && \
>     cp -r /usr/lib64/python* /libs64 && \
>     ln -sfT /lib64 /slash/usr/lib64
> 
> RUN mkdir /libs && \
>     cp -r /usr/lib/rpm* /libs
> 
> # only these binaries in the image to keep it small
> RUN mkdir /bins && \
>     cp -L /bin/sh /bins/ && \
>     cp -L /bin/ls /bins/ && \
>     cp -L /bin/echo /bins/ && \
>     cp -L /usr/bin/cat /bins/ && \
>     cp -L /usr/bin/gdb /bins/ && \
>     cp -L /usr/bin/python3 /bins/ && \
>     cp -L /usr/bin/docker /bins/ && \
>     cp -L /usr/bin/env /bins/ && \
>     cp -rL /usr/share/gdb /slash/usr/share
> 
> # symlinks
> RUN cp -d /bin/migration-test /bins/ && \
>     cp -d /bin/qemu-system-* /bins/
> 
> FROM scratch
> COPY --from=runtime /slash /
> COPY --from=runtime /libs64 /lib64
> COPY --from=runtime /libs /usr/lib
> COPY --from=runtime /bins /bin
> COPY --from=runtime /build /build
> ENV PYTHONPATH="/lib64/python3.13"
> ENV TMPDIR="/tmp/migtmp"
> WORKDIR /build
> 
> # -version helps debugging, this is overriden when calling qemu with
> # proper args
> CMD ["qemu-system-x86_64", "-version"]
> ---
> 
> 
> - The wrappers. One for each of qemu-system-x86_64, sh, migration-test,
>   these live in a build-<version> directory.
> ---
> #!/bin/sh
> 
> # pass the environment into the container 
> echo > "/tmp/migtmp/envfile"
> env | while IFS= read -r line; do
>     case "$line" in
>     QTEST*|QEMU*|PYTHON*)
>         echo "$line" >> "/tmp/migtmp/envfile"
>         ;;
>     esac
> done
> 
> bin=qemu-system-x86_64
> exec docker run --rm -w /build \
>                 -v /tmp/migtmp:/tmp/migtmp \
>                 -v /home/farosas/archive:/archive \
>                 -v /var/run/docker.sock:/var/run/docker.sock \
>                 --network host \
>                 --env-file /tmp/migtmp/envfile \
>                 --cap-add=SYS_PTRACE \
>                 --security-opt seccomp=unconfined \
>                 --device /dev/kvm \
>                 --device /dev/userfaultfd:/dev/userfaultfd \
>                 -it qemu:11.1.0 $bin $(printf "%s\n" "$@")
> ---
> 
> - The snippet of the test script that uses all of this:
> 
> cd ./archive/build-11.0.0
> 
> # container-relative
> CURRENT_QEMU=/archive/build-latest
> 
> if [ $1 = "src" ]; then
>      info "forward..."
>      run_qtest PYTHON=/bin/python3 TMPDIR=/tmp/migtmp \
>      QTEST_QEMU_BINARY=${CURRENT_QEMU}/qemu-system-${ARCH} \
>      QTEST_QEMU_BINARY_SRC=./qemu-system-${ARCH} ./tests/qtest/${TEST}
> else
>      info "backward..."
>      run_qtest PYTHON=/bin/python3 TMPDIR=/tmp/migtmp \
>      QTEST_QEMU_BINARY=${CURRENT_QEMU}/qemu-system-${ARCH} \
>      QTEST_QEMU_BINARY_DST=./qemu-system-${ARCH} ./tests/qtest/${TEST}
> fi
> ---
> 

-- 
Peter Xu

Re: [RFC] container images for migration-test

[Fabiano Rosas]

Peter Xu <peterx@redhat.com> writes:

> On Mon, Sep 14, 2026 at 08:58:09PM -0300, Fabiano Rosas wrote:
>> Hi, I just wanted to share and ask for feedback on some container images
>> I've been building as part of my testing of migration pull-requests.
>> 
>> From a Dockerfile from ./tests/docker/dockerfiles/ I'm producing the
>> following:
>> 
>> $ docker images
>> qemu-build-base:11.0.0                        b6efa198e93a        5.8GB
>> qemu-rootfs-base:11.0.0                       72bcb5418f4d       4.08GB
>> qemu:11.0.0                                   018a3f0f4dcc          2GB
>> 
>> This last image is stripped (for size) of everything but a few libs and
>> binaries (qemu, gdb, migration-test, sh) and has an entrypoint of:
>> 
>>  CMD ["qemu-system-x86_64"]
>> 
>> The point is that I can wrap it in a script called qemu-system-x86_64
>> and run it interchangeably with any other qemu binary during
>> migration-test:
>> 
>>  QTEST_QEMU_BINARY_SRC=./build-11.0.0/qemu-system-x86_64 \
>>  QTEST_QEMU_BINARY=./build-master/qemu-system-x86_64 \
>>  ./tests/qtest/migration-test
>> 
>> (note that migration-test spawns the qemu binaries itself)
>> 
>> I run tests against the previous 3 QEMU releases. At every release I
>> build the container for the new release and at every PR I build the
>> container for that git hash using the latest tag as a base.
>
> I'm not a container person,

Me neither, lol. Last container work I did was back when multiarch
manifests were being added to docker. Maybe 2016? I (clearly) haven't
kept up with the progress in the area.

> e.g. I did test my recent channel fix on
> qemu-7.2 by simply building it with build-7.2/.. but after learning it the
> idea using a link to kickoff the container is very interesting on its own
> already.
>

Yeah, the issue is that `git checkout <old version>` and try to build it
is pretty hit or miss.

>> 
>> The reasons that lead me to this solution are:
>> 
>> 1) Migration compat testing is special, the two migration peers can run
>> in different OS versions, so their QEMU builds can be quite different. I
>> cannot test using *just* my dev machine, neither using *just* a
>> container. I need two hosts. Or two containers running different OSes
>> that can both be reachable by the migration-test binary running in
>> either of the OSes.
>
> True..  One thing I want to make us on the same page is, this is about
> distro, not OS?  Say, the kernel is still the same.
>

Yes, it's about what it takes to get a working QEMU binary. I'm not
trying to cover a realistic migration scenario, I'm simply trying to
test across QEMU versions *somehow*.

> So we have three things (1) QEMU (2) distro (3) kernel/OS.  Here the extra
> coverage is (2) only.
>

It's not really about extra coverage, it's about (what I think should
be) the minimum migration testing done by upstream, i.e. ensuring one
binary we release is migration-compatible with another binary we
release. Today we test n vs. n-1 in the CI and I think that's not
enough.

> My gut feeling is distro specific thing causing migration failure is
> extremely rare; I can't even think of any in my work history.  Meanwhile,
> it's still not the "real migration", because kernel is shared.  Anything
> real will need to happen on two real and different hosts, ultimately, to
> make it cover all things.  It's more likely to me that the kernel
> difference may fail a migration than distro difference.
>

You right, but that feels like another problem for another day. It
starts to get away from what the QEMU community can do. I've seen a lot
of cpu-specific issues in the field actually, so even swapping kernels
would miss a few bugs.

> So I still agree having such variance is better, though the hope is what we
> currently do e.g. with compat migration test is good enough.
>
> Said that, if part of this RFC is to see if we want to introduce that too
> for the compat test, no objections either.  Really, it's a matter of if it
> will help us find more bugs.
>

Right, I'm not sure yet. Just sharing because I feel like keeping this
stuff private is a bad approach. In general, if a maintainer relies on
something to do their work, it should be under the community
umbrella. Of course there are matters of _workflow_, and every person
has a preference, but maybe the tooling could be shared.

>> 
>> 2) Neither the QEMU build nor the distro repos are reliable enough to
>> allow me to just "go back in time" and rebuild at any moment, I need to
>> keep the artifacts cached to avoid rebuilding. Otherwise it comes time
>> to send a PR and I discover that some python piece has changed or the
>> distro messed up the older repository.
>
> This is definitely true.  While studying your test approach, I also noticed
> that there's a "toolbox" tool that should have done many of what your
> script did: "enter" the toolbox will transparently map many things (files
> under /dev/, or $HOME) automatically.  At least when thinking this as
> something I use personally, toolbox seems working.  But again I'm not
> container person, maybe there's some difference, just FYI.
>

I had no idea this existed, I'll take a look, thanks!

>> 
>> Any thoughts on this? Could it be useful to anyone? Maybe the stable
>> branch has some similar issues?
>
> I'd be surprised if Michael would need to test stable branch migrations!
> That's IMHO too much work for him.. if it needs to be more than what CI
> covers.  After all, migration is only a submodule in stable maint pov.
>

Oh, not migration, but the idea of building an older codebase. I know
he's dealt with distro repos being broken as well.

>
> That's a good approach.
>
> If you want to go further, you could directly host it on dockerhub, for
> debian or fedora hosts, so other don't need to rebuild anything.
> And everytime there is a new release, we would just add the new image.
>

It would save storage as well. I'm currently keeping a 7.2 build just
because Peter found a bug in it.

> $ ./scripts/qemu-time-machine.sh qemu-system-aarch64 11.1.2 "$@"
> podman run -it --rm \
> -w $(pwd) -v $(pwd):$(pwd) \
> docker.io/qemu-releases/qemu-system-aarch64:11.1.2 \
> /usr/bin/qemu-system-aarch64 "$@"
>
> You can drastically optimize size by using multi-stage builds.
> All images can be based on a common deps image, which contain all
> libraries qemu depend on (~140MB from what I can see on my debian
> system), and then all qemu-* images would be FROM:deps, and they only
> add a single binary inside.
>

Ah true, so I probably don't need the full distro build. And keeping the
build around is also kinda pointless if it can just be rebuilt. I
haven't yet encountered a scenario where a rebuild was necessary
actually.

> Final result would be 140MB for base deps + binary size per image on
> top, (from 3MB to 40MB) per binary. And we could have the whole
> collection of qemu binaries available for everyone with a single script.
>

Good to know, I haven't put too much effort in reducing the size of the
images. Now I'm thinking that if they get quite small, we could also
host a image just with the (non-unit) tests and maybe use it for some
bisecting once a test fails.

> Regards,
> Pierrick

Re: [RFC] container images for migration-test

[Alex Bennée]

Fabiano Rosas <farosas@suse.de> writes:

> Peter Xu <peterx@redhat.com> writes:
>
>> On Mon, Sep 14, 2026 at 08:58:09PM -0300, Fabiano Rosas wrote:
>>> Hi, I just wanted to share and ask for feedback on some container images
>>> I've been building as part of my testing of migration pull-requests.
>>> 
>>> From a Dockerfile from ./tests/docker/dockerfiles/ I'm producing the
>>> following:
>>> 
>>> $ docker images
>>> qemu-build-base:11.0.0                        b6efa198e93a        5.8GB
>>> qemu-rootfs-base:11.0.0                       72bcb5418f4d       4.08GB
>>> qemu:11.0.0                                   018a3f0f4dcc          2GB
>>> 
>>> This last image is stripped (for size) of everything but a few libs and
>>> binaries (qemu, gdb, migration-test, sh) and has an entrypoint of:
>>> 
>>>  CMD ["qemu-system-x86_64"]
>>> 
>>> The point is that I can wrap it in a script called qemu-system-x86_64
>>> and run it interchangeably with any other qemu binary during
>>> migration-test:
>>> 
>>>  QTEST_QEMU_BINARY_SRC=./build-11.0.0/qemu-system-x86_64 \
>>>  QTEST_QEMU_BINARY=./build-master/qemu-system-x86_64 \
>>>  ./tests/qtest/migration-test
>>> 
>>> (note that migration-test spawns the qemu binaries itself)
>>> 
>>> I run tests against the previous 3 QEMU releases. At every release I
>>> build the container for the new release and at every PR I build the
>>> container for that git hash using the latest tag as a base.
>>
>> I'm not a container person,
>
> Me neither, lol. Last container work I did was back when multiarch
> manifests were being added to docker. Maybe 2016? I (clearly) haven't
> kept up with the progress in the area.
>
>> e.g. I did test my recent channel fix on
>> qemu-7.2 by simply building it with build-7.2/.. but after learning it the
>> idea using a link to kickoff the container is very interesting on its own
>> already.
>>
>
> Yeah, the issue is that `git checkout <old version>` and try to build it
> is pretty hit or miss.
>
>>> 
>>> The reasons that lead me to this solution are:
>>> 
>>> 1) Migration compat testing is special, the two migration peers can run
>>> in different OS versions, so their QEMU builds can be quite different. I
>>> cannot test using *just* my dev machine, neither using *just* a
>>> container. I need two hosts. Or two containers running different OSes
>>> that can both be reachable by the migration-test binary running in
>>> either of the OSes.
>>
>> True..  One thing I want to make us on the same page is, this is about
>> distro, not OS?  Say, the kernel is still the same.
>>
>
> Yes, it's about what it takes to get a working QEMU binary. I'm not
> trying to cover a realistic migration scenario, I'm simply trying to
> test across QEMU versions *somehow*.

You could just consider using the system qemu if installed and then we
get the variance between versions from which OS's we build on. That
would work outside the container and we could include qemu-system in the
containers as well.

It would cover the migrations from the various LTS qemu's to current
master which should be the majority of common use cases.

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: [RFC] container images for migration-test

[Pierrick Bouvier]

On 9/14/2026 4:58 PM, Fabiano Rosas wrote:
> Hi, I just wanted to share and ask for feedback on some container images
> I've been building as part of my testing of migration pull-requests.
> 
> From a Dockerfile from ./tests/docker/dockerfiles/ I'm producing the
> following:
> 
> $ docker images
> qemu-build-base:11.0.0                        b6efa198e93a        5.8GB
> qemu-rootfs-base:11.0.0                       72bcb5418f4d       4.08GB
> qemu:11.0.0                                   018a3f0f4dcc          2GB
> 
> This last image is stripped (for size) of everything but a few libs and
> binaries (qemu, gdb, migration-test, sh) and has an entrypoint of:
> 
>  CMD ["qemu-system-x86_64"]
> 
> The point is that I can wrap it in a script called qemu-system-x86_64
> and run it interchangeably with any other qemu binary during
> migration-test:
> 
>  QTEST_QEMU_BINARY_SRC=./build-11.0.0/qemu-system-x86_64 \
>  QTEST_QEMU_BINARY=./build-master/qemu-system-x86_64 \
>  ./tests/qtest/migration-test
> 
> (note that migration-test spawns the qemu binaries itself)
> 
> I run tests against the previous 3 QEMU releases. At every release I
> build the container for the new release and at every PR I build the
> container for that git hash using the latest tag as a base.
> 
> The reasons that lead me to this solution are:
> 
> 1) Migration compat testing is special, the two migration peers can run
> in different OS versions, so their QEMU builds can be quite different. I
> cannot test using *just* my dev machine, neither using *just* a
> container. I need two hosts. Or two containers running different OSes
> that can both be reachable by the migration-test binary running in
> either of the OSes.
> 
> 2) Neither the QEMU build nor the distro repos are reliable enough to
> allow me to just "go back in time" and rebuild at any moment, I need to
> keep the artifacts cached to avoid rebuilding. Otherwise it comes time
> to send a PR and I discover that some python piece has changed or the
> distro messed up the older repository.
> 
> Any thoughts on this? Could it be useful to anyone? Maybe the stable
> branch has some similar issues?
>

That's a good approach.

If you want to go further, you could directly host it on dockerhub, for
debian or fedora hosts, so other don't need to rebuild anything.
And everytime there is a new release, we would just add the new image.

$ ./scripts/qemu-time-machine.sh qemu-system-aarch64 11.1.2 "$@"
podman run -it --rm \
-w $(pwd) -v $(pwd):$(pwd) \
docker.io/qemu-releases/qemu-system-aarch64:11.1.2 \
/usr/bin/qemu-system-aarch64 "$@"

You can drastically optimize size by using multi-stage builds.
All images can be based on a common deps image, which contain all
libraries qemu depend on (~140MB from what I can see on my debian
system), and then all qemu-* images would be FROM:deps, and they only
add a single binary inside.

Final result would be 140MB for base deps + binary size per image on
top, (from 3MB to 40MB) per binary. And we could have the whole
collection of qemu binaries available for everyone with a single script.

Regards,
Pierrick

Re: [RFC] container images for migration-test

[Pierrick Bouvier]

[-- Attachment #1: Type: text/plain, Size: 482 bytes --]

See attached an example of a multi stage build that creates a container
having only qemu binaries, their libraries, and qemu roms, and nothing
else. It does not even have a shell.

build with:
$ podman build -f qemu.Dockerfile --build-arg QEMU_VERSION=v11.1.0 \
  -t test --squash-all

run with:
$ podman run -it --rm test qemu-system-aarch64 --version
QEMU emulator version 11.1.0 (v11.1.0)
Copyright (c) 2003-2026 Fabrice Bellard and the QEMU Project developers

Regards,
Pierrick

[-- Attachment #2: qemu.Dockerfile --]
[-- Type: text/plain, Size: 1350 bytes --]

FROM debian:13 as build

RUN apt update && apt install -y git build-essential
RUN sed -e 's/Types: deb/Types: deb deb-src/' -i /etc/apt/sources.list.d/debian.sources
RUN apt update && apt build-dep -y --arch-only qemu

ARG QEMU_VERSION
RUN echo "QEMU_VERSION=${QEMU_VERSION:?QEMU_VERSION build arg must be set}"
RUN cd /tmp && \
git clone https://gitlab.com/qemu-project/qemu --depth 1 --single-branch --branch "${QEMU_VERSION}" && \
cd qemu && ./configure --prefix=/opt/qemu && \
ninja -C build install && \
rm -rf /tmp/*

# now retrieve runtime dependencies (libs + loader)
RUN ldd /opt/qemu/bin/* | grep '=>' | sed -e 's/.*=>//' -e 's/(.*//' | sort -u > /opt/qemu/libs.links
RUN find /lib/ld-linux* >> /opt/qemu/libs.links
RUN cat /opt/qemu/libs.links | xargs -n1 readlink -f > /opt/qemu/libs.realpath
# store them in /deps
RUN mkdir /deps && cp --parents $(cat /opt/qemu/libs.*) /deps/

################################################################################
FROM scratch
# FROM debian:13-slim
# in case you want a more complete container with a shell

# copy qemu and all associated artifacts (roms, ...)
COPY --from=build /opt/qemu /usr
COPY --from=build /deps /

# build with:
# podman build -f qemu.Dockerfile --build-arg QEMU_VERSION=v11.1.0 -t test --squash-all
# run with:
# podman run -it --rm test qemu-system-aarch64 --version

Re: [RFC] container images for migration-test

[Pierrick Bouvier]

On 10/8/26 5:48 PM, Pierrick Bouvier wrote:
> On 9/14/2026 4:58 PM, Fabiano Rosas wrote:
>> Hi, I just wanted to share and ask for feedback on some container images
>> I've been building as part of my testing of migration pull-requests.
>>
>>  From a Dockerfile from ./tests/docker/dockerfiles/ I'm producing the
>> following:
>>
>> $ docker images
>> qemu-build-base:11.0.0                        b6efa198e93a        5.8GB
>> qemu-rootfs-base:11.0.0                       72bcb5418f4d       4.08GB
>> qemu:11.0.0                                   018a3f0f4dcc          2GB
>>
>> This last image is stripped (for size) of everything but a few libs and
>> binaries (qemu, gdb, migration-test, sh) and has an entrypoint of:
>>
>>   CMD ["qemu-system-x86_64"]
>>
>> The point is that I can wrap it in a script called qemu-system-x86_64
>> and run it interchangeably with any other qemu binary during
>> migration-test:
>>
>>   QTEST_QEMU_BINARY_SRC=./build-11.0.0/qemu-system-x86_64 \
>>   QTEST_QEMU_BINARY=./build-master/qemu-system-x86_64 \
>>   ./tests/qtest/migration-test
>>
>> (note that migration-test spawns the qemu binaries itself)
>>
>> I run tests against the previous 3 QEMU releases. At every release I
>> build the container for the new release and at every PR I build the
>> container for that git hash using the latest tag as a base.
>>
>> The reasons that lead me to this solution are:
>>
>> 1) Migration compat testing is special, the two migration peers can run
>> in different OS versions, so their QEMU builds can be quite different. I
>> cannot test using *just* my dev machine, neither using *just* a
>> container. I need two hosts. Or two containers running different OSes
>> that can both be reachable by the migration-test binary running in
>> either of the OSes.
>>
>> 2) Neither the QEMU build nor the distro repos are reliable enough to
>> allow me to just "go back in time" and rebuild at any moment, I need to
>> keep the artifacts cached to avoid rebuilding. Otherwise it comes time
>> to send a PR and I discover that some python piece has changed or the
>> distro messed up the older repository.
>>
>> Any thoughts on this? Could it be useful to anyone? Maybe the stable
>> branch has some similar issues?
>>
> 
> That's a good approach.
> 
> If you want to go further, you could directly host it on dockerhub, for
> debian or fedora hosts, so other don't need to rebuild anything.
> And everytime there is a new release, we would just add the new image.
> 
> $ ./scripts/qemu-time-machine.sh qemu-system-aarch64 11.1.2 "$@"
> podman run -it --rm \
> -w $(pwd) -v $(pwd):$(pwd) \
> docker.io/qemu-releases/qemu-system-aarch64:11.1.2 \
> /usr/bin/qemu-system-aarch64 "$@"
> 
> You can drastically optimize size by using multi-stage builds.
> All images can be based on a common deps image, which contain all
> libraries qemu depend on (~140MB from what I can see on my debian
> system), and then all qemu-* images would be FROM:deps, and they only
> add a single binary inside.
> 
> Final result would be 140MB for base deps + binary size per image on
> top, (from 3MB to 40MB) per binary. And we could have the whole
> collection of qemu binaries available for everyone with a single script.
>

I gave a try to this, see:
https://lore.kernel.org/qemu-devel/20261010231648.3105-1-pierrick.bouvier@oss.qualcomm.com/T/#t

Regards,
Pierrick

Re: [RFC] container images for migration-test

[Daniel P. Berrangé]

On Mon, Sep 14, 2026 at 08:58:09PM -0300, Fabiano Rosas wrote:
> Hi, I just wanted to share and ask for feedback on some container images
> I've been building as part of my testing of migration pull-requests.
> 
> From a Dockerfile from ./tests/docker/dockerfiles/ I'm producing the
> following:
> 
> $ docker images
> qemu-build-base:11.0.0                        b6efa198e93a        5.8GB
> qemu-rootfs-base:11.0.0                       72bcb5418f4d       4.08GB
> qemu:11.0.0                                   018a3f0f4dcc          2GB

snip

> 2) Neither the QEMU build nor the distro repos are reliable enough to
> allow me to just "go back in time" and rebuild at any moment, I need to
> keep the artifacts cached to avoid rebuilding. Otherwise it comes time
> to send a PR and I discover that some python piece has changed or the
> distro messed up the older repository.

Yes, in general building old QEMU on new distros periodically
hits trouble.  If building old QEMU, better reliability is
achieved by using a distro release that was from the same
timeframe as QEMU.  Is using containers targetting such old
distro versions not an option ? The tests/docker/dockerfiles
from the old QEMU branch ought to work ?

Our stable release branches have the same needs, and their
CI is using the container images from the old branches.

How far back in time do you need/want to go for migration
testing ?  Is it sufficient to merely target versions for
which mjt is still doing active maint work, so we can
expect the containers to still work.

With regards,
Daniel
-- 
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|