[RFC] container images for migration-test

[Fabiano Rosas <farosas@suse.de>]

Hi, I just wanted to share and ask for feedback on some container images
I've been building as part of my testing of migration pull-requests.

From a Dockerfile from ./tests/docker/dockerfiles/ I'm producing the
following:

$ docker images
qemu-build-base:11.0.0                        b6efa198e93a        5.8GB
qemu-rootfs-base:11.0.0                       72bcb5418f4d       4.08GB
qemu:11.0.0                                   018a3f0f4dcc          2GB

This last image is stripped (for size) of everything but a few libs and
binaries (qemu, gdb, migration-test, sh) and has an entrypoint of:

 CMD ["qemu-system-x86_64"]

The point is that I can wrap it in a script called qemu-system-x86_64
and run it interchangeably with any other qemu binary during
migration-test:

 QTEST_QEMU_BINARY_SRC=./build-11.0.0/qemu-system-x86_64 \
 QTEST_QEMU_BINARY=./build-master/qemu-system-x86_64 \
 ./tests/qtest/migration-test

(note that migration-test spawns the qemu binaries itself)

I run tests against the previous 3 QEMU releases. At every release I
build the container for the new release and at every PR I build the
container for that git hash using the latest tag as a base.

The reasons that lead me to this solution are:

1) Migration compat testing is special, the two migration peers can run
in different OS versions, so their QEMU builds can be quite different. I
cannot test using *just* my dev machine, neither using *just* a
container. I need two hosts. Or two containers running different OSes
that can both be reachable by the migration-test binary running in
either of the OSes.

2) Neither the QEMU build nor the distro repos are reliable enough to
allow me to just "go back in time" and rebuild at any moment, I need to
keep the artifacts cached to avoid rebuilding. Otherwise it comes time
to send a PR and I discover that some python piece has changed or the
distro messed up the older repository.

Any thoughts on this? Could it be useful to anyone? Maybe the stable
branch has some similar issues?

I have a messy script that builds everything, but here are the important
pieces:

- Dockerfile.rootfs.base ==> qemu-rootfs-base-11.1.0
This is just ./tests/docker/dockerfiles/opensuse-leap.docker with a sed
to remove ncurses-devel (suse repo issue)


- Dockerfile.build.base ==> qemu-build-base:11.1.0
FROM qemu-rootfs-base:11.1.0

# actual source, mounted RO
RUN mkdir /src

# not the actual build dir, only the artifacts that are needed
RUN mkdir /build

WORKDIR /build-tmp-11.1.0

RUN --mount=type=cache,target=/build-tmp-11.1.0,rw \
    --mount=type=bind,from=qemu-src-11.1.0,target=/src,ro \
    /src/configure \
    --target-list=x86_64-softmmu,aarch64-softmmu,ppc64-softmmu,s390x-softmmu \
    --disable-plugins --disable-modules --enable-werror --enable-debug \
    --disable-docs --disable-tools && \
    make -j14 && { echo 'build ok'; } || { echo 'build failed'; exit 1; }; \
    cp -rL qemu-bundle /build && \
    cp -rL scripts /build && \
    find . -maxdepth 1 -name "qemu-system-*" -type f -exec cp {} /build \; && \
    find . -maxdepth 1 -name "tests" -type d -exec cp -r {} /build/tests \; ;

RUN --mount=type=cache,target=/build-tmp-11.1.0,rw \
    find /build -maxdepth 1 -name "qemu-system-*" -type f -exec cp -s {} /bin \; && \
    find /build/tests/qtest -maxdepth 1 -name "migration-test" -type f -exec ln -sfT {} /bin/migration-test \; ;
---


- Dockerfile ==> qemu:11.1.0
FROM qemu-build-base:11.1.0 AS runtime

WORKDIR /build

# isolate the library dependencies of qemu, python and gdb
RUN mkdir /libs64 && \
    cp -L $(ldd qemu-system-x86_64 | awk '{print $3}' | xargs) /libs64 && \
    cp -L $(ldd /usr/bin/gdb | awk '{print $3}' | xargs) /libs64 && \
    cp -L $(ldd /usr/bin/python3 | awk '{print $3}' | xargs) /libs64 && \
    cp -L $(ldd /bin/sh | awk '{print $3}' | xargs) /libs64 && \
    cp -L $(ldd /bin/sh | grep ld-linux | awk '{print $1}' | xargs) /libs64 && \
    cp -L $(ldd /usr/lib64/$(readlink $(which python3))/site-packages/rpm/_rpm*.so | awk '{print $3}' | xargs) /libs64
# rpm stuff above is because gdb has its own python runtime

RUN mkdir -p /slash/usr/share && \
    cp -r /usr/lib64/python* /libs64 && \
    ln -sfT /lib64 /slash/usr/lib64

RUN mkdir /libs && \
    cp -r /usr/lib/rpm* /libs

# only these binaries in the image to keep it small
RUN mkdir /bins && \
    cp -L /bin/sh /bins/ && \
    cp -L /bin/ls /bins/ && \
    cp -L /bin/echo /bins/ && \
    cp -L /usr/bin/cat /bins/ && \
    cp -L /usr/bin/gdb /bins/ && \
    cp -L /usr/bin/python3 /bins/ && \
    cp -L /usr/bin/docker /bins/ && \
    cp -L /usr/bin/env /bins/ && \
    cp -rL /usr/share/gdb /slash/usr/share

# symlinks
RUN cp -d /bin/migration-test /bins/ && \
    cp -d /bin/qemu-system-* /bins/

FROM scratch
COPY --from=runtime /slash /
COPY --from=runtime /libs64 /lib64
COPY --from=runtime /libs /usr/lib
COPY --from=runtime /bins /bin
COPY --from=runtime /build /build
ENV PYTHONPATH="/lib64/python3.13"
ENV TMPDIR="/tmp/migtmp"
WORKDIR /build

# -version helps debugging, this is overriden when calling qemu with
# proper args
CMD ["qemu-system-x86_64", "-version"]
---


- The wrappers. One for each of qemu-system-x86_64, sh, migration-test,
  these live in a build-<version> directory.
---
#!/bin/sh

# pass the environment into the container 
echo > "/tmp/migtmp/envfile"
env | while IFS= read -r line; do
    case "$line" in
    QTEST*|QEMU*|PYTHON*)
        echo "$line" >> "/tmp/migtmp/envfile"
        ;;
    esac
done

bin=qemu-system-x86_64
exec docker run --rm -w /build \
                -v /tmp/migtmp:/tmp/migtmp \
                -v /home/farosas/archive:/archive \
                -v /var/run/docker.sock:/var/run/docker.sock \
                --network host \
                --env-file /tmp/migtmp/envfile \
                --cap-add=SYS_PTRACE \
                --security-opt seccomp=unconfined \
                --device /dev/kvm \
                --device /dev/userfaultfd:/dev/userfaultfd \
                -it qemu:11.1.0 $bin $(printf "%s\n" "$@")
---

- The snippet of the test script that uses all of this:

cd ./archive/build-11.0.0

# container-relative
CURRENT_QEMU=/archive/build-latest

if [ $1 = "src" ]; then
     info "forward..."
     run_qtest PYTHON=/bin/python3 TMPDIR=/tmp/migtmp \
     QTEST_QEMU_BINARY=${CURRENT_QEMU}/qemu-system-${ARCH} \
     QTEST_QEMU_BINARY_SRC=./qemu-system-${ARCH} ./tests/qtest/${TEST}
else
     info "backward..."
     run_qtest PYTHON=/bin/python3 TMPDIR=/tmp/migtmp \
     QTEST_QEMU_BINARY=${CURRENT_QEMU}/qemu-system-${ARCH} \
     QTEST_QEMU_BINARY_DST=./qemu-system-${ARCH} ./tests/qtest/${TEST}
fi
---