[PATCH v3 0/3] releases: build and run any QEMU release easily!

[PATCH v3 0/3] releases: build and run any QEMU release easily!

[Pierrick Bouvier]

This series was inspired by a recent post from Fabiano to provide container
images for migration tests:
https://lore.kernel.org/qemu-devel/24d04a1f-344d-4b90-a307-b582dd9a1128@oss.qualcomm.com/T/#m353f119dc1a24fa437dde8a96d1891629e64303c

The idea to be able to build and run any QEMU (past) release with containers was
in my mind for some time, so I gave it a shot. Finally, it was easier than
expected to obtain a single dockerfile (with a few hacks) that works over all
our releases.

The oldest functional distro we can find nowadays is ubuntu 12.04. Because of
this, the oldest QEMU release we can build is v0.12.0. Previous ones require an
environment we can't get easily today.

Images have the absolute minimal content possible. It's only qemu binaries,
artifacts (roms, keymap, ...), and libraries. There is no shell or any other
tool included. As a result, images have a size from 149MB (v0.12.0) to 1.48GB
(v11.1.2).

This series adds 3 pieces:
- a unique dockerfile to build any QEMU release. User has to provide an explicit
  Ubuntu version when building a given QEMU version.
- A script build_release_container.sh which automatically detects which ubuntu
  version a given release should be buildable on (based on date). It works
  great, and the heuristic works for all our versions.
- a script generate_release_links.sh which creates wrapper scripts automatically
  using prebuilt images (hosted on my dockerhub).

TLDR:
$ ./scripts/build_release_container.sh v1.6.1
$ podman run -it --rm qemu/v1.6.1 qemu-system-arm --version

$ ./script/generate_release_links.sh qemu-time-machine
$ ./qemu-time-machine/v1.6.1/qemu-system-arm --version
QEMU emulator version 1.6.1, Copyright (c) 2003-2008 Fabrice Bellard
$ ./qemu-time-machine/v11.0.0/qemu-system-arm --version
QEMU emulator version 11.0.0 (v11.0.0)
Copyright (c) 2003-2026 Fabrice Bellard and the QEMU Project developers

v3
--

- extend dockerfile to support ubuntu 12.04, and lower older revision we can
  build from v1.6.1 to v0.12.0.

v2
--

- fix generate_release_links.sh: was missing userns option for podman
- dockerfile: cleanup "|| true" and replaced with proper ifs

Pierrick Bouvier (3):
  tests/docker/dockerfiles: add release.docker
  scripts: add build_release_container.sh
  scripts: add generate_release_wrappers.sh

 scripts/build_release_container.sh      | 62 ++++++++++++++++++++
 scripts/generate_release_wrappers.sh    | 71 +++++++++++++++++++++++
 tests/docker/dockerfiles/release.docker | 76 +++++++++++++++++++++++++
 3 files changed, 209 insertions(+)
 create mode 100755 scripts/build_release_container.sh
 create mode 100755 scripts/generate_release_wrappers.sh
 create mode 100644 tests/docker/dockerfiles/release.docker

-- 
2.43.0

[PATCH v3 1/3] tests/docker/dockerfiles: add release.docker

[Pierrick Bouvier]

This new dockerfile allows to build a distroless image containing only
qemu stripped release binaries, their lib dependencies, and installed
artifacts (roms, keymap, ...). There is no shell, no python, nothing
else. This is achieved thanks to a multi-stage build, where we add qemu
binaries, artifacts and libraries in a new scratch image. Image has
the minimal size possible.

This is based on Ubuntu, which has the advantage to have working
containers available up to 12.04. Debian old old stable have broken apt
sources for instance. Ubuntu 10.04 binaries segfault due to missing
vsyscall emulation on modern kernels, so 12.04 is the lowest possible.

This unique dockerfile works for all qemu versions, with 0.12.0 being the
oldest we can build today (based on ubuntu 12.04). Previous ones need an
older compiler.

To build the image, you need to provide a qemu and ubuntu version:
$ podman build -f release.Dockerfile --build-arg QEMU_VERSION=v11.1.0 --build-arg UBUNTU_VERSION=24.04 -t qemu/v11.1.0

To run it, as there is no shell, and no defined entrypoint, you need to
choose the binary you want explicitely:
$ podman run -it --rm qemu/v11.1.0 qemu-system-aarch64 --version

Notes: We could optimize a bit further, by creating a base container for
each ubuntu version, that is reused between all qemu versions built with
it. Given that recent QEMU have around 800MB of binaries + artifacts,
and only 100MB of library dependency, I don't think it's worth the extra
complexity.

Notes: Also, I considered create one image per binary. Problem is
that qemu depends on other artifacts (rom, keymap, etc) and it's not
easy to identify what is needed for one binary. Thus, I think it's
better to stick to one container with all binaries per release.

Signed-off-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>
---
 tests/docker/dockerfiles/release.docker | 76 +++++++++++++++++++++++++
 1 file changed, 76 insertions(+)
 create mode 100644 tests/docker/dockerfiles/release.docker

diff --git a/tests/docker/dockerfiles/release.docker b/tests/docker/dockerfiles/release.docker
new file mode 100644
index 00000000000..611ecca9859
--- /dev/null
+++ b/tests/docker/dockerfiles/release.docker
@@ -0,0 +1,76 @@
+ARG UBUNTU_VERSION="UBUNTU_VERSION_build_arg_must_be_set"
+FROM ubuntu:$UBUNTU_VERSION as build
+ARG UBUNTU_VERSION
+
+# add deb-src for apt build-dep
+RUN sed -e 's/Types: deb/Types: deb deb-src/' -i /etc/apt/sources.list.d/* || true
+RUN sed -e 's/# deb-src/deb-src/' -i /etc/apt/sources.list
+# archive.canonical.com is not reachable anymore, and cause a long timeout
+RUN sed -e '/archive.canonical.com/d' -i /etc/apt/sources.list
+# LTS < 14.04 are hosted on https://old-releases.ubuntu.com/releases/
+RUN if [ $(echo $UBUNTU_VERSION | cut -f 1 -d '.') -lt 14 ]; then \
+    sed -e 's/archive/old-releases/g' -i /etc/apt/sources.list; \
+fi
+
+# use apt-get since apt was missing -y option in some ancient distro
+RUN apt-get update && \
+env DEBIAN_FRONTEND=noninteractive apt-get install -y git build-essential
+RUN apt-get update && \
+env DEBIAN_FRONTEND=noninteractive apt-get build-dep -y --arch-only qemu
+# additional dependencies that are missing at some point in history
+RUN apt-get update && \
+env DEBIAN_FRONTEND=noninteractive apt-get install -y ninja-build flex bison libpixman-1-dev
+# python3-venv didn't exist in ancient ubuntu distro
+RUN apt-get update && \
+if apt-cache show python3-venv; then \
+    env DEBIAN_FRONTEND=noninteractive apt-get install -y python3-venv; \
+fi
+
+# patch cc and c++ to inject warning suppression options
+RUN mv /usr/bin/cc /usr/bin/cc.real
+RUN echo "#!/usr/bin/env bash" > /usr/bin/cc && \
+echo "exec /usr/bin/cc.real \"\$@\" -Wno-error -Wno-int-conversion" >> /usr/bin/cc && \
+chmod +x /usr/bin/cc
+RUN mv /usr/bin/c++ /usr/bin/c++.real
+RUN echo "#!/usr/bin/env bash" > /usr/bin/c++ && \
+echo "exec /usr/bin/c++.real \"\$@\" -Wno-error -Wno-int-conversion" >> /usr/bin/c++ && \
+chmod +x /usr/bin/c++
+
+# 2.9.0 => 3.1.0: clone git submodules from expired url, replace them with gitlab.
+# 2.9.0 => 2.11.0: issue with libfdt version, solved with dtc submodule, which
+#                  was removed later.
+# Ubuntu < 14.04: git has no --single-branch, so perform manual fetch after clone.
+ARG QEMU_VERSION
+RUN echo "QEMU_VERSION=${QEMU_VERSION:?QEMU_VERSION build arg must be set}"
+RUN cd /tmp && \
+git clone https://gitlab.com/qemu-project/qemu.git --depth 1 && cd qemu && \
+git fetch origin "${QEMU_VERSION}" --depth 1 && \
+git checkout FETCH_HEAD && \
+sed -e 's#git://git.qemu.org#https://gitlab.com/qemu-project/#' -i .gitmodules && \
+sed -e 's#git://git.qemu-project.org#https://gitlab.com/qemu-project/#' -i .gitmodules && \
+sed -e 's#https://git.qemu.org/git/#https://gitlab.com/qemu-project/#' -i .gitmodules && \
+sed -e 's#git://github.com#https://github.com#' -i .gitmodules && \
+if grep -q dtc .gitmodules; then git submodule update --init dtc; fi && \
+./configure --prefix=/opt/qemu --disable-docs && \
+make install -j $(nproc) && \
+strip /opt/qemu/bin/* && \
+rm -rf /tmp/*
+
+# retrieve runtime dependencies (libs + loader)
+RUN ldd /opt/qemu/bin/* | grep '=>' | sed -e 's/.*=>//' -e 's/(.*//' | sort -u > /opt/qemu/libs.links
+RUN find /lib*/ld-linux* >> /opt/qemu/libs.links
+RUN cat /opt/qemu/libs.links | xargs -n1 readlink -f > /opt/qemu/libs.realpath
+# store them in /deps
+RUN mkdir /deps && cp --parents $(cat /opt/qemu/libs.*) /deps/
+
+################################################################################
+FROM scratch
+
+# copy qemu and all associated artifacts (roms, ...)
+COPY --from=build /opt/qemu /usr
+COPY --from=build /deps /
+
+# build with:
+# podman build -f release.docker --build-arg QEMU_VERSION=v11.1.0 --build-arg UBUNTU_VERSION=24.04 -t qemu/v11.1.0
+# run with:
+# podman run -it --rm qemu/v11.1.0 qemu-system-aarch64 --version
-- 
2.43.0

[PATCH v3 2/3] scripts: add build_release_container.sh

[Pierrick Bouvier]

This script allows to easily build a qemu release container, based on
dockerfile introduced in previous commit.

It filters available versions, names image "qemu/$version", it
automatically detects which ubuntu version should be used to build it.

$ ./scripts/build_release_container.sh v0.12.0
$ podman run -it --rm qemu/v0.12.0 qemu-system-arm --version
QEMU PC emulator version 0.12.0, Copyright (c) 2003-2008 Fabrice Bellard

Signed-off-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>
---
 scripts/build_release_container.sh | 62 ++++++++++++++++++++++++++++++
 1 file changed, 62 insertions(+)
 create mode 100755 scripts/build_release_container.sh

diff --git a/scripts/build_release_container.sh b/scripts/build_release_container.sh
new file mode 100755
index 00000000000..38910f60e8f
--- /dev/null
+++ b/scripts/build_release_container.sh
@@ -0,0 +1,62 @@
+#!/usr/bin/env bash
+# SPDX-License-Identifier: GPL-2.0-or-later
+
+set -euo pipefail
+
+script_dir=$(dirname $(readlink -f $0))
+
+# We use ubuntu LTS containers, as original repo urls still exist for old
+# distros (>= 14.04). On ubuntu <= 10.04, binaries (including apt-get) segfault
+# due to missing vsyscall emulation on modern kernels, which leave us with
+# 12.04 as the oldest distro usable.
+#
+# The oldest buildable release is v0.12.0, as previous ones require gcc 3.X.
+oldest_release=v0.12.0
+
+if [ $# -ne 1 ]; then
+    echo "usage: tag (>= $oldest_release)"
+    exit 1
+fi
+version=$1;shift
+
+tags()
+{
+    git tag --sort=creatordate | grep '^v[0-9]' | grep -A 10000 $oldest_release
+}
+
+if ! tags | grep -q "^$version$"; then
+    echo "version $version is not available, check git tag (>= $oldest_release)"
+    exit 1
+fi
+
+year=$(git log -1 --format='%cd' --date='format:%Y' $version)
+month=$(git log -1 --format='%cd' --date='format:%m' $version)
+
+# Ubuntu LTS are published every (even) two years in April.
+ubuntu=
+if [ $((year % 2)) -eq 1 ]; then
+    # pick latest LTS from previous year
+    ubuntu=$((year - 1))
+elif [ $month -gt 6 ]; then
+    # LTS published this year should work
+    ubuntu=$year
+else
+    # pick latest LTS from two years ago
+    ubuntu=$((year - 2))
+fi
+if [ $ubuntu -lt 2012 ]; then
+    # oldest ubuntu release available with containers is 12.04
+    ubuntu=2012
+fi
+ubuntu=$((ubuntu - 2000))
+ubuntu_version=$ubuntu.04
+
+dockerfile=$script_dir/../tests/docker/dockerfiles/release.docker
+docker=$script_dir/../tests/docker/docker.py
+
+set -x
+$docker build \
+    -f $dockerfile \
+    --build-arg QEMU_VERSION=$version \
+    --build-arg UBUNTU_VERSION=$ubuntu_version \
+    -t qemu/$version
-- 
2.43.0

[PATCH v3 3/3] scripts: add generate_release_wrappers.sh

[Pierrick Bouvier]

Now that it's easy to build containers for all QEMU versions, we can go
further and offer wrapper scripts to all qemu binaries from all
buildable versions.

Note: this is based on prebuilt container images, that are hosted on my
dockerhub (https://hub.docker.com/repository/docker/pboqemu/qemu-releases).
If there is enough interest, I would be happy to relocate that on an
official QEMU registry, or even add it to our CI in case people get
paranoid about where those binaries come from.

$ ./scripts/generate_release_wrappers.sh qemu-releases/
$ ./qemu-releases/v0.12.0/qemu-system-arm --version
QEMU PC emulator version 0.12.0, Copyright (c) 2003-2008 Fabrice Bellard
$ ./qemu-releases/v11.1.2/qemu-system-arm --version
QEMU emulator version 11.1.2 (v11.1.2)
Copyright (c) 2003-2026 Fabrice Bellard and the QEMU Project developers

Container engine automatically download and runs the right image, so no
build is needed on developer machine.

Signed-off-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>
---
 scripts/generate_release_wrappers.sh | 71 ++++++++++++++++++++++++++++
 1 file changed, 71 insertions(+)
 create mode 100755 scripts/generate_release_wrappers.sh

diff --git a/scripts/generate_release_wrappers.sh b/scripts/generate_release_wrappers.sh
new file mode 100755
index 00000000000..3f37ecad4a1
--- /dev/null
+++ b/scripts/generate_release_wrappers.sh
@@ -0,0 +1,71 @@
+#!/usr/bin/env bash
+# SPDX-License-Identifier: GPL-2.0-or-later
+
+set -euo pipefail
+
+script_dir=$(dirname $(readlink -f $0))
+
+if [ $# -ne 1 ]; then
+    echo "usage: out_folder"
+    exit 1
+fi
+
+out=$1;shift
+
+releases()
+{
+    git tag --sort=creatordate |
+        grep '^v[0-9]' |
+        grep -A 10000 'v0\.12\.0' |
+        grep -v '\-rc[0-9]$'
+}
+
+qemu_targets()
+{
+    version=$1;shift
+    git ls-tree -r $version --name-only configs default-configs |
+        grep configs.*mak |
+        sed -e 's#.*/##' -e 's#\.mak$##' |
+        sort -u
+}
+
+softmmu_binaries()
+{
+    qemu_targets "$@" |
+        grep softmmu |
+        sed -e 's/-softmmu//' -e 's/^/qemu-system-/'
+}
+
+linux_user_binaries()
+{
+    qemu_targets "$@" |
+        grep linux-user |
+        sed -e 's/-linux-user//' -e 's/^/qemu-/'
+}
+
+root=$(readlink -f $script_dir/..)
+docker=$($root/tests/docker/docker.py probe)
+userns=
+if echo "$docker" | grep -q podman; then
+    userns="--userns=keep-id"
+fi
+
+registry=docker.io/pboqemu/qemu-releases
+for release in $(releases); do
+    echo "generate wrappers for $release"
+    public_image=$registry:$release
+    mkdir -p $out/$release
+    for bin in $(softmmu_binaries $release) $(linux_user_binaries $release); do
+        script=$out/$release/$bin
+        cat > $script << EOF
+#!/usr/bin/env bash
+
+here=\$(pwd)
+exec $docker run -it --rm -u $UID $userns \\
+     --security-opt label=disable \\
+     -w \$(pwd) -v \$(pwd):\$(pwd) \\
+     $public_image $bin "\$@"
+EOF
+        chmod +x $script
+    done
+done
-- 
2.43.0

Re: [PATCH v3 0/3] releases: build and run any QEMU release easily!

[Fabiano Rosas]

Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com> writes:

> This series was inspired by a recent post from Fabiano to provide container
> images for migration tests:
> https://lore.kernel.org/qemu-devel/24d04a1f-344d-4b90-a307-b582dd9a1128@oss.qualcomm.com/T/#m353f119dc1a24fa437dde8a96d1891629e64303c
>
> The idea to be able to build and run any QEMU (past) release with containers was
> in my mind for some time, so I gave it a shot. Finally, it was easier than
> expected to obtain a single dockerfile (with a few hacks) that works over all
> our releases.
>

Good stuff, thanks for this work. I'm just not sure how much of it fits
upstream in the current shape, let's discuss...

You say this was in your mind for a while, could you elaborate a bit on
how you think this could benefit the project?

To be clear, my use-case is simply to produce a few containers for
testing migration on my workstation. Your series can help a lot with
that, but from my point of view, some parts are clearly extra and some
are missing.

Also beware that what you're touching here is both "infrastructure" and
"publishing". This has implications for how people consume the work of
the project. There's a chance that people see this as "official QEMU
containers" and try to use them in all sorts of weird scenarios that we
as a community are not prepared to maintain. You might get some
push-back on that basis.

> The oldest functional distro we can find nowadays is ubuntu 12.04. Because of
> this, the oldest QEMU release we can build is v0.12.0. Previous ones require an
> environment we can't get easily today.
>

I think the ability to go all the way back is very cool, but unless we
want to look at this from a archival or heritage perspective, I'd say
there's a line somewhere to how old we should go. I don't see an
use-case for very old binaries. Even knowing if they still work would
already incur in a maintenance cost.

> Images have the absolute minimal content possible. It's only qemu binaries,
> artifacts (roms, keymap, ...), and libraries. There is no shell or any other
> tool included. As a result, images have a size from 149MB (v0.12.0) to 1.48GB
> (v11.1.2).
>

No shell means the first thing to do if one wants to debug the image is
to rebuild with a shell in. Not sure how frequent it will be, maybe it's
ok like this.

> This series adds 3 pieces:
> - a unique dockerfile to build any QEMU release. User has to provide an explicit
>   Ubuntu version when building a given QEMU version.

This is the tricky part. We'll be carrying code (script+dockerfile) that
only works on master. If we tag today and tomorrow the old distro repo
breaks, we obviously cannot fix it retroactively. So we need to be
careful of where we direct the maintenance effort.

1) I'd suggest first of all to keep the release image entirely separate
from the ubuntu image. Treat release.docker as we treat the other
dockerfiles and update it to work for the current release without
looking at past ones. Have it clone, checkout, build and gather
deps. Allow the configure to be tweaked via parameter in case we need to
disable something in an older build.

The script would then build a distro image (or dockerfile) with the
knowledge that it has about ubuntu depending on the QEMU tag vs. ubuntu
version combination. This would allow us to track the cludges more
precisely instead of "some ancient distro" as the dockerfile currently
says.

This is more of a layering concern and how easy it will be to drop/add
version-specific fixes.

2) I wonder if the usage of apt-get build-dep might be a convenience
trap. In theory, some of the adaptations that are required (such as
tweaking dependencies) have already been done by us when working on that
development window.  If we take the dockerfiles from the git tag, those
changes don't need to be done again and for the issues that didn't exist
at the time, we can map them to the git tag instead of ubuntu version;
(we tend to know more about QEMU releases than ubuntu releases).

Another issue with buid-dep is that it lists _ubuntu package_
dependencies. The distro can change how it builds the package without us
noticing. There is already some work done by lcitool, I imagine that it
would then be duplicated. Or worse(!) be different because build-dep
might be listing way more packages that what we need for the default
build.

This is more of a push towards using existing infra because those pieces
are already looked after for other purposes.

> - A script build_release_container.sh which automatically detects which ubuntu
>   version a given release should be buildable on (based on date). It works
>   great, and the heuristic works for all our versions.
> - a script generate_release_links.sh which creates wrapper scripts automatically
>   using prebuilt images (hosted on my dockerhub).
>

It would be nice if these two could take any hash, instead of just a
tag. This is one of the things that will depend on what you intend to
achieve with this series.

> TLDR:
> $ ./scripts/build_release_container.sh v1.6.1
> $ podman run -it --rm qemu/v1.6.1 qemu-system-arm --version
>
> $ ./script/generate_release_links.sh qemu-time-machine
> $ ./qemu-time-machine/v1.6.1/qemu-system-arm --version
> QEMU emulator version 1.6.1, Copyright (c) 2003-2008 Fabrice Bellard
> $ ./qemu-time-machine/v11.0.0/qemu-system-arm --version
> QEMU emulator version 11.0.0 (v11.0.0)
> Copyright (c) 2003-2026 Fabrice Bellard and the QEMU Project developers
>
> v3
> --
>
> - extend dockerfile to support ubuntu 12.04, and lower older revision we can
>   build from v1.6.1 to v0.12.0.
>
> v2
> --
>
> - fix generate_release_links.sh: was missing userns option for podman
> - dockerfile: cleanup "|| true" and replaced with proper ifs
>
> Pierrick Bouvier (3):
>   tests/docker/dockerfiles: add release.docker
>   scripts: add build_release_container.sh
>   scripts: add generate_release_wrappers.sh
>
>  scripts/build_release_container.sh      | 62 ++++++++++++++++++++
>  scripts/generate_release_wrappers.sh    | 71 +++++++++++++++++++++++
>  tests/docker/dockerfiles/release.docker | 76 +++++++++++++++++++++++++
>  3 files changed, 209 insertions(+)
>  create mode 100755 scripts/build_release_container.sh
>  create mode 100755 scripts/generate_release_wrappers.sh
>  create mode 100644 tests/docker/dockerfiles/release.docker

Re: [PATCH v3 1/3] tests/docker/dockerfiles: add release.docker

[Fabiano Rosas]

Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com> writes:

> This new dockerfile allows to build a distroless image containing only
> qemu stripped release binaries, their lib dependencies, and installed
> artifacts (roms, keymap, ...). There is no shell, no python, nothing
> else. This is achieved thanks to a multi-stage build, where we add qemu
> binaries, artifacts and libraries in a new scratch image. Image has
> the minimal size possible.
>
> This is based on Ubuntu, which has the advantage to have working
> containers available up to 12.04. Debian old old stable have broken apt
> sources for instance. Ubuntu 10.04 binaries segfault due to missing
> vsyscall emulation on modern kernels, so 12.04 is the lowest possible.
>
> This unique dockerfile works for all qemu versions, with 0.12.0 being the
> oldest we can build today (based on ubuntu 12.04). Previous ones need an
> older compiler.
>
> To build the image, you need to provide a qemu and ubuntu version:
> $ podman build -f release.Dockerfile --build-arg QEMU_VERSION=v11.1.0 --build-arg UBUNTU_VERSION=24.04 -t qemu/v11.1.0
>
> To run it, as there is no shell, and no defined entrypoint, you need to
> choose the binary you want explicitely:
> $ podman run -it --rm qemu/v11.1.0 qemu-system-aarch64 --version
>
> Notes: We could optimize a bit further, by creating a base container for
> each ubuntu version, that is reused between all qemu versions built with
> it. Given that recent QEMU have around 800MB of binaries + artifacts,
> and only 100MB of library dependency, I don't think it's worth the extra
> complexity.
>
> Notes: Also, I considered create one image per binary. Problem is
> that qemu depends on other artifacts (rom, keymap, etc) and it's not
> easy to identify what is needed for one binary. Thus, I think it's
> better to stick to one container with all binaries per release.
>

Yeah, it's fine this way.

> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>
> ---
>  tests/docker/dockerfiles/release.docker | 76 +++++++++++++++++++++++++
>  1 file changed, 76 insertions(+)
>  create mode 100644 tests/docker/dockerfiles/release.docker
>
> diff --git a/tests/docker/dockerfiles/release.docker b/tests/docker/dockerfiles/release.docker
> new file mode 100644
> index 00000000000..611ecca9859
> --- /dev/null
> +++ b/tests/docker/dockerfiles/release.docker
> @@ -0,0 +1,76 @@
> +ARG UBUNTU_VERSION="UBUNTU_VERSION_build_arg_must_be_set"
> +FROM ubuntu:$UBUNTU_VERSION as build

I believe FROM should be capitalized the same as AS.

> +ARG UBUNTU_VERSION

Is this duplicated?

> +
> +# add deb-src for apt build-dep
> +RUN sed -e 's/Types: deb/Types: deb deb-src/' -i /etc/apt/sources.list.d/* || true
> +RUN sed -e 's/# deb-src/deb-src/' -i /etc/apt/sources.list
> +# archive.canonical.com is not reachable anymore, and cause a long timeout
> +RUN sed -e '/archive.canonical.com/d' -i /etc/apt/sources.list
> +# LTS < 14.04 are hosted on https://old-releases.ubuntu.com/releases/
> +RUN if [ $(echo $UBUNTU_VERSION | cut -f 1 -d '.') -lt 14 ]; then \
> +    sed -e 's/archive/old-releases/g' -i /etc/apt/sources.list; \
> +fi

Hm, so I imagine these containers have reached end-of-life? I wonder
whether we should cap how further back we go based on this. It's one
thing to workaround a few installation bugs we list/install our
dependencies, another thing is the distro image poiting to dead URLs.

> +
> +# use apt-get since apt was missing -y option in some ancient distro

Ok, this is a reasonable issue to encounter and fix in the dockerfile. I
pointed at this on the cover letter, but we don't gain much in knowing
exactly which apt version has -y or not.

> +RUN apt-get update && \
> +env DEBIAN_FRONTEND=noninteractive apt-get install -y git build-essential
> +RUN apt-get update && \
> +env DEBIAN_FRONTEND=noninteractive apt-get build-dep -y --arch-only qemu
> +# additional dependencies that are missing at some point in history
> +RUN apt-get update && \
> +env DEBIAN_FRONTEND=noninteractive apt-get install -y ninja-build flex bison libpixman-1-dev

So a discrepancy on which QEMU version and how the distro packaged it at
the time vs. which dependencies we had. Is this something that would
have been solved by lcitool? (Hm, did we even use it back then?)

> +# python3-venv didn't exist in ancient ubuntu distro
> +RUN apt-get update && \
> +if apt-cache show python3-venv; then \
> +    env DEBIAN_FRONTEND=noninteractive apt-get install -y python3-venv; \
> +fi

Maybe good to know which version so we know when to drop this block?

I see some of these check the ubuntu version, some not, I wonder how
easy it will be to break an older image's build when trying to fix a new
one. I find it a bit confusing.

> +
> +# patch cc and c++ to inject warning suppression options
> +RUN mv /usr/bin/cc /usr/bin/cc.real
> +RUN echo "#!/usr/bin/env bash" > /usr/bin/cc && \
> +echo "exec /usr/bin/cc.real \"\$@\" -Wno-error -Wno-int-conversion" >> /usr/bin/cc && \
> +chmod +x /usr/bin/cc
> +RUN mv /usr/bin/c++ /usr/bin/c++.real
> +RUN echo "#!/usr/bin/env bash" > /usr/bin/c++ && \
> +echo "exec /usr/bin/c++.real \"\$@\" -Wno-error -Wno-int-conversion" >> /usr/bin/c++ && \
> +chmod +x /usr/bin/c++
> +

This assumes this dockerfile will not be used to test a current
build. Probably ok.

> +# 2.9.0 => 3.1.0: clone git submodules from expired url, replace them with gitlab.
> +# 2.9.0 => 2.11.0: issue with libfdt version, solved with dtc submodule, which
> +#                  was removed later.
> +# Ubuntu < 14.04: git has no --single-branch, so perform manual fetch after clone.

See how unstructured this has become? A comment here, the code down
there, nothing actually checks the version, the sed happens for all
versions, etc. Dockerfiles are bad at this kind of thing.

> +ARG QEMU_VERSION
> +RUN echo "QEMU_VERSION=${QEMU_VERSION:?QEMU_VERSION build arg must be set}"

In order to build local trees I added at this point:

RUN --mount=type=bind,from=qemu-src,target=/tmp/qemu-src,rw \
cd /tmp && \
if echo "${QEMU_VERSION}" | grep -q "\."; then \
    git clone https://gitlab.com/qemu-project/qemu.git --depth 1 && cd qemu && \
    git fetch origin "${QEMU_VERSION}" --depth 1 && \
    git checkout FETCH_HEAD; \
else \
    git clone qemu-src qemu && cd qemu && \
    git checkout "${QEMU_VERSION}"; \
fi && \


> +RUN cd /tmp && \
> +git clone https://gitlab.com/qemu-project/qemu.git --depth 1 && cd qemu && \
> +git fetch origin "${QEMU_VERSION}" --depth 1 && \
> +git checkout FETCH_HEAD && \
> +sed -e 's#git://git.qemu.org#https://gitlab.com/qemu-project/#' -i .gitmodules && \
> +sed -e 's#git://git.qemu-project.org#https://gitlab.com/qemu-project/#' -i .gitmodules && \
> +sed -e 's#https://git.qemu.org/git/#https://gitlab.com/qemu-project/#' -i .gitmodules && \
> +sed -e 's#git://github.com#https://github.com#' -i .gitmodules && \
> +if grep -q dtc .gitmodules; then git submodule update --init dtc; fi && \
> +./configure --prefix=/opt/qemu --disable-docs && \
> +make install -j $(nproc) && \
> +strip /opt/qemu/bin/* && \
> +rm -rf /tmp/*
> +
> +# retrieve runtime dependencies (libs + loader)
> +RUN ldd /opt/qemu/bin/* | grep '=>' | sed -e 's/.*=>//' -e 's/(.*//' | sort -u > /opt/qemu/libs.links
> +RUN find /lib*/ld-linux* >> /opt/qemu/libs.links
> +RUN cat /opt/qemu/libs.links | xargs -n1 readlink -f > /opt/qemu/libs.realpath
> +# store them in /deps
> +RUN mkdir /deps && cp --parents $(cat /opt/qemu/libs.*) /deps/
> +
> +################################################################################
> +FROM scratch
> +
> +# copy qemu and all associated artifacts (roms, ...)
> +COPY --from=build /opt/qemu /usr
> +COPY --from=build /deps /
> +
> +# build with:
> +# podman build -f release.docker --build-arg QEMU_VERSION=v11.1.0 --build-arg UBUNTU_VERSION=24.04 -t qemu/v11.1.0
> +# run with:
> +# podman run -it --rm qemu/v11.1.0 qemu-system-aarch64 --version

Re: [PATCH v3 1/3] tests/docker/dockerfiles: add release.docker

[Pierrick Bouvier]

On 10/23/2026 9:06 AM, Fabiano Rosas wrote:
> Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com> writes:
> 
>> This new dockerfile allows to build a distroless image containing only
>> qemu stripped release binaries, their lib dependencies, and installed
>> artifacts (roms, keymap, ...). There is no shell, no python, nothing
>> else. This is achieved thanks to a multi-stage build, where we add qemu
>> binaries, artifacts and libraries in a new scratch image. Image has
>> the minimal size possible.
>>
>> This is based on Ubuntu, which has the advantage to have working
>> containers available up to 12.04. Debian old old stable have broken apt
>> sources for instance. Ubuntu 10.04 binaries segfault due to missing
>> vsyscall emulation on modern kernels, so 12.04 is the lowest possible.
>>
>> This unique dockerfile works for all qemu versions, with 0.12.0 being the
>> oldest we can build today (based on ubuntu 12.04). Previous ones need an
>> older compiler.
>>
>> To build the image, you need to provide a qemu and ubuntu version:
>> $ podman build -f release.Dockerfile --build-arg QEMU_VERSION=v11.1.0 --build-arg UBUNTU_VERSION=24.04 -t qemu/v11.1.0
>>
>> To run it, as there is no shell, and no defined entrypoint, you need to
>> choose the binary you want explicitely:
>> $ podman run -it --rm qemu/v11.1.0 qemu-system-aarch64 --version
>>
>> Notes: We could optimize a bit further, by creating a base container for
>> each ubuntu version, that is reused between all qemu versions built with
>> it. Given that recent QEMU have around 800MB of binaries + artifacts,
>> and only 100MB of library dependency, I don't think it's worth the extra
>> complexity.
>>
>> Notes: Also, I considered create one image per binary. Problem is
>> that qemu depends on other artifacts (rom, keymap, etc) and it's not
>> easy to identify what is needed for one binary. Thus, I think it's
>> better to stick to one container with all binaries per release.
>>
> 
> Yeah, it's fine this way.
> 
>> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>
>> ---
>>  tests/docker/dockerfiles/release.docker | 76 +++++++++++++++++++++++++
>>  1 file changed, 76 insertions(+)
>>  create mode 100644 tests/docker/dockerfiles/release.docker
>>
>> diff --git a/tests/docker/dockerfiles/release.docker b/tests/docker/dockerfiles/release.docker
>> new file mode 100644
>> index 00000000000..611ecca9859
>> --- /dev/null
>> +++ b/tests/docker/dockerfiles/release.docker
>> @@ -0,0 +1,76 @@
>> +ARG UBUNTU_VERSION="UBUNTU_VERSION_build_arg_must_be_set"
>> +FROM ubuntu:$UBUNTU_VERSION as build
> 
> I believe FROM should be capitalized the same as AS.
>

Dockerfile instructions are not case sensitive, but yes, AS would be
more consistent, I'll change it.

>> +ARG UBUNTU_VERSION
> 
> Is this duplicated?
>

ARG have to be duplicated for each stage of the build.
In this case:
- pre-stage (to be used in first FROM)
- first-stage (to be used inside this stage)

>> +
>> +# add deb-src for apt build-dep
>> +RUN sed -e 's/Types: deb/Types: deb deb-src/' -i /etc/apt/sources.list.d/* || true
>> +RUN sed -e 's/# deb-src/deb-src/' -i /etc/apt/sources.list
>> +# archive.canonical.com is not reachable anymore, and cause a long timeout
>> +RUN sed -e '/archive.canonical.com/d' -i /etc/apt/sources.list
>> +# LTS < 14.04 are hosted on https://old-releases.ubuntu.com/releases/
>> +RUN if [ $(echo $UBUNTU_VERSION | cut -f 1 -d '.') -lt 14 ]; then \
>> +    sed -e 's/archive/old-releases/g' -i /etc/apt/sources.list; \
>> +fi
> 
> Hm, so I imagine these containers have reached end-of-life? I wonder
> whether we should cap how further back we go based on this. It's one
> thing to workaround a few installation bugs we list/install our
> dependencies, another thing is the distro image poiting to dead URLs.
>

If we want to cap it, then 14.04 is the lowest limit with fixed url.
Note that archive.canonical.com still has to be removed, it's present in
some LTS > 14.04 and create a long timeout for apt update.

>> +
>> +# use apt-get since apt was missing -y option in some ancient distro
> 
> Ok, this is a reasonable issue to encounter and fix in the dockerfile. I
> pointed at this on the cover letter, but we don't gain much in knowing
> exactly which apt version has -y or not.
> 
>> +RUN apt-get update && \
>> +env DEBIAN_FRONTEND=noninteractive apt-get install -y git build-essential
>> +RUN apt-get update && \
>> +env DEBIAN_FRONTEND=noninteractive apt-get build-dep -y --arch-only qemu
>> +# additional dependencies that are missing at some point in history
>> +RUN apt-get update && \
>> +env DEBIAN_FRONTEND=noninteractive apt-get install -y ninja-build flex bison libpixman-1-dev
> 
> So a discrepancy on which QEMU version and how the distro packaged it at
> the time vs. which dependencies we had. Is this something that would
> have been solved by lcitool? (Hm, did we even use it back then?)
>

It's new dependencies added by QEMU over time, that distro didn't
reflect yet at a past date. (ninja introduction, or recently hexagon
target which requires flex/bison for idef tool).

>> +# python3-venv didn't exist in ancient ubuntu distro
>> +RUN apt-get update && \
>> +if apt-cache show python3-venv; then \
>> +    env DEBIAN_FRONTEND=noninteractive apt-get install -y python3-venv; \
>> +fi
> 
> Maybe good to know which version so we know when to drop this block?
> 
> I see some of these check the ubuntu version, some not, I wonder how
> easy it will be to break an older image's build when trying to fix a new
> one. I find it a bit confusing.
>

Good point, I think I had this written *before* trying to go before
14.04, thus the inconsistency.

>> +
>> +# patch cc and c++ to inject warning suppression options
>> +RUN mv /usr/bin/cc /usr/bin/cc.real
>> +RUN echo "#!/usr/bin/env bash" > /usr/bin/cc && \
>> +echo "exec /usr/bin/cc.real \"\$@\" -Wno-error -Wno-int-conversion" >> /usr/bin/cc && \
>> +chmod +x /usr/bin/cc
>> +RUN mv /usr/bin/c++ /usr/bin/c++.real
>> +RUN echo "#!/usr/bin/env bash" > /usr/bin/c++ && \
>> +echo "exec /usr/bin/c++.real \"\$@\" -Wno-error -Wno-int-conversion" >> /usr/bin/c++ && \
>> +chmod +x /usr/bin/c++
>> +
> 
> This assumes this dockerfile will not be used to test a current
> build. Probably ok.
>

Why is it a problem? It just disables werror in a systematic way,
without relying on configure --disable-werror that was not always there.

>> +# 2.9.0 => 3.1.0: clone git submodules from expired url, replace them with gitlab.
>> +# 2.9.0 => 2.11.0: issue with libfdt version, solved with dtc submodule, which
>> +#                  was removed later.
>> +# Ubuntu < 14.04: git has no --single-branch, so perform manual fetch after clone.
> 
> See how unstructured this has become? A comment here, the code down
> there, nothing actually checks the version, the sed happens for all
> versions, etc. Dockerfiles are bad at this kind of thing.
>

This one is just a comment, and the code below has a single
implementation that works over all versions.
We could have ifs with multiple code paths, just a question of personal
taste I suppose.

Out of the if on ubuntu version at the beginning, the general approach
was to have a single code path for all versions of QEMU/Ubuntu, without
ifs per version.

>> +ARG QEMU_VERSION
>> +RUN echo "QEMU_VERSION=${QEMU_VERSION:?QEMU_VERSION build arg must be set}"
> 
> In order to build local trees I added at this point:
> 
> RUN --mount=type=bind,from=qemu-src,target=/tmp/qemu-src,rw \
> cd /tmp && \
> if echo "${QEMU_VERSION}" | grep -q "\."; then \
>     git clone https://gitlab.com/qemu-project/qemu.git --depth 1 && cd qemu && \
>     git fetch origin "${QEMU_VERSION}" --depth 1 && \
>     git checkout FETCH_HEAD; \
> else \
>     git clone qemu-src qemu && cd qemu && \
>     git checkout "${QEMU_VERSION}"; \
> fi && \
>

We can extend to support any sha-1 (it's just a matter of doing full vs
shallow clone).

For the local build, I don't like the idea to build from local state, it
has zero guarantee for reproducibility, which goes against what
dockerfiles are IMHO.

Regards,
Pierrick

Re: [PATCH v3 0/3] releases: build and run any QEMU release easily!

[Pierrick Bouvier]

On 10/23/2026 8:33 AM, Fabiano Rosas wrote:
> Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com> writes:
> 
>> This series was inspired by a recent post from Fabiano to provide container
>> images for migration tests:
>> https://lore.kernel.org/qemu-devel/24d04a1f-344d-4b90-a307-b582dd9a1128@oss.qualcomm.com/T/#m353f119dc1a24fa437dde8a96d1891629e64303c
>>
>> The idea to be able to build and run any QEMU (past) release with containers was
>> in my mind for some time, so I gave it a shot. Finally, it was easier than
>> expected to obtain a single dockerfile (with a few hacks) that works over all
>> our releases.
>>
> 
> Good stuff, thanks for this work. I'm just not sure how much of it fits
> upstream in the current shape, let's discuss...
>
> You say this was in your mind for a while, could you elaborate a bit on
> how you think this could benefit the project?
>

First usage that comes to my mind is that it would greatly help for
migration, allowing automatic testing in CI, by just downloading any
previous QEMU, dump a machine, and restore it on current version. We
could do it for last 10 releases to ensure migration from packaged
versions in all supported distros (red hat, alpine, ubuntu, ...) work as
expected.

It could be a enhancement compared to the manual process done once per
release. I read many times in review "it breaks the migration", that
yes, it would be nice to have CI saying "this series broke the migration
from qemu x.y".

Beyond that, I think we could also offer a "rebase on steroids" tool. We
could extend list of containers published to all top merge commits (git
log --first-parent), and provide a fast rebase, based on download only.
See it as a giant immutable precompiled cache. Dockerhub storage is free
and unlimited.
Some quick math: Since 8.0, there are ~2000 top merge commits. Each
image is < 250MB compressed, so the whole thing would be < 500GB.

Finally, in my personal case, while working on single binary, I had to
compare output of qemu help between versions. I'm also interested to
track QEMU binary size evolution over time. And some personal fun with
software archeology on the project too for anyone.

> To be clear, my use-case is simply to produce a few containers for
> testing migration on my workstation. Your series can help a lot with
> that, but from my point of view, some parts are clearly extra and some
> are missing.
> 
> Also beware that what you're touching here is both "infrastructure" and
> "publishing". This has implications for how people consume the work of
> the project. There's a chance that people see this as "official QEMU
> containers" and try to use them in all sorts of weird scenarios that we
> as a community are not prepared to maintain. You might get some
> push-back on that basis.
>

Sure, it was free to generate all images and push them on my personal
dockerhub, and it's now automated to push any feature release also in a
GitHub nightly job. If we want to make it more official, it should
really be hosted on an official account.

>> The oldest functional distro we can find nowadays is ubuntu 12.04. Because of
>> this, the oldest QEMU release we can build is v0.12.0. Previous ones require an
>> environment we can't get easily today.
>>
> 
> I think the ability to go all the way back is very cool, but unless we
> want to look at this from a archival or heritage perspective, I'd say
> there's a line somewhere to how old we should go. I don't see an
> use-case for very old binaries. Even knowing if they still work would
> already incur in a maintenance cost.
>

Agree, mostly was a fun challenge. I was also curious to see until when
we can go back in past from 2026.

For the "maintenance" a scheduled CI job running nightly is a free thing
for me, but I understand opinions on "maintenance cost" may differ. I
also co-maintain tests/docker with Alex, so I'll be the one taking the
load, and I'm fine with it.
  >> Images have the absolute minimal content possible. It's only qemu
binaries,
>> artifacts (roms, keymap, ...), and libraries. There is no shell or any other
>> tool included. As a result, images have a size from 149MB (v0.12.0) to 1.48GB
>> (v11.1.2).
>>
> 
> No shell means the first thing to do if one wants to debug the image is
> to rebuild with a shell in. Not sure how frequent it will be, maybe it's
> ok like this.
>

We could leave a base system, for only ~100MB per image, which is not
the end of the world (+10% in size). Personally, I appreciate to have no
shell, it reduces possibility of exploit when running an older QEMU
having CVEs.

>> This series adds 3 pieces:
>> - a unique dockerfile to build any QEMU release. User has to provide an explicit
>>   Ubuntu version when building a given QEMU version.
> 
> This is the tricky part. We'll be carrying code (script+dockerfile) that
> only works on master. If we tag today and tomorrow the old distro repo
> breaks, we obviously cannot fix it retroactively. So we need to be
> careful of where we direct the maintenance effort.
>
> 1) I'd suggest first of all to keep the release image entirely separate
> from the ubuntu image. Treat release.docker as we treat the other
> dockerfiles and update it to work for the current release without
> looking at past ones. Have it clone, checkout, build and gather
> deps. Allow the configure to be tweaked via parameter in case we need to
> disable something in an older build.
> 
> The script would then build a distro image (or dockerfile) with the
> knowledge that it has about ubuntu depending on the QEMU tag vs. ubuntu
> version combination. This would allow us to track the cludges more
> precisely instead of "some ancient distro" as the dockerfile currently
> says.
> 
> This is more of a layering concern and how easy it will be to drop/add
> version-specific fixes.
>

IIUC I think that's the approach Fabiano took, to use Dockerfile from
previous versions, and the conclusion was that it requires some seds and
hacks to have something that work. Internet state is not immutable.
IMHO, building previous QEMU with past Dockerfiles is doomed to fail.

Only thing you can control/guarantee is that "current dockerfile on
master works".

> 2) I wonder if the usage of apt-get build-dep might be a convenience
> trap. In theory, some of the adaptations that are required (such as
> tweaking dependencies) have already been done by us when working on that
> development window.  If we take the dockerfiles from the git tag, those
> changes don't need to be done again and for the issues that didn't exist
> at the time, we can map them to the git tag instead of ubuntu version;
> (we tend to know more about QEMU releases than ubuntu releases).
> 
> Another issue with buid-dep is that it lists _ubuntu package_
> dependencies. The distro can change how it builds the package without us
> noticing. There is already some work done by lcitool, I imagine that it
> would then be duplicated. Or worse(!) be different because build-dep
> might be listing way more packages that what we need for the default
> build.
> 
> This is more of a push towards using existing infra because those pieces
> are already looked after for other purposes.
> 
>> - A script build_release_container.sh which automatically detects which ubuntu
>>   version a given release should be buildable on (based on date). It works
>>   great, and the heuristic works for all our versions.
>> - a script generate_release_links.sh which creates wrapper scripts automatically
>>   using prebuilt images (hosted on my dockerhub).
>>

Sure, lcitool could work to express all that. Would I like to spend time
to rewrite all the missing bits in yaml when debian did it for me? not
really.

> 
> It would be nice if these two could take any hash, instead of just a
> tag. This is one of the things that will depend on what you intend to
> achieve with this series.
>

Very easy, it just needs a non shallow clone.

---

After reading and answering to this message, I feel there are two topics
we can discuss:
1. the current implementation
2. what would we like

1. Personally, I just posted this series as an example of "yes it's
possible with very limited amount of code", and I don't mind having
something completely different. I feel it's probably not worth reviewing
every line at this point.

2. The biggest question seems to be the direction with:
'do we want to provide a single dockerfile that covers all versions' (a)
VS 'having a dockerfile at a time of history for current version' (b).

I think (b) is what we have now (current ubuntu/suse/debian/alpine image
works for current revision, since it's used in CI), and it was already
proven it fails from Fabiano's feedback.
It seems to be that's what you propose also from your answer above.

This series goes in another direction (a), adding a unique dockerfile
covering all versions, independently of all the dockerfiles we have now.

If we agree (a) is a better direction, unique dockerfile approach can be
vastly simplified by restricting ourselves to qemu >= 3.1.0. If not, no
hard feelings, it's just a proposal.

Regards,
Pierrick

Re: [PATCH v3 1/3] tests/docker/dockerfiles: add release.docker

[Fabiano Rosas]

Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com> writes:

> On 10/23/2026 9:06 AM, Fabiano Rosas wrote:
>> Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com> writes:
>> 
>>> This new dockerfile allows to build a distroless image containing only
>>> qemu stripped release binaries, their lib dependencies, and installed
>>> artifacts (roms, keymap, ...). There is no shell, no python, nothing
>>> else. This is achieved thanks to a multi-stage build, where we add qemu
>>> binaries, artifacts and libraries in a new scratch image. Image has
>>> the minimal size possible.
>>>
>>> This is based on Ubuntu, which has the advantage to have working
>>> containers available up to 12.04. Debian old old stable have broken apt
>>> sources for instance. Ubuntu 10.04 binaries segfault due to missing
>>> vsyscall emulation on modern kernels, so 12.04 is the lowest possible.
>>>
>>> This unique dockerfile works for all qemu versions, with 0.12.0 being the
>>> oldest we can build today (based on ubuntu 12.04). Previous ones need an
>>> older compiler.
>>>
>>> To build the image, you need to provide a qemu and ubuntu version:
>>> $ podman build -f release.Dockerfile --build-arg QEMU_VERSION=v11.1.0 --build-arg UBUNTU_VERSION=24.04 -t qemu/v11.1.0
>>>
>>> To run it, as there is no shell, and no defined entrypoint, you need to
>>> choose the binary you want explicitely:
>>> $ podman run -it --rm qemu/v11.1.0 qemu-system-aarch64 --version
>>>
>>> Notes: We could optimize a bit further, by creating a base container for
>>> each ubuntu version, that is reused between all qemu versions built with
>>> it. Given that recent QEMU have around 800MB of binaries + artifacts,
>>> and only 100MB of library dependency, I don't think it's worth the extra
>>> complexity.
>>>
>>> Notes: Also, I considered create one image per binary. Problem is
>>> that qemu depends on other artifacts (rom, keymap, etc) and it's not
>>> easy to identify what is needed for one binary. Thus, I think it's
>>> better to stick to one container with all binaries per release.
>>>
>> 
>> Yeah, it's fine this way.
>> 
>>> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>
>>> ---
>>>  tests/docker/dockerfiles/release.docker | 76 +++++++++++++++++++++++++
>>>  1 file changed, 76 insertions(+)
>>>  create mode 100644 tests/docker/dockerfiles/release.docker
>>>
>>> diff --git a/tests/docker/dockerfiles/release.docker b/tests/docker/dockerfiles/release.docker
>>> new file mode 100644
>>> index 00000000000..611ecca9859
>>> --- /dev/null
>>> +++ b/tests/docker/dockerfiles/release.docker
>>> @@ -0,0 +1,76 @@
>>> +ARG UBUNTU_VERSION="UBUNTU_VERSION_build_arg_must_be_set"
>>> +FROM ubuntu:$UBUNTU_VERSION as build
>> 
>> I believe FROM should be capitalized the same as AS.
>>
>
> Dockerfile instructions are not case sensitive, but yes, AS would be
> more consistent, I'll change it.
>

Docker emits a warning about the mismatch.

>>> +ARG UBUNTU_VERSION
>> 
>> Is this duplicated?
>>
>
> ARG have to be duplicated for each stage of the build.
> In this case:
> - pre-stage (to be used in first FROM)
> - first-stage (to be used inside this stage)
>

Got it.

>>> +
>>> +# add deb-src for apt build-dep
>>> +RUN sed -e 's/Types: deb/Types: deb deb-src/' -i /etc/apt/sources.list.d/* || true
>>> +RUN sed -e 's/# deb-src/deb-src/' -i /etc/apt/sources.list
>>> +# archive.canonical.com is not reachable anymore, and cause a long timeout
>>> +RUN sed -e '/archive.canonical.com/d' -i /etc/apt/sources.list
>>> +# LTS < 14.04 are hosted on https://old-releases.ubuntu.com/releases/
>>> +RUN if [ $(echo $UBUNTU_VERSION | cut -f 1 -d '.') -lt 14 ]; then \
>>> +    sed -e 's/archive/old-releases/g' -i /etc/apt/sources.list; \
>>> +fi
>> 
>> Hm, so I imagine these containers have reached end-of-life? I wonder
>> whether we should cap how further back we go based on this. It's one
>> thing to workaround a few installation bugs we list/install our
>> dependencies, another thing is the distro image poiting to dead URLs.
>>
>
> If we want to cap it, then 14.04 is the lowest limit with fixed url.
> Note that archive.canonical.com still has to be removed, it's present in
> some LTS > 14.04 and create a long timeout for apt update.
>
>>> +
>>> +# use apt-get since apt was missing -y option in some ancient distro
>> 
>> Ok, this is a reasonable issue to encounter and fix in the dockerfile. I
>> pointed at this on the cover letter, but we don't gain much in knowing
>> exactly which apt version has -y or not.
>> 
>>> +RUN apt-get update && \
>>> +env DEBIAN_FRONTEND=noninteractive apt-get install -y git build-essential
>>> +RUN apt-get update && \
>>> +env DEBIAN_FRONTEND=noninteractive apt-get build-dep -y --arch-only qemu
>>> +# additional dependencies that are missing at some point in history
>>> +RUN apt-get update && \
>>> +env DEBIAN_FRONTEND=noninteractive apt-get install -y ninja-build flex bison libpixman-1-dev
>> 
>> So a discrepancy on which QEMU version and how the distro packaged it at
>> the time vs. which dependencies we had. Is this something that would
>> have been solved by lcitool? (Hm, did we even use it back then?)
>>
>
> It's new dependencies added by QEMU over time, that distro didn't
> reflect yet at a past date. (ninja introduction, or recently hexagon
> target which requires flex/bison for idef tool).
>

Right, and they may never add it, because it's either already in the
past or because they simply don't need that dependency for their
package.

If we're around release time and for some reason they change what's in
their package, it would be annoying to have to figure this out to
unbreak CI or something similar.

It's a situation that tends to degenerate to that last line simply
becoming the full list of dependencies. There's also the whack-a-mole-y
job of removing packages (for tidyness) from the last line if they end
up being provided by build-dep.

Granted it's containers and there may be little expectation of things
working indefinitely without code changes, so I'll let the maintainers
do what they prefer.

>>> +# python3-venv didn't exist in ancient ubuntu distro
>>> +RUN apt-get update && \
>>> +if apt-cache show python3-venv; then \
>>> +    env DEBIAN_FRONTEND=noninteractive apt-get install -y python3-venv; \
>>> +fi
>> 
>> Maybe good to know which version so we know when to drop this block?
>> 
>> I see some of these check the ubuntu version, some not, I wonder how
>> easy it will be to break an older image's build when trying to fix a new
>> one. I find it a bit confusing.
>>
>
> Good point, I think I had this written *before* trying to go before
> 14.04, thus the inconsistency.
>
>>> +
>>> +# patch cc and c++ to inject warning suppression options
>>> +RUN mv /usr/bin/cc /usr/bin/cc.real
>>> +RUN echo "#!/usr/bin/env bash" > /usr/bin/cc && \
>>> +echo "exec /usr/bin/cc.real \"\$@\" -Wno-error -Wno-int-conversion" >> /usr/bin/cc && \
>>> +chmod +x /usr/bin/cc
>>> +RUN mv /usr/bin/c++ /usr/bin/c++.real
>>> +RUN echo "#!/usr/bin/env bash" > /usr/bin/c++ && \
>>> +echo "exec /usr/bin/c++.real \"\$@\" -Wno-error -Wno-int-conversion" >> /usr/bin/c++ && \
>>> +chmod +x /usr/bin/c++
>>> +
>> 
>> This assumes this dockerfile will not be used to test a current
>> build. Probably ok.
>>
>
> Why is it a problem? It just disables werror in a systematic way,
> without relying on configure --disable-werror that was not always there.
>

Depends on how we use this image. If it runs early in some CI it might
be better to let it catch the warnings.

>>> +# 2.9.0 => 3.1.0: clone git submodules from expired url, replace them with gitlab.

This sentence may need rewording.

>>> +# 2.9.0 => 2.11.0: issue with libfdt version, solved with dtc submodule, which
>>> +#                  was removed later.
>>> +# Ubuntu < 14.04: git has no --single-branch, so perform manual fetch after clone.
>> 
>> See how unstructured this has become? A comment here, the code down
>> there, nothing actually checks the version, the sed happens for all
>> versions, etc. Dockerfiles are bad at this kind of thing.
>>
>
> This one is just a comment, and the code below has a single
> implementation that works over all versions.
> We could have ifs with multiple code paths, just a question of personal
> taste I suppose.
>

Could be.

> Out of the if on ubuntu version at the beginning, the general approach
> was to have a single code path for all versions of QEMU/Ubuntu, without
> ifs per version.
>

Ok, makes sense.

>>> +ARG QEMU_VERSION
>>> +RUN echo "QEMU_VERSION=${QEMU_VERSION:?QEMU_VERSION build arg must be set}"
>> 
>> In order to build local trees I added at this point:
>> 
>> RUN --mount=type=bind,from=qemu-src,target=/tmp/qemu-src,rw \
>> cd /tmp && \
>> if echo "${QEMU_VERSION}" | grep -q "\."; then \
>>     git clone https://gitlab.com/qemu-project/qemu.git --depth 1 && cd qemu && \
>>     git fetch origin "${QEMU_VERSION}" --depth 1 && \
>>     git checkout FETCH_HEAD; \
>> else \
>>     git clone qemu-src qemu && cd qemu && \
>>     git checkout "${QEMU_VERSION}"; \
>> fi && \
>>
>
> We can extend to support any sha-1 (it's just a matter of doing full vs
> shallow clone).
>

+1

> For the local build, I don't like the idea to build from local state, it
> has zero guarantee for reproducibility, which goes against what
> dockerfiles are IMHO.
>

Could this be crafted to take the git repo url as argument, then? Your
point is valid, but once this image is used for something that exposes a
bug, it's natural to want to re-execute with a modified codebase to
debug or test a fix.

> Regards,
> Pierrick

Re: [PATCH v3 1/3] tests/docker/dockerfiles: add release.docker

[Pierrick Bouvier]

On 10/26/2026 6:31 AM, Fabiano Rosas wrote:
> Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com> writes:
> 
>> On 10/23/2026 9:06 AM, Fabiano Rosas wrote:
>>> Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com> writes:
>>>
>>>> This new dockerfile allows to build a distroless image containing only
>>>> qemu stripped release binaries, their lib dependencies, and installed
>>>> artifacts (roms, keymap, ...). There is no shell, no python, nothing
>>>> else. This is achieved thanks to a multi-stage build, where we add qemu
>>>> binaries, artifacts and libraries in a new scratch image. Image has
>>>> the minimal size possible.
>>>>
>>>> This is based on Ubuntu, which has the advantage to have working
>>>> containers available up to 12.04. Debian old old stable have broken apt
>>>> sources for instance. Ubuntu 10.04 binaries segfault due to missing
>>>> vsyscall emulation on modern kernels, so 12.04 is the lowest possible.
>>>>
>>>> This unique dockerfile works for all qemu versions, with 0.12.0 being the
>>>> oldest we can build today (based on ubuntu 12.04). Previous ones need an
>>>> older compiler.
>>>>
>>>> To build the image, you need to provide a qemu and ubuntu version:
>>>> $ podman build -f release.Dockerfile --build-arg QEMU_VERSION=v11.1.0 --build-arg UBUNTU_VERSION=24.04 -t qemu/v11.1.0
>>>>
>>>> To run it, as there is no shell, and no defined entrypoint, you need to
>>>> choose the binary you want explicitely:
>>>> $ podman run -it --rm qemu/v11.1.0 qemu-system-aarch64 --version
>>>>
>>>> Notes: We could optimize a bit further, by creating a base container for
>>>> each ubuntu version, that is reused between all qemu versions built with
>>>> it. Given that recent QEMU have around 800MB of binaries + artifacts,
>>>> and only 100MB of library dependency, I don't think it's worth the extra
>>>> complexity.
>>>>
>>>> Notes: Also, I considered create one image per binary. Problem is
>>>> that qemu depends on other artifacts (rom, keymap, etc) and it's not
>>>> easy to identify what is needed for one binary. Thus, I think it's
>>>> better to stick to one container with all binaries per release.
>>>>
>>>
>>> Yeah, it's fine this way.
>>>
>>>> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>
>>>> ---
>>>>  tests/docker/dockerfiles/release.docker | 76 +++++++++++++++++++++++++
>>>>  1 file changed, 76 insertions(+)
>>>>  create mode 100644 tests/docker/dockerfiles/release.docker
>>>>
>>>> diff --git a/tests/docker/dockerfiles/release.docker b/tests/docker/dockerfiles/release.docker
>>>> new file mode 100644
>>>> index 00000000000..611ecca9859
>>>> --- /dev/null
>>>> +++ b/tests/docker/dockerfiles/release.docker
>>>> @@ -0,0 +1,76 @@
>>>> +ARG UBUNTU_VERSION="UBUNTU_VERSION_build_arg_must_be_set"
>>>> +FROM ubuntu:$UBUNTU_VERSION as build
>>>
>>> I believe FROM should be capitalized the same as AS.
>>>
>>
>> Dockerfile instructions are not case sensitive, but yes, AS would be
>> more consistent, I'll change it.
>>
> 
> Docker emits a warning about the mismatch.
>

👍

>>>> +ARG UBUNTU_VERSION
>>>
>>> Is this duplicated?
>>>
>>
>> ARG have to be duplicated for each stage of the build.
>> In this case:
>> - pre-stage (to be used in first FROM)
>> - first-stage (to be used inside this stage)
>>
> 
> Got it.
> 
>>>> +
>>>> +# add deb-src for apt build-dep
>>>> +RUN sed -e 's/Types: deb/Types: deb deb-src/' -i /etc/apt/sources.list.d/* || true
>>>> +RUN sed -e 's/# deb-src/deb-src/' -i /etc/apt/sources.list
>>>> +# archive.canonical.com is not reachable anymore, and cause a long timeout
>>>> +RUN sed -e '/archive.canonical.com/d' -i /etc/apt/sources.list
>>>> +# LTS < 14.04 are hosted on https://old-releases.ubuntu.com/releases/
>>>> +RUN if [ $(echo $UBUNTU_VERSION | cut -f 1 -d '.') -lt 14 ]; then \
>>>> +    sed -e 's/archive/old-releases/g' -i /etc/apt/sources.list; \
>>>> +fi
>>>
>>> Hm, so I imagine these containers have reached end-of-life? I wonder
>>> whether we should cap how further back we go based on this. It's one
>>> thing to workaround a few installation bugs we list/install our
>>> dependencies, another thing is the distro image poiting to dead URLs.
>>>
>>
>> If we want to cap it, then 14.04 is the lowest limit with fixed url.
>> Note that archive.canonical.com still has to be removed, it's present in
>> some LTS > 14.04 and create a long timeout for apt update.
>>
>>>> +
>>>> +# use apt-get since apt was missing -y option in some ancient distro
>>>
>>> Ok, this is a reasonable issue to encounter and fix in the dockerfile. I
>>> pointed at this on the cover letter, but we don't gain much in knowing
>>> exactly which apt version has -y or not.
>>>
>>>> +RUN apt-get update && \
>>>> +env DEBIAN_FRONTEND=noninteractive apt-get install -y git build-essential
>>>> +RUN apt-get update && \
>>>> +env DEBIAN_FRONTEND=noninteractive apt-get build-dep -y --arch-only qemu
>>>> +# additional dependencies that are missing at some point in history
>>>> +RUN apt-get update && \
>>>> +env DEBIAN_FRONTEND=noninteractive apt-get install -y ninja-build flex bison libpixman-1-dev
>>>
>>> So a discrepancy on which QEMU version and how the distro packaged it at
>>> the time vs. which dependencies we had. Is this something that would
>>> have been solved by lcitool? (Hm, did we even use it back then?)
>>>
>>
>> It's new dependencies added by QEMU over time, that distro didn't
>> reflect yet at a past date. (ninja introduction, or recently hexagon
>> target which requires flex/bison for idef tool).
>>
> 
> Right, and they may never add it, because it's either already in the
> past or because they simply don't need that dependency for their
> package.
> 
> If we're around release time and for some reason they change what's in
> their package, it would be annoying to have to figure this out to
> unbreak CI or something similar.
> 
> It's a situation that tends to degenerate to that last line simply
> becoming the full list of dependencies. There's also the whack-a-mole-y
> job of removing packages (for tidyness) from the last line if they end
> up being provided by build-dep.
>
It's only needed in the time frame where a new dependency is added until
next LTS catches up with it, so it will never be the full list of
dependencies. As you can see in current patch, we are talking about 5
packages over 15 years of history.

Also, all "extra" packages installed manually are still valid today. I
feel confident that 15 years of history is enough to prove it's a
sustainable approach, with a result more maintainable than a manual list
of dependencies for each release.

> Granted it's containers and there may be little expectation of things
> working indefinitely without code changes, so I'll let the maintainers
> do what they prefer.
> 
>>>> +# python3-venv didn't exist in ancient ubuntu distro
>>>> +RUN apt-get update && \
>>>> +if apt-cache show python3-venv; then \
>>>> +    env DEBIAN_FRONTEND=noninteractive apt-get install -y python3-venv; \
>>>> +fi
>>>
>>> Maybe good to know which version so we know when to drop this block?
>>>
>>> I see some of these check the ubuntu version, some not, I wonder how
>>> easy it will be to break an older image's build when trying to fix a new
>>> one. I find it a bit confusing.
>>>
>>
>> Good point, I think I had this written *before* trying to go before
>> 14.04, thus the inconsistency.
>>
>>>> +
>>>> +# patch cc and c++ to inject warning suppression options
>>>> +RUN mv /usr/bin/cc /usr/bin/cc.real
>>>> +RUN echo "#!/usr/bin/env bash" > /usr/bin/cc && \
>>>> +echo "exec /usr/bin/cc.real \"\$@\" -Wno-error -Wno-int-conversion" >> /usr/bin/cc && \
>>>> +chmod +x /usr/bin/cc
>>>> +RUN mv /usr/bin/c++ /usr/bin/c++.real
>>>> +RUN echo "#!/usr/bin/env bash" > /usr/bin/c++ && \
>>>> +echo "exec /usr/bin/c++.real \"\$@\" -Wno-error -Wno-int-conversion" >> /usr/bin/c++ && \
>>>> +chmod +x /usr/bin/c++
>>>> +
>>>
>>> This assumes this dockerfile will not be used to test a current
>>> build. Probably ok.
>>>
>>
>> Why is it a problem? It just disables werror in a systematic way,
>> without relying on configure --disable-werror that was not always there.
>>
> 
> Depends on how we use this image. If it runs early in some CI it might
> be better to let it catch the warnings.
>

Even in this case, I think that ignoring the warnings is the right thing
to do, as you want this to build at any point of history.

Classical example is a new LTS release is published, and new warnings
appear breaking compilation. It's something we want to deal with
existing image containers (ubuntu, fedora, ...) and not with a unified
dockerfile for past releases.

>>>> +# 2.9.0 => 3.1.0: clone git submodules from expired url, replace them with gitlab.
> 
> This sentence may need rewording.
> 
>>>> +# 2.9.0 => 2.11.0: issue with libfdt version, solved with dtc submodule, which
>>>> +#                  was removed later.
>>>> +# Ubuntu < 14.04: git has no --single-branch, so perform manual fetch after clone.
>>>
>>> See how unstructured this has become? A comment here, the code down
>>> there, nothing actually checks the version, the sed happens for all
>>> versions, etc. Dockerfiles are bad at this kind of thing.
>>>
>>
>> This one is just a comment, and the code below has a single
>> implementation that works over all versions.
>> We could have ifs with multiple code paths, just a question of personal
>> taste I suppose.
>>
> 
> Could be.
> 
>> Out of the if on ubuntu version at the beginning, the general approach
>> was to have a single code path for all versions of QEMU/Ubuntu, without
>> ifs per version.
>>
> 
> Ok, makes sense.
>

From the conversation on v2, I feel that the right thing to do is to
restrict this to versions >= v3.1.0, which simplifies a lot the code of
the Dockerfile (no if on UBUNTU_VERSION, no submodule hacks, and
checkout process gets unified).

>>>> +ARG QEMU_VERSION
>>>> +RUN echo "QEMU_VERSION=${QEMU_VERSION:?QEMU_VERSION build arg must be set}"
>>>
>>> In order to build local trees I added at this point:
>>>
>>> RUN --mount=type=bind,from=qemu-src,target=/tmp/qemu-src,rw \
>>> cd /tmp && \
>>> if echo "${QEMU_VERSION}" | grep -q "\."; then \
>>>     git clone https://gitlab.com/qemu-project/qemu.git --depth 1 && cd qemu && \
>>>     git fetch origin "${QEMU_VERSION}" --depth 1 && \
>>>     git checkout FETCH_HEAD; \
>>> else \
>>>     git clone qemu-src qemu && cd qemu && \
>>>     git checkout "${QEMU_VERSION}"; \
>>> fi && \
>>>
>>
>> We can extend to support any sha-1 (it's just a matter of doing full vs
>> shallow clone).
>>
> 
> +1
>

I'll make sure this can take any version.

>> For the local build, I don't like the idea to build from local state, it
>> has zero guarantee for reproducibility, which goes against what
>> dockerfiles are IMHO.
>>
> 
> Could this be crafted to take the git repo url as argument, then? Your
> point is valid, but once this image is used for something that exposes a
> bug, it's natural to want to re-execute with a modified codebase to
> debug or test a fix.
>

Yes that's easy to add, I'll add an option for it.

Regards,
Pierrick

Re: [PATCH v3 0/3] releases: build and run any QEMU release easily!

[Fabiano Rosas]

Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com> writes:

> On 10/23/2026 8:33 AM, Fabiano Rosas wrote:
>> Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com> writes:
>> 
>>> This series was inspired by a recent post from Fabiano to provide container
>>> images for migration tests:
>>> https://lore.kernel.org/qemu-devel/24d04a1f-344d-4b90-a307-b582dd9a1128@oss.qualcomm.com/T/#m353f119dc1a24fa437dde8a96d1891629e64303c
>>>
>>> The idea to be able to build and run any QEMU (past) release with containers was
>>> in my mind for some time, so I gave it a shot. Finally, it was easier than
>>> expected to obtain a single dockerfile (with a few hacks) that works over all
>>> our releases.
>>>
>> 
>> Good stuff, thanks for this work. I'm just not sure how much of it fits
>> upstream in the current shape, let's discuss...
>>
>> You say this was in your mind for a while, could you elaborate a bit on
>> how you think this could benefit the project?
>>
>
> First usage that comes to my mind is that it would greatly help for
> migration, allowing automatic testing in CI, by just downloading any
> previous QEMU, dump a machine, and restore it on current version.

Don't we also build QEMU several times over in CI? Maybe I'm wrong, but
it might allow us to decouple a few tests from the build steps.

> We could do it for last 10 releases to ensure migration from packaged
> versions in all supported distros (red hat, alpine, ubuntu, ...) work
> as expected.
>

It's a nice idea but potentially a costly one. I don't think there is a
guarantee that the downstream package is migration-compatible with the
upstream package.

> It could be a enhancement compared to the manual process done once per
> release. I read many times in review "it breaks the migration", that
> yes, it would be nice to have CI saying "this series broke the migration
> from qemu x.y".
>

Most of the "breaks migration" issues are due to device migration. What
we have today in terms of migration testing cannot detect most of those.

> Beyond that, I think we could also offer a "rebase on steroids" tool. We
> could extend list of containers published to all top merge commits (git
> log --first-parent), and provide a fast rebase, based on download only.

Bisect requires a test of some sort. Will this cause the opposite
problem than what migration-test encounters? I.e. instead of "old QEMU
needs old OS" you'd have "new test needs QEMU from new OS". One
alternative would be to build the tests into an image as well. Or
perhaps it wouldn't be a problem in practice.

> See it as a giant immutable precompiled cache. Dockerhub storage is free
> and unlimited.
> Some quick math: Since 8.0, there are ~2000 top merge commits. Each
> image is < 250MB compressed, so the whole thing would be < 500GB.
>
> Finally, in my personal case, while working on single binary, I had to
> compare output of qemu help between versions. I'm also interested to
> track QEMU binary size evolution over time. And some personal fun with
> software archeology on the project too for anyone.
>
>> To be clear, my use-case is simply to produce a few containers for
>> testing migration on my workstation. Your series can help a lot with
>> that, but from my point of view, some parts are clearly extra and some
>> are missing.
>> 
>> Also beware that what you're touching here is both "infrastructure" and
>> "publishing". This has implications for how people consume the work of
>> the project. There's a chance that people see this as "official QEMU
>> containers" and try to use them in all sorts of weird scenarios that we
>> as a community are not prepared to maintain. You might get some
>> push-back on that basis.
>>
>
> Sure, it was free to generate all images and push them on my personal
> dockerhub, and it's now automated to push any feature release also in a
> GitHub nightly job. If we want to make it more official, it should
> really be hosted on an official account.
>
>>> The oldest functional distro we can find nowadays is ubuntu 12.04. Because of
>>> this, the oldest QEMU release we can build is v0.12.0. Previous ones require an
>>> environment we can't get easily today.
>>>
>> 
>> I think the ability to go all the way back is very cool, but unless we
>> want to look at this from a archival or heritage perspective, I'd say
>> there's a line somewhere to how old we should go. I don't see an
>> use-case for very old binaries. Even knowing if they still work would
>> already incur in a maintenance cost.
>>
>
> Agree, mostly was a fun challenge. I was also curious to see until when
> we can go back in past from 2026.
>
> For the "maintenance" a scheduled CI job running nightly is a free thing
> for me, but I understand opinions on "maintenance cost" may differ. I
> also co-maintain tests/docker with Alex, so I'll be the one taking the
> load, and I'm fine with it.

Fair enough.

>   >> Images have the absolute minimal content possible. It's only qemu
> binaries,
>>> artifacts (roms, keymap, ...), and libraries. There is no shell or any other
>>> tool included. As a result, images have a size from 149MB (v0.12.0) to 1.48GB
>>> (v11.1.2).
>>>
>> 
>> No shell means the first thing to do if one wants to debug the image is
>> to rebuild with a shell in. Not sure how frequent it will be, maybe it's
>> ok like this.
>>
>
> We could leave a base system, for only ~100MB per image, which is not
> the end of the world (+10% in size). Personally, I appreciate to have no
> shell, it reduces possibility of exploit when running an older QEMU
> having CVEs.
>
>>> This series adds 3 pieces:
>>> - a unique dockerfile to build any QEMU release. User has to provide an explicit
>>>   Ubuntu version when building a given QEMU version.
>> 
>> This is the tricky part. We'll be carrying code (script+dockerfile) that
>> only works on master. If we tag today and tomorrow the old distro repo
>> breaks, we obviously cannot fix it retroactively. So we need to be
>> careful of where we direct the maintenance effort.
>>
>> 1) I'd suggest first of all to keep the release image entirely separate
>> from the ubuntu image. Treat release.docker as we treat the other
>> dockerfiles and update it to work for the current release without
>> looking at past ones. Have it clone, checkout, build and gather
>> deps. Allow the configure to be tweaked via parameter in case we need to
>> disable something in an older build.
>> 
>> The script would then build a distro image (or dockerfile) with the
>> knowledge that it has about ubuntu depending on the QEMU tag vs. ubuntu
>> version combination. This would allow us to track the cludges more
>> precisely instead of "some ancient distro" as the dockerfile currently
>> says.
>> 
>> This is more of a layering concern and how easy it will be to drop/add
>> version-specific fixes.
>>
>
> IIUC I think that's the approach Fabiano took, to use Dockerfile from
> previous versions, and the conclusion was that it requires some seds and
> hacks to have something that work. Internet state is not immutable.
> IMHO, building previous QEMU with past Dockerfiles is doomed to fail.
>

That's not what I'm suggesting here. I'm saying that "current
dockerfile" should have less tweaks and "current
build_release_container.sh" should be the one responsible for doing the
ubuntu-version-to-tweak mapping.

But maybe you meant to reply to the second point below. In that case,
yes the past dockerfiles require tweaks, but they also provide most of
the time the exact list of dependencies for the previous QEMU, and that
work was already done by the people from the past.

> Only thing you can control/guarantee is that "current dockerfile on
> master works".
>
>> 2) I wonder if the usage of apt-get build-dep might be a convenience
>> trap. In theory, some of the adaptations that are required (such as
>> tweaking dependencies) have already been done by us when working on that
>> development window.  If we take the dockerfiles from the git tag, those
>> changes don't need to be done again and for the issues that didn't exist
>> at the time, we can map them to the git tag instead of ubuntu version;
>> (we tend to know more about QEMU releases than ubuntu releases).
>> 
>> Another issue with buid-dep is that it lists _ubuntu package_
>> dependencies. The distro can change how it builds the package without us
>> noticing. There is already some work done by lcitool, I imagine that it
>> would then be duplicated. Or worse(!) be different because build-dep
>> might be listing way more packages that what we need for the default
>> build.
>> 
>> This is more of a push towards using existing infra because those pieces
>> are already looked after for other purposes.
>> 
>>> - A script build_release_container.sh which automatically detects which ubuntu
>>>   version a given release should be buildable on (based on date). It works
>>>   great, and the heuristic works for all our versions.
>>> - a script generate_release_links.sh which creates wrapper scripts automatically
>>>   using prebuilt images (hosted on my dockerhub).
>>>
>
> Sure, lcitool could work to express all that. Would I like to spend time
> to rewrite all the missing bits in yaml when debian did it for me? not
> really.
>

I may have misled you. I'm not suggesting to use lcitool for the missing
bits, I'm suggesting we consume what's already in
tests/docker/dockerfiles. That point you already got.

>> 
>> It would be nice if these two could take any hash, instead of just a
>> tag. This is one of the things that will depend on what you intend to
>> achieve with this series.
>>
>
> Very easy, it just needs a non shallow clone.
>
> ---
>
> After reading and answering to this message, I feel there are two topics
> we can discuss:
> 1. the current implementation
> 2. what would we like
>
> 1. Personally, I just posted this series as an example of "yes it's
> possible with very limited amount of code", and I don't mind having
> something completely different. I feel it's probably not worth reviewing
> every line at this point.
>
> 2. The biggest question seems to be the direction with:
> 'do we want to provide a single dockerfile that covers all versions' (a)
> VS 'having a dockerfile at a time of history for current version' (b).
>

It's probably (b), because the "all versions" part spontaneously spawned
to life. If you take a recent-ish QEMU version there will be few or no
tweaks at all and then it all becomes a question of "why base this on
ubuntu when we already have dockerfiles written for several
distros?".

UPDATE: You just replied that you think build-dep is consistently close
to the real deps enough that it would be less work to maintain. You
could very well be right and we're approaching the hair-splitting zone
around this topic. I suggest you carry on using your best judgement and
let's leave it to time.

> I think (b) is what we have now (current ubuntu/suse/debian/alpine image
> works for current revision, since it's used in CI), and it was already
> proven it fails from Fabiano's feedback.

But the tweaks may be necessary either way. This series adds an
indirection to choose an ubuntu version just to be able to issue (some)
commands unconditionally in the dockerfile. It could just as well check
the qemu version directly.

(yes, there's potentially dealing with all the other distros and lcitool
and all _that_, I get it)

> It seems to be that's what you propose also from your answer above.
>

Fabiano and I are the same person. =)

> This series goes in another direction (a), adding a unique dockerfile
> covering all versions, independently of all the dockerfiles we have now.
>
> If we agree (a) is a better direction, unique dockerfile approach can be
> vastly simplified by restricting ourselves to qemu >= 3.1.0. If not, no
> hard feelings, it's just a proposal.
>
> Regards,
> Pierrick

Re: [PATCH v3 0/3] releases: build and run any QEMU release easily!

[Pierrick Bouvier]

On 10/26/2026 8:32 AM, Fabiano Rosas wrote:
> Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com> writes:
> 
>> On 10/23/2026 8:33 AM, Fabiano Rosas wrote:
>>> Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com> writes:
>>>
>>>> This series was inspired by a recent post from Fabiano to provide container
>>>> images for migration tests:
>>>> https://lore.kernel.org/qemu-devel/24d04a1f-344d-4b90-a307-b582dd9a1128@oss.qualcomm.com/T/#m353f119dc1a24fa437dde8a96d1891629e64303c
>>>>
>>>> The idea to be able to build and run any QEMU (past) release with containers was
>>>> in my mind for some time, so I gave it a shot. Finally, it was easier than
>>>> expected to obtain a single dockerfile (with a few hacks) that works over all
>>>> our releases.
>>>>
>>>
>>> Good stuff, thanks for this work. I'm just not sure how much of it fits
>>> upstream in the current shape, let's discuss...
>>>
>>> You say this was in your mind for a while, could you elaborate a bit on
>>> how you think this could benefit the project?
>>>
>>
>> First usage that comes to my mind is that it would greatly help for
>> migration, allowing automatic testing in CI, by just downloading any
>> previous QEMU, dump a machine, and restore it on current version.
> 
> Don't we also build QEMU several times over in CI? Maybe I'm wrong, but
> it might allow us to decouple a few tests from the build steps.
>

This is for current version, not past ones. I'm not sure it's worth
adding an identical build for past versions on every pipeline.

>> We could do it for last 10 releases to ensure migration from packaged
>> versions in all supported distros (red hat, alpine, ubuntu, ...) work
>> as expected.
>>
> 
> It's a nice idea but potentially a costly one. I don't think there is a
> guarantee that the downstream package is migration-compatible with the
> upstream package.
>

What would be costly? It seems it would just be a download + savevm
through monitor, and a loadvm from current version tested.

>> It could be a enhancement compared to the manual process done once per
>> release. I read many times in review "it breaks the migration", that
>> yes, it would be nice to have CI saying "this series broke the migration
>> from qemu x.y".
>>
> 
> Most of the "breaks migration" issues are due to device migration. What
> we have today in terms of migration testing cannot detect most of those.
>

Ok.

>> Beyond that, I think we could also offer a "rebase on steroids" tool. We
>> could extend list of containers published to all top merge commits (git
>> log --first-parent), and provide a fast rebase, based on download only.
> 
> Bisect requires a test of some sort. Will this cause the opposite
> problem than what migration-test encounters? I.e. instead of "old QEMU
> needs old OS" you'd have "new test needs QEMU from new OS". One
> alternative would be to build the tests into an image as well. Or
> perhaps it wouldn't be a problem in practice.
>

I don't think adding tests in image has a specific value. What we want
is to keep the same test source, but just change the binary used. In
this case, the container just acts as a binary.

>> See it as a giant immutable precompiled cache. Dockerhub storage is free
>> and unlimited.
>> Some quick math: Since 8.0, there are ~2000 top merge commits. Each
>> image is < 250MB compressed, so the whole thing would be < 500GB.
>>
>> Finally, in my personal case, while working on single binary, I had to
>> compare output of qemu help between versions. I'm also interested to
>> track QEMU binary size evolution over time. And some personal fun with
>> software archeology on the project too for anyone.
>>
>>> To be clear, my use-case is simply to produce a few containers for
>>> testing migration on my workstation. Your series can help a lot with
>>> that, but from my point of view, some parts are clearly extra and some
>>> are missing.
>>>
>>> Also beware that what you're touching here is both "infrastructure" and
>>> "publishing". This has implications for how people consume the work of
>>> the project. There's a chance that people see this as "official QEMU
>>> containers" and try to use them in all sorts of weird scenarios that we
>>> as a community are not prepared to maintain. You might get some
>>> push-back on that basis.
>>>
>>
>> Sure, it was free to generate all images and push them on my personal
>> dockerhub, and it's now automated to push any feature release also in a
>> GitHub nightly job. If we want to make it more official, it should
>> really be hosted on an official account.
>>
>>>> The oldest functional distro we can find nowadays is ubuntu 12.04. Because of
>>>> this, the oldest QEMU release we can build is v0.12.0. Previous ones require an
>>>> environment we can't get easily today.
>>>>
>>>
>>> I think the ability to go all the way back is very cool, but unless we
>>> want to look at this from a archival or heritage perspective, I'd say
>>> there's a line somewhere to how old we should go. I don't see an
>>> use-case for very old binaries. Even knowing if they still work would
>>> already incur in a maintenance cost.
>>>
>>
>> Agree, mostly was a fun challenge. I was also curious to see until when
>> we can go back in past from 2026.
>>
>> For the "maintenance" a scheduled CI job running nightly is a free thing
>> for me, but I understand opinions on "maintenance cost" may differ. I
>> also co-maintain tests/docker with Alex, so I'll be the one taking the
>> load, and I'm fine with it.
> 
> Fair enough.
> 
>>   >> Images have the absolute minimal content possible. It's only qemu
>> binaries,
>>>> artifacts (roms, keymap, ...), and libraries. There is no shell or any other
>>>> tool included. As a result, images have a size from 149MB (v0.12.0) to 1.48GB
>>>> (v11.1.2).
>>>>
>>>
>>> No shell means the first thing to do if one wants to debug the image is
>>> to rebuild with a shell in. Not sure how frequent it will be, maybe it's
>>> ok like this.
>>>
>>
>> We could leave a base system, for only ~100MB per image, which is not
>> the end of the world (+10% in size). Personally, I appreciate to have no
>> shell, it reduces possibility of exploit when running an older QEMU
>> having CVEs.
>>
>>>> This series adds 3 pieces:
>>>> - a unique dockerfile to build any QEMU release. User has to provide an explicit
>>>>   Ubuntu version when building a given QEMU version.
>>>
>>> This is the tricky part. We'll be carrying code (script+dockerfile) that
>>> only works on master. If we tag today and tomorrow the old distro repo
>>> breaks, we obviously cannot fix it retroactively. So we need to be
>>> careful of where we direct the maintenance effort.
>>>
>>> 1) I'd suggest first of all to keep the release image entirely separate
>>> from the ubuntu image. Treat release.docker as we treat the other
>>> dockerfiles and update it to work for the current release without
>>> looking at past ones. Have it clone, checkout, build and gather
>>> deps. Allow the configure to be tweaked via parameter in case we need to
>>> disable something in an older build.
>>>
>>> The script would then build a distro image (or dockerfile) with the
>>> knowledge that it has about ubuntu depending on the QEMU tag vs. ubuntu
>>> version combination. This would allow us to track the cludges more
>>> precisely instead of "some ancient distro" as the dockerfile currently
>>> says.
>>>
>>> This is more of a layering concern and how easy it will be to drop/add
>>> version-specific fixes.
>>>
>>
>> IIUC I think that's the approach Fabiano took, to use Dockerfile from
>> previous versions, and the conclusion was that it requires some seds and
>> hacks to have something that work. Internet state is not immutable.
>> IMHO, building previous QEMU with past Dockerfiles is doomed to fail.
>>
> 
> That's not what I'm suggesting here. I'm saying that "current
> dockerfile" should have less tweaks and "current
> build_release_container.sh" should be the one responsible for doing the
> ubuntu-version-to-tweak mapping.
>

Got you.

> But maybe you meant to reply to the second point below. In that case,
> yes the past dockerfiles require tweaks, but they also provide most of
> the time the exact list of dependencies for the previous QEMU, and that
> work was already done by the people from the past.
> 
>> Only thing you can control/guarantee is that "current dockerfile on
>> master works".
>>
>>> 2) I wonder if the usage of apt-get build-dep might be a convenience
>>> trap. In theory, some of the adaptations that are required (such as
>>> tweaking dependencies) have already been done by us when working on that
>>> development window.  If we take the dockerfiles from the git tag, those
>>> changes don't need to be done again and for the issues that didn't exist
>>> at the time, we can map them to the git tag instead of ubuntu version;
>>> (we tend to know more about QEMU releases than ubuntu releases).
>>>
>>> Another issue with buid-dep is that it lists _ubuntu package_
>>> dependencies. The distro can change how it builds the package without us
>>> noticing. There is already some work done by lcitool, I imagine that it
>>> would then be duplicated. Or worse(!) be different because build-dep
>>> might be listing way more packages that what we need for the default
>>> build.
>>>
>>> This is more of a push towards using existing infra because those pieces
>>> are already looked after for other purposes.
>>>
>>>> - A script build_release_container.sh which automatically detects which ubuntu
>>>>   version a given release should be buildable on (based on date). It works
>>>>   great, and the heuristic works for all our versions.
>>>> - a script generate_release_links.sh which creates wrapper scripts automatically
>>>>   using prebuilt images (hosted on my dockerhub).
>>>>
>>
>> Sure, lcitool could work to express all that. Would I like to spend time
>> to rewrite all the missing bits in yaml when debian did it for me? not
>> really.
>>
> 
> I may have misled you. I'm not suggesting to use lcitool for the missing
> bits, I'm suggesting we consume what's already in
> tests/docker/dockerfiles. That point you already got.
>

Oh I see ok. Like extract the list of packages, and inject that.
Well, I'm not sure it will be simpler than just doing an apt build-dep
to be honest, and not sure what we gain from taking this information
from our sources.

>>>
>>> It would be nice if these two could take any hash, instead of just a
>>> tag. This is one of the things that will depend on what you intend to
>>> achieve with this series.
>>>
>>
>> Very easy, it just needs a non shallow clone.
>>
>> ---
>>
>> After reading and answering to this message, I feel there are two topics
>> we can discuss:
>> 1. the current implementation
>> 2. what would we like
>>
>> 1. Personally, I just posted this series as an example of "yes it's
>> possible with very limited amount of code", and I don't mind having
>> something completely different. I feel it's probably not worth reviewing
>> every line at this point.
>>
>> 2. The biggest question seems to be the direction with:
>> 'do we want to provide a single dockerfile that covers all versions' (a)
>> VS 'having a dockerfile at a time of history for current version' (b).
>>
> 
> It's probably (b), because the "all versions" part spontaneously spawned
> to life. If you take a recent-ish QEMU version there will be few or no
> tweaks at all and then it all becomes a question of "why base this on
> ubuntu when we already have dockerfiles written for several
> distros?".
> 
> UPDATE: You just replied that you think build-dep is consistently close
> to the real deps enough that it would be less work to maintain. You
> could very well be right and we're approaching the hair-splitting zone
> around this topic. I suggest you carry on using your best judgement and
> let's leave it to time.
>

We can also decide to settle on fedora, alpine, debian, suse, or any
distro. In terms of images, ubuntu was just the most convenient (tried
with debian first, which required updating apt sources for older versions).

>> I think (b) is what we have now (current ubuntu/suse/debian/alpine image
>> works for current revision, since it's used in CI), and it was already
>> proven it fails from Fabiano's feedback.
> 
> But the tweaks may be necessary either way. This series adds an
> indirection to choose an ubuntu version just to be able to issue (some)
> commands unconditionally in the dockerfile. It could just as well check
> the qemu version directly.
>

That's maybe where the philosophy differs. Key idea of this series is to
tweak the environment so it works all time, and not do specific things
per version.

> (yes, there's potentially dealing with all the other distros and lcitool
> and all _that_, I get it)
> 
>> It seems to be that's what you propose also from your answer above.
>>
> 
> Fabiano and I are the same person. =)
>

Sorry, got confused as Peter answered on v2, and I assumed he was the
one who answered on v3 also.

>> This series goes in another direction (a), adding a unique dockerfile
>> covering all versions, independently of all the dockerfiles we have now.
>>
>> If we agree (a) is a better direction, unique dockerfile approach can be
>> vastly simplified by restricting ourselves to qemu >= 3.1.0. If not, no
>> hard feelings, it's just a proposal.

Regards,
Pierrick

Re: [PATCH v3 0/3] releases: build and run any QEMU release easily!

[Pierrick Bouvier]

Sent v4, simplifying release dockerfile, and supporting to build any
version with any git repository:
https://lore.kernel.org/qemu-devel/20261026210801.179790-1-pierrick.bouvier@oss.qualcomm.com/T/#t

Regards,
Pierrick

Re: [PATCH v3 0/3] releases: build and run any QEMU release easily!

[Fabiano Rosas]

Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com> writes:

> On 10/26/2026 8:32 AM, Fabiano Rosas wrote:
>> Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com> writes:
>> 
>>> On 10/23/2026 8:33 AM, Fabiano Rosas wrote:
>>>> Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com> writes:
>>>>
>>>>> This series was inspired by a recent post from Fabiano to provide container
>>>>> images for migration tests:
>>>>> https://lore.kernel.org/qemu-devel/24d04a1f-344d-4b90-a307-b582dd9a1128@oss.qualcomm.com/T/#m353f119dc1a24fa437dde8a96d1891629e64303c
>>>>>
>>>>> The idea to be able to build and run any QEMU (past) release with containers was
>>>>> in my mind for some time, so I gave it a shot. Finally, it was easier than
>>>>> expected to obtain a single dockerfile (with a few hacks) that works over all
>>>>> our releases.
>>>>>
>>>>
>>>> Good stuff, thanks for this work. I'm just not sure how much of it fits
>>>> upstream in the current shape, let's discuss...
>>>>
>>>> You say this was in your mind for a while, could you elaborate a bit on
>>>> how you think this could benefit the project?
>>>>
>>>
>>> First usage that comes to my mind is that it would greatly help for
>>> migration, allowing automatic testing in CI, by just downloading any
>>> previous QEMU, dump a machine, and restore it on current version.
>> 
>> Don't we also build QEMU several times over in CI? Maybe I'm wrong, but
>> it might allow us to decouple a few tests from the build steps.
>>
>
> This is for current version, not past ones. I'm not sure it's worth
> adding an identical build for past versions on every pipeline.
>

I thought we re-built current qemu a few times, no? We have containers
for various distros and then make; make check inside of them. The
question was whether a step 0 to build the binary container for this
release would allow some builds to be skipped.

>>> We could do it for last 10 releases to ensure migration from packaged
>>> versions in all supported distros (red hat, alpine, ubuntu, ...) work
>>> as expected.
>>>
>> 
>> It's a nice idea but potentially a costly one. I don't think there is a
>> guarantee that the downstream package is migration-compatible with the
>> upstream package.
>>
>
> What would be costly? It seems it would just be a download + savevm
> through monitor, and a loadvm from current version tested.
>

Setting it up would be simple. Working with it, not so much. It's a test
that once it fails, then it becomes a tedious job of figuring out
whether the issue is a failure on our side or if it's merely an
incompatibility between upstream vs. downstream. The packaged QEMU can
have all sorts of tweaks, extra patches, fewer patches, different code
entirely, different machine type ABI, etc (their own QMP commands
even). The distro can guarantee it all works inside their environment.

When a problem happens on the intersection between the two binaries, if
it's on our side, we'd need to determine whether to fix it or tell the
distro "don't do that". If it's on their side then we open a bug report
and hope that this is even a priority for them.

So overall I'm in favor, but I don't expect it to be game changing.

>>> It could be a enhancement compared to the manual process done once per
>>> release. I read many times in review "it breaks the migration", that
>>> yes, it would be nice to have CI saying "this series broke the migration
>>> from qemu x.y".
>>>
>> 
>> Most of the "breaks migration" issues are due to device migration. What
>> we have today in terms of migration testing cannot detect most of those.
>>
>
> Ok.
>
>>> Beyond that, I think we could also offer a "rebase on steroids" tool. We
>>> could extend list of containers published to all top merge commits (git
>>> log --first-parent), and provide a fast rebase, based on download only.
>> 
>> Bisect requires a test of some sort. Will this cause the opposite
>> problem than what migration-test encounters? I.e. instead of "old QEMU
>> needs old OS" you'd have "new test needs QEMU from new OS". One
>> alternative would be to build the tests into an image as well. Or
>> perhaps it wouldn't be a problem in practice.
>>
>
> I don't think adding tests in image has a specific value. What we want
> is to keep the same test source, but just change the binary used. In
> this case, the container just acts as a binary.
>

There are requirements such as QEMU command line, external binaries,
maybe even the code itself, in the case of unit tests. Exchanging the
QEMU binary while keeping the same test (whether binary or not) may not
work.

- At some point we added the -audio command line option and updated
libqtest to include the option by default. Now older binaries can't
start when called by the new test code.

- If a test is sensitive to qemu output, deprecation warnings and error
message format changes might break it.

- qemu-img and qemu-io might have implicit dependencies on a range of
versions.

If we're being rigorous, two commits with completely different test and
QEMU code could still test the same functionality. That's how we do
normal bisect anyway. The requirement of test compatibility across QEMU
versions is something we probably never thought about before. I have
just recently started thinking about what can be done to libqtest to
allow more flexibility in this case.

>>> See it as a giant immutable precompiled cache. Dockerhub storage is free
>>> and unlimited.
>>> Some quick math: Since 8.0, there are ~2000 top merge commits. Each
>>> image is < 250MB compressed, so the whole thing would be < 500GB.
>>>
>>> Finally, in my personal case, while working on single binary, I had to
>>> compare output of qemu help between versions. I'm also interested to
>>> track QEMU binary size evolution over time. And some personal fun with
>>> software archeology on the project too for anyone.
>>>
>>>> To be clear, my use-case is simply to produce a few containers for
>>>> testing migration on my workstation. Your series can help a lot with
>>>> that, but from my point of view, some parts are clearly extra and some
>>>> are missing.
>>>>
>>>> Also beware that what you're touching here is both "infrastructure" and
>>>> "publishing". This has implications for how people consume the work of
>>>> the project. There's a chance that people see this as "official QEMU
>>>> containers" and try to use them in all sorts of weird scenarios that we
>>>> as a community are not prepared to maintain. You might get some
>>>> push-back on that basis.
>>>>
>>>
>>> Sure, it was free to generate all images and push them on my personal
>>> dockerhub, and it's now automated to push any feature release also in a
>>> GitHub nightly job. If we want to make it more official, it should
>>> really be hosted on an official account.
>>>
>>>>> The oldest functional distro we can find nowadays is ubuntu 12.04. Because of
>>>>> this, the oldest QEMU release we can build is v0.12.0. Previous ones require an
>>>>> environment we can't get easily today.
>>>>>
>>>>
>>>> I think the ability to go all the way back is very cool, but unless we
>>>> want to look at this from a archival or heritage perspective, I'd say
>>>> there's a line somewhere to how old we should go. I don't see an
>>>> use-case for very old binaries. Even knowing if they still work would
>>>> already incur in a maintenance cost.
>>>>
>>>
>>> Agree, mostly was a fun challenge. I was also curious to see until when
>>> we can go back in past from 2026.
>>>
>>> For the "maintenance" a scheduled CI job running nightly is a free thing
>>> for me, but I understand opinions on "maintenance cost" may differ. I
>>> also co-maintain tests/docker with Alex, so I'll be the one taking the
>>> load, and I'm fine with it.
>> 
>> Fair enough.
>> 
>>>   >> Images have the absolute minimal content possible. It's only qemu
>>> binaries,
>>>>> artifacts (roms, keymap, ...), and libraries. There is no shell or any other
>>>>> tool included. As a result, images have a size from 149MB (v0.12.0) to 1.48GB
>>>>> (v11.1.2).
>>>>>
>>>>
>>>> No shell means the first thing to do if one wants to debug the image is
>>>> to rebuild with a shell in. Not sure how frequent it will be, maybe it's
>>>> ok like this.
>>>>
>>>
>>> We could leave a base system, for only ~100MB per image, which is not
>>> the end of the world (+10% in size). Personally, I appreciate to have no
>>> shell, it reduces possibility of exploit when running an older QEMU
>>> having CVEs.
>>>
>>>>> This series adds 3 pieces:
>>>>> - a unique dockerfile to build any QEMU release. User has to provide an explicit
>>>>>   Ubuntu version when building a given QEMU version.
>>>>
>>>> This is the tricky part. We'll be carrying code (script+dockerfile) that
>>>> only works on master. If we tag today and tomorrow the old distro repo
>>>> breaks, we obviously cannot fix it retroactively. So we need to be
>>>> careful of where we direct the maintenance effort.
>>>>
>>>> 1) I'd suggest first of all to keep the release image entirely separate
>>>> from the ubuntu image. Treat release.docker as we treat the other
>>>> dockerfiles and update it to work for the current release without
>>>> looking at past ones. Have it clone, checkout, build and gather
>>>> deps. Allow the configure to be tweaked via parameter in case we need to
>>>> disable something in an older build.
>>>>
>>>> The script would then build a distro image (or dockerfile) with the
>>>> knowledge that it has about ubuntu depending on the QEMU tag vs. ubuntu
>>>> version combination. This would allow us to track the cludges more
>>>> precisely instead of "some ancient distro" as the dockerfile currently
>>>> says.
>>>>
>>>> This is more of a layering concern and how easy it will be to drop/add
>>>> version-specific fixes.
>>>>
>>>
>>> IIUC I think that's the approach Fabiano took, to use Dockerfile from
>>> previous versions, and the conclusion was that it requires some seds and
>>> hacks to have something that work. Internet state is not immutable.
>>> IMHO, building previous QEMU with past Dockerfiles is doomed to fail.
>>>
>> 
>> That's not what I'm suggesting here. I'm saying that "current
>> dockerfile" should have less tweaks and "current
>> build_release_container.sh" should be the one responsible for doing the
>> ubuntu-version-to-tweak mapping.
>>
>
> Got you.
>
>> But maybe you meant to reply to the second point below. In that case,
>> yes the past dockerfiles require tweaks, but they also provide most of
>> the time the exact list of dependencies for the previous QEMU, and that
>> work was already done by the people from the past.
>> 
>>> Only thing you can control/guarantee is that "current dockerfile on
>>> master works".
>>>
>>>> 2) I wonder if the usage of apt-get build-dep might be a convenience
>>>> trap. In theory, some of the adaptations that are required (such as
>>>> tweaking dependencies) have already been done by us when working on that
>>>> development window.  If we take the dockerfiles from the git tag, those
>>>> changes don't need to be done again and for the issues that didn't exist
>>>> at the time, we can map them to the git tag instead of ubuntu version;
>>>> (we tend to know more about QEMU releases than ubuntu releases).
>>>>
>>>> Another issue with buid-dep is that it lists _ubuntu package_
>>>> dependencies. The distro can change how it builds the package without us
>>>> noticing. There is already some work done by lcitool, I imagine that it
>>>> would then be duplicated. Or worse(!) be different because build-dep
>>>> might be listing way more packages that what we need for the default
>>>> build.
>>>>
>>>> This is more of a push towards using existing infra because those pieces
>>>> are already looked after for other purposes.
>>>>
>>>>> - A script build_release_container.sh which automatically detects which ubuntu
>>>>>   version a given release should be buildable on (based on date). It works
>>>>>   great, and the heuristic works for all our versions.
>>>>> - a script generate_release_links.sh which creates wrapper scripts automatically
>>>>>   using prebuilt images (hosted on my dockerhub).
>>>>>
>>>
>>> Sure, lcitool could work to express all that. Would I like to spend time
>>> to rewrite all the missing bits in yaml when debian did it for me? not
>>> really.
>>>
>> 
>> I may have misled you. I'm not suggesting to use lcitool for the missing
>> bits, I'm suggesting we consume what's already in
>> tests/docker/dockerfiles. That point you already got.
>>
>
> Oh I see ok. Like extract the list of packages, and inject that.
> Well, I'm not sure it will be simpler than just doing an apt build-dep
> to be honest, and not sure what we gain from taking this information
> from our sources.
>

Just so you're aware of my reasoning:

It's information we control, add/remove packages, pin package versions,
write a big commit message explaining why, etc. While apt build-dep is
simple and it works great, it's still a little piece in the hands of a
third party that has their own policies and goals. No one on their side
(whatever distro it is) is thinking "what happens with QEMU CI if I
change this". It makes the project more robust if we limit that kind of
exposure.

>>>>
>>>> It would be nice if these two could take any hash, instead of just a
>>>> tag. This is one of the things that will depend on what you intend to
>>>> achieve with this series.
>>>>
>>>
>>> Very easy, it just needs a non shallow clone.
>>>
>>> ---
>>>
>>> After reading and answering to this message, I feel there are two topics
>>> we can discuss:
>>> 1. the current implementation
>>> 2. what would we like
>>>
>>> 1. Personally, I just posted this series as an example of "yes it's
>>> possible with very limited amount of code", and I don't mind having
>>> something completely different. I feel it's probably not worth reviewing
>>> every line at this point.
>>>
>>> 2. The biggest question seems to be the direction with:
>>> 'do we want to provide a single dockerfile that covers all versions' (a)
>>> VS 'having a dockerfile at a time of history for current version' (b).
>>>
>> 
>> It's probably (b), because the "all versions" part spontaneously spawned
>> to life. If you take a recent-ish QEMU version there will be few or no
>> tweaks at all and then it all becomes a question of "why base this on
>> ubuntu when we already have dockerfiles written for several
>> distros?".
>> 
>> UPDATE: You just replied that you think build-dep is consistently close
>> to the real deps enough that it would be less work to maintain. You
>> could very well be right and we're approaching the hair-splitting zone
>> around this topic. I suggest you carry on using your best judgement and
>> let's leave it to time.
>>
>
> We can also decide to settle on fedora, alpine, debian, suse, or any
> distro. In terms of images, ubuntu was just the most convenient (tried
> with debian first, which required updating apt sources for older versions).
>
>>> I think (b) is what we have now (current ubuntu/suse/debian/alpine image
>>> works for current revision, since it's used in CI), and it was already
>>> proven it fails from Fabiano's feedback.
>> 
>> But the tweaks may be necessary either way. This series adds an
>> indirection to choose an ubuntu version just to be able to issue (some)
>> commands unconditionally in the dockerfile. It could just as well check
>> the qemu version directly.
>>
>
> That's maybe where the philosophy differs. Key idea of this series is to
> tweak the environment so it works all time, and not do specific things
> per version.
>

Sure, but the "all the time" part doesn't convince much. Anyway, that's
only one point and we've discussed it enough. If it fails miserably
you'll learn something, otherwise, I will. =)

>> (yes, there's potentially dealing with all the other distros and lcitool
>> and all _that_, I get it)
>> 
>>> It seems to be that's what you propose also from your answer above.
>>>
>> 
>> Fabiano and I are the same person. =)
>>
>
> Sorry, got confused as Peter answered on v2, and I assumed he was the
> one who answered on v3 also.
>

I'm flattered, thanks.

>>> This series goes in another direction (a), adding a unique dockerfile
>>> covering all versions, independently of all the dockerfiles we have now.
>>>
>>> If we agree (a) is a better direction, unique dockerfile approach can be
>>> vastly simplified by restricting ourselves to qemu >= 3.1.0. If not, no
>>> hard feelings, it's just a proposal.
>
> Regards,
> Pierrick

Re: [PATCH v3 0/3] releases: build and run any QEMU release easily!

[Pierrick Bouvier]

On 10/27/2026 7:36 AM, Fabiano Rosas wrote:
> Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com> writes:
> 
>> On 10/26/2026 8:32 AM, Fabiano Rosas wrote:
>>> Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com> writes:
>>>
>>>> On 10/23/2026 8:33 AM, Fabiano Rosas wrote:
>>>>> Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com> writes:
>>>>>
>>>>>> This series was inspired by a recent post from Fabiano to provide container
>>>>>> images for migration tests:
>>>>>> https://lore.kernel.org/qemu-devel/24d04a1f-344d-4b90-a307-b582dd9a1128@oss.qualcomm.com/T/#m353f119dc1a24fa437dde8a96d1891629e64303c
>>>>>>
>>>>>> The idea to be able to build and run any QEMU (past) release with containers was
>>>>>> in my mind for some time, so I gave it a shot. Finally, it was easier than
>>>>>> expected to obtain a single dockerfile (with a few hacks) that works over all
>>>>>> our releases.
>>>>>>
>>>>>
>>>>> Good stuff, thanks for this work. I'm just not sure how much of it fits
>>>>> upstream in the current shape, let's discuss...
>>>>>
>>>>> You say this was in your mind for a while, could you elaborate a bit on
>>>>> how you think this could benefit the project?
>>>>>
>>>>
>>>> First usage that comes to my mind is that it would greatly help for
>>>> migration, allowing automatic testing in CI, by just downloading any
>>>> previous QEMU, dump a machine, and restore it on current version.
>>>
>>> Don't we also build QEMU several times over in CI? Maybe I'm wrong, but
>>> it might allow us to decouple a few tests from the build steps.
>>>
>>
>> This is for current version, not past ones. I'm not sure it's worth
>> adding an identical build for past versions on every pipeline.
>>
> 
> I thought we re-built current qemu a few times, no? We have containers
> for various distros and then make; make check inside of them. The
> question was whether a step 0 to build the binary container for this
> release would allow some builds to be skipped.
>

I see. To some extent, yes. However, if you take a look more closely,
you'll notice most of the builds differ by:
- set of targets built
- configure options
- distro target

While we could maybe optimize some parts, that still requires duplication.

>>>> We could do it for last 10 releases to ensure migration from packaged
>>>> versions in all supported distros (red hat, alpine, ubuntu, ...) work
>>>> as expected.
>>>>
>>>
>>> It's a nice idea but potentially a costly one. I don't think there is a
>>> guarantee that the downstream package is migration-compatible with the
>>> upstream package.
>>>
>>
>> What would be costly? It seems it would just be a download + savevm
>> through monitor, and a loadvm from current version tested.
>>
> 
> Setting it up would be simple. Working with it, not so much. It's a test
> that once it fails, then it becomes a tedious job of figuring out
> whether the issue is a failure on our side or if it's merely an
> incompatibility between upstream vs. downstream. The packaged QEMU can
> have all sorts of tweaks, extra patches, fewer patches, different code
> entirely, different machine type ABI, etc (their own QMP commands
> even). The distro can guarantee it all works inside their environment.
> 
> When a problem happens on the intersection between the two binaries, if
> it's on our side, we'd need to determine whether to fix it or tell the
> distro "don't do that". If it's on their side then we open a bug report
> and hope that this is even a priority for them.
> 
> So overall I'm in favor, but I don't expect it to be game changing.
>

Do we have a concrete example in recent migration history of distro
doing a custom downstream patch that broke migration?

This would be series author's job/time, not maintainer's.

Also, I think it's precisely why it's interesting to investigate those.
What is the point to maintain any kind of compatibility, if it's not for
our main users?
Preserving compatibility for versions built manually from source is a
niche use case.

>>>> It could be a enhancement compared to the manual process done once per
>>>> release. I read many times in review "it breaks the migration", that
>>>> yes, it would be nice to have CI saying "this series broke the migration
>>>> from qemu x.y".
>>>>
>>>
>>> Most of the "breaks migration" issues are due to device migration. What
>>> we have today in terms of migration testing cannot detect most of those.
>>>
>>
>> Ok.
>>
>>>> Beyond that, I think we could also offer a "rebase on steroids" tool. We
>>>> could extend list of containers published to all top merge commits (git
>>>> log --first-parent), and provide a fast rebase, based on download only.
>>>
>>> Bisect requires a test of some sort. Will this cause the opposite
>>> problem than what migration-test encounters? I.e. instead of "old QEMU
>>> needs old OS" you'd have "new test needs QEMU from new OS". One
>>> alternative would be to build the tests into an image as well. Or
>>> perhaps it wouldn't be a problem in practice.
>>>
>>
>> I don't think adding tests in image has a specific value. What we want
>> is to keep the same test source, but just change the binary used. In
>> this case, the container just acts as a binary.
>>
> 
> There are requirements such as QEMU command line, external binaries,
> maybe even the code itself, in the case of unit tests. Exchanging the
> QEMU binary while keeping the same test (whether binary or not) may not
> work.
> 
> - At some point we added the -audio command line option and updated
> libqtest to include the option by default. Now older binaries can't
> start when called by the new test code.
> 
> - If a test is sensitive to qemu output, deprecation warnings and error
> message format changes might break it.
> 
> - qemu-img and qemu-io might have implicit dependencies on a range of
> versions.
> 
> If we're being rigorous, two commits with completely different test and
> QEMU code could still test the same functionality. That's how we do
> normal bisect anyway. The requirement of test compatibility across QEMU
> versions is something we probably never thought about before. I have
> just recently started thinking about what can be done to libqtest to
> allow more flexibility in this case.
>

I see. For me it's a bit surprising to change both the source code and
the test code, but if we are used to it, I'm happy to follow the culture.

In all cases, when running a git bisect, the previous version of the
test will be checkout too. My proposal is just to skip the build part by
relying on prebuilt images, but if we follow classic git bisect, test
will be updated accordingly during bisection.

>>>> See it as a giant immutable precompiled cache. Dockerhub storage is free
>>>> and unlimited.
>>>> Some quick math: Since 8.0, there are ~2000 top merge commits. Each
>>>> image is < 250MB compressed, so the whole thing would be < 500GB.
>>>>
>>>> Finally, in my personal case, while working on single binary, I had to
>>>> compare output of qemu help between versions. I'm also interested to
>>>> track QEMU binary size evolution over time. And some personal fun with
>>>> software archeology on the project too for anyone.
>>>>
>>>>> To be clear, my use-case is simply to produce a few containers for
>>>>> testing migration on my workstation. Your series can help a lot with
>>>>> that, but from my point of view, some parts are clearly extra and some
>>>>> are missing.
>>>>>
>>>>> Also beware that what you're touching here is both "infrastructure" and
>>>>> "publishing". This has implications for how people consume the work of
>>>>> the project. There's a chance that people see this as "official QEMU
>>>>> containers" and try to use them in all sorts of weird scenarios that we
>>>>> as a community are not prepared to maintain. You might get some
>>>>> push-back on that basis.
>>>>>
>>>>
>>>> Sure, it was free to generate all images and push them on my personal
>>>> dockerhub, and it's now automated to push any feature release also in a
>>>> GitHub nightly job. If we want to make it more official, it should
>>>> really be hosted on an official account.
>>>>
>>>>>> The oldest functional distro we can find nowadays is ubuntu 12.04. Because of
>>>>>> this, the oldest QEMU release we can build is v0.12.0. Previous ones require an
>>>>>> environment we can't get easily today.
>>>>>>
>>>>>
>>>>> I think the ability to go all the way back is very cool, but unless we
>>>>> want to look at this from a archival or heritage perspective, I'd say
>>>>> there's a line somewhere to how old we should go. I don't see an
>>>>> use-case for very old binaries. Even knowing if they still work would
>>>>> already incur in a maintenance cost.
>>>>>
>>>>
>>>> Agree, mostly was a fun challenge. I was also curious to see until when
>>>> we can go back in past from 2026.
>>>>
>>>> For the "maintenance" a scheduled CI job running nightly is a free thing
>>>> for me, but I understand opinions on "maintenance cost" may differ. I
>>>> also co-maintain tests/docker with Alex, so I'll be the one taking the
>>>> load, and I'm fine with it.
>>>
>>> Fair enough.
>>>
>>>>   >> Images have the absolute minimal content possible. It's only qemu
>>>> binaries,
>>>>>> artifacts (roms, keymap, ...), and libraries. There is no shell or any other
>>>>>> tool included. As a result, images have a size from 149MB (v0.12.0) to 1.48GB
>>>>>> (v11.1.2).
>>>>>>
>>>>>
>>>>> No shell means the first thing to do if one wants to debug the image is
>>>>> to rebuild with a shell in. Not sure how frequent it will be, maybe it's
>>>>> ok like this.
>>>>>
>>>>
>>>> We could leave a base system, for only ~100MB per image, which is not
>>>> the end of the world (+10% in size). Personally, I appreciate to have no
>>>> shell, it reduces possibility of exploit when running an older QEMU
>>>> having CVEs.
>>>>
>>>>>> This series adds 3 pieces:
>>>>>> - a unique dockerfile to build any QEMU release. User has to provide an explicit
>>>>>>   Ubuntu version when building a given QEMU version.
>>>>>
>>>>> This is the tricky part. We'll be carrying code (script+dockerfile) that
>>>>> only works on master. If we tag today and tomorrow the old distro repo
>>>>> breaks, we obviously cannot fix it retroactively. So we need to be
>>>>> careful of where we direct the maintenance effort.
>>>>>
>>>>> 1) I'd suggest first of all to keep the release image entirely separate
>>>>> from the ubuntu image. Treat release.docker as we treat the other
>>>>> dockerfiles and update it to work for the current release without
>>>>> looking at past ones. Have it clone, checkout, build and gather
>>>>> deps. Allow the configure to be tweaked via parameter in case we need to
>>>>> disable something in an older build.
>>>>>
>>>>> The script would then build a distro image (or dockerfile) with the
>>>>> knowledge that it has about ubuntu depending on the QEMU tag vs. ubuntu
>>>>> version combination. This would allow us to track the cludges more
>>>>> precisely instead of "some ancient distro" as the dockerfile currently
>>>>> says.
>>>>>
>>>>> This is more of a layering concern and how easy it will be to drop/add
>>>>> version-specific fixes.
>>>>>
>>>>
>>>> IIUC I think that's the approach Fabiano took, to use Dockerfile from
>>>> previous versions, and the conclusion was that it requires some seds and
>>>> hacks to have something that work. Internet state is not immutable.
>>>> IMHO, building previous QEMU with past Dockerfiles is doomed to fail.
>>>>
>>>
>>> That's not what I'm suggesting here. I'm saying that "current
>>> dockerfile" should have less tweaks and "current
>>> build_release_container.sh" should be the one responsible for doing the
>>> ubuntu-version-to-tweak mapping.
>>>
>>
>> Got you.
>>
>>> But maybe you meant to reply to the second point below. In that case,
>>> yes the past dockerfiles require tweaks, but they also provide most of
>>> the time the exact list of dependencies for the previous QEMU, and that
>>> work was already done by the people from the past.
>>>
>>>> Only thing you can control/guarantee is that "current dockerfile on
>>>> master works".
>>>>
>>>>> 2) I wonder if the usage of apt-get build-dep might be a convenience
>>>>> trap. In theory, some of the adaptations that are required (such as
>>>>> tweaking dependencies) have already been done by us when working on that
>>>>> development window.  If we take the dockerfiles from the git tag, those
>>>>> changes don't need to be done again and for the issues that didn't exist
>>>>> at the time, we can map them to the git tag instead of ubuntu version;
>>>>> (we tend to know more about QEMU releases than ubuntu releases).
>>>>>
>>>>> Another issue with buid-dep is that it lists _ubuntu package_
>>>>> dependencies. The distro can change how it builds the package without us
>>>>> noticing. There is already some work done by lcitool, I imagine that it
>>>>> would then be duplicated. Or worse(!) be different because build-dep
>>>>> might be listing way more packages that what we need for the default
>>>>> build.
>>>>>
>>>>> This is more of a push towards using existing infra because those pieces
>>>>> are already looked after for other purposes.
>>>>>
>>>>>> - A script build_release_container.sh which automatically detects which ubuntu
>>>>>>   version a given release should be buildable on (based on date). It works
>>>>>>   great, and the heuristic works for all our versions.
>>>>>> - a script generate_release_links.sh which creates wrapper scripts automatically
>>>>>>   using prebuilt images (hosted on my dockerhub).
>>>>>>
>>>>
>>>> Sure, lcitool could work to express all that. Would I like to spend time
>>>> to rewrite all the missing bits in yaml when debian did it for me? not
>>>> really.
>>>>
>>>
>>> I may have misled you. I'm not suggesting to use lcitool for the missing
>>> bits, I'm suggesting we consume what's already in
>>> tests/docker/dockerfiles. That point you already got.
>>>
>>
>> Oh I see ok. Like extract the list of packages, and inject that.
>> Well, I'm not sure it will be simpler than just doing an apt build-dep
>> to be honest, and not sure what we gain from taking this information
>> from our sources.
>>
> 
> Just so you're aware of my reasoning:
> 
> It's information we control, add/remove packages, pin package versions,
> write a big commit message explaining why, etc. While apt build-dep is
> simple and it works great, it's still a little piece in the hands of a
> third party that has their own policies and goals. No one on their side
> (whatever distro it is) is thinking "what happens with QEMU CI if I
> change this". It makes the project more robust if we limit that kind of
> exposure.
>

That's true. On the other hand, they have to get the deps right, or they
can't build latest version. So I tend to think they need to get the
right set of deps by design, not by choice.

Only exception is when new deps are introduced (like ninja, or
flex/bison recently). But considering all versions since v3.1.0,
flex/bison is the only exception.

>>>>>
>>>>> It would be nice if these two could take any hash, instead of just a
>>>>> tag. This is one of the things that will depend on what you intend to
>>>>> achieve with this series.
>>>>>
>>>>
>>>> Very easy, it just needs a non shallow clone.
>>>>
>>>> ---
>>>>
>>>> After reading and answering to this message, I feel there are two topics
>>>> we can discuss:
>>>> 1. the current implementation
>>>> 2. what would we like
>>>>
>>>> 1. Personally, I just posted this series as an example of "yes it's
>>>> possible with very limited amount of code", and I don't mind having
>>>> something completely different. I feel it's probably not worth reviewing
>>>> every line at this point.
>>>>
>>>> 2. The biggest question seems to be the direction with:
>>>> 'do we want to provide a single dockerfile that covers all versions' (a)
>>>> VS 'having a dockerfile at a time of history for current version' (b).
>>>>
>>>
>>> It's probably (b), because the "all versions" part spontaneously spawned
>>> to life. If you take a recent-ish QEMU version there will be few or no
>>> tweaks at all and then it all becomes a question of "why base this on
>>> ubuntu when we already have dockerfiles written for several
>>> distros?".
>>>
>>> UPDATE: You just replied that you think build-dep is consistently close
>>> to the real deps enough that it would be less work to maintain. You
>>> could very well be right and we're approaching the hair-splitting zone
>>> around this topic. I suggest you carry on using your best judgement and
>>> let's leave it to time.
>>>
>>
>> We can also decide to settle on fedora, alpine, debian, suse, or any
>> distro. In terms of images, ubuntu was just the most convenient (tried
>> with debian first, which required updating apt sources for older versions).
>>
>>>> I think (b) is what we have now (current ubuntu/suse/debian/alpine image
>>>> works for current revision, since it's used in CI), and it was already
>>>> proven it fails from Fabiano's feedback.
>>>
>>> But the tweaks may be necessary either way. This series adds an
>>> indirection to choose an ubuntu version just to be able to issue (some)
>>> commands unconditionally in the dockerfile. It could just as well check
>>> the qemu version directly.
>>>
>>
>> That's maybe where the philosophy differs. Key idea of this series is to
>> tweak the environment so it works all time, and not do specific things
>> per version.
>>
> 
> Sure, but the "all the time" part doesn't convince much. Anyway, that's
> only one point and we've discussed it enough. If it fails miserably
> you'll learn something, otherwise, I will. =)
>

Sure, and count on me to send a fix if it breaks, so it will keep on
working. I'll catch this with automatic build for new releases.

>>> (yes, there's potentially dealing with all the other distros and lcitool
>>> and all _that_, I get it)
>>>
>>>> It seems to be that's what you propose also from your answer above.
>>>>
>>>
>>> Fabiano and I are the same person. =)
>>>
>>
>> Sorry, got confused as Peter answered on v2, and I assumed he was the
>> one who answered on v3 also.
>>
> 
> I'm flattered, thanks.
> 
>>>> This series goes in another direction (a), adding a unique dockerfile
>>>> covering all versions, independently of all the dockerfiles we have now.
>>>>
>>>> If we agree (a) is a better direction, unique dockerfile approach can be
>>>> vastly simplified by restricting ourselves to qemu >= 3.1.0. If not, no
>>>> hard feelings, it's just a proposal.
>>
>> Regards,
>> Pierrick

Feel free to review v4 or say if anything else is missing for you, I
would be happy to add it.

For now, it can build any version (sha-1 or tag) with any git repo URL.

Regards,
Pierrick

Re: [PATCH v3 0/3] releases: build and run any QEMU release easily!

[Fabiano Rosas]

Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com> writes:

> On 10/27/2026 7:36 AM, Fabiano Rosas wrote:
>> Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com> writes:
>> 
>>> On 10/26/2026 8:32 AM, Fabiano Rosas wrote:
>>>> Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com> writes:
>>>>
>>>>> On 10/23/2026 8:33 AM, Fabiano Rosas wrote:
>>>>>> Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com> writes:
>>>>>>
>>>>>>> This series was inspired by a recent post from Fabiano to provide container
>>>>>>> images for migration tests:
>>>>>>> https://lore.kernel.org/qemu-devel/24d04a1f-344d-4b90-a307-b582dd9a1128@oss.qualcomm.com/T/#m353f119dc1a24fa437dde8a96d1891629e64303c
>>>>>>>
>>>>>>> The idea to be able to build and run any QEMU (past) release with containers was
>>>>>>> in my mind for some time, so I gave it a shot. Finally, it was easier than
>>>>>>> expected to obtain a single dockerfile (with a few hacks) that works over all
>>>>>>> our releases.
>>>>>>>
>>>>>>
>>>>>> Good stuff, thanks for this work. I'm just not sure how much of it fits
>>>>>> upstream in the current shape, let's discuss...
>>>>>>
>>>>>> You say this was in your mind for a while, could you elaborate a bit on
>>>>>> how you think this could benefit the project?
>>>>>>
>>>>>
>>>>> First usage that comes to my mind is that it would greatly help for
>>>>> migration, allowing automatic testing in CI, by just downloading any
>>>>> previous QEMU, dump a machine, and restore it on current version.
>>>>
>>>> Don't we also build QEMU several times over in CI? Maybe I'm wrong, but
>>>> it might allow us to decouple a few tests from the build steps.
>>>>
>>>
>>> This is for current version, not past ones. I'm not sure it's worth
>>> adding an identical build for past versions on every pipeline.
>>>
>> 
>> I thought we re-built current qemu a few times, no? We have containers
>> for various distros and then make; make check inside of them. The
>> question was whether a step 0 to build the binary container for this
>> release would allow some builds to be skipped.
>>
>
> I see. To some extent, yes. However, if you take a look more closely,
> you'll notice most of the builds differ by:
> - set of targets built
> - configure options
> - distro target
>
> While we could maybe optimize some parts, that still requires duplication.
>
>>>>> We could do it for last 10 releases to ensure migration from packaged
>>>>> versions in all supported distros (red hat, alpine, ubuntu, ...) work
>>>>> as expected.
>>>>>
>>>>
>>>> It's a nice idea but potentially a costly one. I don't think there is a
>>>> guarantee that the downstream package is migration-compatible with the
>>>> upstream package.
>>>>
>>>
>>> What would be costly? It seems it would just be a download + savevm
>>> through monitor, and a loadvm from current version tested.
>>>
>> 
>> Setting it up would be simple. Working with it, not so much. It's a test
>> that once it fails, then it becomes a tedious job of figuring out
>> whether the issue is a failure on our side or if it's merely an
>> incompatibility between upstream vs. downstream. The packaged QEMU can
>> have all sorts of tweaks, extra patches, fewer patches, different code
>> entirely, different machine type ABI, etc (their own QMP commands
>> even). The distro can guarantee it all works inside their environment.
>> 
>> When a problem happens on the intersection between the two binaries, if
>> it's on our side, we'd need to determine whether to fix it or tell the
>> distro "don't do that". If it's on their side then we open a bug report
>> and hope that this is even a priority for them.
>> 
>> So overall I'm in favor, but I don't expect it to be game changing.
>>
>
> Do we have a concrete example in recent migration history of distro
> doing a custom downstream patch that broke migration?
>

No because no one tries to migrate from the distro packaged QEMU to the
upstream QEMU (i.e. apt install qemu vs. git clone qemu).

The downstream package contains whatever upstream version the downstream
maintainer wanted to include (e.g. a stable branch) with their own
customizations added on top. These are experienced people that make
sensible choices, so things are not wild, but there's still a solid
layer of separation.

The (mostly enterprise) distros will also potentially have more than one
stream, e.g. qemu-11.0.0 and qemu-11.0.0-A, where this "A" is something
that fits their own policies (e.g. a version that only receives updates
for CVE fixes; or a version with backports that only ships as part of a
particular product).

The downstream companies might also have their own special sauce to deal
with migration. Red Hat for instance defines machine types that don't
even exist upstream.

The general point here is that it's already a lot of work for the
downstream, it might not be a good use of the community time to have us
also taking a look at all those details. And of course in some cases
there's a conflict of interest.

> This would be series author's job/time, not maintainer's.
>

It should be, yes. More complex problems or ones that for some reason
weren't detected right away will still take other people's time.

> Also, I think it's precisely why it's interesting to investigate those.
> What is the point to maintain any kind of compatibility, if it's not for
> our main users?
> Preserving compatibility for versions built manually from source is a
> niche use case.
>

You're thinking in the correct direction, however migration specifically
is very dependent on hardware, both emulated and real. The right people
to test compatibility between distro version X vs. distro version X+1 is
the downstream QA team. Companies have access to hardware that closely
matches their customers', usually outside of the range of what the
community can get their hands on.

We could maybe attempt a joint effort where upstream provides the CI and
tests while the downstreams add their own hardware and test jobs. Take
git HEAD, package it into a .rpm/.deb, etc and run tests on it.

Note that there's:

A) the migration/ subsystem, which performs the migration and provides
the APIs and core code.

  This is where we test features and compatibility mechanisms.
  I.e. qtest/migration-test, the migration-compat job in CI and what I
  test in every PR.  This is well covered. We don't add new features or
  major fixes without tests.

B) the vmstate code for a device, IOW, the thing being migrated.

  This is where compatibility issues are more likely to occur (think
  developer adds new emulation feature and forgets to update vmstate
  code).

  There are some tests for this under qemu-iotests, there's static
  checking using -dump-vmstate, some parts in functional tests, some in
  qtests/qom-test.

  This is pooly covered for a bunch of reasons:

  One needs to think carefully about the device, craft tests that
  include it, configure it and do something that is prone to be affected
  by migration. Some bugs will only trigger at the OS layer, so guest
  kernel and drivers come into play.

  There's a spec for the hardware, there's no spec for what happens when
  the device state and OS magically change because it's actually QEMU
  running v8.2.0 and migration took 1s more to converge than usual. So
  writing the device and tests for it tends to be quicker than writing a
  good migration test; most people don't bother.

  We also like to keep backwards migration compatibility because distros
  usually provide that. It's a completely different test and sometimes
  hard to justify when the device writer doesn't particularly care about
  the cloud computing scenarios where migration is used.

>>>>> It could be a enhancement compared to the manual process done once per
>>>>> release. I read many times in review "it breaks the migration", that
>>>>> yes, it would be nice to have CI saying "this series broke the migration
>>>>> from qemu x.y".
>>>>>
>>>>
>>>> Most of the "breaks migration" issues are due to device migration. What
>>>> we have today in terms of migration testing cannot detect most of those.
>>>>
>>>
>>> Ok.
>>>
>>>>> Beyond that, I think we could also offer a "rebase on steroids" tool. We
>>>>> could extend list of containers published to all top merge commits (git
>>>>> log --first-parent), and provide a fast rebase, based on download only.
>>>>
>>>> Bisect requires a test of some sort. Will this cause the opposite
>>>> problem than what migration-test encounters? I.e. instead of "old QEMU
>>>> needs old OS" you'd have "new test needs QEMU from new OS". One
>>>> alternative would be to build the tests into an image as well. Or
>>>> perhaps it wouldn't be a problem in practice.
>>>>
>>>
>>> I don't think adding tests in image has a specific value. What we want
>>> is to keep the same test source, but just change the binary used. In
>>> this case, the container just acts as a binary.
>>>
>> 
>> There are requirements such as QEMU command line, external binaries,
>> maybe even the code itself, in the case of unit tests. Exchanging the
>> QEMU binary while keeping the same test (whether binary or not) may not
>> work.
>> 
>> - At some point we added the -audio command line option and updated
>> libqtest to include the option by default. Now older binaries can't
>> start when called by the new test code.
>> 
>> - If a test is sensitive to qemu output, deprecation warnings and error
>> message format changes might break it.
>> 
>> - qemu-img and qemu-io might have implicit dependencies on a range of
>> versions.
>> 
>> If we're being rigorous, two commits with completely different test and
>> QEMU code could still test the same functionality. That's how we do
>> normal bisect anyway. The requirement of test compatibility across QEMU
>> versions is something we probably never thought about before. I have
>> just recently started thinking about what can be done to libqtest to
>> allow more flexibility in this case.
>>
>
> I see. For me it's a bit surprising to change both the source code and
> the test code, but if we are used to it, I'm happy to follow the culture.
>

Disconsider, I was describing what happens if the test does not follow
the code (that's how migration compat tests work).

You're expecting to match a pre-built QEMU to a bisect-time built
test. The environment _then_ doesn't match the environment when the
pre-built QEMU was built. But this is likely to never have an issue, so
ok.

> In all cases, when running a git bisect, the previous version of the
> test will be checkout too. My proposal is just to skip the build part by
> relying on prebuilt images, but if we follow classic git bisect, test
> will be updated accordingly during bisection.
>
>>>>> See it as a giant immutable precompiled cache. Dockerhub storage is free
>>>>> and unlimited.
>>>>> Some quick math: Since 8.0, there are ~2000 top merge commits. Each
>>>>> image is < 250MB compressed, so the whole thing would be < 500GB.
>>>>>
>>>>> Finally, in my personal case, while working on single binary, I had to
>>>>> compare output of qemu help between versions. I'm also interested to
>>>>> track QEMU binary size evolution over time. And some personal fun with
>>>>> software archeology on the project too for anyone.
>>>>>
>>>>>> To be clear, my use-case is simply to produce a few containers for
>>>>>> testing migration on my workstation. Your series can help a lot with
>>>>>> that, but from my point of view, some parts are clearly extra and some
>>>>>> are missing.
>>>>>>
>>>>>> Also beware that what you're touching here is both "infrastructure" and
>>>>>> "publishing". This has implications for how people consume the work of
>>>>>> the project. There's a chance that people see this as "official QEMU
>>>>>> containers" and try to use them in all sorts of weird scenarios that we
>>>>>> as a community are not prepared to maintain. You might get some
>>>>>> push-back on that basis.
>>>>>>
>>>>>
>>>>> Sure, it was free to generate all images and push them on my personal
>>>>> dockerhub, and it's now automated to push any feature release also in a
>>>>> GitHub nightly job. If we want to make it more official, it should
>>>>> really be hosted on an official account.
>>>>>
>>>>>>> The oldest functional distro we can find nowadays is ubuntu 12.04. Because of
>>>>>>> this, the oldest QEMU release we can build is v0.12.0. Previous ones require an
>>>>>>> environment we can't get easily today.
>>>>>>>
>>>>>>
>>>>>> I think the ability to go all the way back is very cool, but unless we
>>>>>> want to look at this from a archival or heritage perspective, I'd say
>>>>>> there's a line somewhere to how old we should go. I don't see an
>>>>>> use-case for very old binaries. Even knowing if they still work would
>>>>>> already incur in a maintenance cost.
>>>>>>
>>>>>
>>>>> Agree, mostly was a fun challenge. I was also curious to see until when
>>>>> we can go back in past from 2026.
>>>>>
>>>>> For the "maintenance" a scheduled CI job running nightly is a free thing
>>>>> for me, but I understand opinions on "maintenance cost" may differ. I
>>>>> also co-maintain tests/docker with Alex, so I'll be the one taking the
>>>>> load, and I'm fine with it.
>>>>
>>>> Fair enough.
>>>>
>>>>>   >> Images have the absolute minimal content possible. It's only qemu
>>>>> binaries,
>>>>>>> artifacts (roms, keymap, ...), and libraries. There is no shell or any other
>>>>>>> tool included. As a result, images have a size from 149MB (v0.12.0) to 1.48GB
>>>>>>> (v11.1.2).
>>>>>>>
>>>>>>
>>>>>> No shell means the first thing to do if one wants to debug the image is
>>>>>> to rebuild with a shell in. Not sure how frequent it will be, maybe it's
>>>>>> ok like this.
>>>>>>
>>>>>
>>>>> We could leave a base system, for only ~100MB per image, which is not
>>>>> the end of the world (+10% in size). Personally, I appreciate to have no
>>>>> shell, it reduces possibility of exploit when running an older QEMU
>>>>> having CVEs.
>>>>>
>>>>>>> This series adds 3 pieces:
>>>>>>> - a unique dockerfile to build any QEMU release. User has to provide an explicit
>>>>>>>   Ubuntu version when building a given QEMU version.
>>>>>>
>>>>>> This is the tricky part. We'll be carrying code (script+dockerfile) that
>>>>>> only works on master. If we tag today and tomorrow the old distro repo
>>>>>> breaks, we obviously cannot fix it retroactively. So we need to be
>>>>>> careful of where we direct the maintenance effort.
>>>>>>
>>>>>> 1) I'd suggest first of all to keep the release image entirely separate
>>>>>> from the ubuntu image. Treat release.docker as we treat the other
>>>>>> dockerfiles and update it to work for the current release without
>>>>>> looking at past ones. Have it clone, checkout, build and gather
>>>>>> deps. Allow the configure to be tweaked via parameter in case we need to
>>>>>> disable something in an older build.
>>>>>>
>>>>>> The script would then build a distro image (or dockerfile) with the
>>>>>> knowledge that it has about ubuntu depending on the QEMU tag vs. ubuntu
>>>>>> version combination. This would allow us to track the cludges more
>>>>>> precisely instead of "some ancient distro" as the dockerfile currently
>>>>>> says.
>>>>>>
>>>>>> This is more of a layering concern and how easy it will be to drop/add
>>>>>> version-specific fixes.
>>>>>>
>>>>>
>>>>> IIUC I think that's the approach Fabiano took, to use Dockerfile from
>>>>> previous versions, and the conclusion was that it requires some seds and
>>>>> hacks to have something that work. Internet state is not immutable.
>>>>> IMHO, building previous QEMU with past Dockerfiles is doomed to fail.
>>>>>
>>>>
>>>> That's not what I'm suggesting here. I'm saying that "current
>>>> dockerfile" should have less tweaks and "current
>>>> build_release_container.sh" should be the one responsible for doing the
>>>> ubuntu-version-to-tweak mapping.
>>>>
>>>
>>> Got you.
>>>
>>>> But maybe you meant to reply to the second point below. In that case,
>>>> yes the past dockerfiles require tweaks, but they also provide most of
>>>> the time the exact list of dependencies for the previous QEMU, and that
>>>> work was already done by the people from the past.
>>>>
>>>>> Only thing you can control/guarantee is that "current dockerfile on
>>>>> master works".
>>>>>
>>>>>> 2) I wonder if the usage of apt-get build-dep might be a convenience
>>>>>> trap. In theory, some of the adaptations that are required (such as
>>>>>> tweaking dependencies) have already been done by us when working on that
>>>>>> development window.  If we take the dockerfiles from the git tag, those
>>>>>> changes don't need to be done again and for the issues that didn't exist
>>>>>> at the time, we can map them to the git tag instead of ubuntu version;
>>>>>> (we tend to know more about QEMU releases than ubuntu releases).
>>>>>>
>>>>>> Another issue with buid-dep is that it lists _ubuntu package_
>>>>>> dependencies. The distro can change how it builds the package without us
>>>>>> noticing. There is already some work done by lcitool, I imagine that it
>>>>>> would then be duplicated. Or worse(!) be different because build-dep
>>>>>> might be listing way more packages that what we need for the default
>>>>>> build.
>>>>>>
>>>>>> This is more of a push towards using existing infra because those pieces
>>>>>> are already looked after for other purposes.
>>>>>>
>>>>>>> - A script build_release_container.sh which automatically detects which ubuntu
>>>>>>>   version a given release should be buildable on (based on date). It works
>>>>>>>   great, and the heuristic works for all our versions.
>>>>>>> - a script generate_release_links.sh which creates wrapper scripts automatically
>>>>>>>   using prebuilt images (hosted on my dockerhub).
>>>>>>>
>>>>>
>>>>> Sure, lcitool could work to express all that. Would I like to spend time
>>>>> to rewrite all the missing bits in yaml when debian did it for me? not
>>>>> really.
>>>>>
>>>>
>>>> I may have misled you. I'm not suggesting to use lcitool for the missing
>>>> bits, I'm suggesting we consume what's already in
>>>> tests/docker/dockerfiles. That point you already got.
>>>>
>>>
>>> Oh I see ok. Like extract the list of packages, and inject that.
>>> Well, I'm not sure it will be simpler than just doing an apt build-dep
>>> to be honest, and not sure what we gain from taking this information
>>> from our sources.
>>>
>> 
>> Just so you're aware of my reasoning:
>> 
>> It's information we control, add/remove packages, pin package versions,
>> write a big commit message explaining why, etc. While apt build-dep is
>> simple and it works great, it's still a little piece in the hands of a
>> third party that has their own policies and goals. No one on their side
>> (whatever distro it is) is thinking "what happens with QEMU CI if I
>> change this". It makes the project more robust if we limit that kind of
>> exposure.
>>
>
> That's true. On the other hand, they have to get the deps right, or they
> can't build latest version. So I tend to think they need to get the
> right set of deps by design, not by choice.
>
> Only exception is when new deps are introduced (like ninja, or
> flex/bison recently). But considering all versions since v3.1.0,
> flex/bison is the only exception.
>
>>>>>>
>>>>>> It would be nice if these two could take any hash, instead of just a
>>>>>> tag. This is one of the things that will depend on what you intend to
>>>>>> achieve with this series.
>>>>>>
>>>>>
>>>>> Very easy, it just needs a non shallow clone.
>>>>>
>>>>> ---
>>>>>
>>>>> After reading and answering to this message, I feel there are two topics
>>>>> we can discuss:
>>>>> 1. the current implementation
>>>>> 2. what would we like
>>>>>
>>>>> 1. Personally, I just posted this series as an example of "yes it's
>>>>> possible with very limited amount of code", and I don't mind having
>>>>> something completely different. I feel it's probably not worth reviewing
>>>>> every line at this point.
>>>>>
>>>>> 2. The biggest question seems to be the direction with:
>>>>> 'do we want to provide a single dockerfile that covers all versions' (a)
>>>>> VS 'having a dockerfile at a time of history for current version' (b).
>>>>>
>>>>
>>>> It's probably (b), because the "all versions" part spontaneously spawned
>>>> to life. If you take a recent-ish QEMU version there will be few or no
>>>> tweaks at all and then it all becomes a question of "why base this on
>>>> ubuntu when we already have dockerfiles written for several
>>>> distros?".
>>>>
>>>> UPDATE: You just replied that you think build-dep is consistently close
>>>> to the real deps enough that it would be less work to maintain. You
>>>> could very well be right and we're approaching the hair-splitting zone
>>>> around this topic. I suggest you carry on using your best judgement and
>>>> let's leave it to time.
>>>>
>>>
>>> We can also decide to settle on fedora, alpine, debian, suse, or any
>>> distro. In terms of images, ubuntu was just the most convenient (tried
>>> with debian first, which required updating apt sources for older versions).
>>>
>>>>> I think (b) is what we have now (current ubuntu/suse/debian/alpine image
>>>>> works for current revision, since it's used in CI), and it was already
>>>>> proven it fails from Fabiano's feedback.
>>>>
>>>> But the tweaks may be necessary either way. This series adds an
>>>> indirection to choose an ubuntu version just to be able to issue (some)
>>>> commands unconditionally in the dockerfile. It could just as well check
>>>> the qemu version directly.
>>>>
>>>
>>> That's maybe where the philosophy differs. Key idea of this series is to
>>> tweak the environment so it works all time, and not do specific things
>>> per version.
>>>
>> 
>> Sure, but the "all the time" part doesn't convince much. Anyway, that's
>> only one point and we've discussed it enough. If it fails miserably
>> you'll learn something, otherwise, I will. =)
>>
>
> Sure, and count on me to send a fix if it breaks, so it will keep on
> working. I'll catch this with automatic build for new releases.
>
>>>> (yes, there's potentially dealing with all the other distros and lcitool
>>>> and all _that_, I get it)
>>>>
>>>>> It seems to be that's what you propose also from your answer above.
>>>>>
>>>>
>>>> Fabiano and I are the same person. =)
>>>>
>>>
>>> Sorry, got confused as Peter answered on v2, and I assumed he was the
>>> one who answered on v3 also.
>>>
>> 
>> I'm flattered, thanks.
>> 
>>>>> This series goes in another direction (a), adding a unique dockerfile
>>>>> covering all versions, independently of all the dockerfiles we have now.
>>>>>
>>>>> If we agree (a) is a better direction, unique dockerfile approach can be
>>>>> vastly simplified by restricting ourselves to qemu >= 3.1.0. If not, no
>>>>> hard feelings, it's just a proposal.
>>>
>>> Regards,
>>> Pierrick
>
> Feel free to review v4 or say if anything else is missing for you, I
> would be happy to add it.
>
> For now, it can build any version (sha-1 or tag) with any git repo URL.
>
> Regards,
> Pierrick

Re: [PATCH v3 0/3] releases: build and run any QEMU release easily!

[Pierrick Bouvier]

On 10/27/2026 2:54 PM, Fabiano Rosas wrote:
> Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com> writes:
> 
>> On 10/27/2026 7:36 AM, Fabiano Rosas wrote:
>>> Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com> writes:
>>>
>>>> On 10/26/2026 8:32 AM, Fabiano Rosas wrote:
>>>>> Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com> writes:
>>>>>
>>>>>> On 10/23/2026 8:33 AM, Fabiano Rosas wrote:
>>>>>>> Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com> writes:
>>>>>>>
>>>>>>>> This series was inspired by a recent post from Fabiano to provide container
>>>>>>>> images for migration tests:
>>>>>>>> https://lore.kernel.org/qemu-devel/24d04a1f-344d-4b90-a307-b582dd9a1128@oss.qualcomm.com/T/#m353f119dc1a24fa437dde8a96d1891629e64303c
>>>>>>>>
>>>>>>>> The idea to be able to build and run any QEMU (past) release with containers was
>>>>>>>> in my mind for some time, so I gave it a shot. Finally, it was easier than
>>>>>>>> expected to obtain a single dockerfile (with a few hacks) that works over all
>>>>>>>> our releases.
>>>>>>>>
>>>>>>>
>>>>>>> Good stuff, thanks for this work. I'm just not sure how much of it fits
>>>>>>> upstream in the current shape, let's discuss...
>>>>>>>
>>>>>>> You say this was in your mind for a while, could you elaborate a bit on
>>>>>>> how you think this could benefit the project?
>>>>>>>
>>>>>>
>>>>>> First usage that comes to my mind is that it would greatly help for
>>>>>> migration, allowing automatic testing in CI, by just downloading any
>>>>>> previous QEMU, dump a machine, and restore it on current version.
>>>>>
>>>>> Don't we also build QEMU several times over in CI? Maybe I'm wrong, but
>>>>> it might allow us to decouple a few tests from the build steps.
>>>>>
>>>>
>>>> This is for current version, not past ones. I'm not sure it's worth
>>>> adding an identical build for past versions on every pipeline.
>>>>
>>>
>>> I thought we re-built current qemu a few times, no? We have containers
>>> for various distros and then make; make check inside of them. The
>>> question was whether a step 0 to build the binary container for this
>>> release would allow some builds to be skipped.
>>>
>>
>> I see. To some extent, yes. However, if you take a look more closely,
>> you'll notice most of the builds differ by:
>> - set of targets built
>> - configure options
>> - distro target
>>
>> While we could maybe optimize some parts, that still requires duplication.
>>
>>>>>> We could do it for last 10 releases to ensure migration from packaged
>>>>>> versions in all supported distros (red hat, alpine, ubuntu, ...) work
>>>>>> as expected.
>>>>>>
>>>>>
>>>>> It's a nice idea but potentially a costly one. I don't think there is a
>>>>> guarantee that the downstream package is migration-compatible with the
>>>>> upstream package.
>>>>>
>>>>
>>>> What would be costly? It seems it would just be a download + savevm
>>>> through monitor, and a loadvm from current version tested.
>>>>
>>>
>>> Setting it up would be simple. Working with it, not so much. It's a test
>>> that once it fails, then it becomes a tedious job of figuring out
>>> whether the issue is a failure on our side or if it's merely an
>>> incompatibility between upstream vs. downstream. The packaged QEMU can
>>> have all sorts of tweaks, extra patches, fewer patches, different code
>>> entirely, different machine type ABI, etc (their own QMP commands
>>> even). The distro can guarantee it all works inside their environment.
>>>
>>> When a problem happens on the intersection between the two binaries, if
>>> it's on our side, we'd need to determine whether to fix it or tell the
>>> distro "don't do that". If it's on their side then we open a bug report
>>> and hope that this is even a priority for them.
>>>
>>> So overall I'm in favor, but I don't expect it to be game changing.
>>>
>>
>> Do we have a concrete example in recent migration history of distro
>> doing a custom downstream patch that broke migration?
>>
> 
> No because no one tries to migrate from the distro packaged QEMU to the
> upstream QEMU (i.e. apt install qemu vs. git clone qemu).
> 
> The downstream package contains whatever upstream version the downstream
> maintainer wanted to include (e.g. a stable branch) with their own
> customizations added on top. These are experienced people that make
> sensible choices, so things are not wild, but there's still a solid
> layer of separation.
> 
> The (mostly enterprise) distros will also potentially have more than one
> stream, e.g. qemu-11.0.0 and qemu-11.0.0-A, where this "A" is something
> that fits their own policies (e.g. a version that only receives updates
> for CVE fixes; or a version with backports that only ships as part of a
> particular product).
> 
> The downstream companies might also have their own special sauce to deal
> with migration. Red Hat for instance defines machine types that don't
> even exist upstream.
> 
> The general point here is that it's already a lot of work for the
> downstream, it might not be a good use of the community time to have us
> also taking a look at all those details. And of course in some cases
> there's a conflict of interest.
>

Right, I see your point. We indeed can't do this job for the distros.

>> This would be series author's job/time, not maintainer's.
>>
> 
> It should be, yes. More complex problems or ones that for some reason
> weren't detected right away will still take other people's time.
> 
>> Also, I think it's precisely why it's interesting to investigate those.
>> What is the point to maintain any kind of compatibility, if it's not for
>> our main users?
>> Preserving compatibility for versions built manually from source is a
>> niche use case.
>>
> 
> You're thinking in the correct direction, however migration specifically
> is very dependent on hardware, both emulated and real. The right people
> to test compatibility between distro version X vs. distro version X+1 is
> the downstream QA team. Companies have access to hardware that closely
> matches their customers', usually outside of the range of what the
> community can get their hands on.
> 
> We could maybe attempt a joint effort where upstream provides the CI and
> tests while the downstreams add their own hardware and test jobs. Take
> git HEAD, package it into a .rpm/.deb, etc and run tests on it.
> 
> Note that there's:
> 
> A) the migration/ subsystem, which performs the migration and provides
> the APIs and core code.
> 
>   This is where we test features and compatibility mechanisms.
>   I.e. qtest/migration-test, the migration-compat job in CI and what I
>   test in every PR.  This is well covered. We don't add new features or
>   major fixes without tests.
> 
> B) the vmstate code for a device, IOW, the thing being migrated.
> 
>   This is where compatibility issues are more likely to occur (think
>   developer adds new emulation feature and forgets to update vmstate
>   code).
> 
>   There are some tests for this under qemu-iotests, there's static
>   checking using -dump-vmstate, some parts in functional tests, some in
>   qtests/qom-test.
> 
>   This is pooly covered for a bunch of reasons:
> 
>   One needs to think carefully about the device, craft tests that
>   include it, configure it and do something that is prone to be affected
>   by migration. Some bugs will only trigger at the OS layer, so guest
>   kernel and drivers come into play.
> 
>   There's a spec for the hardware, there's no spec for what happens when
>   the device state and OS magically change because it's actually QEMU
>   running v8.2.0 and migration took 1s more to converge than usual. So
>   writing the device and tests for it tends to be quicker than writing a
>   good migration test; most people don't bother.
> 
>   We also like to keep backwards migration compatibility because distros
>   usually provide that. It's a completely different test and sometimes
>   hard to justify when the device writer doesn't particularly care about
>   the cloud computing scenarios where migration is used.
>

Thanks for the insight, I understand better why migration is poorly
covered in our test suite.

Regards,
Pierrick