Re: [RFC] single-binary: merge experimental qemu-system before filtering

[Chao Liu <chao.liu@processmission.com>]

On Fri, Aug 28, 2026 at 03:44:43PM +0800, Yonggang Luo wrote:
> Hi,
> 
> This is a design RFC, not a patch series. It follows review of
> 
>   [PATCH 00/12] single-binary: link ARM and RISC-V into qemu-system
>   https://patchew.org/QEMU/20260823150731.896-1-luoyonggang@gmail.com/
> 
>   [PATCH v2 00/13] single-binary: link ARM and RISC-V into qemu-system
>   https://patchew.org/QEMU/20260826184229.1145-1-luoyonggang@gmail.com/
> 
> and Daniel P. Berrange's measurements of that v1.
> 
> What v2 already did
> ===================
> 
> v2 of that series already takes the v1 review that got maintainer
> support:
> 
> - Keep real qemu-system-$TARGET (and $TARGETw on Windows). v1
>   installed per-arch names as symlinks to qemu-system; v2 does
>   not. qemu-system is an extra binary from the ARM+RISC-V union,
>   behind a meson option.
> - argv[0] still selects the target for qemu-system-aarch64.
> - Drop v1 little- and big-endian TargetInfo variants
>   (target-info-<name>-le/be, -target aarch64-le). One TargetInfo
>   per target_name. Endian variants are later work.
> - -target ? lists target_name only. Unknown names point at
>   -target ? rather than printing the list and exit(1).
> - Do not build qemu-systemw for the combined binary, so Windows
>   data exports (qdev_prop_array / qdev_prop_*) do not block
>   enabling modules later.
> - TargetCpuOps (b444b4fb) for QMP CPU queries, dump, and ARM/RISC-V
>   semihosting, still as one process-wide table filtered at
>   register time. This RFC revises that into a map.
> 
> v2 still required -target on bare qemu-system, still used
> TYPE_TARGET_SPECIFIC on the combined binary, and did not mark
> qemu-system experimental. Those are what this RFC asks to freeze
> before posting the next series: extra binary stays, filtering is
> not a gate, proposal A vs B. Heterogeneous (no target: on -M)
> stays an error until later.
> 
> Why this RFC now
> ================
> 
> We are very close to a linked qemu-system that contains more than
> one target (ARM + RISC-V; see v2 above). v2 already dropped the
> symlink. The remaining delay is not "can it link", it is unresolved
> product questions: -target vs -M only, complete filtering before
> the first binary exists, experimental union vs looking like
> $TARGET.
> 
> This RFC is to freeze those questions so the next series can land
> an experimental qemu-system quickly, instead of another patch bomb
> that still debates the end result. Filtering, heterogeneous
> machines, modules for RSS, and CLI polish can follow against a
> real binary. Please comment on the split and on proposal A vs B;
> that is what unblocks the merge.
> 
> Thesis
> ======
> 
> Land an extra, experimental qemu-system first, without completing
> target/device filtering. Use that binary as the test vehicle for
> filtering. Keep today's qemu-system-$TARGET as real binaries, not
> symlinks.
> 
> qemu.git has no combined qemu-system today. Filtering, uniquify,
> and CPU-op dispatch are therefore untested assumptions. Completing
> TYPE_TARGET_SPECIFIC against qemu-system-riscv64 (a single-target
> link) does not prove the union case. We need a real qemu-system in
> the tree before filtering can be designed against measured
> behavior.
> 
> Berrange compared a v1 qemu-system-riscv64 *symlink* to today's
> standalone binary:
> 
> - qom-list-types went from about 770 classes to about 1530 (ARM
>   types leak into RISC-V)
> - -device help grew
> - RSS for -nodefaults -display none -M none went from 17.3 MB to
>   20.7 MB
> - A full-union guess was much worse for RISC-V
> 
> That is a packaging blocker if $TARGET becomes a symlink. Extra RAM
> on a *new* experimental qemu-system is acceptable if we document
> that this binary exposes the union. It is also not the last word
> on RSS.
> 
> Once modules are always enabled for qemu-system, unused machines
> and devices can live in .so / .dll / .dylib and stay out of the
> startup working set. One simple shape is one module per target, so
> qemu-system -M arm:virt (or -target aarch64) loads the ARM module
> and does not map RISC-V boards. That is later work, not a blocker
> for the first extra binary (Mark Cave-Ayland already suggested
> mandatory modules for qemu-system; Berrange did not want to block
> on that). The first merge still links the union into one
> executable so we have something to test.
> 
> This split is meant to take both review positions at the layer they
> apply to:
> 
> - Berrange / Paolo: do not replace qemu-system-$TARGET with a
>   symlink. A new qemu-system can be its own thing.
> - Pierrick: select TargetInfo when the user wants a single-target
>   view; TYPE_TARGET_SPECIFIC as the filtering mechanism. Matching
>   $TARGET introspection is the *goal* of a later filtering series,
>   not a gate on the first link.
> 
> Two CLI proposals for how to select one TargetInfo are below.
> They share argv[0] and -M target:machine. They differ on whether
> -target exists. A run with no selected TargetInfo (heterogeneous /
> SYS_EMU_TARGET_HETEROGENEOUS) is deferred: the first series only
> prints an error.
> 
> experimental-qemu-system
> ========================
> 
> Keep building real qemu-system-$TARGET as today (and $TARGETw on
> Windows). Add qemu-system alongside them, from the union of ARM and
> RISC-V for the first pair, behind a meson option, marked
> experimental.
> 
> Do not install per-arch names as symlinks to qemu-system. Distro
> packaging of qemu-system-riscv64 stays on the old RSS and QOM list.
> 
> The combined binary is documented as exposing everything that was
> linked. Incomplete device filtering is an accepted experimental
> defect, not a release blocker for this first binary. -M help /
> -device help with no selected TargetInfo may still list the union.
> Starting a machine that way is not supported yet (error; see
> below).
> 
> Binding TargetInfo
> ==================
> 
> A combined process has more than one TargetInfo (target_name, page
> bits, long width, CPU_RESOLVING_TYPE, endianness).
> 
> In this mail, single-target means the process has one selected
> TargetInfo / SysEmuTarget, like qemu-system-$TARGET today.
> Heterogeneous would mean it does not (SYS_EMU_TARGET_HETEROGENEOUS).
> That mode is not implemented yet.
> 
> Single-target runs still pick one TargetInfo before the chosen
> machine and accelerators initialize.
> 
> Shared in both proposals:
> 
> - argv[0]: qemu-system-aarch64 infers aarch64. Existing $TARGET
>   command lines stay valid. Single-target. -M virt works as today.
> - -M arm:virt on bare qemu-system: single-target. The target:
>   prefix selects TargetInfo arm and machine virt together. The
>   token is target_name (arm, aarch64, riscv32, riscv64).
>   Properties stay after the machine name:
>   -M aarch64:virt,gic-version=3.
> - Bare qemu-system and -M has no target: prefix: not supported
>   yet. Fail with an error that this would be a heterogeneous
>   machine (SYS_EMU_TARGET_HETEROGENEOUS), which is deferred, and
>   point at -M arm:virt and (proposal A) -target. Do not start
>   the VM. -M help / -device help with no selected TargetInfo may
>   still list the union so users can pick a qualified name.
> 
> If argv[0] and a target: prefix are both given, they must match.
> Unknown prefix names fail. -M help may also print qualified names
> (arm:virt, riscv64:virt) so users can opt into a single-target run.
> 
> Proposal A: -target plus -M target:machine
> ------------------------------------------
> 
> Pierrick suggested -target. Keep it as an optional way to select
> TargetInfo on the combined binary. It is not mandatory on
> qemu-system.
> 
> - -target aarch64: single-target. Later filtering can make this
>   look like qemu-system-aarch64. -M virt works as today.
> - If both -target and a target: prefix are given, they must match.
>   Unknown -target names fail and point at -target ?.
> 
>   qemu-system -M help
>   qemu-system -M virt              # error: heterogeneous not supported
>   qemu-system -M arm:virt ...
>   qemu-system -target aarch64 -M virt ...
>   qemu-system-riscv64 -M virt ...
> 
> Proposal B: target only in -M or argv[0] (alternative)
> -----------------------------------------------------
> 
> Do not add -target. The target is always embedded in the machine
> argument, or inferred from argv[0].
> 
> - qemu-system-aarch64 -M virt: single-target from argv[0].
> - qemu-system -M arm:virt: single-target from -M.
> - qemu-system -M virt: error (heterogeneous not supported yet).
> 
> There is no global option that selects TargetInfo before -M.
> Early option parsing still has to read -M (and argv[0]) before
> machines initialize, the same as proposal A reads -target.
> 
>   qemu-system -M help
>   qemu-system -M virt              # error: heterogeneous not supported
>   qemu-system -M arm:virt ...
>   qemu-system -M aarch64:virt,gic-version=3 ...
>   qemu-system-riscv64 -M virt ...

> 
> Compared with A: one place to write the target on the combined
> binary (-M), no new flag, no -target vs -M mismatch cases. The
> cost is that qemu-system -M virt is never a hidden alias for
> aarch64; colliding names must be qualified, and a single-target
> view of the union binary is qemu-system -M aarch64:virt, not
> qemu-system -target aarch64 -M virt.
> 
> Please pick A or B; the rest of this RFC is the same either way.
> 
I think Proposal B is clearer and easier to merge upstream. It does
not add another option, and the <target>: prefix in -M can select
the right TargetInfo.

We only need to check that -M <target>:<machine-name> is valid.
We do not need to rename existing machine names: parse the prefix
first, then keep the existing -M parser for the rest.

Another point to discuss is what should happen when -M is missing.
For the bare qemu-system, it should exit with an error instead
of choosing a default machine, because the target is unclear.

BTW, I haven't looked closely at the earlier patch series,
so I'm not sure how much this would affect the follow-up work.

Just my two cents.

Thanks,
Chao
> SYS_EMU_TARGET_HETEROGENEOUS (deferred)
> --------------------------------------
> 
> QAPI SysEmuTarget today is the configure list (aarch64, arm,
> riscv64, ...). A later process with no selected TargetInfo and no
> target: on -M would need a value that is not a single linked
> target: add heterogeneous (C name SYS_EMU_TARGET_HETEROGENEOUS).
> It would be a process mode, not a ./configure target and not
> qemu-system-heterogeneous.
> 
> The first series does not add that enum and does not run that
> mode. qemu-system -M virt with no argv[0] target and no -target
> fails with an error: heterogeneous machines are not supported
> yet; use -M arm:virt (and -target in proposal A). How such a
> machine would name per-CPU / per-device targets is later work.
> 
> target-info-def.c
> =================
> 
> The minimum compile unit for several targets in one process is
> target-info-def.c built once per target (COMPILING_PER_TARGET).
> Each translation unit registers one TargetInfo and a
> TYPE_TARGET_INFO QOM class. meson already does that for
> qemu-system-$TARGET. Combined qemu-system extracts those objects
> from every participating target, plus arch objects from a primary
> per base architecture (aarch64 and riscv64 for the first pair).
> 
> That is enough to have several TargetInfo values. Selecting one
> TargetInfo (-M arm:virt, argv[0], and -target in proposal A) picks
> one value for the run. With none of those, the first series errors
> out; it does not invent SYS_EMU_TARGET_HETEROGENEOUS. That is not
> enough to look like a single-target binary. That is the point of
> the first merge.
> 
> TYPE_TARGET_SPECIFIC
> ====================
> 
> Pierrick's series "single-binary: implement dynamic filtering for
> machine types" is a prerequisite and is already upstream. For a
> usable combined binary we need it at least for machines: without
> is_available(), -M virt in a union link is ambiguous or can start
> the wrong board against the selected TargetInfo.
> 
> Help and introspection depend on whether TargetInfo is already
> selected:
> 
> - No selected TargetInfo and no target: on -M: -M help /
>   -device help may list the union (qualified names). Starting a
>   machine is an error (heterogeneous not supported yet).
> - Selected TargetInfo (argv[0], -M arm:virt, or -target in
>   proposal A): filter with is_available() as today. -M virt is
>   unambiguous once TargetInfo is selected.
> 
> We do not need to finish filtering every QOM class before the first
> qemu-system exists. Machine-level TYPE_TARGET_SPECIFIC (already
> wired for ARM and RISC-V boards) is enough to select a board after
> TargetInfo is selected. Device and class leakage with -target is what
> Berrange measured; on experimental qemu-system that is accepted
> until filtering-against-qemu-system. Without a selected TargetInfo
> and without a target: prefix on -M, do not run; error.
> 
> New TYPE_TARGET_SPECIFIC annotations in the first series should be
> limited to what the combined link needs. Not a campaign to tag
> every class.
> 
> TargetCpuOps must be a map
> ==========================
> 
> A combined link cannot keep one global C symbol per architecture
> for query-cpu-definitions, dump notes, or Angel semihosting.
> Commit b444b4fb6ecb ("target-info: dispatch CPU QMP, dump, and
> arm/riscv semihosting") moved those handlers onto TargetCpuOps and
> TARGET_INFO_CPU_OP (one member at a time, type_init). That
> direction is required.
> 
> The implementation is not complete. It uses a single process-wide
> 
>   static TargetCpuOps target_cpu_ops;
> 
> and target_info_register_cpu_op() stores a member only when
> cpu_type matches the already selected TargetInfo. Other
> architectures are dropped. That depends on MODULE_INIT_QOM running
> after target_info_qom_set_target(). It is the same class of
> assumption we cannot test until a combined binary exists, and it
> is the wrong shape even for qemu-system-*: one global filled or
> ignored by init order, instead of a table keyed by the selected
> target.
> 
> Proposal: keep TARGET_INFO_CPU_OP. Replace the single global with
> a map from the selected target to its TargetCpuOps. Lookup uses
> target_arch() (SysEmuTarget):
> 
>   target_info_cpu_ops() -> &map[target_arch()]
> 
> Registration always fills that architecture's slot. It does not
> compare against the currently selected TargetInfo and drop the
> rest.
> 
> qemu-system-aarch64 has one occupied slot. qemu-system has one
> slot per linked target. Same API, same callers in hw/core and
> semihosting. Help paths with no selected TargetInfo
> (qemu-system -M help) list the union and must not dispatch CPU
> QMP / dump / semihosting through a single-arch ops table. A VM
> start still needs argv[0], -M arm:virt, or -target (proposal A).
> There is no map[SYS_EMU_TARGET_HETEROGENEOUS] until that mode
> exists.
> 
> CPU_RESOLVING_TYPE is not a unique map key. TYPE_ARM_CPU is shared
> by arm and aarch64, TYPE_RISCV_CPU by riscv32 and riscv64,
> TYPE_X86_CPU by i386 and x86_64. Those siblings share one object
> file in the combined link (the primary softmmu target).
> Registration fills the compiling TARGET_ARCH, then aliases the
> same TargetCpuOps to other SysEmuTarget values whose
> TargetInfo.cpu_type matches. Lookup is still map[target_arch()].
> 
> What the first series is not
> ============================
> 
> - Completing Kconfig / target / device filtering
> - Implementing a heterogeneous machine. First series: error only.
>   SYS_EMU_TARGET_HETEROGENEOUS and how it names per-CPU /
>   per-device targets come later.
> - Mandatory modules / splitting machines and devices (or a whole
>   target) into .so / .dll / .dylib. That is the RSS follow-up
>   once qemu-system exists, not a gate on linking it.
> - A new experimental CLI (-nodefaults default, no default machine,
>   dropping -hda / -drive, ordered args)
> - Replacing qemu-system-$TARGET with a symlink
> 
> Symbol uniquify (QOM names, helpers, TYPE_ACCEL_CPU) remains
> necessary to *link*. That is engineering, not a change in user
> semantics.
> 
> filtering-against-qemu-system
> =============================
> 
> Once qemu-system exists in the tree, filtering can be implemented
> and tested for real:
> 
>   qemu-system -M riscv64:virt ...
>   qemu-system -target riscv64 ...   # proposal A only
>   qemu-system-riscv64 ...
> 
> That is the series that should drive qom-list-types, -device help,
> and RSS toward the single-target numbers Berrange measured, for
> the single-target path. Modules (always on for qemu-system, unused
> targets not mapped) are the other RSS tool, after the binary
> exists. Distros keep shipping qemu-system-$TARGET until they opt
> into the combined binary.
> 
> The no-selected-TargetInfo path is an error until heterogeneous
> support exists. Filtering must not make qemu-system -M help hide
> other arches.
> 
> Please comment on this split, and on proposal A vs B. We are
> close enough that a decision here is what lets the next series
> land experimental qemu-system instead of waiting on filtering.
> 
> --
>          此致
> 礼
> 罗勇刚
> Yours
>     sincerely,
> Yonggang Luo