Re: [PATCH v5 0/2] Add crosvm recipe to meta-virtualization

Re: [PATCH v5 0/2] Add crosvm recipe to meta-virtualization

[Bruce Ashfield]

Hi Keerthivasan,

Quick caveat before the per-patch reviews land: I see from the list
traffic that you sent a v1 series, but I can't find it in my inbox.
So my comments on 1/2 and 2/2 (sent separately as replies under
those messages) are based on v5. If anything I raise has already
been addressed in v1, please just say so and I'll move on from
those specific items.

Bruce

On Thu, May 28, 2026 at 00:40 +0530, Keerthivasan Raghavan wrote:
> This patch series introduces support for crosvm, the ChromeOS Virtual
> Machine Monitor, into the meta-virtualization layer. The goal is to
> provide a modern, security-oriented alternative to QEMU for
> virtualization workloads.
> 
> QEMU continues to be the industry-standard VMM, offering extensive device
> models, multi-architecture emulation, and a highly flexible subsystem
> architecture. However, QEMU's strength and breadth contributes to a
> large and complex monolithic codebase. As documented in QEMU's own
> security guidance, the system architecture places numerous components
> within a single process. This increases the trusted computing base and
> expands the range of possible exploit surfaces. Maintaining device models
> and legacy emulation paths further adds to complexity and potential
> vulnerability exposure.
> 
> crosvm approaches virtualization from a different perspective. Its design
> principles center on minimalism, isolation, and security. Instead of
> supporting full-system emulation, crosvm relies exclusively on KVM for
> hardware-assisted virtualization. It is implemented in Rust, enabling
> strong memory safety guarantees and eliminating many classes of bugs that
> commonly arise in large C codebases.
> 
> A key architectural distinction is crosvm's process separation model.
> Each device backend is offloaded into its own tightly sandboxed process
> using seccomp filters and Linux namespaces. This “process-per-device”
> layout shrinks the trusted computing base and prevents a compromise in one
> device model from affecting the rest of the system. These isolation
> boundaries align closely with secure-by-design principles and provide a
> more predictable attack surface.
> 
> For embedded, edge, and containerized environments—where minimalism,
> stability, and strict isolation are essential—crosvm offers a compelling
> VMM alternative. By integrating crosvm into meta-virtualization, we allow
> Yocto users to choose between a feature-rich, broad-emulation platform
> (QEMU) and a streamlined, security-hardened VMM (crosvm) best suited for
> KVM-first workloads.
> 
> Patches follow.
> 
> Keerthivasan Raghavan (2):
>   crosvm: add recipe for ChromeOS Virtual Machine Monitor (VMM)
>   crosvm-image-minimal: add a reference image for crosvm demo
> 
>  recipes-devtools/crosvm/crosvm-crates.inc     | 966 ++++++++++++++++++
>  recipes-devtools/crosvm/crosvm_0.1.0.bb       |  57 ++
>  recipes-extended/images/README-crosvm.md      |  68 ++
>  .../images/crosvm-image-minimal.bb            |  68 ++
>  4 files changed, 1159 insertions(+)
>  create mode 100644 recipes-devtools/crosvm/crosvm-crates.inc
>  create mode 100644 recipes-devtools/crosvm/crosvm_0.1.0.bb
>  create mode 100644 recipes-extended/images/README-crosvm.md
>  create mode 100644 recipes-extended/images/crosvm-image-minimal.bb
> 

Re: [PATCH v5 1/2] crosvm: add recipe for ChromeOS Virtual Machine Monitor (VMM)

[Bruce Ashfield]

Hi Keerthivasan,

Inline review of 1/2. Same caveat as on the cover: if any of these are
already addressed in your v1 (which I can't see), please ignore and
let me know.

On Thu, May 28, 2026 at 00:40 +0530, Keerthivasan Raghavan wrote:
> crosvm is a lightweight, Rust-based virtual
> machine monitor originally developed for ChromeOS,
> and provides an alternative to traditional VMMs such
> as QEMU.
> 
> Signed-off-by: Keerthivasan Raghavan <kraghava@qti.qualcomm.com>
> ---
>  recipes-devtools/crosvm/crosvm-crates.inc | 966 ++++++++++++++++++++++
>  recipes-devtools/crosvm/crosvm_0.1.0.bb   |  57 ++
>  2 files changed, 1023 insertions(+)
>  create mode 100644 recipes-devtools/crosvm/crosvm-crates.inc
>  create mode 100644 recipes-devtools/crosvm/crosvm_0.1.0.bb
> 
> diff --git a/recipes-devtools/crosvm/crosvm-crates.inc b/recipes-devtools/crosvm/crosvm-crates.inc
> [... 966 lines of autogenerated crates.inc elided ...]
> diff --git a/recipes-devtools/crosvm/crosvm_0.1.0.bb b/recipes-devtools/crosvm/crosvm_0.1.0.bb
> new file mode 100644
> index 00000000..00b492b0
> --- /dev/null
> +++ b/recipes-devtools/crosvm/crosvm_0.1.0.bb
> @@ -0,0 +1,57 @@
> +SUMMARY = "crosvm is a secure, lightweight, and performant Virtual Machine Monitor (VMM) written in Rust."
> +
> +DESCRIPTION = "\
> +crosvm is a Rust-based virtual machine monitor (VMM) originally developed \
> +for ChromeOS. It uses KVM acceleration and process-level isolation to run guest VMs \
> +with sandboxed device emulation.\
> +"
> +
> +HOMEPAGE = "https://github.com/google/crosvm"
> +
> +LICENSE = "BSD-3-Clause-Clear"
> +LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/BSD-3-Clause-Clear;md5=7a434440b651f4a472ca93716d01033a"
> +
> +inherit cargo pkgconfig cargo-update-recipe-crates features_check
> +
> +SRC_URI = " \
> +    git://chromium.googlesource.com/crosvm/crosvm.git;branch=main;protocol=https;name=crosvm \
> +    git://chromium.googlesource.com/chromiumos/platform/minijail;branch=main;protocol=https;name=minijail;destsuffix=${BB_GIT_DEFAULT_DESTSUFFIX}/third_party/minijail \
> +"
> +
> +SRCREV_crosvm = "b04c13a65b93034010b20612e3566d4f9d83c4f0"
> +SRCREV_minijail = "bfd22f25fd2302fe4ae5121d80c836e0f124e742"
> +
> +SRCREV_FORMAT = "crosvm_minijail"
> +
> +PV = "0.1.0+git"
> +
> +DEPENDS += "libcap wayland wayland-native protobuf-native wayland-protocols"
> +
> +REQUIRED_DISTRO_FEATURES = "kvm"
> +
> +COMPATIBLE_HOST = "(aarch64|x86_64).*-linux.*"
> +
> +BBCLASSEXTEND = "native"
> +
> +# cargo.bbclass adds SRC_URI entries with "name" and "destsuffix" to
> +# ${CARGO_HOME}/config.toml as Cargo patches. It uses the SRC_URI path as-is
> +# and does not account for "subdir".
> +# This can make Cargo try to build a directory that has no Cargo.toml. Remove
> +# the generated entry from ${CARGO_HOME}/config.toml. The crosvm Cargo.toml
> +# already points to the correct patch path relative to the source root.
> +do_filter_minijail_cargo_config() {
> +
> +    cfg="${CARGO_HOME}/config.toml"
> +
> +    if [ ! -f "$cfg" ]; then
> +        bbwarn "Skipping missing Cargo config: $cfg"
> +        exit 0
> +    fi
> +
> +    bbnote "Processing Cargo config: $cfg"
> +    sed -i '/minijail/d' "$cfg"
> +}
> +
> +addtask filter_minijail_cargo_config after do_configure before do_compile
> +
> +require crosvm-crates.inc

A few things:

1. recipes-devtools/ vs recipes-extended/

The other VMMs in this layer (libvirt, xen, xen-tools) all live under
recipes-extended/. recipes-devtools/ in meta-virt is mostly go-mod
tooling (go, go-distribution, etc.). crosvm fits the VMM category, so
recipes-extended/crosvm/ would match convention better.

Not strictly blocking, but worth moving in v2 (or whatever the next
spin is called).

2. do_filter_minijail_cargo_config — root cause, please

The recipe carries this workaround:

> +do_filter_minijail_cargo_config() {
> +    cfg="${CARGO_HOME}/config.toml"
> +    if [ ! -f "$cfg" ]; then
> +        bbwarn "Skipping missing Cargo config: $cfg"
> +        exit 0
> +    fi
> +    bbnote "Processing Cargo config: $cfg"
> +    sed -i '/minijail/d' "$cfg"
> +}
> +addtask filter_minijail_cargo_config after do_configure before do_compile

The comment says cargo.bbclass writes a [patches.crates-io] entry to
${CARGO_HOME}/config.toml using the SRC_URI path as-is, without
honouring the "subdir=" in your minijail SRC_URI entry. So cargo tries
to build a path that has no Cargo.toml and fails.

This concerns me because meta-virt has a number of other cargo
recipes — netavark, aardvark-dns, podlet, fuse-overlayfs — and none
of them carry an equivalent workaround. So either:

  a) they all benefit from the same problem somehow (e.g. their
     SRC_URI entries don't use subdir=), or

  b) they do trigger the same cargo.bbclass behaviour but happen to
     not break because their secondary repos do contain a Cargo.toml
     at the SRC_URI root, or

  c) cargo.bbclass has been fixed in oe-core master since this recipe
     was written, and the workaround is now stale.

Could you dig into the root cause and confirm which of (a/b/c) applies?
If (c), we can just drop the workaround. If (a) or (b), the fix
probably belongs in cargo.bbclass upstream — and we'd carry a
documented local workaround until that lands, rather than a sed-script
postfunc with no commentary on why it's needed only for crosvm.

3. SRC_URI source

> +    git://chromium.googlesource.com/crosvm/crosvm.git;branch=main;protocol=https;name=crosvm \

chromium.googlesource.com is the canonical source. The
github.com/google/crosvm mirror is typically more reliable from a
yocto build infrastructure standpoint (the chromium.org hosts have
periodically been touchy with cgit-style fetches for me in the past).
Did you consider the github mirror? If there's a reason chromium is
preferred (e.g. it gets fixes first, or has signed tags), worth a
one-line comment in the recipe.

4. PV = "0.1.0+git"

> +PV = "0.1.0+git"

crosvm doesn't tag releases, so "0.1.0" is arbitrary. That's fine —
we have precedent in the layer (kvmtool's PV is similarly cosmetic).
Just calling it out so we don't get bug reports asking why we're
shipping a "0.1.0" of a project that has never had a 1.0.

5. BBCLASSEXTEND = "native"

> +BBCLASSEXTEND = "native"

Per offline discussion with you previously, I understand this is for
a future "runqemu"-like runner that can boot a guest from a host
build. Nothing to do today; leave it.

Bruce

Re: [PATCH v5 1/2] crosvm: add recipe for ChromeOS Virtual Machine Monitor (VMM)

[Keerthivasan Raghavan]

Hi Bruce,

> The other VMMs in this layer (libvirt, xen, xen-tools) all live under
> recipes-extended/. recipes-devtools/ in meta-virt is mostly go-mod
> tooling (go, go-distribution, etc.). crosvm fits the VMM category, so
> recipes-extended/crosvm/ would match convention better.

> Not strictly blocking, but worth moving in v2 (or whatever the next
> spin is called).

Ack. Will move it to recipes-extended.

> The comment says cargo.bbclass writes a [patches.crates-io] entry to
> ${CARGO_HOME}/config.toml using the SRC_URI path as-is, without
> honouring the "subdir=" in your minijail SRC_URI entry. So cargo tries
> to build a path that has no Cargo.toml and fails.
>
> This concerns me because meta-virt has a number of other cargo
> recipes — netavark, aardvark-dns, podlet, fuse-overlayfs — and none
> of them carry an equivalent workaround. So either:
>
>   a) they all benefit from the same problem somehow (e.g. their
>      SRC_URI entries don't use subdir=), or
>
>   b) they do trigger the same cargo.bbclass behaviour but happen to
>      not break because their secondary repos do contain a Cargo.toml
>      at the SRC_URI root, or
>
>   c) cargo.bbclass has been fixed in oe-core master since this recipe
>      was written, and the workaround is now stale.
>
> Could you dig into the root cause and confirm which of (a/b/c) applies?
> If (c), we can just drop the workaround. If (a) or (b), the fix
> probably belongs in cargo.bbclass upstream — and we'd carry a
> documented local workaround until that lands, rather than a sed-script
> postfunc with no commentary on why it's needed only for crosvm.

I re-checked this in current trees and the result is: (a)+(b), not (c).

  - cargo_common_do_patch_paths() is still active in oe-core and is run from do_configure (cargo_common.bbclass:207, commit
    684a8af41c5).
  - It only creates Cargo [patch."..."] entries for git/gitsm URLs that have both name and destsuffix (cargo_common.bbclass:149-152,
    commits f871d9d6094 + 684a8af41c5), and writes those entries to config.toml (cargo_common.bbclass:165-167, commit 684a8af41c5).
  - crosvm is exactly in that path: it has an extra git dependency for minijail with name=minijail and destsuffix=...
    (crosvm_0.1.0.bb:18, commit 52c76179d), and therefore gets an auto-generated Cargo patch entry. The local sed filter is removing
    that generated entry (crosvm_0.1.0.bb:36-55, commit 52c76179d).
  - netavark, aardvark-dns, and podlet do not use this problematic secondary-git pattern; they use one top-level git source and crat
    es from crate://... includes:
      - netavark_1.17.2.bb:11-13:11, commits 8e45045c8/8c599baea/498a53037
      - netavark-crates.inc:4-6:4, commits 498a53037/8e45045c8
      - aardvark-dns_1.17.0.bb:11-13:11, commit 508517af3/cd6d4550a
      - aardvark-dns-crates.inc:4-6:4, commits cd6d4550a/c08591fd7
      - podlet_git.bb:9-12:9, commit 7ab697991
      - podlet-crates.inc:4-6:4, commit 7ab697991
  - fuse-overlayfs is not a cargo recipe (inherit autotools pkgconfig) (fuse-overlayfs_1.16.bb:13, commit 19a66be0d).

  Additional conflicting observation (important):

  - oe-core commit e955099a2e added subdir handling by appending it to destsuffix in cargo_common (cargo_common.bbclass:157-159,
    commit e955099a2ee).
  - But BitBake git unpack semantics are destdir = join(destdir, subdir) first, then join(destdir, destsuffix) (git.py:670-681,
    commits 66953f06fe, 3e7f8afeac, 25e0b0bc50).
  - So when both are used, cargo path construction and actual unpack layout still disagree.

  Conclusion: until oe-core lands a proper fix in cargo_common.bbclass, keeping the crosvm sed workaround is justified and should
  stay with explicit rationale in the recipe.

subdir is interpreted differently between the two code paths. In BitBake’s git fetcher, unpack destination is built as <UNPACKDIR>/
<subdir>/<destsuffix> (apply subdir first, then append destsuffix) in git.py:670-681. In cargo_common_do_patch_paths(), the
generated Cargo patch path is built as <UNPACKDIR>/<destsuffix>/<subdir> (destsuffix = os.path.join(destsuffix, subdir)) in
cargo_common.bbclass:157-160.
So the same SRC_URI parameters resolve to different filesystem paths, and Cargo [patch] can point to a location different from
where BitBake actually unpacked the repo.

> chromium.googlesource.com is the canonical source. The
> github.com/google/crosvm mirror is typically more reliable from a
> yocto build infrastructure standpoint (the chromium.org hosts have
> periodically been touchy with cgit-style fetches for me in the past).
> Did you consider the github mirror? If there's a reason chromium is
> preferred (e.g. it gets fixes first, or has signed tags), worth a
> one-line comment in the recipe.

Ack. Will be changed to github url.

> crosvm doesn't tag releases, so "0.1.0" is arbitrary. That's fine —
> we have precedent in the layer (kvmtool's PV is similarly cosmetic).
> Just calling it out so we don't get bug reports asking why we're
> shipping a "0.1.0" of a project that has never had a 1.0.

Ack. Comment will be added.

Thank you,
Keerthivasan Raghavan

Re: [PATCH v5 2/2] crosvm-image-minimal: add a reference image for crosvm demo

[Bruce Ashfield]

Hi Keerthivasan,

Inline review of 2/2. Same caveat as before — if any of these are
addressed in v1, please ignore.

On Thu, May 28, 2026 at 00:40 +0530, Keerthivasan Raghavan wrote:
> A reference image called `crosvm-image-minimal` is provided
> for testing/validation on supported machines(arm64 and x86-64).
> Instructions are provided to test on QEMU(x86-64 only).
> 
> Testing was done on x86-64 by achieving boot to shell on a crosvm vm
> instance running inside a vm instance on QEMU, i.e nested virtualization
> with machine as qemux86-64.
> Testing was done on arm64 by achieving boot to shell on a crosvm vm
> instance running directly on meta-qcom machine qcs9100-ride-sx.
> 
> Signed-off-by: Keerthivasan Raghavan <kraghava@qti.qualcomm.com>
> ---
>  recipes-extended/images/README-crosvm.md      | 68 +++++++++++++++++++
>  .../images/crosvm-image-minimal.bb            | 68 +++++++++++++++++++
>  2 files changed, 136 insertions(+)
>  create mode 100644 recipes-extended/images/README-crosvm.md
>  create mode 100644 recipes-extended/images/crosvm-image-minimal.bb
> 
> diff --git a/recipes-extended/images/README-crosvm.md b/recipes-extended/images/README-crosvm.md
> new file mode 100644
> index 00000000..2c36da7d
> --- /dev/null
> +++ b/recipes-extended/images/README-crosvm.md
> @@ -0,0 +1,68 @@
> +# crosvm-image-minimal: build and test
> +
> +Steps to test/validate crosvm are given below.
> +A reference image `crosvm-image-minimal` is provided
> +for this purpose.
> +
> +## local.conf example
> +
> +```conf
> +# virtualization
> +MACHINE ??= "qemux86-64"
> +DISTRO ??= "nodistro"
> +BBMULTICONFIG ?= ""
> +
> +DISTRO_FEATURES += " kvm virtualization"
> +IMAGE_FEATURES += " empty-root-password allow-empty-password allow-root-login"
> +IMAGE_FSTYPES = "ext4"
> +IMAGE_FSTYPES:remove = "ext4.zst"
> +```
> +
> +## Build
> +
> +```bash
> +bitbake crosvm-image-minimal
> +```
> +
> +## Boot in QEMU
> +
> +When the target `MACHINE` is `qemux86-64`, the host running QEMU must support:
> +- nested virtualization
> +- Intel VT-x or AMD SVM
> +- exposed `/dev/kvm`
> +
> +Run from `tmp/deploy/images/qemux86-64/` (or adjust paths):
> +
> +```bash
> +qemu-system-x86_64 \
> +  -enable-kvm \
> +  -machine q35 \
> +  -cpu host \
> +  -smp 4 \
> +  -m 4096 \
> +  -kernel bzImage \
> +  -drive file=crosvm-image-minimal-qemux86-64.rootfs.ext4,format=raw,if=virtio \
> +  -append "root=/dev/vda rw console=ttyS0 earlyprintk=serial nokaslr" \
> +  -nographic
> +```
> +
> +## Run crosvm inside the guest
> +
> +The image bundles:
> +
> +- `/var/lib/crosvm/images/guest-kernel`
> +- `/var/lib/crosvm/images/guest.ext4`
> +
> +```bash
> +crosvm --log-level "debug,disk=off" run \
> +  --disable-sandbox  /var/lib/crosvm/images/guest-kernel \
> +  --block /var/lib/crosvm/images/guest.ext4,root \
> +  -p "root=/dev/vda rw console=ttyS0 earlyprintk=serial nokaslr"
> +```
> +
> +## Quick sanity checks
> +
> +```bash
> +test -e /dev/kvm && echo "/dev/kvm present"
> +ls -l /var/lib/crosvm/images/
> +```
> diff --git a/recipes-extended/images/crosvm-image-minimal.bb b/recipes-extended/images/crosvm-image-minimal.bb
> new file mode 100644
> index 00000000..de503a99
> --- /dev/null
> +++ b/recipes-extended/images/crosvm-image-minimal.bb
> @@ -0,0 +1,68 @@
> +SUMMARY = "A minimal image containing crosvm to demo its uage."
> +
> +KVM_MODULES = ""
> +
> +KVM_MODULES:x86-64 = " \
> +    kernel-module-kvm \
> +    kernel-module-kvm-intel \
> +    kernel-module-kvm-amd \
> +"
> +
> +IMAGE_INSTALL = "packagegroup-core-boot crosvm"
> +IMAGE_INSTALL:append = " ${KVM_MODULES}"
> +
> +IMAGE_LINGUAS = " "
> +
> +LICENSE = "MIT"
> +
> +inherit core-image
> +
> +# Guest artifact source settings (can be overridden in local.conf).
> +CROSVM_GUEST_IMAGE_RECIPE ?= "core-image-minimal"
> +CROSVM_GUEST_IMAGE_FSTYPE ?= "ext4"
> +CROSVM_GUEST_ROOTFS ?= "${CROSVM_GUEST_IMAGE_RECIPE}-${MACHINE}.rootfs.${CROSVM_GUEST_IMAGE_FSTYPE}"
> +CROSVM_GUEST_KERNEL ?= "${KERNEL_IMAGETYPE}"
> +
> +# Final location inside the host/root image.
> +CROSVM_GUEST_TARGET_DIR ?= "/var/lib/crosvm/images"
> +CROSVM_GUEST_TARGET_ROOTFS_NAME ?= "guest.${CROSVM_GUEST_IMAGE_FSTYPE}"
> +CROSVM_GUEST_TARGET_KERNEL_NAME ?= "guest-kernel"
> +
> +# Ensure guest image and kernel are available before rootfs postprocess runs.
> +# The same kernel is used for both the host and guest.
> +do_rootfs[depends] += "${CROSVM_GUEST_IMAGE_RECIPE}:do_image_complete virtual/kernel:do_deploy"
> +
> +# Copy guest rootfs+kernel into the host rootfs for crosvm demo runtime.
> +bundle_crosvm_guest() {
> +    set -e
> +
> +    local deploy_dir="${DEPLOY_DIR_IMAGE}"
> +    local rootfs_src="${deploy_dir}/${CROSVM_GUEST_ROOTFS}"
> +    local kernel_src="${deploy_dir}/${CROSVM_GUEST_KERNEL}"
> +    local target_dir="${IMAGE_ROOTFS}${CROSVM_GUEST_TARGET_DIR}"
> +
> +    if [ ! -e "$rootfs_src" ]; then
> +        bbfatal "Cannot find guest rootfs. Tried: \
> +                ${deploy_dir}/${CROSVM_GUEST_ROOTFS} and \
> +                ${deploy_dir}/${CROSVM_GUEST_IMAGE_RECIPE}-${MACHINE}.rootfs.${CROSVM_GUEST_IMAGE_FSTYPE}"
> +    fi
> +
> +    if [ ! -e "$kernel_src" ]; then
> +        bbfatal "Cannot find guest kernel. Tried: \
> +                ${deploy_dir}/${CROSVM_GUEST_KERNEL}"
> +    fi
> +
> +    install -d "$target_dir"
> +
> +    install -m 0644 "$(readlink -f "$rootfs_src")" \
> +                    "$target_dir/${CROSVM_GUEST_TARGET_ROOTFS_NAME}"
> +
> +    install -m 0644 "$(readlink -f "$kernel_src")" \
> +                    "$target_dir/${CROSVM_GUEST_TARGET_KERNEL_NAME}"
> +}
> +
> +ROOTFS_POSTPROCESS_COMMAND += "bundle_crosvm_guest;"
> +
> +IMAGE_ROOTFS_SIZE = "0"
> +
> +KERNEL_MODULE_AUTOLOAD:append:x86-64 = " kvm kvm_amd kvm_intel"

A few observations:

1. README is appreciated; I asked for this on v4 and it's here.
   Skimmed it, looks fine. Will check it more carefully once I'm
   actually building.

2. Test coverage gap

> Testing was done on x86-64 by achieving boot to shell on a crosvm vm
> instance running inside a vm instance on QEMU, i.e nested virtualization
> with machine as qemux86-64.
> Testing was done on arm64 by achieving boot to shell on a crosvm vm
> instance running directly on meta-qcom machine qcs9100-ride-sx.

Good progress over v4 — x86_64 nested under qemu lets me actually try
it locally, which is what I needed. Thank you for that.

The remaining gap is qemuarm64. I can't reproduce qcs9100-ride-sx
without the hardware, so the arm64 path is effectively un-testable
from my side right now. Did you try qemuarm64 with nested
virtualization (the way you did qemux86-64), and if so what
happened? Even a "I tried, it didn't boot because X" would tell me
whether we need to ask upstream for qemuarm64 support or whether it
just needs some Yocto config wiring.

3. Image recipe — bundle_crosvm_guest postprocess

> +# Ensure guest image and kernel are available before rootfs postprocess runs.
> +# The same kernel is used for both the host and guest.
> +do_rootfs[depends] += "${CROSVM_GUEST_IMAGE_RECIPE}:do_image_complete virtual/kernel:do_deploy"

> +bundle_crosvm_guest() {
> +    set -e
[...]
> +    install -m 0644 "$(readlink -f "$rootfs_src")" \
> +                    "$target_dir/${CROSVM_GUEST_TARGET_ROOTFS_NAME}"
[...]
> +ROOTFS_POSTPROCESS_COMMAND += "bundle_crosvm_guest;"

This is the same pattern we use elsewhere (container-cross-install
does similar bundle-into-rootfs work). Worth being aware that
DEPLOY_DIR_IMAGE-as-input has the sstate sharp-edge we documented
recently — the recipe that produces ${CROSVM_GUEST_ROOTFS} can hit
sstate while its deployed output is stale from a prior run, and
then this image hits sstate with the stale guest baked in. Same
class of bug we fixed for vcontainer-tarball with explicit
mcdepends on rootfs-image:do_image_complete (commit 6fc1aa83
on master).

Not necessarily blocking — the do_rootfs[depends] line is doing
the right thing for ordering — but worth being aware of for when
this becomes a CI build and stale-guest reports start appearing.

4. KVM modules layout

> +KVM_MODULES = ""
> +
> +KVM_MODULES:x86-64 = " \
> +    kernel-module-kvm \
> +    kernel-module-kvm-intel \
> +    kernel-module-kvm-amd \
> +"
[...]
> +KERNEL_MODULE_AUTOLOAD:append:x86-64 = " kvm kvm_amd kvm_intel"

x86_64 is handled. arm64 is empty (KVM_MODULES = "" with no
:aarch64 override). On arm64, KVM is built into the kernel and
not a loadable module on most boards — so an empty list is right.
But should we add explicit a :aarch64 = "" comment or some
documentation so the reader doesn't wonder if it's a bug?

5. CROSVM_GUEST_IMAGE_RECIPE default

> +CROSVM_GUEST_IMAGE_RECIPE ?= "core-image-minimal"

Sensible default. The PARSE-TIME do_rootfs[depends] += that uses
this variable means changing it in local.conf actually works
correctly (because += happens at parse time, before the dep is
read). Worth a comment in the recipe to that effect so users know
they can override without surprises.

Bruce

Re: [meta-virtualization] [PATCH v5 2/2] crosvm-image-minimal: add a reference image for crosvm demo

[Keerthivasan Raghavan]

Hi Bruce,

> The remaining gap is qemuarm64. I can't reproduce qcs9100-ride-sx
> without the hardware, so the arm64 path is effectively un-testable
> from my side right now. Did you try qemuarm64 with nested
> virtualization (the way you did qemux86-64), and if so what
> happened? Even a "I tried, it didn't boot because X" would tell me
> whether we need to ask upstream for qemuarm64 support or whether it
> just needs some Yocto config wiring.

Nested virtualization via qemu64 was not supported on the native arm64 server I had access to nor on the qcs9100-rise-sx device. With x86-64 host running an arm64 guest which again runs a vm inside it, the setup is extremely slow, it did not boot after 8+ hrs wait time. The only sane solution given these constraints was to test on physical hw if the guestvm would boot. You are right in stating upstream qemu64 needs to do something about providing nested virtualization support that is moderately fast for arm64. Both the physical arm64 device and the arm64 server I had access to did not support native kvm accl(there was cpu support but qemu backed out) forcing me to use tcg on x86-64 which did not boot the 2nd guest(using tcg on arm64 yields the same result), its too slow or rather just stuck when trying to boot the 2nd guest from within the 1st guest. It may benefit to discuss this in quemu forums ?

The qemu cmdline I used on an x86-64 host:

qemu-system-aarch64 -cpu cortex-a57 -machine virt,virtualization=on -smp 8 -m 16376 -kernel Image -drive file=crosvm-image-minimal-qemuarm64.rootfs.ext4,format=raw,if=none,id=rootfs -device virtio-blk-pci,drive=rootfs -append "root=/dev/vda rw rootwait console=ttyAMA0 earlycon" -nographic

Last few logs(if it helps) when booting the 2nd vm using crosvm from the 1st vm:

[2026-05-26T10:04:34.465695943+00:00 DEBUG devices::usb::xhci::device_slot] resetting device slot 1!
[2026-05-26T10:04:34.467123732+00:00 DEBUG devices::usb::xhci::device_slot] resetting device slot 2!
[2026-05-26T10:04:34.468690776+00:00 DEBUG devices::usb::xhci::device_slot] resetting device slot 3!
[2026-05-26T10:04:34.469804614+00:00 DEBUG devices::usb::xhci::device_slot] resetting device slot 4!
[2026-05-26T10:04:34.471032068+00:00 DEBUG devices::usb::xhci::device_slot] resetting device slot 5!
[2026-05-26T10:04:34.472381430+00:00 DEBUG devices::usb::xhci::device_slot] resetting device slot 6!
[2026-05-26T10:04:34.473522822+00:00 DEBUG devices::usb::xhci::device_slot] resetting device slot 7!
[2026-05-26T10:04:34.474818881+00:00 DEBUG devices::usb::xhci::device_slot] resetting device slot 8!
[2026-05-26T10:04:34.476900802+00:00 DEBUG devices::usb::xhci::device_slot] resetting device slot 9!
[2026-05-26T10:04:34.478251681+00:00 DEBUG devices::usb::xhci::device_slot] resetting device slot 10!
[2026-05-26T10:04:34.479289071+00:00 DEBUG devices::usb::xhci::device_slot] resetting device slot 11!
[2026-05-26T10:04:34.480722260+00:00 DEBUG devices::usb::xhci::device_slot] resetting device slot 12!
[2026-05-26T10:04:34.482246479+00:00 DEBUG devices::usb::xhci::device_slot] resetting device slot 13!
[2026-05-26T10:04:34.483322573+00:00 DEBUG devices::usb::xhci::device_slot] resetting device slot 14!
[2026-05-26T10:04:34.484896724+00:00 DEBUG devices::usb::xhci::device_slot] resetting device slot 15!
[2026-05-26T10:04:34.486036774+00:00 DEBUG devices::usb::xhci::device_slot] resetting device slot 16!
[2026-05-26T10:04:34.532898423+00:00 DEBUG devices::usb::xhci] xhci_controller: halt device
[2026-05-26T10:04:34.534384706+00:00 DEBUG devices::usb::xhci] xhci_controller: interrupter enable?: false
[2026-05-26T10:04:34.538624229+00:00 DEBUG devices::usb::xhci] xhci_controller: clear halt bits
[2026-05-26T10:04:34.540089300+00:00 DEBUG devices::usb::xhci] xhci_controller: interrupter enable?: true
[28761.296705] hrtimer: interrupt took 9440429 ns
[635866.980323] sched: DL replenish lagged too much
Killed

Note: I killed the process after giving up on wait time.

qemu cmd line I used on arm64 server/device:

qemu-system-aarch64   -machine virt,accel=kvm,gic-version=host,virtual
ization=on   -cpu host   -smp 4   -m 4096   -kernel Image   -drive file=crosvm-image-minimal-qemuarm64.rootfs.ext4,format=raw,if=none
,id=rootfs   -device virtio-blk-pci,drive=rootfs   -append "root=/dev/vda rw rootwait console=ttyAMA0 earlycon"   -nographic

Logs if it helps:

qemu-system-aarch64: mach-virt: KVM does not support providing Virtualization extensions to the guest CPU

> x86_64 is handled. arm64 is empty (KVM_MODULES = "" with no
> :aarch64 override). On arm64, KVM is built into the kernel and
> not a loadable module on most boards — so an empty list is right.
> But should we add explicit a :aarch64 = "" comment or some
> documentation so the reader doesn't wonder if it's a bug?

Ack. Will add the explicit override for arm64/aarch64.

> Sensible default. The PARSE-TIME do_rootfs[depends] += that uses
> this variable means changing it in local.conf actually works
> correctly (because += happens at parse time, before the dep is
> read). Worth a comment in the recipe to that effect so users know
> they can override without surprises.

Ack. Will add the comment in the bb recipe.

Thank you,
Keerthivasan Raghavan