Re: [RFC v4 00/24] Add Realm support to QEMU-VMM

[Mathieu Poirier <mathieu.poirier@linaro.org>]

Good day,

On Mon, Sep 21, 2026 at 02:05:19PM +1000, Gavin Shan wrote:
> Hi Mathieu,
> 
> On 9/4/26 5:35 AM, Mathieu Poirier wrote:
> > This patchset provides minimal functionality to start a Realm VM
> > from an Arm RME capable host using the following command line:
> > 
> > qemu-system-aarch64 \
> >   -M confidential-guest-support=rme0,memory-backend=ram0 \
> >   -object rme-guest,id=rme0,convert-in-place=on \
> >   -object memory-backend-guest-memfd,id=ram0,size=1G,share=on \
> >   -cpu host -M virt -enable-kvm -M gic-version=3,its=on -nodefaults ..
> > 
> > It is a refactoring of Jean-Philippe Brucker's initial work dating from a while
> > back.  It is compatible with Steven Price's v16 revision [1] of his work adding
> > CCA support to KVM.
> > 
> > This revision is based on Michael Roth's work on guest memfd in-place memory
> > conversion [2], rebased to QEMU master.  I left out the AMD specific changes
> > from Michael's work because I can't test the result.  Other than the
> > modications to the linux headers, this series is identical to V3.
> > 
> > It was tested on the QEMU SBSA machine.  For convenience, a repository is hosted
> > here [3], along with the TF-A [4] and RMM [5] for the SBSA machine (compatible
> > with Steven's v16 patchset).  The Repository for the kernel [6] is also
> > available.  Note that for the latter, defines for
> > KVM_CAP_GUEST_MEMFD_MEMORY_ATTRIBUTES and KVM_CAP_ARM_RMI have been modified to
> > follow QEMU master.
> > 
> > Instructions to compile and run the entire stack can be found here [7].
> > 
> > Device Assignment is not included.
> > 
> > Thanks,
> > Mathieu
> > 
> > [1]. https://lore.kernel.org/kvm/20260803134403.80630-1-steven.price@arm.com/
> > [2]. https://lore.kernel.org/kvm/20260528000416.8161-1-michael.roth@amd.com/
> > [3]. https://gitlab.com/Linaro/cca-public/qemu/-/tree/upstream-v4?ref_type=heads
> > [4]. https://gitlab.com/Linaro/cca-public/tf-a/trusted-firmware-a/-/tree/cca/v13?ref_type=heads
> > [5]. https://gitlab.com/Linaro/cca-public/rmm/-/tree/cca/v16?ref_type=heads
> > [6]. https://gitlab.com/Linaro/cca-public/linux/-/tree/upstream-v4?ref_type=heads
> > [7]. https://gitlab.com/Linaro/cca-public/build-instructions
> > 
> 
> Give this series and (v19) arm/cca host series [1] a try, and found the following
> issues. I'm sharing the found issues that I need take chance to look at them
> closely in future.

Thank you for taking the time.

> 
> [1] https://git.gitlab.arm.com/linux-arm/linux-cca.git  (branch: cca/cca-host/kvm-v19/integration)
> 
> Note that the host is emulated using qemu (TCG mode) via 'virt' platform, using
> the following command lines

I am currently focused on the SBSA machine but will try to reproduce your
manipulations using 'upstream-v5' that I just posted.

Note that upstream-v5 doesn't address any of the comments you posted today.
Since upstream-v4 was based on v16, I wanted to send another revision that is
based on the very latest (v21) to keep pace with Suzuki and Steven.

I am off to Linux Plumbers next week but rest assured that I will look into your
comments (and the content herein) when I'm back.  

Thanks,
Mathieu

> 
> laptop$ /home/gshan/sandbox/cca/host/qemu/build/qemu-system-aarch64                 \
>         -M virt,virtualization=on,secure=on,gic-version=3,acpi=off                  \
>         -cpu max,x-rme=on -m 4G -smp 8                                              \
>         -serial mon:stdio -monitor none -nographic -nodefaults                      \
>         -bios /home/gshan/sandbox/cca/host/tf-a/flash.bin                           \
>         -kernel /home/gshan/sandbox/cca/host/linux/arch/arm64/boot/Image            \
>         -initrd /home/gshan/sandbox/cca/host/buildroot/output/images/rootfs.cpio.xz \
>         -append kvm-arm.mode=rmm                                                    \
>         -device pcie-root-port,bus=pcie.0,chassis=1,id=pcie.1                       \
>         -device pcie-root-port,bus=pcie.0,chassis=2,id=pcie.2                       \
>         -device pcie-root-port,bus=pcie.0,chassis=3,id=pcie.3                       \
>         -device pcie-root-port,bus=pcie.0,chassis=4,id=pcie.4                       \
>         -device virtio-9p-device,fsdev=shr0,mount_tag=shr0                          \
>         -fsdev local,security_model=none,path=/home/gshan/sandbox/cca/guest,id=shr0 \
>         -netdev tap,id=tap1,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown       \
>         -device virtio-net-pci,bus=pcie.2,netdev=tap1,mac=b8:3f:d2:1d:3e:80
> 
> The realm VM is started using the following command lines.
> 
> host$ qemu-system-aarch64 -enable-kvm -object rme-guest,id=rme0,convert-in-place=on \
>       -machine virt,gic-version=3,confidential-guest-support=rme0 -m 2G -cpu host   \
>       -smp maxcpus=4,cpus=4,sockets=1,clusters=1,cores=2,threads=2                  \
>       -object memory-backend-guest-memfd,id=mem0,size=2G,share=on                   \
>       -numa node,nodeid=0,cpus=0-3,memdev=mem0                                      \
>       -serial mon:stdio -monitor none -nographic -nodefaults                        \
>       -kernel /mnt/linux/arch/arm64/boot/Image                                      \
>       -initrd /mnt/buildroot/output/images/rootfs.cpio.xz                           \
>       -append earlycon=pl011,mmio,0x10009000000
> 
> Found issues
> ============
> 
> (1) qemu is terminiated on qmp command 'dump-guest-memory'
> 
> (qemu) dump-guest-memory -p "/tmp/mem" 0x40000000 0x10000000
> **
> ERROR:../system/memory_mapping.c:215:guest_phys_block_add_section: assertion failed: (predecessor->target_end <= target_start)
> Bail out! ERROR:../system/memory_mapping.c:215:guest_phys_block_add_section: assertion failed: (predecessor->target_end <= target_start)
> 
> (2) qemu emulation can't be stopped and resumed
> 
> (qemu) stop
> (qemu) cont
> Failed to set KVM_REG_ARM_TIMER_CNT
> 
> (3) qemu is terminiated on qmp command 'system_reset'
> 
> (qemu) system_reset
> cpus are not resettable, terminating
> 
> (4) 'reboot' from the guest doesn't work
> 
> root@guest:~# reboot
>    :
> Requesting system reboot
> [  123.264893] reboot: Restarting system
> qemu-system-aarch64: cpus are not resettable, terminating
> 
> (5) Incompatible to virtio-iommu-pci. The realm guest doesn't boot up if a virtio-iommu-pci
>     device is specified in the command lines
> 
> host$ qemu-system-aarch64 -enable-kvm                                          \
> -object rme-guest,id=rme0,convert-in-place=on                                  \
> -machine virt,gic-version=3,confidential-guest-support=rme0                    \
> -m 2G -cpu host -smp maxcpus=4,cpus=4,sockets=1,clusters=1,cores=2,threads=2   \
> -object memory-backend-guest-memfd,id=mem0,size=2G,share=on                    \
> -numa node,nodeid=0,cpus=0-3,memdev=mem0                                       \
> -serial mon:stdio -monitor none -nographic -nodefaults                         \
> -kernel /mnt/linux/arch/arm64/boot/Image                                       \
> -initrd /mnt/buildroot/output/images/rootfs.cpio.xz                            \
> -append earlycon=pl011,mmio,0x10009000000                                      \
> -device pcie-root-port,bus=pcie.0,chassis=1,id=pcie.1                          \
> -device pcie-root-port,bus=pcie.0,chassis=2,id=pcie.2                          \
> -device virtio-iommu-pci,bus=pcie.0                                            \
> -drive file=/mnt/disk.qcow2,if=none,id=drive0                                  \
> -device virtio-blk-pci,id=virtblk0,bus=pcie.1,drive=drive0,num-queues=4        \
> -netdev tap,id=tap1,vhost=on,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown \
> -device virtio-net-pci,bus=pcie.2,netdev=tap1,mac=b8:3f:d2:1d:3e:c0
>      :
> [    0.000000] Booting Linux on physical CPU 0x0000000000 [0x000f0510]
> [    0.000000] Linux version 7.2.0-rc5-gavin-gf5098b6bae76 (gshan@nvidia-grace-hopper-01.khw.eng.bos2.dc.redhat.com) (gcc (GCC) 14.3.1 20251022 (Red Hat 14.3.1-4), GNU ld version 2.41-65.el10) #47 SMP PREEMPT Mon Jul 27 02:29:03 EDT 2026
> [    0.000000] KASLR enabled
> [    0.000000] Machine model: linux,dummy-virt
> [    0.000000] earlycon: pl11 at MMIO 0x0000010009000000 (options '')
> [    0.000000] printk: legacy bootconsole [pl11] enabled
> [    0.000000] efi: UEFI not found.
> [    0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT
> [    0.000000] NODE_DATA(0) allocated [mem 0xbfbecb00-0xbfbf09bf]
> [    0.000000] psci: probing for conduit method from DT.
> [    0.000000] psci: PSCIv1.1 detected in firmware.
> [    0.000000] psci: Using standard PSCI v0.2 function IDs
> [    0.000000] psci: MIGRATE_INFO_TYPE not supported.
> [    0.000000] psci: SMC Calling Convention v1.2
> [    0.000000] RME: Using RSI version 1.0
> [    0.000000] Zone ranges:
> [    0.000000]   DMA      [mem 0x0000000040000000-0x00000000bfffffff]
> [    0.000000]   DMA32    empty
> [    0.000000]   Normal   empty
> [    0.000000] Movable zone start for each node
>      :
> [    6.664423] Serial: AMBA PL011 UART driver
> [    7.151312] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1
> [    7.213181] printk: console [ttyAMA0] enabled
> [    8.536454] HugeTLB: registered 1.00 Gi
> <...The realm guest stops here, no output since then...>
> 
> Thanks,
> Gavin
>