From: Mathieu Poirier <mathieu.poirier@linaro.org>
To: Kohei Enju <enju.kohei@fujitsu.com>
Cc: Gavin Shan <gshan@redhat.com>,
berrange@redhat.com, kchamart@redhat.com,
pierrick.bouvier@oss.qualcomm.com, peter.maydell@linaro.org,
mst@redhat.com, cohuck@redhat.com, pbonzini@redhat.com,
eblake@redhat.com, armbru@redhat.com,
lorenzo.pieralisi@linaro.org, qemu-devel@nongnu.org,
qemu-arm@nongnu.org, kvm@vger.kernel.org
Subject: Re: [RFC v4 00/24] Add Realm support to QEMU-VMM
Date: Tue, 17 Nov 2026 15:18:52 -0600 [thread overview]
Message-ID: <avzEvGXSEcgVdytB@p14s> (raw)
In-Reply-To: <avLM3UdwUcgUHLgP@FCCLS0092175.localdomain>
On Tue, Nov 10, 2026 at 05:01:14PM +0900, Kohei Enju wrote:
> On 10/23 14:23, Mathieu Poirier wrote:
> > On Fri, Oct 23, 2026 at 09:00:12PM +1000, Gavin Shan wrote:
> > > Hi Mathieu,
> > >
> > > On 10/22/26 7:03 AM, Mathieu Poirier wrote:
> > > > Hi Gavin,
> > > >
> > > > On Mon, Sep 21, 2026 at 02:05:19PM +1000, Gavin Shan wrote:
> > > > > Hi Mathieu,
> > > > >
> > > > > On 9/4/26 5:35 AM, Mathieu Poirier wrote:
> > > > > > This patchset provides minimal functionality to start a Realm VM
> > > > > > from an Arm RME capable host using the following command line:
> > > > > >
> > > > > > qemu-system-aarch64 \
> > > > > > -M confidential-guest-support=rme0,memory-backend=ram0 \
> > > > > > -object rme-guest,id=rme0,convert-in-place=on \
> > > > > > -object memory-backend-guest-memfd,id=ram0,size=1G,share=on \
> > > > > > -cpu host -M virt -enable-kvm -M gic-version=3,its=on -nodefaults ..
> > > > > >
> > > > > > It is a refactoring of Jean-Philippe Brucker's initial work dating from a while
> > > > > > back. It is compatible with Steven Price's v16 revision [1] of his work adding
> > > > > > CCA support to KVM.
> > > > > >
> > > > > > This revision is based on Michael Roth's work on guest memfd in-place memory
> > > > > > conversion [2], rebased to QEMU master. I left out the AMD specific changes
> > > > > > from Michael's work because I can't test the result. Other than the
> > > > > > modications to the linux headers, this series is identical to V3.
> > > > > >
> > > > > > It was tested on the QEMU SBSA machine. For convenience, a repository is hosted
> > > > > > here [3], along with the TF-A [4] and RMM [5] for the SBSA machine (compatible
> > > > > > with Steven's v16 patchset). The Repository for the kernel [6] is also
> > > > > > available. Note that for the latter, defines for
> > > > > > KVM_CAP_GUEST_MEMFD_MEMORY_ATTRIBUTES and KVM_CAP_ARM_RMI have been modified to
> > > > > > follow QEMU master.
> > > > > >
> > > > > > Instructions to compile and run the entire stack can be found here [7].
> > > > > >
> > > > > > Device Assignment is not included.
> > > > > >
> > > > > > Thanks,
> > > > > > Mathieu
> > > > > >
> > > > > > [1]. https://lore.kernel.org/kvm/20260803134403.80630-1-steven.price@arm.com/
> > > > > > [2]. https://lore.kernel.org/kvm/20260528000416.8161-1-michael.roth@amd.com/
> > > > > > [3]. https://gitlab.com/Linaro/cca-public/qemu/-/tree/upstream-v4?ref_type=heads
> > > > > > [4]. https://gitlab.com/Linaro/cca-public/tf-a/trusted-firmware-a/-/tree/cca/v13?ref_type=heads
> > > > > > [5]. https://gitlab.com/Linaro/cca-public/rmm/-/tree/cca/v16?ref_type=heads
> > > > > > [6]. https://gitlab.com/Linaro/cca-public/linux/-/tree/upstream-v4?ref_type=heads
> > > > > > [7]. https://gitlab.com/Linaro/cca-public/build-instructions
> > > > > >
> > > > >
> > > > > Give this series and (v19) arm/cca host series [1] a try, and found the following
> > > > > issues. I'm sharing the found issues that I need take chance to look at them
> > > > > closely in future.
> > > > >
> > > > > [1] https://git.gitlab.arm.com/linux-arm/linux-cca.git (branch: cca/cca-host/kvm-v19/integration)
> > > > >
> > > > > Note that the host is emulated using qemu (TCG mode) via 'virt' platform, using
> > > > > the following command lines
> > > > >
> > > > > laptop$ /home/gshan/sandbox/cca/host/qemu/build/qemu-system-aarch64 \
> > > > > -M virt,virtualization=on,secure=on,gic-version=3,acpi=off \
> > > > > -cpu max,x-rme=on -m 4G -smp 8 \
> > > > > -serial mon:stdio -monitor none -nographic -nodefaults \
> > > > > -bios /home/gshan/sandbox/cca/host/tf-a/flash.bin \
> > > > > -kernel /home/gshan/sandbox/cca/host/linux/arch/arm64/boot/Image \
> > > > > -initrd /home/gshan/sandbox/cca/host/buildroot/output/images/rootfs.cpio.xz \
> > > > > -append kvm-arm.mode=rmm \
> > > > > -device pcie-root-port,bus=pcie.0,chassis=1,id=pcie.1 \
> > > > > -device pcie-root-port,bus=pcie.0,chassis=2,id=pcie.2 \
> > > > > -device pcie-root-port,bus=pcie.0,chassis=3,id=pcie.3 \
> > > > > -device pcie-root-port,bus=pcie.0,chassis=4,id=pcie.4 \
> > > > > -device virtio-9p-device,fsdev=shr0,mount_tag=shr0 \
> > > > > -fsdev local,security_model=none,path=/home/gshan/sandbox/cca/guest,id=shr0 \
> > > > > -netdev tap,id=tap1,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown \
> > > > > -device virtio-net-pci,bus=pcie.2,netdev=tap1,mac=b8:3f:d2:1d:3e:80
> > > > >
> > > > > The realm VM is started using the following command lines.
> > > > >
> > > > > host$ qemu-system-aarch64 -enable-kvm -object rme-guest,id=rme0,convert-in-place=on \
> > > > > -machine virt,gic-version=3,confidential-guest-support=rme0 -m 2G -cpu host \
> > > > > -smp maxcpus=4,cpus=4,sockets=1,clusters=1,cores=2,threads=2 \
> > > > > -object memory-backend-guest-memfd,id=mem0,size=2G,share=on \
> > > > > -numa node,nodeid=0,cpus=0-3,memdev=mem0 \
> > > > > -serial mon:stdio -monitor none -nographic -nodefaults \
> > > > > -kernel /mnt/linux/arch/arm64/boot/Image \
> > > > > -initrd /mnt/buildroot/output/images/rootfs.cpio.xz \
> > > > > -append earlycon=pl011,mmio,0x10009000000
> > > > >
> > > > > Found issues
> > > > > ============
> > > >
> > > > I was able to do the same manipulations on my side, but using the SBSA machine
> > > > for system emulation, with results as described below:
> > > >
> > >
> > > Thanks for double check on your side. Just a head up: I will re-run the broken test
> > > cases with (RFCv5) qemu series and share the result. I expect we will have more
> > > discussions on those broken test cases under RFCv5 series if you agree. I don't
> > > know when I get a chance to try RFCv5 series, but hopefully next week or early
> > > of the week after that.
> > >
> >
> > I omitted to mention that I used RFCv5 for my manipulations on SBSA, so I expect
> > you'll find the same problems as with RFCv4. No need to rush to do the test
> > since my next revision will _not_ address any of those issues. For RFCv6 I want
> > to concentrate on the other comments you and Kohei made and keep the bigger
> > issues (the ones you raised here) for future revision. That way we make progess
> > on this patchset and keep synchronized with the KVM patches Suzuki and Steven
> > release.
> >
> > We can discuss further at the KVM forum if you are there.
> >
> > > Cheers,
> > > Gavin
> > >
> > > > >
> > > > > (1) qemu is terminiated on qmp command 'dump-guest-memory'
> > > > >
> > > > > (qemu) dump-guest-memory -p "/tmp/mem" 0x40000000 0x10000000
> > > > > **
> > > > > ERROR:../system/memory_mapping.c:215:guest_phys_block_add_section: assertion failed: (predecessor->target_end <= target_start)
> > > > > Bail out! ERROR:../system/memory_mapping.c:215:guest_phys_block_add_section: assertion failed: (predecessor->target_end <= target_start)
> > > > >
> > > >
> > > > On my side this command causes the Realm to abort, as I expected it to be.
>
> Hi Mathieu, Gavin.
>
> I've tested RFC v5, and I reproduced issue (1) on my side as well:
>
> (qemu) dump-guest-memory -p "/tmp/mem" 0x40000000 0x10000000
> Failed to set KVM_REG_ARM_TIMER_CNT
> ./qemu-ram-realm.sh: line 21: 77580 Aborted (core dumped) $QEMU -no-reboot -M confidential-guest-support=rme0,memory-backend=ram0 ...
>
> This was resolved by adding kvm-no-adjvtime=on.
>
> > > > Realm memory should not be observable from the host. There may be a debug mode
> > > > to enable this kind of operation but I will have to look at the documentation
> > > > for details.
> > > > > (2) qemu emulation can't be stopped and resumed
> > > > >
> > > > > (qemu) stop
> > > > > (qemu) cont
> > > > > Failed to set KVM_REG_ARM_TIMER_CNT
> > > > >
> > > >
> > > > That too I was expecting. Anything that can comprimise a Realm will cause the
> > > > HW to abort. I'll investigate the "Failed to set KVM_REG_ARM_TIMER_CNT" error
> > > > message though.
>
> kvm-no-adjvtime=on also resolved issue (2) above.
>
> Setting KVM_REG_ARM_TIMER_CNT for Realm VMs is not allowed, so we should
> either add this option or consider making it the default for Realm VMs
> in the future. What do you think?
Apologies for the late reply, I was away. I'll look into the
'kvm-no-adjvtime=on' tomorrow and will get back to you.
Many thanks for the input,
Mathieu
>
> Thanks,
> Kohei
>
> > > >
> > > > > (3) qemu is terminiated on qmp command 'system_reset'
> > > > >
> > > > > (qemu) system_reset
> > > > > cpus are not resettable, terminating
> > > > >
> > > > > (4) 'reboot' from the guest doesn't work
> > > > >
> > > > > root@guest:~# reboot
> > > > > :
> > > > > Requesting system reboot
> > > > > [ 123.264893] reboot: Restarting system
> > > > > qemu-system-aarch64: cpus are not resettable, terminating
> > > > >
> > > >
> > > > I suspect 'system_reset' from the monitor and a 'reboot' from the command line
> > > > will, at some point, end up following the same sequence. I need to think about
> > > > the 'reboot' use case further.
> > > >
> > > >
> > > > > (5) Incompatible to virtio-iommu-pci. The realm guest doesn't boot up if a virtio-iommu-pci
> > > > > device is specified in the command lines
> > > > >
> > > > > host$ qemu-system-aarch64 -enable-kvm \
> > > > > -object rme-guest,id=rme0,convert-in-place=on \
> > > > > -machine virt,gic-version=3,confidential-guest-support=rme0 \
> > > > > -m 2G -cpu host -smp maxcpus=4,cpus=4,sockets=1,clusters=1,cores=2,threads=2 \
> > > > > -object memory-backend-guest-memfd,id=mem0,size=2G,share=on \
> > > > > -numa node,nodeid=0,cpus=0-3,memdev=mem0 \
> > > > > -serial mon:stdio -monitor none -nographic -nodefaults \
> > > > > -kernel /mnt/linux/arch/arm64/boot/Image \
> > > > > -initrd /mnt/buildroot/output/images/rootfs.cpio.xz \
> > > > > -append earlycon=pl011,mmio,0x10009000000 \
> > > > > -device pcie-root-port,bus=pcie.0,chassis=1,id=pcie.1 \
> > > > > -device pcie-root-port,bus=pcie.0,chassis=2,id=pcie.2 \
> > > > > -device virtio-iommu-pci,bus=pcie.0 \
> > > > > -drive file=/mnt/disk.qcow2,if=none,id=drive0 \
> > > > > -device virtio-blk-pci,id=virtblk0,bus=pcie.1,drive=drive0,num-queues=4 \
> > > > > -netdev tap,id=tap1,vhost=on,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown \
> > > > > -device virtio-net-pci,bus=pcie.2,netdev=tap1,mac=b8:3f:d2:1d:3e:c0
> > > >
> > > >
> > > > I added the following to my Realm's boot command parameters:
> > > >
> > > > -device pcie-root-port,bus=pcie.0,chassis=1,id=pcie.1 \
> > > > -device virtio-iommu-pci,bus=pcie.0 \
> > > >
> > > >
> > > > > :
> > > > > [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x000f0510]
> > > > > [ 0.000000] Linux version 7.2.0-rc5-gavin-gf5098b6bae76 (gshan@nvidia-grace-hopper-01.khw.eng.bos2.dc.redhat.com) (gcc (GCC) 14.3.1 20251022 (Red Hat 14.3.1-4), GNU ld version 2.41-65.el10) #47 SMP PREEMPT Mon Jul 27 02:29:03 EDT 2026
> > > > > [ 0.000000] KASLR enabled
> > > > > [ 0.000000] Machine model: linux,dummy-virt
> > > > > [ 0.000000] earlycon: pl11 at MMIO 0x0000010009000000 (options '')
> > > > > [ 0.000000] printk: legacy bootconsole [pl11] enabled
> > > > > [ 0.000000] efi: UEFI not found.
> > > > > [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT
> > > > > [ 0.000000] NODE_DATA(0) allocated [mem 0xbfbecb00-0xbfbf09bf]
> > > > > [ 0.000000] psci: probing for conduit method from DT.
> > > > > [ 0.000000] psci: PSCIv1.1 detected in firmware.
> > > > > [ 0.000000] psci: Using standard PSCI v0.2 function IDs
> > > > > [ 0.000000] psci: MIGRATE_INFO_TYPE not supported.
> > > > > [ 0.000000] psci: SMC Calling Convention v1.2
> > > > > [ 0.000000] RME: Using RSI version 1.0
> > > > > [ 0.000000] Zone ranges:
> > > > > [ 0.000000] DMA [mem 0x0000000040000000-0x00000000bfffffff]
> > > > > [ 0.000000] DMA32 empty
> > > > > [ 0.000000] Normal empty
> > > > > [ 0.000000] Movable zone start for each node
> > > > > :
> > > > > [ 6.664423] Serial: AMBA PL011 UART driver
> > > > > [ 7.151312] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1
> > > > > [ 7.213181] printk: console [ttyAMA0] enabled
> > > > > [ 8.536454] HugeTLB: registered 1.00 Gi
> > > > > <...The realm guest stops here, no output since then...>
> > > > >
> > > >
> > > > On my side I get to user space but virtio-serial and virtio-net don't work. I'm
> > > > guessing virtio-iommu-pci does the same thing as this patch [1], but virtio
> > > > devices are disrupted because it is not configured properly. I need to
> > > > investigate to find the best approach to follow, i.e implicitly use (and
> > > > configure) a virtio-iommu-pci or make sure that [1] and a virtio-iommu-pci
> > > > device can play nice together.
> > > >
> > > > Thanks,
> > > > Mathieu
> > > >
> > > > [1]. [RFC v4 21/24] target/arm/kvm-rme: Add DMA remapping for the shared memory region
> > > >
> > > >
> > > > > Thanks,
> > > > > Gavin
> > > > >
> > > >
> > >
prev parent reply other threads:[~2026-11-17 21:18 UTC|newest]
Thread overview: 84+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-09-03 19:35 [RFC v4 00/24] Add Realm support to QEMU-VMM Mathieu Poirier
2026-09-03 19:35 ` [RFC v4 01/24] linux-headers: Add RME related definitions Mathieu Poirier
2026-09-04 6:07 ` Gavin Shan
2026-09-04 16:24 ` Mathieu Poirier
2026-09-03 19:35 ` [RFC v4 02/24] target/arm/kvm: Return immediately on error in kvm_arch_init() Mathieu Poirier
2026-09-15 21:01 ` Michael Roth
2026-09-30 21:24 ` Mathieu Poirier
2026-09-03 19:35 ` [RFC v4 03/24] target/arm: Add confidential guest support Mathieu Poirier
2026-09-07 8:49 ` Markus Armbruster
2026-09-15 21:27 ` Michael Roth
2026-09-22 8:01 ` Lorenzo Pieralisi
2026-09-24 3:12 ` Kohei Enju
2026-09-30 21:28 ` Mathieu Poirier
2026-10-19 20:46 ` Mathieu Poirier
2026-10-20 11:02 ` Lorenzo Pieralisi
2026-10-20 17:27 ` Mathieu Poirier
2026-10-20 18:12 ` Michael Roth
2026-10-27 14:36 ` Lorenzo Pieralisi
2026-10-29 23:01 ` Michael Roth
2026-10-21 13:42 ` Gavin Shan
2026-10-26 18:06 ` Lorenzo Pieralisi
2026-10-28 4:51 ` Kohei Enju
2026-10-29 21:14 ` Mathieu Poirier
2026-10-29 22:09 ` Mathieu Poirier
2026-09-03 19:35 ` [RFC v4 04/24] target/arm/kvm-rme: Add mechanic to initialize realms Mathieu Poirier
2026-10-01 23:42 ` Gavin Shan
2026-09-03 19:35 ` [RFC v4 05/24] target/arm/kvm: Split kvm_arch_get/put_registers Mathieu Poirier
2026-09-03 19:35 ` [RFC v4 06/24] target/arm/kvm-rme: Initialize vCPU Mathieu Poirier
2026-10-02 0:53 ` Gavin Shan
2026-10-15 20:29 ` Mathieu Poirier
2026-09-03 19:35 ` [RFC v4 07/24] target/arm/kvm: Create scratch Realm VM when requested Mathieu Poirier
2026-09-04 3:09 ` Kohei Enju
2026-09-04 16:25 ` Mathieu Poirier
2026-10-02 1:05 ` Gavin Shan
2026-10-15 20:31 ` Mathieu Poirier
2026-09-03 19:35 ` [RFC v4 08/24] target/arm/kvm: Use kvm_vm_check_extension() where necessary Mathieu Poirier
2026-09-03 19:35 ` [RFC v4 09/24] hw/core/loader: Add a ROM loader notifier Mathieu Poirier
2026-09-15 22:36 ` Michael Roth
2026-10-02 1:15 ` Gavin Shan
2026-10-15 20:31 ` Mathieu Poirier
2026-09-03 19:35 ` [RFC v4 10/24] target/arm/kvm-rme: Keep track of images loaded in Realm memory Mathieu Poirier
2026-09-15 22:50 ` Michael Roth
2026-09-16 21:40 ` Mathieu Poirier
2026-10-02 1:23 ` Gavin Shan
2026-10-16 17:14 ` Mathieu Poirier
2026-09-03 19:35 ` [RFC v4 11/24] target/arm/kvm-rme: Populate Realm with runtime images Mathieu Poirier
2026-10-02 3:45 ` Gavin Shan
2026-10-02 9:27 ` Lorenzo Pieralisi
2026-10-16 20:08 ` Mathieu Poirier
2026-10-17 1:06 ` Gavin Shan
2026-10-17 7:46 ` Lorenzo Pieralisi
2026-10-19 20:23 ` Mathieu Poirier
2026-10-21 9:47 ` Lorenzo Pieralisi
2026-10-21 12:24 ` Gavin Shan
2026-10-22 14:34 ` Lorenzo Pieralisi
2026-10-22 23:30 ` Gavin Shan
2026-10-23 6:58 ` Lorenzo Pieralisi
2026-10-23 20:25 ` Mathieu Poirier
2026-09-03 19:35 ` [RFC v4 12/24] target/arm/cpu: Set number of breakpoints and watchpoints in KVM Mathieu Poirier
2026-10-02 4:06 ` Gavin Shan
2026-10-16 22:53 ` Mathieu Poirier
2026-09-03 19:36 ` [RFC v4 13/24] target/arm/cpu: Set number of PMU counters " Mathieu Poirier
2026-09-03 19:36 ` [RFC v4 14/24] target/arm/cpu: Don't read Realm registers Mathieu Poirier
2026-09-03 19:36 ` [RFC v4 15/24] hw/arm/virt: Set proper conduit method for Realms Mathieu Poirier
2026-09-03 19:36 ` [RFC v4 16/24] hw/arm/virt: Embed Realm VM type with IPA address space Mathieu Poirier
2026-10-02 4:16 ` Gavin Shan
2026-09-03 19:36 ` [RFC v4 17/24] hw/arm/virt: Reserve one bit of guest physical address for RME Mathieu Poirier
2026-10-02 4:19 ` Gavin Shan
2026-10-19 20:00 ` Mathieu Poirier
2026-09-03 19:36 ` [RFC v4 18/24] hw/arm/virt: Disable DTB randomness for confidential VMs Mathieu Poirier
2026-09-03 19:36 ` [RFC v4 19/24] hw/arm/virt: Move virt_flash_create() to machvirt_init() Mathieu Poirier
2026-09-03 19:36 ` [RFC v4 20/24] hw/arm/virt: Use RAM instead of flash for confidential guest firmware Mathieu Poirier
2026-09-03 19:36 ` [RFC v4 21/24] target/arm/kvm-rme: Add DMA remapping for the shared memory region Mathieu Poirier
2026-09-03 19:36 ` [RFC v4 22/24] docs/interop/firmware.json: Add arm-rme firmware feature Mathieu Poirier
2026-09-03 19:36 ` [RFC v4 23/24] hw/arm/boot: Load DTB as is for confidential VMs Mathieu Poirier
2026-09-03 19:36 ` [RFC v4 24/24] hw/arm/boot: Skip bootloader for confidential guests Mathieu Poirier
2026-09-21 4:05 ` [RFC v4 00/24] Add Realm support to QEMU-VMM Gavin Shan
2026-10-02 20:31 ` Mathieu Poirier
2026-10-21 21:03 ` Mathieu Poirier
2026-10-23 11:00 ` Gavin Shan
2026-10-23 20:23 ` Mathieu Poirier
2026-11-10 8:01 ` Kohei Enju
2026-11-11 8:56 ` Lorenzo Pieralisi
2026-11-17 21:18 ` Mathieu Poirier [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=avzEvGXSEcgVdytB@p14s \
--to=mathieu.poirier@linaro.org \
--cc=armbru@redhat.com \
--cc=berrange@redhat.com \
--cc=cohuck@redhat.com \
--cc=eblake@redhat.com \
--cc=enju.kohei@fujitsu.com \
--cc=gshan@redhat.com \
--cc=kchamart@redhat.com \
--cc=kvm@vger.kernel.org \
--cc=lorenzo.pieralisi@linaro.org \
--cc=mst@redhat.com \
--cc=pbonzini@redhat.com \
--cc=peter.maydell@linaro.org \
--cc=pierrick.bouvier@oss.qualcomm.com \
--cc=qemu-arm@nongnu.org \
--cc=qemu-devel@nongnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox