Re: [PATCH v13 00/48] arm64: Support for Arm CCA in KVM

Suzuki K Poulose
2026-03-26T11:24:01Z

Hi Gavin,

On 26/03/2026 00:48, Gavin Shan wrote:
> Hi Suzuki,
>
> On 3/25/26 8:16 PM, Suzuki K Poulose wrote:
>> On 25/03/2026 06:37, Gavin Shan wrote:
>>> On 3/21/26 2:45 AM, Steven Price wrote:
>
> [...]
>
>>>
>>> In upstream TF-A repository [1], I don't see the config option
>>> 'RMM_V1_COMPAT'.
>>> would it be something else?
>>>
>>> [1] git@github.com:ARM-software/arm-trusted-firmware.git    (branch:
>>> master)
>>>
>>
>> suzuki@ewhatever:trusted-firmware-a$ git grep RMM_V1_COMPAT
>> Makefile:       RMM_V1_COMPAT \
>> Makefile:       RMM_V1_COMPAT \
>> docs/getting_started/build-options.rst:-  ``RMM_V1_COMPAT``: Boolean
>> flag to enable support for RMM v1.x compatibility
>> include/services/rmmd_svc.h:#if RMM_V1_COMPAT
>> include/services/rmmd_svc.h:#endif /* RMM_V1_COMPAT */
>> make_helpers/defaults.mk:RMM_V1_COMPAT                  := 1
>> services/std_svc/rmmd/rmmd_main.c:#if RMM_V1_COMPAT
>> services/std_svc/rmmd/rmmd_main.c:#if RMM_V1_COMPAT
>> services/std_svc/rmmd/rmmd_main.c:#if !RMM_V1_COMPAT
>> services/std_svc/rmmd/rmmd_main.c:#if RMM_V1_COMPAT
>> services/std_svc/rmmd/rmmd_main.c:#if RMM_V1_COMPAT
>> services/std_svc/rmmd/rmmd_main.c:#if RMM_V1_COMPAT
>> services/std_svc/rmmd/rmmd_rmm_lfa.c:#if RMM_V1_COMPAT
>> services/std_svc/rmmd/rmmd_rmm_lfa.c:#if RMM_V1_COMPAT
>> services/std_svc/rmmd/rmmd_rmm_lfa.c:#if RMM_V1_COMPAT
>> services/std_svc/rmmd/rmmd_rmm_lfa.c:#if RMM_V1_COMPAT
>> suzuki@ewhatever:trusted-firmware-a$ git log --oneline -1
>> 8dae0862c (HEAD, origin/master, origin/integration, origin/HEAD) Merge
>> changes from topic "qti_lemans_evk" into integration
>> suzuki@ewhatever:trusted-firmware-a$ git remote get-url origin
>> https://git.trustedfirmware.org/TF-A/trusted-firmware-a.git
>>
>
> Thanks for the details. It turned out that I used the wrong TF-A
> repository. In
> the proposed repository, I'm able to see the option 'RMM_V1_COMPAT' and
> the EL3-RMM
> interface compatible issue disappears. However, there are more issues
> popped up.
>
> I build everything manually where the host is emulated by QEMU instead
> of shrinkwrap
> and FVP model. It's used to work well before. Maybe it's time to switch
> to shinkwrap
> and FVP model since device assignment (DA) isn't supported by an
> emulated host
> by QEMU and shrinkwrap and FVP model seems the only option. I need to
> learn how
> to do that later.

Thanks for the update. Yes, QEMU TF-RMM support is in progress, @Mathieu
Poirier is looking into it

>
> There are two issues I can see with the following combinations. Details
> are provided
> like below.
>
>     QEMU:      https://git.qemu.org/git/
> qemu.git                            (branch: stable-9.2)
>     TF-RMM:    https://git.trustedfirmware.org/TF-RMM/tf-
> rmm.git            (branch: topics/rmm-v2.0-poc)
>     EDK2:      git@github.com:tianocore/
> edk2.git                            (tag:    edk2-stable202411)
>     TF-A:      https://git.trustedfirmware.org/TF-A/trusted-firmware-
> a.git  (branch: master)
>     HOST:      https://git.gitlab.arm.com/linux-arm/linux-
> cca.git           (branch: cca-host/v13)
>     BUILDROOT: https://github.com/buildroot/
> buildroot                       (branch: master)
>     KVMTOOL:   https://gitlab.arm.com/linux-arm/kvmtool-
> cca                 (branch: cca/v11)
>     GUEST:     https://github.com/torvalds/
> linux.git                        (branch: master)
>
> (1) The emulated host is started by the following command lines.
>
>     sudo /home/gshan/sandbox/cca/host/qemu/build/qemu-system-
> aarch64                  \
>     -M virt,virtualization=on,secure=on,gic-
> version=3,acpi=off                        \
>     -cpu max,x-rme=on -m 8G -smp
> 8                                                    \
>     -serial mon:stdio -monitor none -nographic -
> nodefaults                            \
>     -bios /home/gshan/sandbox/cca/host/tf-a/
> flash.bin                                 \
>     -kernel /home/gshan/sandbox/cca/host/linux/arch/arm64/boot/
> Image                  \
>     -initrd /home/gshan/sandbox/cca/host/buildroot/output/images/
> rootfs.cpio.xz       \
>     -device pcie-root-
> port,bus=pcie.0,chassis=1,id=pcie.1                             \
>     -device pcie-root-
> port,bus=pcie.0,chassis=2,id=pcie.2                             \
>     -device pcie-root-
> port,bus=pcie.0,chassis=3,id=pcie.3                             \
>     -device pcie-root-
> port,bus=pcie.0,chassis=4,id=pcie.4                             \
>     -device virtio-9p-
> device,fsdev=shr0,mount_tag=shr0                                \
>     -fsdev local,security_model=none,path=/home/gshan/sandbox/cca/
> guest,id=shr0       \
>     -netdev tap,id=tap1,script=/etc/qemu-ifup-gshan,downscript=/etc/
> qemu-ifdown-gshan \
>     -device virtio-net-pci,bus=pcie.2,netdev=tap1,mac=b8:3f:d2:1d:3e:f1
>
> (2) Issue-1: TF-RMM complains about the root complex list is invalid.
> This error is
>     raised in TF-RMM::setup_root_complex_list() where the error code is
> still set to
>     0 (SUCCESS) in this failing case. The TF-RMM initialization is
> terminated early,
>     but TF-A still thinks the initialization has been completely done.
>
>     INFO:    BL31: Initializing RMM
>     INFO:    RMM init start.
>     RMM EL3 compat memory reservation enabled.
>     Dynamic VA pool base address: 0xc0000000
>     Reserved 20 pages. Remaining: 3615 pages
>     Reserve mem: 20 pages at PA: 0x401f2000 (alignment 0x1000)
>     Static Low VA initialized. xlat tables allocated: 20 used: 7
>     Reserved 514 pages. Remaining: 3101 pages
>     Reserve mem: 514 pages at PA: 0x40206000 (alignment 0x1000)
>     Dynamic Low VA initialized. xlat tables allocated: 514 used: 514
>     Invalid: Root Complex list
> <<<<< ERROR
>     INFO:    RMM init end.
>
> (3) Issue-2: The host kernel gets stuck in rmi_check_version() where
> SMC_RMI_VERSION
>     is issued to TF-A, but it can't be forwarded to TF-RMM because its
> initialization
>     isn't completely done (issue-1).
>
>     [   37.438253] Unpacking initramfs...
>     [   37.563460] kvm [1]: nv: 570 coarse grained trap handlers
>     [   37.581139] kvm [1]: nv: 664 fine grained trap handlers
>     <... system becomes stuck here ...>
>
> So my workaround is to skip fetching root complex list from the EL3-RMM
> manifest data
> in TF-RMM::setup_root_complex_list() since it's not provided for the
> qemu platform by

^^ This may have to do with the RMM<->TF-A Manifest changes

> TF-A. With this workaround, the host can boot up into shell prompt and
> the guest can
> be started by kvmtool.
>
>     host$ uname -r
>     7.0.0-rc1-gavin-gd62aa44b2590
>     host$ lkvm run --realm -c 2 -m 256                   \
>           -k /mnt/linux/arch/arm64/boot/Image            \
>           -i /mnt/buildroot/output/images/rootfs.cpio.xz
>           -p earlycon=uart,mmio,0x101000000
>     Info: # lkvm run -k /mnt/linux/arch/arm64/boot/Image -m 256 -c 2 --
> name guest-163
>     Info: Enabling Guest memfd for confidential guest
>     Warning: The maximum recommended amount of VCPUs is 1
>     [    0.000000] Booting Linux on physical CPU 0x0000000000 [0x000f0510]
>     [    0.000000] Linux version 7.0.0-rc2-gavin-g0031c06807cf
> (gshan@nvidia-grace-hopper-01.khw.eng.bos2.dc.redhat.com) (gcc (GCC)
> 14.3.1 20251022 (Red Hat 14.3.1-4), GNU ld version 2.41-64.el10) #2 SMP
> PREEMPT Wed Mar 25 20:28:05 EDT 2026
>     [    0.000000] KASLR enabled
>          :
>     [  267.578060] Freeing initrd memory: 4728K
>     [  267.921865] Warning: unable to open an initial console.
>     [  270.327960] Freeing unused kernel memory: 1792K
>     [  270.669368] Run /init as init process
>

Cool, thanks!

Suzuki

> Thanks,
> Gavin
>