Re: [PATCH v13 00/48] arm64: Support for Arm CCA in KVM

[Suzuki K Poulose <suzuki.poulose@arm.com>]

Hi Gavin,

On 26/03/2026 00:48, Gavin Shan wrote:
> Hi Suzuki,
> 
> On 3/25/26 8:16 PM, Suzuki K Poulose wrote:
>> On 25/03/2026 06:37, Gavin Shan wrote:
>>> On 3/21/26 2:45 AM, Steven Price wrote:
> 
> [...]
> 
>>>
>>> In upstream TF-A repository [1], I don't see the config option 
>>> 'RMM_V1_COMPAT'.
>>> would it be something else?
>>>
>>> [1] git@github.com:ARM-software/arm-trusted-firmware.git    (branch: 
>>> master)
>>>
>>
>> suzuki@ewhatever:trusted-firmware-a$ git grep RMM_V1_COMPAT
>> Makefile:       RMM_V1_COMPAT \
>> Makefile:       RMM_V1_COMPAT \
>> docs/getting_started/build-options.rst:-  ``RMM_V1_COMPAT``: Boolean 
>> flag to enable support for RMM v1.x compatibility
>> include/services/rmmd_svc.h:#if RMM_V1_COMPAT
>> include/services/rmmd_svc.h:#endif /* RMM_V1_COMPAT */
>> make_helpers/defaults.mk:RMM_V1_COMPAT                  := 1
>> services/std_svc/rmmd/rmmd_main.c:#if RMM_V1_COMPAT
>> services/std_svc/rmmd/rmmd_main.c:#if RMM_V1_COMPAT
>> services/std_svc/rmmd/rmmd_main.c:#if !RMM_V1_COMPAT
>> services/std_svc/rmmd/rmmd_main.c:#if RMM_V1_COMPAT
>> services/std_svc/rmmd/rmmd_main.c:#if RMM_V1_COMPAT
>> services/std_svc/rmmd/rmmd_main.c:#if RMM_V1_COMPAT
>> services/std_svc/rmmd/rmmd_rmm_lfa.c:#if RMM_V1_COMPAT
>> services/std_svc/rmmd/rmmd_rmm_lfa.c:#if RMM_V1_COMPAT
>> services/std_svc/rmmd/rmmd_rmm_lfa.c:#if RMM_V1_COMPAT
>> services/std_svc/rmmd/rmmd_rmm_lfa.c:#if RMM_V1_COMPAT
>> suzuki@ewhatever:trusted-firmware-a$ git log --oneline -1
>> 8dae0862c (HEAD, origin/master, origin/integration, origin/HEAD) Merge 
>> changes from topic "qti_lemans_evk" into integration
>> suzuki@ewhatever:trusted-firmware-a$ git remote get-url origin
>> https://git.trustedfirmware.org/TF-A/trusted-firmware-a.git
>>
> 
> Thanks for the details. It turned out that I used the wrong TF-A 
> repository. In
> the proposed repository, I'm able to see the option 'RMM_V1_COMPAT' and 
> the EL3-RMM
> interface compatible issue disappears. However, there are more issues 
> popped up.
> 
> I build everything manually where the host is emulated by QEMU instead 
> of shrinkwrap
> and FVP model. It's used to work well before. Maybe it's time to switch 
> to shinkwrap
> and FVP model since device assignment (DA) isn't supported by an 
> emulated host
> by QEMU and shrinkwrap and FVP model seems the only option. I need to 
> learn how
> to do that later.

Thanks for the update. Yes, QEMU TF-RMM support is in progress, @Mathieu 
Poirier is looking into it

> 
> There are two issues I can see with the following combinations. Details 
> are provided
> like below.
> 
>      QEMU:      https://git.qemu.org/git/ 
> qemu.git                            (branch: stable-9.2)
>      TF-RMM:    https://git.trustedfirmware.org/TF-RMM/tf- 
> rmm.git            (branch: topics/rmm-v2.0-poc)
>      EDK2:      git@github.com:tianocore/ 
> edk2.git                            (tag:    edk2-stable202411)
>      TF-A:      https://git.trustedfirmware.org/TF-A/trusted-firmware- 
> a.git  (branch: master)
>      HOST:      https://git.gitlab.arm.com/linux-arm/linux- 
> cca.git           (branch: cca-host/v13)
>      BUILDROOT: https://github.com/buildroot/ 
> buildroot                       (branch: master)
>      KVMTOOL:   https://gitlab.arm.com/linux-arm/kvmtool- 
> cca                 (branch: cca/v11)
>      GUEST:     https://github.com/torvalds/ 
> linux.git                        (branch: master)
> 
> (1) The emulated host is started by the following command lines.
> 
>      sudo /home/gshan/sandbox/cca/host/qemu/build/qemu-system- 
> aarch64                  \
>      -M virt,virtualization=on,secure=on,gic- 
> version=3,acpi=off                        \
>      -cpu max,x-rme=on -m 8G -smp 
> 8                                                    \
>      -serial mon:stdio -monitor none -nographic - 
> nodefaults                            \
>      -bios /home/gshan/sandbox/cca/host/tf-a/ 
> flash.bin                                 \
>      -kernel /home/gshan/sandbox/cca/host/linux/arch/arm64/boot/ 
> Image                  \
>      -initrd /home/gshan/sandbox/cca/host/buildroot/output/images/ 
> rootfs.cpio.xz       \
>      -device pcie-root- 
> port,bus=pcie.0,chassis=1,id=pcie.1                             \
>      -device pcie-root- 
> port,bus=pcie.0,chassis=2,id=pcie.2                             \
>      -device pcie-root- 
> port,bus=pcie.0,chassis=3,id=pcie.3                             \
>      -device pcie-root- 
> port,bus=pcie.0,chassis=4,id=pcie.4                             \
>      -device virtio-9p- 
> device,fsdev=shr0,mount_tag=shr0                                \
>      -fsdev local,security_model=none,path=/home/gshan/sandbox/cca/ 
> guest,id=shr0       \
>      -netdev tap,id=tap1,script=/etc/qemu-ifup-gshan,downscript=/etc/ 
> qemu-ifdown-gshan \
>      -device virtio-net-pci,bus=pcie.2,netdev=tap1,mac=b8:3f:d2:1d:3e:f1
> 
> (2) Issue-1: TF-RMM complains about the root complex list is invalid. 
> This error is
>      raised in TF-RMM::setup_root_complex_list() where the error code is 
> still set to
>      0 (SUCCESS) in this failing case. The TF-RMM initialization is 
> terminated early,
>      but TF-A still thinks the initialization has been completely done.
> 
>      INFO:    BL31: Initializing RMM
>      INFO:    RMM init start.
>      RMM EL3 compat memory reservation enabled.
>      Dynamic VA pool base address: 0xc0000000
>      Reserved 20 pages. Remaining: 3615 pages
>      Reserve mem: 20 pages at PA: 0x401f2000 (alignment 0x1000)
>      Static Low VA initialized. xlat tables allocated: 20 used: 7
>      Reserved 514 pages. Remaining: 3101 pages
>      Reserve mem: 514 pages at PA: 0x40206000 (alignment 0x1000)
>      Dynamic Low VA initialized. xlat tables allocated: 514 used: 514
>      Invalid: Root Complex list                                         
> <<<<< ERROR
>      INFO:    RMM init end.
> 
> (3) Issue-2: The host kernel gets stuck in rmi_check_version() where 
> SMC_RMI_VERSION
>      is issued to TF-A, but it can't be forwarded to TF-RMM because its 
> initialization
>      isn't completely done (issue-1).
> 
>      [   37.438253] Unpacking initramfs...
>      [   37.563460] kvm [1]: nv: 570 coarse grained trap handlers
>      [   37.581139] kvm [1]: nv: 664 fine grained trap handlers
>      <... system becomes stuck here ...>
> 
> So my workaround is to skip fetching root complex list from the EL3-RMM 
> manifest data
> in TF-RMM::setup_root_complex_list() since it's not provided for the 
> qemu platform by

^^ This may have to do with the RMM<->TF-A Manifest changes


> TF-A. With this workaround, the host can boot up into shell prompt and 
> the guest can
> be started by kvmtool.
> 
>      host$ uname -r
>      7.0.0-rc1-gavin-gd62aa44b2590
>      host$ lkvm run --realm -c 2 -m 256                   \
>            -k /mnt/linux/arch/arm64/boot/Image            \
>            -i /mnt/buildroot/output/images/rootfs.cpio.xz
>            -p earlycon=uart,mmio,0x101000000
>      Info: # lkvm run -k /mnt/linux/arch/arm64/boot/Image -m 256 -c 2 -- 
> name guest-163
>      Info: Enabling Guest memfd for confidential guest
>      Warning: The maximum recommended amount of VCPUs is 1
>      [    0.000000] Booting Linux on physical CPU 0x0000000000 [0x000f0510]
>      [    0.000000] Linux version 7.0.0-rc2-gavin-g0031c06807cf 
> (gshan@nvidia-grace-hopper-01.khw.eng.bos2.dc.redhat.com) (gcc (GCC) 
> 14.3.1 20251022 (Red Hat 14.3.1-4), GNU ld version 2.41-64.el10) #2 SMP 
> PREEMPT Wed Mar 25 20:28:05 EDT 2026
>      [    0.000000] KASLR enabled
>           :
>      [  267.578060] Freeing initrd memory: 4728K
>      [  267.921865] Warning: unable to open an initial console.
>      [  270.327960] Freeing unused kernel memory: 1792K
>      [  270.669368] Run /init as init process
> 

Cool, thanks!

Suzuki

> Thanks,
> Gavin
>