Re: [PATCH v16 00/45] arm64: Support for Arm CCA in KVM

[Gavin Shan <gshan@redhat.com>]

On 8/11/26 9:12 PM, Suzuki K Poulose wrote:
> On 11/08/2026 05:44, Gavin Shan wrote:
>> On 8/7/26 8:05 AM, Suzuki K Poulose wrote:
>>
>> [...]
>>
>>>
>>> Here is a cleaned up version, rebased on to Will's kvmtool master
>>> branch:
>>>
>>> https://gitlab.arm.com/linux-arm/kvmtool-cca cca/kvm-v16
>>>
>>> (This works with the v15 of the KVM series too)
>>> Build on top of Fuad's Guest memfd support patches.
>>>
>>
>> With the following combination, I'm able boot up the realm guest.
>>
>>    tf-rmm:   https://git.trustedfirmware.org/TF-RMM/tf- rmm.git            (branch: topics/rmm-v2.0-poc_2)
> 
> 
> Please be aware that CCA KVM v15 onwards, the above branch is not compatible. You should be using rmm-v2.0-poc_3. Please could you
> confirm if this is still an issue ?
> 

With tf-rmm/topics/rmm-v2.0-poc_2 + cca/host-v16 + kvmtool/cca/v16, there is no issue
and the realm guest can boot up successfully.

When tf-rmm/topics/rmm-v2.0-poc_3 is used, the realm guest boot gets stuck as I reported
earlier. Note the host is emulated by QEMU (TCG mode).

As the following calltrace indicates, -EAGAIN is returned from tf-rmm::update_ripas()
because true is returned from s2tte_drain_pending() for the S2TTE corresponding to
IPA 0x80000000. Linux host received error (RMI_ERROR_RTT, level=3) in ripas_change().
Upon this specific error and the IPA range [0x80000000 0x90000000], find_map_level()
returns level of 2, and realm_create_rtt_levels() returns 0 without populating any
RTTs. After that, rmi_rtt_set_ripas() is re-executed and the above loop starts over
again.

   Linux host
   ==========
   kvm_arch_vcpu_ioctl_run                     // cca/host-v16
     check_vcpu_requests
       kvm_check_request
         kvm_rec_handle_request
           kvm_complete_ripas_change
             realm_set_ipa_state
               ripas_change
                 rmi_rtt_set_ripas
                   SMC_RMI_RTT_SET_RIPAS

   TF-RMM
   ======
   SMC_RMI_RTT_SET_RIPAS                      // tf-rmm/topics/rmm-v2.0-poc_3
     smc_rtt_set_ripas
       s2tt_walk_lock_unlock
       rtt_set_ripas_range
         update_ripas
           s2tte_drain_pending                // true, returns -EAGAIN

The problem is the pending-bit for RTE corresponding IPA address 0x80000000 isn't cleared
when SMC_RMI_RTT_SET_RIPAS is invoked. I didn't figure out how this bit is set and why
it's not cleared in time.

Thanks,
Gavin

> 
> Cheers
> Suzuki
> 
> 
>>    tf-a:     https://git.trustedfirmware.org/TF-A/trusted-firmware- a.git  (branch: master)
>>    host:     https://git.gitlab.arm.com/linux-arm/linux- cca.git           (branch: cca-host/v16)
>>    kvmtool:  https://gitlab.arm.com/linux-arm/kvmtool- cca                 (branch: cca/kvm-v16)
>>
>> However, the guest can't boot up and become stuck in SMC_RSI_IPA_STATE_SET request,
>> which can't completed by the host.
>>
>>    tf-rmm:   https://git.trustedfirmware.org/TF-RMM/tf- rmm.git            (branch: topics/rmm-v2.0-poc_3)
>>
>> (1) Login the emulated host
>>
>> machine$ ssh -o StrictHostKeyChecking=no root@10.26.1.240
>>
>> (2) Start realm guest using kvmtool
>>
>> root@host:~# lkvm run --realm -c 1 -m 256                   \
>>               -k /mnt/linux/arch/arm64/boot/Image            \
>>               -i /mnt/buildroot/output/images/rootfs.cpio.xz \
>>               -p earlycon=uart,mmio,0x101000000
>>                   :
>> [    0.000000] Booting Linux on physical CPU 0x0000000000 [0x000f0510]
>> [    0.000000] Linux version 7.2.0-rc5-gavin-gf5098b6bae76 (gshan@nvidia-grace-hopper-01.khw.eng.bos2.dc.redhat.com) (gcc (GCC) 14.3.1 20251022 (Red Hat 14.3.1-4), GNU ld version 2.41-65.el10) #47 SMP PREEMPT Mon Jul 27 02:29:03 EDT 2026
>> [    0.000000] KASLR enabled
>> [    0.000000] Machine model: linux,dummy-virt
>> [    0.000000] earlycon: uart0 at MMIO 0x0000000101000000 (options '')
>> [    0.000000] printk: legacy bootconsole [uart0] enabled
>> [    0.000000] efi: UEFI not found.
>> [    0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT
>> [    0.000000] NUMA: Faking a node at [mem 0x0000000080000000-0x000000008fffffff]
>> [    0.000000] NODE_DATA(0) allocated [mem 0x8ff76dc0-0x8ff7ac7f]
>> [    0.000000] psci: probing for conduit method from DT.
>> [    0.000000] psci: PSCIv1.1 detected in firmware.
>> [    0.000000] psci: Using standard PSCI v0.2 function IDs
>> [    0.000000] psci: MIGRATE_INFO_TYPE not supported.
>> [    0.000000] psci: SMC Calling Convention v1.2
>> [    0.000000] RME: Using RSI version 1.0
>> <... no more output from the guest ...>
>>
>>
>> (3) The output from host's serial console
>>
>> SMC_RMI_REALM_CREATE              12a4c7000 12a4c6000 > RMI_INCOMPLETE 0 24 0 0
>>        SMC_RMI_OP_MEM_DONATE       0 1002c7098 9 > RMI_INCOMPLETE 9 0 0 0
>>        SMC_RMI_OP_CONTINUE         0 0 > RMI_SUCCESS 0 0
>> SMC_RMI_RTT_UNPROT_UNMAP          12a4c7000 18f85c000 18fe00000 0 0 > RMI_ERROR_RTT 2 0 0 0 0
>> SMC_RMI_RTT_UNPROT_UNMAP          12a4c7000 18fe00000 18fe10000 0 0 > RMI_ERROR_RTT 2 0 0 0 0
>> SMC_RMI_RTT_UNPROT_UNMAP          12a4c7000 18fe00000 18fe10000 0 0 > RMI_ERROR_RTT 2 0 0 0 0
>> SMC_RMI_REC_CREATE                12a4c7000 12a392000 12a391000 > RMI_INCOMPLETE 0 40 0 0
>>        SMC_RMI_OP_MEM_DONATE       0 106406098 10 > RMI_INCOMPLETE 10 0 0 0
>>        SMC_RMI_OP_CONTINUE         0 0 > RMI_SUCCESS 0 0
>> SMC_RMI_REALM_ACTIVATE            12a4c7000 > RMI_SUCCESS
>> Unhandled write S2_0_C0_C2_2
>> SMC_RMI_RTT_DATA_MAP              12a4c7000 8ffb4000 8ffb5000 1 129f1d004 > RMI_INCOMPLETE 0 0 0 1
>>        SMC_RMI_OP_CONTINUE         0 0 > RMI_SUCCESS 8ffb5000 0
>> PSCI_84000000                     0 0 0 0 0 0 0 > 10001 0 0 0
>> PSCI_84000006                     0 0 0 0 0 0 0 > ffffffffffffffff 0 0 0
>> PSCI_8400000a                     80000000 0 0 0 0 0 0 > 0 0 0 0
>> SMC_80000000                      0 0 0 0 0 0 0 > 10002 0 0 0
>> SMC_84000050                      1 0 0 ffffacd67d1e8000 ffffacd67d124000 0 0 > ffffffffffffffff 0 0 0
>> SMC_80000001                      80000002 ffff 0 ffffacd67d1e8000 ffffacd67d124000 ffffacd67cd5d000 ffffacd67cd5dd38 > ffffffffffffffff 0 0 0
>> PSCI_8400000a                     c4000001 0 0 0 0 0 0 > 0 0 0 0
>> PSCI_8400000a                     c4000012 0 0 0 0 0 0 > ffffffffffffffff 0 0 0
>> PSCI_8400000a                     c4000015 0 0 0 0 0 0 > ffffffffffffffff 0 0 0
>> SMC_8600ff01                      ffffacd67cfc0740 10002 0 0 0 0 ffffacd67cfb3cc8 > ffffffffffffffff 0 0 0
>> SMC_RSI_VERSION                   10000 > RSI_SUCCESS 10000 10001
>> SMC_RSI_REALM_CONFIG              81385000 > RSI_SUCCESS
>> SMC_RSI_IPA_STATE_SET             80000000 90000000 1 0
>> SMC_RMI_RTT_SET_RIPAS             12a4c7000 12a392000 80000000 90000000  > RMI_ERROR_RTT 3
>> SMC_RMI_RTT_SET_RIPAS             12a4c7000 12a392000 80000000 90000000  > RMI_ERROR_RTT 3
>> SMC_RMI_RTT_SET_RIPAS             12a4c7000 12a392000 80000000 90000000  > RMI_ERROR_RTT 3
>> SMC_RMI_RTT_SET_RIPAS             12a4c7000 12a392000 80000000 90000000  > RMI_ERROR_RTT 3
>> SMC_RMI_RTT_SET_RIPAS             12a4c7000 12a392000 80000000 90000000  > RMI_ERROR_RTT 3
>> SMC_RMI_RTT_SET_RIPAS             12a4c7000 12a392000 80000000 90000000  > RMI_ERROR_RTT 3
>> <... the last message repeats ...>
>>
>> Thanks,
>> Gavin
>>
>>
>