Re: [RFC PATCH v4 03/16] iommu/arm-smmu-v3: Add initial pSMMU realm viommu plumbing

[Aneesh Kumar K.V <aneesh.kumar@kernel.org>]

Jason Gunthorpe <jgg@ziepe.ca> writes:

> On Wed, Sep 02, 2026 at 06:45:42PM +0530, Aneesh Kumar K.V wrote:
>
>> To reiterate, for this configuration:
>> 
>> - The viommu will use a stage-1 bypass configuration.
>> - A new IOMMU_VIOMMU_TYPE_ARM_REALM_SMMUV3 type will create the viommu.
>>   The psmmu will be activated at this point to avoid creating psmmu
>>   objects early. We will reference-count it to ensure that the same
>>   psmmu is shared across realm guests.
>> - Creating a vdevice will invoke SMC_RMI_PSMMU_ST_L2_CREATE.
>> 
>> I am unclear about the vdev_create suggestion. Creating a vdevice
>> requires an RD, which is created later in the flow above. How do you
>> suggest linking vdevice_alloc to vdev_create?
>
> I was thinking we'd make sure the KVM is associated with the viommu
> and/or possibly the S2 domain. That was always sort of broadly the
> idea in this space. There are several topics unrelated to CC that
> needed this.
> In any case, when you create the iommufd viommu you should also do
> VSMMU_CREATE which needs the RD. It seems reasonable to assume the RD
> is available during vdevice create.
>
> [There is an aside here I will mention: several other use cases need
> this idea of an "external" domain where the HWPT would be created but
> not controlled by iommufd or the iommu subsystem. Xen and Hyperv for
> example. There is probably some merit in thinking more about exactly
> what the nested parent domain should be for this viommu, but it isn't
> critical.]
>
>> From the RMM's perspective, the sequence is as follows:
>>
>> viommu alloc
>>
>> [   rmm ] SMC_RMI_PSMMU_ACTIVATE            2b400000 8819bb000 > RMI_INCOMPLETE 0 10008
>> [   rmm ]       SMC_RMI_OP_MEM_DONATE       0 881bac098 1 > RMI_INCOMPLETE 2 10004
>> [   rmm ]       SMC_RMI_OP_MEM_DONATE       0 881bac098 1 > RMI_INCOMPLETE 1 10004
>> [   rmm ]       SMC_RMI_OP_MEM_DONATE       0 881bac098 1 > RMI_INCOMPLETE 1 0
>> [   rmm ] L1 StrTab: PA 0x881baa000 VA 0x80003c0000 size 0x2000
>> [   rmm ] CMDQ: PA 0x88066a000 VA 0x80003c2000
>> [   rmm ] EVTQ: PA 0x8815c5000 VA 0x80003c3000
>> [   rmm ] PSMMU 0x2b400000 activated
>> [   rmm ]       SMC_RMI_OP_CONTINUE         0 0 > RMI_SUCCESS 0 0
>>
>> [   rmm ] SMC_RMI_PSMMU_ST_L2_CREATE        2b400000 300 > RMI_INCOMPLETE 0 10004
>> [   rmm ]       SMC_RMI_OP_MEM_DONATE       0 881bac098 1 > RMI_INCOMPLETE 1 0
>> [   rmm ] smmu->strtab_base[12] 0x0 @0x80003c0060
>> [   rmm ] L1STD[12] 0x8819bb007 for SID 0x300: L2 table VA 0x80003d0000 PA 0x8819bb000
>> [   rmm ]       SMC_RMI_OP_CONTINUE         0 0 > RMI_SUCCESS 0 0
>
> What was this one for during viommu alloc?
>
>> vdevice alloc
>> 
>> [   rmm ] SMC_RMI_PSMMU_ST_L2_CREATE        2b400000 200 > RMI_INCOMPLETE 0 10004
>> [   rmm ]       SMC_RMI_OP_MEM_DONATE       0 882648098 1 > RMI_INCOMPLETE 1 0
>> [   rmm ] smmu->strtab_base[8] 0x0 @0x80003c0040
>> [   rmm ] L1STD[8] 0x882506007 for SID 0x200: L2 table VA 0x80003cc000 PA 0x882506000
>> [   rmm ]       SMC_RMI_OP_CONTINUE         0 0 > RMI_SUCCESS 0 0
>> 
>> RD gets allocated here
>> 
>> [   rmm ] SMC_RMI_REALM_CREATE              881b03000 8819a1000 > RMI_INCOMPLETE 0 24
>> [   rmm ]       SMC_RMI_OP_MEM_DONATE       0 881605098 9 > RMI_INCOMPLETE 9 0
>> [   rmm ]       SMC_RMI_OP_CONTINUE         0 0 > RMI_SUCCESS 0 0
>
> Why? The VMM needs to create the kvm before starting the iommu stuff.
>
> I would expect that KVM knows it is going to a realm very early on? I
> had assumed the RD would also be created early by KVM?
>
> What triggers RD creation? Why can't the VMM do it earlier?
>
> Your followup said:
>   - The realm is created during viommu allocation, ensuring that the
>     vSMMU can be created here.
>
> Do you mean the SMMUv3 driver triggers RD creation? That feels
> wrong. Did you mean the VMM just does it earlier?
>

Currently, CCA creates the realm lazily as part of another operation.
Realm creation is triggered by kvm_arm_rmi_populate() or
kvm_arch_vcpu_run_pid_change()

The change looks like this:

modified   arch/arm64/include/asm/kvm_rmi.h
@@ -102,6 +102,14 @@ u64 kvm_realm_reset_id_aa64dfr0_el1(const struct kvm_vcpu *vcpu, u64 val);
 bool kvm_rmi_supports_sve(void);
 
 int kvm_init_realm(struct kvm *kvm);
+#ifdef CONFIG_KVM
+int kvm_realm_ensure_created(struct kvm *kvm);
+#else
+static inline int kvm_realm_ensure_created(struct kvm *kvm)
+{
+	return -EOPNOTSUPP;
+}
+#endif
 int kvm_activate_realm(struct kvm *kvm);
 void kvm_destroy_realm(struct kvm *kvm);
 int kvm_realm_teardown_stage2(struct kvm *kvm);
modified   arch/arm64/kvm/rmi.c
@@ -1120,6 +1120,20 @@ static int realm_ensure_created(struct kvm *kvm)
 	return realm_create_rd(kvm);
 }
 
+int kvm_realm_ensure_created(struct kvm *kvm)
+{
+	int ret;
+
+	if (!kvm_is_realm(kvm))
+		return -EINVAL;
+
+	guard(mutex)(&kvm->arch.config_lock);
+	ret = realm_ensure_created(kvm);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(kvm_realm_ensure_created);
+
 int kvm_arm_rmi_init_ripas(struct kvm *kvm,
 			   struct kvm_arm_rmi_init_ripas *args)
 {
modified   drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-realm.c
@@ -253,8 +253,9 @@ int arm_realm_smmu_v3_init(struct iommufd_viommu *viommu,
 		return -EINVAL;
 
 	kvm = viommu->kvm_file->private_data;
-	if (!kvm_is_realm(kvm))
-		return -EINVAL;
+	ret = kvm_realm_ensure_created(kvm);
+	if (ret)
+		return ret;
 
 	if (!(smmu->features & ARM_SMMU_FEAT_RME))
 		return -EOPNOTSUPP;


>
> The draft in the next message looks promising, did you discover any
> other gotchas when exploring it?
>

Nothing significant. I still have questions about moving TDI flows such
as vdev creation and guest requests into the SMMU driver instead of
keeping them in arm-cca-host.ko, but I will follow up in the other
thread.

>
> When you repost this can you take some care to explain the general
> idea of this modelling in the commit messages so AMD and Intel can
> confirm they can use it for both their non-viommu and viommu cases?
>
> Thanks,
> Jason

-aneesh