Re: [RFC PATCH v3 06/15] hw/arm/smmuv3-accel: Restrict accelerated SMMUv3 to vfio-pci endpoints with iommufd

[Eric Auger <eric.auger@redhat.com>]

Hi Shameer,

On 7/16/25 10:06 AM, Shameerali Kolothum Thodi wrote:
>
>> -----Original Message-----
>> From: Nicolin Chen <nicolinc@nvidia.com>
>> Sent: Tuesday, July 15, 2025 6:59 PM
>> To: Duan, Zhenzhong <zhenzhong.duan@intel.com>
>> Cc: Shameerali Kolothum Thodi
>> <shameerali.kolothum.thodi@huawei.com>; qemu-arm@nongnu.org;
>> qemu-devel@nongnu.org; eric.auger@redhat.com;
>> peter.maydell@linaro.org; jgg@nvidia.com; ddutile@redhat.com;
>> berrange@redhat.com; nathanc@nvidia.com; mochs@nvidia.com;
>> smostafa@google.com; Linuxarm <linuxarm@huawei.com>; Wangzhou (B)
>> <wangzhou1@hisilicon.com>; jiangkunkun <jiangkunkun@huawei.com>;
>> Jonathan Cameron <jonathan.cameron@huawei.com>;
>> zhangfei.gao@linaro.org; shameerkolothum@gmail.com
>> Subject: Re: [RFC PATCH v3 06/15] hw/arm/smmuv3-accel: Restrict
>> accelerated SMMUv3 to vfio-pci endpoints with iommufd
> ...
>
>>>> +    if (pdev && !smmuv3_accel_pdev_allowed(pdev, &vfio_pci)) {
>>>> +        error_report("Device(%s) not allowed. Only PCIe root complex
>>>> devices "
>>>> +                     "or PCI bridge devices or vfio-pci endpoint devices
>>>> with "
>>>> +                     "iommufd as backend is allowed with
>>>> arm-smmuv3,accel=on",
>>>> +                     pdev->name);
>>>> +        exit(1);
>>> Seems aggressive for a hotplug, could we fail hotplug instead of kill
>> QEMU?
> That's right. I will try to see whether it is possible to do a dev->hotplugged
> check here.
>  
>> Hotplug will unlikely be supported well, as it would introduce
>> too much complication.
>>
>> With iommufd, a vIOMMU object is allocated per device (vfio). If
>> the device fd (cdev) is not yet given to the QEMU. It isn't able
>> to allocate a vIOMMU object when creating a VM.
>>
>> While a vIOMMU object can be allocated at a later stage once the
>> device is hotplugged. But things like IORT mappings aren't able
>> to get refreshed since the OS is likely already booted.
> Why do we need IORT mappings to be refreshed during hotplug?
> AFAICS, the mappings are created per host bridge Ids. And how is this
> different from a host machine doing hotplug?
>
>  Even an
>> IOMMU capability sync via the hw_info ioctl will be difficult to
>> do at the runtime post the guest iommu driver's initialization.
> We had some discussion on this "at least one vfio-pci" restriction
> for accelerated mode previously here.
> https://lore.kernel.org/qemu-devel/Z6TtCLQ35UI12T77@redhat.com/#t
>
> I am not sure we reached any consensus on that. The 3 different approaches
> discussed were,
>
> 1. The current one used here. At least one cold plugged vfio-pci device
>    so that  we can retrieve the host SMMUV3 HW_INFO as per current
>   IOMMUFD APIs.

I do not get why you can't wait for the 1st device to be attached to
"freeze" the settings. Is it because you may also have some bridges /
root ports also attached to the same viommu. As those latter do not have
any adherence to the host SMMU, is that a problem?
>
> 2. A new IOMMUFD API to retrieve HW_INFO without a device. 
this can only be possible if, on the command line you connect the vsmmu
to a sysfs path to the host iommu (or maybe this is what you meant in
3). This would be another option we also evoked in the past. But this is
not very user friendly for the guy who launches the VM to care both the
device and the associated physical SMMU. Logically we could build that
relationship automatically.
>
> 3. A fully specified vSMMUv3 through Qemu command line so that we
>    don't need HW_INFO from kernel.

I don't think this is sensible as there may be plenty of those, each
requirement a libvirt adaptation

Eric
>
> We're going with option one for now, but completely blocking hotplug
> because of it  feels a bit too restrictive to me.
>
> The real issue (for now), as I see it, is that we need some way to remember
> the Guest SMMUv3 <-> Host SMMUv3 mapping after the guest has booted.
> That way, even if all devices tied to a Guest SMMUv3 get hot-unplugged,
> QEMU can still block attaching a device that belongs to a different Host
> SMMUv3.
>
> Thanks,
> Shameer
>
>> I am not 100% sure. But I think Intel model could have a similar
>> problem if the guest boots with zero cold-plugged device and then
>> hot-plugs a PASID-capable device at the runtime, when the guest-
>> level IOMMU driver is already inited?
>>
>> FWIW, Shameer's cover-letter has the following line:
>>  "At least one vfio-pci device must currently be cold-plugged to
>>   a PCIe root complex associated with arm-smmuv3,accel=on."
>>
>> Perhaps there should be a similar highlight in this smmuv3-accel
>> file as well (@Shameer).
>>
>> Nicolin