Re: [PATCH v4 05/12] vfio/iommufd: Introduce auto domain creation

[Joao Martins <joao.m.martins@oracle.com>]

On 18/07/2024 08:44, Duan, Zhenzhong wrote:
>>>>> If existing hwpt doesn't support dirty tracking.
>>>>> Another device supporting dirty tracking attaches to that hwpt, what
>> will
>>>> happen?
>>>>>
>>>>
>>>> Hmm, It succeeds as there's no incompatbility. At the very least I plan on
>>>> blocking migration if the device neither has VF dirty tracking, nor IOMMU
>>>> dirty
>>>> tracking (and patch 11 needs to be adjusted to check hwpt_flags instead
>> of
>>>> container).
>>>
>>> When bcontainer->dirty_pages_supported is true, I think that container
>> should only contains hwpt list that support dirty tracking. All hwpt not
>> supporting dirty tracking should be in other container.
>>>
>> Well but we are adopting this auto domains scheme and works for any
>> device,
>> dirty tracking or not. We already track hwpt flags so we know which ones
>> support
>> dirty tracking. This differentiation would (IMHO) complicate more and I am
>> not
>> sure the gain
> 
> OK, I was trying to make bcontainer->dirty_pages_supported  accurate because it is used in many functions such as vfio_get_dirty_bitmap() which require an accurate value. If there is mix of hwpt in that container, that's impossible.
> 
> But as you say you want to address the mix issue in a follow-up and presume all are homogeneous hw for now, then OK, there is no conflict.
> 

Right

>>
>>> If device supports dirty tracking, it should bypass attaching container that
>> doesn't support dirty tracking. Vise versa.
>>> This way we can support the mixing environment.
>>>
>>
>> It's not that easy as the whole flow doesn't handle this mixed mode (even
>> excluding this series). We would to have device-dirty-tracking start all
>> non-disabled device trackers first [and stop them as well], and then we
>> would
>> always iterate those first (if device dirty trackers are active), and then defer
>> to IOMMU tracker for those who don't.
> 
> Why is device-dirty-tracking preferred over IOMMU dirty tracking?
> Imagine if many devices attached to same domain.
> 

The heuristic or expectation is that device dirty tracking doesn't involve a
compromise for SW because it can a) perform lowest granularity of IOVA range
being dirty with b) no DMA penalty. With IOMMU though, SW needs to worry about
managing page tables to dictate the granularity and those take time to walk the
deeper the level we descend into. I used to think that IOMMU we have DMA penalty
(because of the IOTLB flushes to clear dirty bit, and IOTLB cache misses) but I
haven't yet that materialized in the field yet (at least for 100Gbit/s rates).

TL;DR At the end of the day with device dirty tracking you have less to worry
about, and it's the VF doing most of the heavy lifting. In theory with device
dirty tracking you could even perform sub basepage tracking if the device allows
it to do so.