Re: [RFC] iommu_ops->domain_alloc_paging() enhancement to support AMD IOMMU driver

[Yi Liu <yi.l.liu@intel.com>]

On 2024/7/11 18:15, Vasant Hegde wrote:
> Hi Yi,
> 
> 
> On 7/3/2024 12:27 PM, Yi Liu wrote:
>> On 2024/7/3 13:42, Vasant Hegde wrote:
>>> Jason,
>>>
>>>
>>> On 7/1/2024 10:56 PM, Jason Gunthorpe wrote:
>>>> On Mon, Jul 01, 2024 at 04:18:01PM +0530, Vasant Hegde wrote:
>>>>
>>>>>> Then VFIO will just request the V2 domain anyhow and you are right
>>>>>> back to the starting problem again.
>>>>>
>>>>> Let me see if I can put together here.
>>>>>
>>>>>
>>>>> - Basically we want domain_alloc_paging() to explicitly tell domain requirement
>>>>> (ex: allocate PASID capable UNMANAGED domain)
>>>>>     If we do that then we will fix ordering issue I described earlier.
>>>>> - Fix attach device path so that for 'PASID capable UNMANAGED domain' :
>>>>>     non-PASID capable device will use AMD V2 page table with GCR3[0]
>>>>>     PASID capable device will setup its GCR3 table as well and it should work
>>>>> fine.
>>>>>
>>>>> Net we need a enhancement to domain_alloc_paging(). Otherwise I don't see a way
>>>>> to reliable implement things.
>>>>
>>>> Yes, some flag at domain allocation time is clearly needed
>>>
>>> Right. I will try to get this going and then we can fine tune with Robin's
>>> suggestion.
>>>
>>>>
>>>> But you need the whole story, including when will VFIO set the flag,
>>>> which requires a uAPI someplace.
>>>>
>>>>> But these two can go independently?
>>>>
>>>> If PASID goes with a default-on uAPI then you loose what you are
>>>> asking for as VFIO will always request PASID.
>>>>
>>>> So the overall plan has to be sorted out now..
>>>
>>> Is this the Yi's thread you were referring ?
>>>
>>> https://lore.kernel.org/linux-iommu/20240628090557.50898-1-yi.l.liu@intel.com/
>>
>> This is part of the whole work. What Jason refers may be the below one.
>> It includes how VFIO report the PASID capability to userspace. There is
>> an open to be closed though. I'll loop you in the next version. :)
> 
> Thank you Yi. I will start looking into this series.
> 
> 
>>
>> https://lore.kernel.org/kvm/20240412082121.33382-1-yi.l.liu@intel.com/
>>
>> BTW. For AMD, can a single device be attached to multiple V1 page table?
>> IIRC. looks like AMD has only one V1 page table pointer per BDF. But there
>> is a PASID table which has multiple cr3 page table pointers.
> 
> Jason covered this part in his response. Basically AMD has two page table :
>    V1 which doesn't support PASID.
>    V2 which support PASID
> 
> By default when host is booted it will be in V1 page table. What we are planning
> is we will implement domain_alloc_paging() interface in AMD driver. As part of that:
>    - PASID capable device will be placed in V2 page table, rest will be in V1
> page table

looks like the format of V1 and V2 page tables are different. Can the V2
page table be attached to a device (RID) and used to do address translation
for the DMAs w/o PASID? I have an impression that a RID can only use V1
page table. I may be wrong here. :)

> For UNMANAGED domain (like vfio-pci), we would need a indication from core layer
> saying whether we want to allocate PASID capable domain or not. So that
> internally we can allocate appropriate page table type.
> 
> That means from VFIO layer we need to pass this detail to IOMMU layer.
> For this part, I will start watching your patch. May be we can start discussion
> in that thread itself.
> 

yeah. Looks like you need to pass a hint to iommu driver for domain
allocation. For the usage vfio+iommufd, it is fine as the domain_alloc_user
accepts a flag. For legacy vfio (using vfio iommu type1) usage, it's a bit
tough as the domain allocation is within kernel, userspace does not have
any input on it so far. And vfio type1 does not expose the domain concept
to userspace, it only exposes container which is kind of an address space
object. This means no good way for userspace to control the domain
allocation.

If it's only a matter when PASID is enabled, perhaps we can skip vfio iommu
type1 as it is not supposed to support PASID.

-- 
Regards,
Yi Liu