Re: [RFC] iommu_ops->domain_alloc_paging() enhancement to support AMD IOMMU driver

[Vasant Hegde <vasant.hegde@amd.com>]

Hi Yi,


On 7/3/2024 12:27 PM, Yi Liu wrote:
> On 2024/7/3 13:42, Vasant Hegde wrote:
>> Jason,
>>
>>
>> On 7/1/2024 10:56 PM, Jason Gunthorpe wrote:
>>> On Mon, Jul 01, 2024 at 04:18:01PM +0530, Vasant Hegde wrote:
>>>
>>>>> Then VFIO will just request the V2 domain anyhow and you are right
>>>>> back to the starting problem again.
>>>>
>>>> Let me see if I can put together here.
>>>>
>>>>
>>>> - Basically we want domain_alloc_paging() to explicitly tell domain requirement
>>>> (ex: allocate PASID capable UNMANAGED domain)
>>>>    If we do that then we will fix ordering issue I described earlier.
>>>> - Fix attach device path so that for 'PASID capable UNMANAGED domain' :
>>>>    non-PASID capable device will use AMD V2 page table with GCR3[0]
>>>>    PASID capable device will setup its GCR3 table as well and it should work
>>>> fine.
>>>>
>>>> Net we need a enhancement to domain_alloc_paging(). Otherwise I don't see a way
>>>> to reliable implement things.
>>>
>>> Yes, some flag at domain allocation time is clearly needed
>>
>> Right. I will try to get this going and then we can fine tune with Robin's
>> suggestion.
>>
>>>
>>> But you need the whole story, including when will VFIO set the flag,
>>> which requires a uAPI someplace.
>>>
>>>> But these two can go independently?
>>>
>>> If PASID goes with a default-on uAPI then you loose what you are
>>> asking for as VFIO will always request PASID.
>>>
>>> So the overall plan has to be sorted out now..
>>
>> Is this the Yi's thread you were referring ?
>>
>> https://lore.kernel.org/linux-iommu/20240628090557.50898-1-yi.l.liu@intel.com/
> 
> This is part of the whole work. What Jason refers may be the below one.
> It includes how VFIO report the PASID capability to userspace. There is
> an open to be closed though. I'll loop you in the next version. :)

Thank you Yi. I will start looking into this series.


> 
> https://lore.kernel.org/kvm/20240412082121.33382-1-yi.l.liu@intel.com/
> 
> BTW. For AMD, can a single device be attached to multiple V1 page table?
> IIRC. looks like AMD has only one V1 page table pointer per BDF. But there
> is a PASID table which has multiple cr3 page table pointers.

Jason covered this part in his response. Basically AMD has two page table :
  V1 which doesn't support PASID.
  V2 which support PASID

By default when host is booted it will be in V1 page table. What we are planning
is we will implement domain_alloc_paging() interface in AMD driver. As part of that:
  - PASID capable device will be placed in V2 page table, rest will be in V1
page table

For UNMANAGED domain (like vfio-pci), we would need a indication from core layer
saying whether we want to allocate PASID capable domain or not. So that
internally we can allocate appropriate page table type.

That means from VFIO layer we need to pass this detail to IOMMU layer.
For this part, I will start watching your patch. May be we can start discussion
in that thread itself.

-Vasant